[Engine-devel] [UX] how to design a bar/line chart?

Tomas Jelinek tjelinek at redhat.com
Wed Nov 13 07:21:56 UTC 2013 


----- Original Message -----
> From: "Alexander Wels" <awels at redhat.com>
> To: "Tomas Jelinek" <tjelinek at redhat.com>
> Cc: engine-devel at ovirt.org, "Michal Skrivanek" <michal.skrivanek at redhat.com>, "Eldan Hildesheim"
> <ehildesh at redhat.com>, "info" <info at eldanet.com>
> Sent: Tuesday, November 12, 2013 4:52:12 PM
> Subject: Re: [Engine-devel] [UX] how to design a bar/line chart?
> 
> On Tuesday, November 12, 2013 10:21:15 AM Tomas Jelinek wrote:
> > ----- Original Message -----
> > 
> > > From: "Alexander Wels" <awels at redhat.com>
> > > To: "Tomas Jelinek" <tjelinek at redhat.com>
> > > Cc: engine-devel at ovirt.org, "Michal Skrivanek"
> > > <michal.skrivanek at redhat.com>, "Eldan Hildesheim" <ehildesh at redhat.com>,
> > > "info" <info at eldanet.com>
> > > Sent: Tuesday, November 12, 2013 4:03:14 PM
> > > Subject: Re: [Engine-devel] [UX] how to design a bar/line chart?
> > > 
> > > On Tuesday, November 12, 2013 09:50:07 AM Tomas Jelinek wrote:
> > > > ----- Original Message -----
> > > > 
> > > > > From: "Alexander Wels" <awels at redhat.com>
> > > > > To: engine-devel at ovirt.org
> > > > > Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "Michal Skrivanek"
> > > > > <michal.skrivanek at redhat.com>, "Eldan Hildesheim"
> > > > > <ehildesh at redhat.com>,
> > > > > "info" <info at eldanet.com>
> > > > > Sent: Tuesday, November 12, 2013 3:33:56 PM
> > > > > Subject: Re: [Engine-devel] [UX] how to design a bar/line chart?
> > > > > 
> > > > > On Tuesday, November 12, 2013 09:25:34 AM Tomas Jelinek wrote:
> > > > > > Hey all,
> > > > > > 
> > > > > > let me conclude what has been written in this thread to one
> > > > > > proposal:
> > > > > > 
> > > > > > == From the UX perspective the behavior ==
> > > > > > - each time the FE will receive new data and this data are
> > > > > > different
> > > > > > from
> > > > > > the old ones, visualize it in the chart It means if you will keep
> > > > > > pressing
> > > > > > the refresh button but the data will not change, no new data will
> > > > > > be
> > > > > > visualized (an exception will be the 0 usage) - the amount of data
> > > > > > visualized will depend on the size of the widget (since the tables
> > > > > > are
> > > > > > resizable). It means that if you make the widget bigger you will
> > > > > > not
> > > > > > see
> > > > > > the same chart bigger but more data. - If you make the widget
> > > > > > bigger,
> > > > > > only
> > > > > > then the amount of data will start to increase: e.g.
> > > > > > 
> > > > > > before resize:
> > > > > > | /-------\ |
> > > > > > |
> > > > > > |/         \|
> > > > > > 
> > > > > > after resize:
> > > > > > | /-------\          |
> > > > > > |
> > > > > > |/         \         |
> > > > > > 
> > > > > > and only now the new data will start to appear
> > > > > > 
> > > > > > == From FE technical point of view ==
> > > > > > - since I have not found any GWT library which would be acceptable
> > > > > > (e.g.
> > > > > > actively developed and without the need to connect to google
> > > > > > servers)
> > > > > > and
> > > > > > given that the required chart is quite simple I guess it would be
> > > > > > ok
> > > > > > to
> > > > > > write it by myself. - according to Einav's mail it is ok to use
> > > > > > HTML5
> > > > > > canvas so I would go with writing a new widget using HTML5 canvas
> > > > > 
> > > > > Just to throw out something to think about, we could also in theory
> > > > > generate an image on the server side and simply display that image
> > > > > inside
> > > > > the grid (so no need for HTML5 canvas or other things like that). The
> > > > > idea being basically that when the grid refreshes it makes a request
> > > > > for
> > > > > a new image on the back- end with the appropriate timeframe and the
> > > > > back-end generates the image which is easy to embed inside the grid.
> > > > > 
> > > > > pros:
> > > > > * Easy to embed inside grid (just an image tag).
> > > > > * Works on all browsers, even ones without HTML5 canvas support.
> > > > > cons:
> > > > > * More load on the back-end.
> > > > > * Extra round trips to back-end on refresh.
> > > > > * Not 'hot' like HTML5 canvas.
> > > > > * No interactivity if that is something we are interested in.
> > > > 
> > > > some more cons:
> > > > * need to remember the statistics on server in the memory. For
> > > > thousands
> > > > of
> > > > VMs it is not something we would like to do * lots of overhead to
> > > > retrieve
> > > > all the images on each refresh. If you have 100 VMs on a page and
> > > > refresh
> > > > each 5 seconds, it is 100 images transmitted from engine to frontend
> > > > each 5
> > > > seconds per one client (and we can have more of them of course) * FE
> > > > logic
> > > > on Server is in general not awesome
> > > 
> > > I would expect the statistics to be stored in the database somewhere,
> > > that
> > > way
> > > we can pull them for reports and things of that nature (like charts).
> > > Obviously we wouldn't do 100 round trips for the image, we would generate
> > > a
> > > single image sprite that would contain all the images in a single request
> > > and display the appropriate part of the image in the grid.
> > > 
> > > You are right in general front-end logic is not done on the back-end.
> > > However we must consider if we are really doing front end logic here, or
> > > if we are just displaying some reporting information as part of the grid.
> > > 
> > > If we are not storing the statistics anywhere, then this is a terrible
> > > plan, and we should do the logic on the client, but if we are, it is
> > > something to consider.
> > 
> > We store only the actual value. The statistics are stored only by DWH
> > but that is a different application. Engine itself does not have it so we
> > would have to implement it.
> > 
> 
> Which as you mentioned is not a desirable thing to do for thousands of VMs,
> but does bring up the question, if we only aggregate the statistics on the
> client, do we care if that information is lost when the user logs
> out/switches
> tab/etc. Since essentially we stop requesting that information from the back-
> end if the current active tab is not the VM main tab.

I would say it is not a problem. The VM tab is not for detailed statistics or history data. 
For this kind of data we have the reports portal.

On the other hand it may be painful to go to the VM tab and see no statistics and get them updated
each 15 seconds. So after couple of minutes you will maybe see something useful. Just don't click on any
other tab, otherwise you will loose that all :)

@Michal,Einav: Is this acceptable?

> 
> > > > > > == From L&F point of view ==
> > > > > > - would look pretty much like the one proposed by Malini:
> > > > > > http://style.org/chartapi/sparklines/
> > > > > > 
> > > > > > == From data point of view ==
> > > > > > - do not do any averages on VDSM side (since it already does them
> > > > > > for
> > > > > > CPU
> > > > > > and network and the memory is stable enough) - do not do any
> > > > > > averages
> > > > > > on
> > > > > > engine side (since would have to be done for each FE separately and
> > > > > > stored
> > > > > > in session which is a bit overcomplicated. If the user wants to see
> > > > > > more
> > > > > > accurate data, he/she can change the refresh rate) - do not do any
> > > > > > interpolation since the data are already averaged and we will show
> > > > > > only
> > > > > > new
> > > > > > ones
> > > > > > 
> > > > > > @Malini,Einav,Eldan,Michal what do you think?
> > > > > > 
> > > > > > Tomas
> > > > > > 
> > > > > > ----- Original Message -----
> > > > > > 
> > > > > > > From: "Michal Skrivanek" <michal.skrivanek at redhat.com>
> > > > > > > To: "Tomas Jelinek" <tjelinek at redhat.com>, "Dan Kenigsberg"
> > > > > > > <danken at redhat.com> Cc: "Malini Rao" <mrao at redhat.com>, "Eldan
> > > > > > > Hildesheim" <ehildesh at redhat.com>, engine-devel at ovirt.org, "info"
> > > > > > > <info at eldanet.com>
> > > > > > > Sent: Tuesday, November 12, 2013 11:49:12 AM
> > > > > > > Subject: Re: [Engine-devel] [UX] how to design a bar/line chart?
> > > > > > > 
> > > > > > > On Nov 12, 2013, at 11:18 , Tomas Jelinek <tjelinek at redhat.com>
> > > > > > > 
> > > > > > > wrote:
> > > > > > > > So we have looked into the resource usage sampling with mbetak
> > > > > > > > and
> > > > > > > > also
> > > > > > > > with Michal and it seems that
> > > > > > > > 
> > > > > > > > for the CPU usage:
> > > > > > > > - VDSM polls libvirt to get the runtime statistics of the VM
> > > > > > > > regularly.
> > > > > > > > The
> > > > > > > > pooling interval is configured in
> > > > > > > > 
> > > > > > > >  vdsm.conf as vm_sample_cpu_interval and by default it is 15
> > > > > > > >  seconds
> > > > > > > > 
> > > > > > > > - libvirt returns something we than use to calculate the
> > > > > > > > average
> > > > > > > > CPU
> > > > > > > > usage
> > > > > > > > since the last poll
> > > > > > > > - engine polls VDSM once in 15 seconds to get the current
> > > > > > > > statistics
> > > > > > > > (the
> > > > > > > > same 15 seconds is just a coincidence and we can not count on
> > > > > > > > this)
> > > > > > > > - than the frontend polls the engine each 5-60 seconds (depends
> > > > > > > > on
> > > > > > > > the
> > > > > > > > refresh rate) and gets the current value from the engine
> > > > > > > > - the user can press the refresh button anytime to poll the
> > > > > > > > engine
> > > > > > > > again
> > > > > > > > 
> > > > > > > > For network usage:
> > > > > > > > - it should be pretty much the same as the CPU just the VDSM
> > > > > > > > poll
> > > > > > > > interval
> > > > > > > > is configured as vm_sample_net_interval and by default it is 5
> > > > > > > > seconds
> > > > > > > 
> > > > > > > Dan, since we poll only every 15s and cpu info is 15s wouldn't it
> > > > > > > make
> > > > > > > sense to change the default for network monitoring to 15s as
> > > > > > > well?
> > > > > > > it's 2
> > > > > > > libvirt rounds trip for nothing really. Or does it serve some
> > > > > > > other
> > > > > > > purpose?>
> > > > > > > 
> > > > > > > > For memory usage:
> > > > > > > > - guest agent sends a message to VDSM with the memory usage
> > > > > > > > regularly.
> > > > > > > > The
> > > > > > > > interval is set in ovirt-guest-agent.conf as heart_beat_rate
> > > > > > > > 
> > > > > > > >  and by default it is 5 seconds
> > > > > > > > 
> > > > > > > > - the actual value sent by ovirt-guest-agent is the actual
> > > > > > > > value
> > > > > > > > at
> > > > > > > > the
> > > > > > > > time when the value is sent (e.g. for Linux taken from "cat
> > > > > > > > /proc/meminfo")
> > > > > > > > - vdsm is doing no statistics on top of it, just remembers the
> > > > > > > > last
> > > > > > > > value
> > > > > > > > taken from ovirt-guest-agent
> > > > > > > 
> > > > > > > which is fine, it doesn't change so often and there are typically
> > > > > > > no
> > > > > > > spikes
> > > > > > > 
> > > > > > > > - the rest of the poling is the same
> > > > > > > > 
> > > > > > > > So, visualizing this in some usable form will be quite
> > > > > > > > challenging
> > > > > > > > ;)
> > > > > > > > I see the following problems:
> > > > > > > > - if the VDSM gets the data faster than the engine polls it
> > > > > > > > (and
> > > > > > > > most
> > > > > > > > often
> > > > > > > > it does) than the info in between will be lost.
> > > > > > > > 
> > > > > > > >  The question is how big this problem is and if it is worth
> > > > > > > >  solving
> > > > > > > >  (I
> > > > > > > >  would say not for CPU which are averages but maybe yes for
> > > > > > > >  memory).
> > > > > > > >  Other question if there is a way how to solve it since the
> > > > > > > >  VDSM
> > > > > > > >  can
> > > > > > > >  be
> > > > > > > >  polled by anyone and it does not really care if someone polls
> > > > > > > >  it...
> > > > > > > >  (Michal?)
> > > > > > > 
> > > > > > > I'd say not solve it and try to keep it in sync on vdsm side with
> > > > > > > engine
> > > > > > > poll, to save unnecessary libvirt calls
> > > > > > > 
> > > > > > > > - we can lost some data between frontend<->engine if the
> > > > > > > > polling
> > > > > > > > interval
> > > > > > > > of the FE is slower than the polling interval of the engine.
> > > > > > > > This
> > > > > > > > is
> > > > > > > > something
> > > > > > > > 
> > > > > > > >  not really worth solving because the user can set this
> > > > > > > >  according
> > > > > > > >  to
> > > > > > > >  the
> > > > > > > >  level of detail he/she wants
> > > > > > > 
> > > > > > > well, you should average the values in engine in case the FE
> > > > > > > refresh
> > > > > > > is
> > > > > > > 
> > > > > > > >15s. Or add (refresh/15) of them
> > > > > > 
> > > > > > It is not that simple since you can have more frontends and not
> > > > > > sure
> > > > > > if
> > > > > > it
> > > > > > would be a good idea to put this into the session...
> > > > > > 
> > > > > > > > - since we will get new info once in ~15 seconds, and the
> > > > > > > > polling
> > > > > > > > of
> > > > > > > > the
> > > > > > > > FE
> > > > > > > > is by default 5 seconds, do we want to do some interpolation?
> > > > > > > > Or
> > > > > > > > just
> > > > > > > > show
> > > > > > > > the
> > > > > > > > 
> > > > > > > >  same value 3 times? Or be smart and show only changed values?
> > > > > > > >  (this
> > > > > > > >  is
> > > > > > > >  tricky since there is a chance that it did not change - e.g.
> > > > > > > >  constant
> > > > > > > >  0
> > > > > > > >  mem usage if you have no guest agent)
> > > > > > > > 
> > > > > > > > - What if the user starts clicking to the refresh button? Do we
> > > > > > > > want
> > > > > > > > to
> > > > > > > > keep appending the same value if the engine still has only the
> > > > > > > > old
> > > > > > > > ones?
> > > > > > > 
> > > > > > > just add a new line/point every 15s should be ok
> > > > > > > 
> > > > > > > Thanks,
> > > > > > > michal
> > > > > > > 
> > > > > > > > Tomas
> > > > > > > > 
> > > > > > > > ----- Original Message -----
> > > > > > > > 
> > > > > > > >> From: "Tomas Jelinek" <tjelinek at redhat.com>
> > > > > > > >> To: "Malini Rao" <mrao at redhat.com>
> > > > > > > >> Cc: "Eldan Hildesheim" <ehildesh at redhat.com>,
> > > > > > > >> engine-devel at ovirt.org,
> > > > > > > >> "info" <info at eldanet.com>
> > > > > > > >> Sent: Monday, November 11, 2013 4:23:09 PM
> > > > > > > >> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >> chart?
> > > > > > > >> 
> > > > > > > >> 
> > > > > > > >> 
> > > > > > > >> ----- Original Message -----
> > > > > > > >> 
> > > > > > > >>> From: "Malini Rao" <mrao at redhat.com>
> > > > > > > >>> To: "Eldan Hildesheim" <ehildesh at redhat.com>
> > > > > > > >>> Cc: "Tomas Jelinek" <tjelinek at redhat.com>, "info"
> > > > > > > >>> <info at eldanet.com>,
> > > > > > > >>> engine-devel at ovirt.org
> > > > > > > >>> Sent: Monday, November 11, 2013 4:15:50 PM
> > > > > > > >>> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >>> chart?
> > > > > > > >>> 
> > > > > > > >>> Is this going to fit in a row of a table? Or are we talking
> > > > > > > >>> of
> > > > > > > >>> a
> > > > > > > >>> more
> > > > > > > >>> detailed view?
> > > > > > > >> 
> > > > > > > >> it should fit into one cell of the table
> > > > > > > >> 
> > > > > > > >>> ----- Original Message -----
> > > > > > > >>> From: "Eldan Hildesheim" <ehildesh at redhat.com>
> > > > > > > >>> To: "Tomas Jelinek" <tjelinek at redhat.com>
> > > > > > > >>> Cc: "info" <info at eldanet.com>, engine-devel at ovirt.org
> > > > > > > >>> Sent: Monday, November 11, 2013 8:01:07 AM
> > > > > > > >>> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >>> chart?
> > > > > > > >>> 
> > > > > > > >>> Throw this gif into a browser. This is more or less what I
> > > > > > > >>> thought.
> > > > > > > >>> Eldan
> > > > > > > >>> 
> > > > > > > >>> ----- Original Message -----
> > > > > > > >>> From: "Tomas Jelinek" <tjelinek at redhat.com>
> > > > > > > >>> To: "Eldan Hildesheim" <ehildesh at redhat.com>
> > > > > > > >>> Cc: "Einav Cohen" <ecohen at redhat.com>, "info"
> > > > > > > >>> <info at eldanet.com>,
> > > > > > > >>> engine-devel at ovirt.org
> > > > > > > >>> Sent: Monday, November 11, 2013 12:03:15 PM
> > > > > > > >>> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >>> chart?
> > > > > > > >>> 
> > > > > > > >>> 
> > > > > > > >>> 
> > > > > > > >>> ----- Original Message -----
> > > > > > > >>> 
> > > > > > > >>>> From: "Eldan Hildesheim" <ehildesh at redhat.com>
> > > > > > > >>>> To: "Einav Cohen" <ecohen at redhat.com>
> > > > > > > >>>> Cc: "info" <info at eldanet.com>, engine-devel at ovirt.org
> > > > > > > >>>> Sent: Sunday, November 10, 2013 3:56:57 PM
> > > > > > > >>>> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >>>> chart?
> > > > > > > >>>> 
> > > > > > > >>>> Hello all,
> > > > > > > >>>> We use to have a good solution in the period pre-WPF.
> > > > > > > >>>> A line chart (used to be in flash) that works like a
> > > > > > > >>>> plotter:
> > > > > > > >>>> The Line Bar (not bar) had a small animation that shifted
> > > > > > > >>>> all
> > > > > > > >>>> the
> > > > > > > >>>> bar
> > > > > > > >>>> to
> > > > > > > >>>> the
> > > > > > > >>>> left.
> > > > > > > >>>> When a new data arrived it just added a new line (to the
> > > > > > > >>>> right)
> > > > > > > >>>> and
> > > > > > > >>>> as I
> > > > > > > >>>> said
> > > > > > > >>>> before, in parallel it always shifted slowly to the left.
> > > > > > > >>> 
> > > > > > > >>> Any chance you still have some screenshot or mockup so I can
> > > > > > > >>> imagine
> > > > > > > >>> it
> > > > > > > >>> better?
> > > > > > > >>> 
> > > > > > > >>>> The animation gives the impression that data is streaming
> > > > > > > >>>> and
> > > > > > > >>>> when a
> > > > > > > >>>> real
> > > > > > > >>>> new
> > > > > > > >>>> data arrives the user gets it very fast.
> > > > > > > >>>> We have to sync between the animation and the rate of the
> > > > > > > >>>> arrival
> > > > > > > >>>> of
> > > > > > > >>>> the
> > > > > > > >>>> data
> > > > > > > >>>> but this is easy.
> > > > > > > >>>> If we can't find a good framework it can be created from
> > > > > > > >>>> scratch
> > > > > > > >>>> with
> > > > > > > >>>> JS,
> > > > > > > >>>> svg
> > > > > > > >>>> or canvas.
> > > > > > > >>> 
> > > > > > > >>> We need to be careful about what we will use. oVirt is
> > > > > > > >>> supposed
> > > > > > > >>> to
> > > > > > > >>> work
> > > > > > > >>> on
> > > > > > > >>> FF
> > > > > > > >>> 17 [1]
> > > > > > > >>> but the HTML5 canvas works only since FF23 [2].
> > > > > > > >>> 
> > > > > > > >>> @Einav:
> > > > > > > >>> Is there a chance that we could start support only FF23+ and
> > > > > > > >>> IE9+
> > > > > > > >>> (this
> > > > > > > >>> one
> > > > > > > >>> is already OK)
> > > > > > > >>> because of this feature?
> > > > > > > >>> 
> > > > > > > >>>> Now regarding its position:
> > > > > > > >>>> Rollover is good but not enough, we should somehow put it in
> > > > > > > >>>> the
> > > > > > > >>>> lower
> > > > > > > >>>> panel
> > > > > > > >>>> under general or even another tab - (live data).
> > > > > > > >>> 
> > > > > > > >>> This is a bit different requirement. The point of this
> > > > > > > >>> specific
> > > > > > > >>> is
> > > > > > > >>> to
> > > > > > > >>> give
> > > > > > > >>> a
> > > > > > > >>> better
> > > > > > > >>> overview in the main tab. If it will be done we can decide if
> > > > > > > >>> we
> > > > > > > >>> want
> > > > > > > >>> to
> > > > > > > >>> give
> > > > > > > >>> more
> > > > > > > >>> details in sub tabs.
> > > > > > > >>> 
> > > > > > > >>>> We could later on have a (live data Tab) in other places as
> > > > > > > >>>> well
> > > > > > > >>>> like
> > > > > > > >>>> host,
> > > > > > > >>>> cluster...
> > > > > > > >>>> Eldan
> > > > > > > >>> 
> > > > > > > >>> [1]: http://www.ovirt.org/Download
> > > > > > > >>> [2]: http://caniuse.com/#feat=canvas
> > > > > > > >>> 
> > > > > > > >>>> ----- Original Message -----
> > > > > > > >>>> From: "Einav Cohen" <ecohen at redhat.com>
> > > > > > > >>>> To: "Ewoud Kohl van Wijngaarden"
> > > > > > > >>>> <ewoud+ovirt at kohlvanwijngaarden.nl>
> > > > > > > >>>> Cc: "Alexander Wels" <awels at redhat.com>, "Eldan Hildesheim"
> > > > > > > >>>> <ehildesh at redhat.com>, engine-devel at ovirt.org, "info"
> > > > > > > >>>> <info at eldanet.com>
> > > > > > > >>>> Sent: Friday, November 8, 2013 10:50:10 PM
> > > > > > > >>>> Subject: Re: [Engine-devel] [UX] how to design a bar/line
> > > > > > > >>>> chart?
> > > > > > > >>>> 
> > > > > > > >>>>> ----- Original Message -----
> > > > > > > >>>>> From: "Ewoud Kohl van Wijngaarden"
> > > > > > > >>>>> <ewoud+ovirt at kohlvanwijngaarden.nl>
> > > > > > > >>>>> Sent: Thursday, November 7, 2013 11:44:07 AM
> > > > > > > >>>>> 
> > > > > > > >>>>> On Wed, Nov 06, 2013 at 11:45:36AM -0500, Alexander Wels
> wrote:
> > > > > > > >>>>>> I suppose we need to answer a few questions before we can
> > > > > > > >>>>>> go
> > > > > > > >>>>>> into
> > > > > > > >>>>>> which
> > > > > > > >>>>>> library is better:
> > > > > > > >>>>>> 
> > > > > > > >>>>>> 1. Do we mind sending data over to Google so Google can
> > > > > > > >>>>>> render
> > > > > > > >>>>>> images
> > > > > > > >>>>>> for
> > > > > > > >>>>>> us.
> > > > > > > >>>>> 
> > > > > > > >>>>> I'd say no. Even from a reliability point of view since
> > > > > > > >>>>> users
> > > > > > > >>>>> may
> > > > > > > >>>>> have
> > > > > > > >>>>> systems that aren't connected to the internet.
> > > > > > > >>>> 
> > > > > > > >>>> +1
> > > > > > > >>>> 
> > > > > > > >>>>> (Though I don't know how well oVirt handles this
> > > > > > > >>>>> currently.)
> > > > > > > >>>> 
> > > > > > > >>>> AFAIK - oVirt is handling it ('it' ==  having no internet
> > > > > > > >>>> connection)
> > > > > > > >>>> well.
> > > > > > > >>>> _______________________________________________
> > > > > > > >>>> Engine-devel mailing list
> > > > > > > >>>> Engine-devel at ovirt.org
> > > > > > > >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > > > >>> 
> > > > > > > >>> _______________________________________________
> > > > > > > >>> Engine-devel mailing list
> > > > > > > >>> Engine-devel at ovirt.org
> > > > > > > >>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > > > >> 
> > > > > > > >> _______________________________________________
> > > > > > > >> Engine-devel mailing list
> > > > > > > >> Engine-devel at ovirt.org
> > > > > > > >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > > 
> > > > > > _______________________________________________
> > > > > > Engine-devel mailing list
> > > > > > Engine-devel at ovirt.org
> > > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
>

More information about the Devel mailing list