[Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9

Ofri Masad omasad at redhat.com
Thu Apr 18 08:31:28 UTC 2013 

We run the query each time the host is moving to UP state.
Which means, we query all the hosts on engine restart. if the host was unreachable or down for any reason - we will query it again before moving to UP state. 

Ofri

----- Original Message -----
> From: "Doron Fediuck" <dfediuck at redhat.com>
> To: "Wei D Chen" <wei.d.chen at intel.com>
> Cc: "Omer Frenkel" <ofrenkel at redhat.com>, "Ofri Masad" <omasad at redhat.com>, "Oved Ourfalli" <ovedo at redhat.com>,
> engine-devel at ovirt.org
> Sent: Thursday, April 18, 2013 11:23:47 AM
> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9
> 
> Generally speaking I agree, we can drop the periodic check is
> this is the way we expect it to work (ie- change trust level only during
> reboot).
> 
> The only thing I'd like to verify is what happens is we miss something.
> ie- let's assume the engine crashed. During the engine down time a host
> reboots and becomes untrusted. Now what? The same goes for network
> disconnect.
> I'd expect to run a query in such cases to make sure we do not miss
> a state change.
> 
> 
> ----- Original Message -----
> > From: "Wei D Chen" <wei.d.chen at intel.com>
> > To: "Omer Frenkel" <ofrenkel at redhat.com>, "Ofri Masad" <omasad at redhat.com>,
> > "Doron Fediuck" <dfediuck at redhat.com>
> > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > Sent: Thursday, April 18, 2013 10:42:10 AM
> > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > integration with oVirt in 4/9
> > 
> > Yes, the host must be rebooted to take effect. Doron, what do you think?
> > 
> > Best Regards,
> > Dave Chen
> > 
> > 
> > -----Original Message-----
> > From: Omer Frenkel [mailto:ofrenkel at redhat.com]
> > Sent: Thursday, April 18, 2013 3:20 PM
> > To: Ofri Masad
> > Cc: Chen, Wei D; Oved Ourfalli; engine-devel at ovirt.org
> > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > integration with oVirt in 4/9
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Ofri Masad" <omasad at redhat.com>
> > > To: "Wei D Chen" <wei.d.chen at intel.com>
> > > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > Sent: Thursday, April 18, 2013 9:38:26 AM
> > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > integration with oVirt in 4/9
> > > 
> > > Hi Dave,
> > > 
> > > Can't a host become untrusted  without being rebooted?
> > > If that is really the case, there is no need for a periodic check -
> > > the trigger for the check would be the host rebooting (which is
> > > visible to the engine).
> > > 
> > 
> > +1
> > 
> > > Thanks,
> > > Ofri
> > > 
> > > ----- Original Message -----
> > > > From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > To: "Itamar Heim" <iheim at redhat.com>, "Ofri Masad"
> > > > <omasad at redhat.com>
> > > > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > Sent: Thursday, April 18, 2013 9:19:03 AM
> > > > Subject: RE: [Engine-devel] minutes for sync up on Open Attestation
> > > > integration with oVirt in 4/9
> > > > 
> > > > I think it's more sensible, the initial status should be the real
> > > > status for this host (trusted / untrusted) only the untrusted host
> > > > will be set to non-operational.
> > > > we just need poll this host instead of all of the hosts in the
> > > > cluster if this can be done in "InitVdsOnUpCommand", and we suppose
> > > > this is the first time when this host add into the cluster.
> > > > Follow these steps (conclusion based on your suggestion).
> > > > 
> > > > -	The first time when one host is added into a trust cluster then poll
> > > > this
> > > > host, the host will be in "up" status if get "trusted" result from
> > > > attestation server, or else, set this host as non-operational status.
> > > > -	The periodic check will poll all of the hosts only once as this will
> > > > cut
> > > > down a lot of time needed instead of poll each host one by one, this
> > > > conclusion is based on our experiments because most of time consumed
> > > > is in parallel.
> > > > -	When host is down for a different reason and up again, we suppose
> > > > "InitVdsOnUpCommand" will be triggered again (right?), so we will
> > > > poll this host again to get the real status(trusted / untrusted).
> > > > Then mark the host as up or non-operational.
> > 
> > right
> > 
> > > > 
> > > > As a trusted host flip to untrusted and take effective only under
> > > > the condition of this host has been hacked and rebooted, so we
> > > > proposal no need to add periodic check if any host's reboot will invoke
> > > > "InitVdsOnUpCommand"
> > > > and we add attestation logic in "InitVdsOnUpCommand" is enough.
> > > > 
> > > > My proposal would be like this (no periodic check is needed, will
> > > > simply our
> > > > integration)
> > > > 
> > > > -	The first time when one host is added into a trust cluster then poll
> > > > this
> > > > host, the host will be in "up" status if get "trusted" result from
> > > > attestation server, or else, set this host as non-operational status.
> > > > -	The periodic check will poll all of the hosts only once as this will
> > > > cut
> > > > down a lot of time needed instead of poll each host one by one, this
> > > > conclusion is based on our experiments because most of time consumed
> > > > is in parallel.
> > > > -	(up -> nonoperationl / trusted -> untrusted)When host is down for a
> > > > different reason and up again, we suppose "InitVdsOnUpCommand" will
> > > > be triggered again (right?) and we also suppose the host will be
> > > > non-operational if host is down for some reason (right), so we will
> > > > poll this host again to get the real status(trusted / untrusted)
> > > > after host's rebooting.
> > > > -	(nonoperationl -> up / untrusted -> trusted) Admin will activate this
> > > > host
> > > > manually after admin fix the issue of this host. Attestation logic
> > > > will be added into "VdsManager:activate ()" as the clue you give me
> > > > before.
> > > > 
> > > > Any suggestion?
> > > > 
> > > > Best Regards,
> > > > Dave Chen
> > > > 
> > > > 
> > > > -----Original Message-----
> > > > From: Itamar Heim [mailto:iheim at redhat.com]
> > > > Sent: Wednesday, April 17, 2013 8:31 PM
> > > > To: Ofri Masad
> > > > Cc: Chen, Wei D; Oved Ourfalli; engine-devel at ovirt.org
> > > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > > integration with oVirt in 4/9
> > > > 
> > > > On 04/17/2013 10:23 AM, Ofri Masad wrote:
> > > > > Hi Dave,
> > > > > The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a
> > > > > good place to call the attestation host.
> > > > > Instead, like we suggested earlier, create a new Quartz job (like
> > > > > the one I've sent you in the QuotaManager class) which run every
> > > > > couple of minutes and update the hosts state.
> > > > > You can also add the check to InitVdsOnUpCommand, so that if a
> > > > > host was down for a different reason, it will be rechecked for
> > > > > attestation.
> > > > > You can add a UI to allow manual "refresh attestation".
> > > > >
> > > > 
> > > > so the new thread will poll all for all hosts and update the db,
> > > > then during the normal monitoring "detect" the issue?
> > > > 
> > > > > Ofri
> > > > >
> > > > > ----- Original Message -----
> > > > >> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > >> To: "Omer Frenkel" <ofrenkel at redhat.com>, "Doron Fediuck"
> > > > >> <dfediuck at redhat.com>
> > > > >> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > >> Sent: Monday, April 15, 2013 5:53:34 PM
> > > > >> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > >> Attestation integration with oVirt in 4/9
> > > > >>
> > > > >> Good approach, thanks all. How long it needs to invoke periodic
> > > > >> check in the class of VdsUpdateRunTimeInfo? Whether this time
> > > > >> configurable?
> > > > >> My concern is if the initial status of each host added into
> > > > >> trusted cluster is non-operational, we need wait a long time for
> > > > >> the invoking of this periodic check, so a trusted host's with
> > > > >> initial status of non-operational will take a long time to flip to a
> > > > >> trusted host.
> > > > >>
> > > > >> As I have not seen the configuration of periodic check in
> > > > >> VdsUpdateRunTimeInfo, would you give me some tips if you has some
> > > > >> clue.
> > > > >> Thanks a lot.
> > > > >>
> > > > >> Best Regards,
> > > > >> Dave Chen
> > > > >>
> > > > >> -----Original Message-----
> > > > >> From: engine-devel-bounces at ovirt.org
> > > > >> [mailto:engine-devel-bounces at ovirt.org]
> > > > >> On Behalf Of Omer Frenkel
> > > > >> Sent: Monday, April 15, 2013 7:01 PM
> > > > >> To: Doron Fediuck
> > > > >> Cc: Oved Ourfalli; engine-devel at ovirt.org
> > > > >> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > >> Attestation integration with oVirt in 4/9
> > > > >>
> > > > >>
> > > > >>
> > > > >> ----- Original Message -----
> > > > >>> From: "Doron Fediuck" <dfediuck at redhat.com>
> > > > >>> To: "Itamar Heim" <iheim at redhat.com>
> > > > >>> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > >>> Sent: Monday, April 15, 2013 10:05:57 AM
> > > > >>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > >>> Attestation integration with oVirt in 4/9
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> ----- Original Message -----
> > > > >>>> From: "Itamar Heim" <iheim at redhat.com>
> > > > >>>> To: "Oved Ourfalli" <ovedo at redhat.com>
> > > > >>>> Cc: engine-devel at ovirt.org
> > > > >>>> Sent: Monday, April 15, 2013 9:49:12 AM
> > > > >>>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > >>>> Attestation integration with oVirt in 4/9
> > > > >>>>
> > > > >>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote:
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> ----- Original Message -----
> > > > >>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > >>>>>> To: "Doron Fediuck" <dfediuck at redhat.com>, "Ofri Masad"
> > > > >>>>>> <omasad at redhat.com>
> > > > >>>>>> Cc: engine-devel at ovirt.org
> > > > >>>>>> Sent: Monday, April 15, 2013 8:54:18 AM
> > > > >>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > >>>>>> Attestation integration with oVirt in 4/9
> > > > >>>>>>
> > > > >>>>>> Hi Doron and Ofri,
> > > > >>>>>>
> > > > >>>>>> Thanks for your reply, here is some other question.
> > > > >>>>>>
> > > > >>>>>> ii.  When adding a host into the trusted cluster, the host
> > > > >>>>>> will be attested via OAT service, only trusted hosted can be
> > > > >>>>>> added.
> > > > >>>>>>
> > > > >>>>>> Would you also kindly tell me if there is any mandatory logic
> > > > >>>>>> check when adding a host into a cluster, so we can also put
> > > > >>>>>> attestation logic with these mandatory check together? Some
> > > > >>>>>> example or code location is better.
> > > > >>>>>> Thanks a lot.
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>> I think there are two approaches, depending on the use case.
> > > > >>>>> #1:
> > > > >>>>> If the host trusted property is static, then you can narrow
> > > > >>>>> the tests to two locations:
> > > > >>>>> * AddVdsCommand - (Vds = Host) - This command is triggered
> > > > >>>>> when a new host is added to the system. You can use the
> > > > >>>>> canDoAction method (which we have on all commands, and it
> > > > >>>>> determines whether an action can be run or not), and there, if
> > > > >>>>> cluster requires trusted hosts only, you can test whether this
> > > > >>>>> host is trusted or not.
> > > > >>>>> * ChangeVdsClusterCommand - this command is used when you
> > > > >>>>> change the cluster of a host. So, if the target cluster
> > > > >>>>> requires tursted hosts, you can do a similar test in its
> > > > >>>>> canDoAction method.
> > > > >>>>>
> > > > >>>>> #2:
> > > > >>>>> If the host trusted property can change, then I'd suggest
> > > > >>>>> using the NonOperational property of a host.
> > > > >>>>> We have a class called VdsUpdateRunTimeInfo that does periodic
> > > > >>>>> tests of hosts, deciding what's their status, according to
> > > > >>>>> specific flags.
> > > > >>>>> The code there is quite complex, and would require refactoring
> > > > >>>>> at some point, but in the meantime you can use the
> > > > >>>>> MonitoringStrategy interface that is being used there, and
> > > > >>>>> implement a new monitoring strategy for Open Attestation.
> > > > >>>>>
> > > > >>>>> There, in the processHardwareCapabilities, you can test that
> > > > >>>>> the host is trusted.
> > > > >>>>>
> > > > >>>>> When creating the strategy, using the
> > > > >>>>> MonitoringStrategyFactory, you'll need to create the appropriate
> > > > >>>>> strategy.
> > > > >>>>> Currently we support either virt strategy or gluster strategy
> > > > >>>>> or both. In your case you would need virt+attestation /
> > > > >>>>> gluster+attestation /
> > > > >>>>> virt+gluster+attestation, according to the services deployed
> > > > >>>>> virt+gluster+on the
> > > > >>>>> cluster.
> > > > >>>>
> > > > >>>> I'd go with the 2nd approach. i.e., not block the host from
> > > > >>>> being added, only from being used, based on monitoring / non-op
> > > > >>>> status
> > > > >>>>
> > > > >>> +1.
> > > > >>> It means that the admin can add the host, but the host will not
> > > > >>> be operational until we get a positive indication from the
> > > > >>> attestation service.
> > > > >>>
> > > > >> +1
> > > > >> if we want to test attestation only once, every time before host
> > > > >> moves to up.
> > > > >> the right place for it, imo, is InitVdsOnUpCommand if its
> > > > >> periodic also after host is up, then it should be somewhere in
> > > > >> the monitoring code
> > > > >>
> > > > >>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> Does that make sense?
> > > > >>>>> Doron and Ofri, what did you have in mind for this feature?
> > > > >>>>>
> > > > >>>>> Regards,
> > > > >>>>> Oved
> > > > >>>>>
> > > > >>>>>>
> > > > >>>>>> Best Regards,
> > > > >>>>>> Dave Chen
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> -----Original Message-----
> > > > >>>>>> From: Doron Fediuck [mailto:dfediuck at redhat.com]
> > > > >>>>>> Sent: Wednesday, April 10, 2013 8:03 PM
> > > > >>>>>> To: Ofri Masad
> > > > >>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan
> > > > >>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > >>>>>> integration with oVirt in 4/9
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> ----- Original Message -----
> > > > >>>>>>> From: "Ofri Masad" <omasad at redhat.com>
> > > > >>>>>>> To: "Gang Wei" <gang.wei at intel.com>
> > > > >>>>>>> Cc: "Wei D Chen" <wei.d.chen at intel.com>, "Buddy Cao"
> > > > >>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > >>>>>>> <lijuan.zhang at intel.com>, "Doron Fediuck"
> > > > >>>>>>> <dfediuck at redhat.com>
> > > > >>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM
> > > > >>>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > >>>>>>> integration with oVirt in 4/9
> > > > >>>>>>>
> > > > >>>>>>> Hi,
> > > > >>>>>>>
> > > > >>>>>>> answers inside the mail body.
> > > > >>>>>>>
> > > > >>>>>>> ----- Original Message -----
> > > > >>>>>>>> From: "Doron Fediuck" <dfediuck at redhat.com>
> > > > >>>>>>>> To: "Wei D Chen" <wei.d.chen at intel.com>
> > > > >>>>>>>> Cc: "Gang Wei" <gang.wei at intel.com>, "Buddy Cao"
> > > > >>>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > >>>>>>>> <lijuan.zhang at intel.com>, "Ofri Masad" <omasad at redhat.com>
> > > > >>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM
> > > > >>>>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > >>>>>>>> integration with oVirt in 4/9
> > > > >>>>>>>>
> > > > >>>>>>>> Hi Dave,
> > > > >>>>>>>> Adding Ofri to answer your questions.
> > > > >>>>>>>>
> > > > >>>>>>>> ----- Original Message -----
> > > > >>>>>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > >>>>>>>>> To: "Gang Wei" <gang.wei at intel.com>, "Doron Fediuck"
> > > > >>>>>>>>> <dfediuck at redhat.com>,
> > > > >>>>>>>>> "Buddy Cao" <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > >>>>>>>>> <lijuan.zhang at intel.com>
> > > > >>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM
> > > > >>>>>>>>> Subject: RE: minutes for sync up on Open Attestation
> > > > >>>>>>>>> integration with oVirt in 4/9
> > > > >>>>>>>>>
> > > > >>>>>>>>> Hi Doron,
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> iii.    Then periodic trust check will be added into
> > > > >>>>>>>>> background
> > > > >>>>>>>>> process
> > > > >>>>>>>>> for
> > > > >>>>>>>>> all existing hosts, once becoming non-trusted, mark it as
> > > > >>>>>>>>> non-operational.
> > > > >>>>>>>>>
> > > > >>>>>>>>> -          [Dave] Would you give me some details about the
> > > > >>>>>>>>> “background
> > > > >>>>>>>>> process” mentioned in our sync up meeting? What’s the
> > > > >>>>>>>>> process and where is related code?
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> The use Quartz Job to scheduled background processes.
> > > > >>>>>>> An example can be found in:
> > > > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line:
> > > > >>>>>>> 222)
> > > > >>>>>>>
> > > > >>>>>>> This code calls a method that is located in:
> > > > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/o
> > > > >>>>>>> virt /engin e/core/bll/quota/QuotaManager.java
> > > > >>>>>>> (Line: 1000)
> > > > >>>>>>> the @OnTimerMethodAnnotation annotation and the unique job
> > > > >>>>>>> name connects the two in runtime.
> > > > >>>>>>>
> > > > >>>>>>> The job can query the attestation server for all the host
> > > > >>>>>>> and then set untrusted hosts to Non-Operational.
> > > > >>>>>>> the best way would be to call SetNonOperationalVdsCommand
> > > > >>>>>>> with a new NonOperationalReason.
> > > > >>>>>>> This command tries to migrate all VMs from the host and then
> > > > >>>>>>> set it non operational.
> > > > >>>>>>> (We need to think about non-migrational VMs - should we
> > > > >>>>>>> close those VMs or keep the host running)
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> One more thing you may want to consider; If we get a positive
> > > > >>>>>> response for a host which is currently non-operational, issue
> > > > >>>>>> an activate command to try and make it operational again.
> > > > >>>>>> Alternatively, you may decide not to handle it, assuming
> > > > >>>>>> untrusted hosts can only be activated manually (see below
> > > > >>>>>> Ofri's response on activating a host).
> > > > >>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> iv.     If admin fixed the non-operational node and try to
> > > > >>>>>>>>> switch
> > > > >>>>>>>>> it
> > > > >>>>>>>>> to
> > > > >>>>>>>>> operational mode again, a trust check will happen to gate
> > > > >>>>>>>>> the switching.
> > > > >>>>>>>>>
> > > > >>>>>>>>> -          [Dave] If there is a button provided from UI  for
> > > > >>>>>>>>> checking
> > > > >>>>>>>>> the
> > > > >>>>>>>>> logic? or we need add an extra-button for attestation check
> > > > >>>>>>>>> only?
> > > > >>>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> In the UI we already have the "Activate" option. What still
> > > > >>>>>>> needs to be added is the check with the attestation provider
> > > > >>>>>>> to make sure that if the cluster is defined 'Trusted'. If
> > > > >>>>>>> so, only trusted host will succeed in the activate command
> > > > >>>>>>> (and, of course, a proper Audit Log error in case the host
> > > > >>>>>>> failed to activate due to trust issue.
> > > > >>>>>>>
> > > > >>>>>>> the correct place to add this check would be in:
> > > > >>>>>>> ovirt-engine/backend/manager/modules/vdsbroker/src/main/java
> > > > >>>>>>> /org
> > > > >>>>>>> /ovirt
> > > > >>>>>>> /engine/core/vdsbroker/VdsManager.java:activate()
> > > > >>>>>>> this is where the re-activation process begins.
> > > > >>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks very much.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> Best Regards,
> > > > >>>>>>>>> Dave Chen
> > > > >>>>>> _______________________________________________
> > > > >>>>>> Engine-devel mailing list
> > > > >>>>>> Engine-devel at ovirt.org
> > > > >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >>>>>>
> > > > >>>>> _______________________________________________
> > > > >>>>> Engine-devel mailing list
> > > > >>>>> Engine-devel at ovirt.org
> > > > >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >>>>>
> > > > >>>>
> > > > >>>> _______________________________________________
> > > > >>>> Engine-devel mailing list
> > > > >>>> Engine-devel at ovirt.org
> > > > >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >>>>
> > > > >>> _______________________________________________
> > > > >>> Engine-devel mailing list
> > > > >>> Engine-devel at ovirt.org
> > > > >>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >>>
> > > > >> _______________________________________________
> > > > >> Engine-devel mailing list
> > > > >> Engine-devel at ovirt.org
> > > > >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >> _______________________________________________
> > > > >> Engine-devel mailing list
> > > > >> Engine-devel at ovirt.org
> > > > >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >>
> > > > > _______________________________________________
> > > > > Engine-devel mailing list
> > > > > Engine-devel at ovirt.org
> > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > >
> > > > 
> > > > 
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > 
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
>

More information about the Engine-devel mailing list