[Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9

Doron Fediuck dfediuck at redhat.com
Thu Apr 18 08:35:40 UTC 2013 

True, I just want folks to be aware of it.
So this should resolve it all.
Now just make sure to optimize the attestation call.

----- Original Message -----
> From: "Ofri Masad" <omasad at redhat.com>
> To: "Doron Fediuck" <dfediuck at redhat.com>
> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> Sent: Thursday, April 18, 2013 11:31:28 AM
> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9
> 
> We run the query each time the host is moving to UP state.
> Which means, we query all the hosts on engine restart. if the host was
> unreachable or down for any reason - we will query it again before moving to
> UP state.
> 
> Ofri
> 
> ----- Original Message -----
> > From: "Doron Fediuck" <dfediuck at redhat.com>
> > To: "Wei D Chen" <wei.d.chen at intel.com>
> > Cc: "Omer Frenkel" <ofrenkel at redhat.com>, "Ofri Masad" <omasad at redhat.com>,
> > "Oved Ourfalli" <ovedo at redhat.com>,
> > engine-devel at ovirt.org
> > Sent: Thursday, April 18, 2013 11:23:47 AM
> > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > integration with oVirt in 4/9
> > 
> > Generally speaking I agree, we can drop the periodic check is
> > this is the way we expect it to work (ie- change trust level only during
> > reboot).
> > 
> > The only thing I'd like to verify is what happens is we miss something.
> > ie- let's assume the engine crashed. During the engine down time a host
> > reboots and becomes untrusted. Now what? The same goes for network
> > disconnect.
> > I'd expect to run a query in such cases to make sure we do not miss
> > a state change.
> > 
> > 
> > ----- Original Message -----
> > > From: "Wei D Chen" <wei.d.chen at intel.com>
> > > To: "Omer Frenkel" <ofrenkel at redhat.com>, "Ofri Masad"
> > > <omasad at redhat.com>,
> > > "Doron Fediuck" <dfediuck at redhat.com>
> > > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > Sent: Thursday, April 18, 2013 10:42:10 AM
> > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > integration with oVirt in 4/9
> > > 
> > > Yes, the host must be rebooted to take effect. Doron, what do you think?
> > > 
> > > Best Regards,
> > > Dave Chen
> > > 
> > > 
> > > -----Original Message-----
> > > From: Omer Frenkel [mailto:ofrenkel at redhat.com]
> > > Sent: Thursday, April 18, 2013 3:20 PM
> > > To: Ofri Masad
> > > Cc: Chen, Wei D; Oved Ourfalli; engine-devel at ovirt.org
> > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > integration with oVirt in 4/9
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Ofri Masad" <omasad at redhat.com>
> > > > To: "Wei D Chen" <wei.d.chen at intel.com>
> > > > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > Sent: Thursday, April 18, 2013 9:38:26 AM
> > > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > > integration with oVirt in 4/9
> > > > 
> > > > Hi Dave,
> > > > 
> > > > Can't a host become untrusted  without being rebooted?
> > > > If that is really the case, there is no need for a periodic check -
> > > > the trigger for the check would be the host rebooting (which is
> > > > visible to the engine).
> > > > 
> > > 
> > > +1
> > > 
> > > > Thanks,
> > > > Ofri
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > > To: "Itamar Heim" <iheim at redhat.com>, "Ofri Masad"
> > > > > <omasad at redhat.com>
> > > > > Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > > Sent: Thursday, April 18, 2013 9:19:03 AM
> > > > > Subject: RE: [Engine-devel] minutes for sync up on Open Attestation
> > > > > integration with oVirt in 4/9
> > > > > 
> > > > > I think it's more sensible, the initial status should be the real
> > > > > status for this host (trusted / untrusted) only the untrusted host
> > > > > will be set to non-operational.
> > > > > we just need poll this host instead of all of the hosts in the
> > > > > cluster if this can be done in "InitVdsOnUpCommand", and we suppose
> > > > > this is the first time when this host add into the cluster.
> > > > > Follow these steps (conclusion based on your suggestion).
> > > > > 
> > > > > -	The first time when one host is added into a trust cluster then
> > > > > poll
> > > > > this
> > > > > host, the host will be in "up" status if get "trusted" result from
> > > > > attestation server, or else, set this host as non-operational status.
> > > > > -	The periodic check will poll all of the hosts only once as this
> > > > > will
> > > > > cut
> > > > > down a lot of time needed instead of poll each host one by one, this
> > > > > conclusion is based on our experiments because most of time consumed
> > > > > is in parallel.
> > > > > -	When host is down for a different reason and up again, we suppose
> > > > > "InitVdsOnUpCommand" will be triggered again (right?), so we will
> > > > > poll this host again to get the real status(trusted / untrusted).
> > > > > Then mark the host as up or non-operational.
> > > 
> > > right
> > > 
> > > > > 
> > > > > As a trusted host flip to untrusted and take effective only under
> > > > > the condition of this host has been hacked and rebooted, so we
> > > > > proposal no need to add periodic check if any host's reboot will
> > > > > invoke
> > > > > "InitVdsOnUpCommand"
> > > > > and we add attestation logic in "InitVdsOnUpCommand" is enough.
> > > > > 
> > > > > My proposal would be like this (no periodic check is needed, will
> > > > > simply our
> > > > > integration)
> > > > > 
> > > > > -	The first time when one host is added into a trust cluster then
> > > > > poll
> > > > > this
> > > > > host, the host will be in "up" status if get "trusted" result from
> > > > > attestation server, or else, set this host as non-operational status.
> > > > > -	The periodic check will poll all of the hosts only once as this
> > > > > will
> > > > > cut
> > > > > down a lot of time needed instead of poll each host one by one, this
> > > > > conclusion is based on our experiments because most of time consumed
> > > > > is in parallel.
> > > > > -	(up -> nonoperationl / trusted -> untrusted)When host is down for a
> > > > > different reason and up again, we suppose "InitVdsOnUpCommand" will
> > > > > be triggered again (right?) and we also suppose the host will be
> > > > > non-operational if host is down for some reason (right), so we will
> > > > > poll this host again to get the real status(trusted / untrusted)
> > > > > after host's rebooting.
> > > > > -	(nonoperationl -> up / untrusted -> trusted) Admin will activate
> > > > > this
> > > > > host
> > > > > manually after admin fix the issue of this host. Attestation logic
> > > > > will be added into "VdsManager:activate ()" as the clue you give me
> > > > > before.
> > > > > 
> > > > > Any suggestion?
> > > > > 
> > > > > Best Regards,
> > > > > Dave Chen
> > > > > 
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Itamar Heim [mailto:iheim at redhat.com]
> > > > > Sent: Wednesday, April 17, 2013 8:31 PM
> > > > > To: Ofri Masad
> > > > > Cc: Chen, Wei D; Oved Ourfalli; engine-devel at ovirt.org
> > > > > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> > > > > integration with oVirt in 4/9
> > > > > 
> > > > > On 04/17/2013 10:23 AM, Ofri Masad wrote:
> > > > > > Hi Dave,
> > > > > > The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a
> > > > > > good place to call the attestation host.
> > > > > > Instead, like we suggested earlier, create a new Quartz job (like
> > > > > > the one I've sent you in the QuotaManager class) which run every
> > > > > > couple of minutes and update the hosts state.
> > > > > > You can also add the check to InitVdsOnUpCommand, so that if a
> > > > > > host was down for a different reason, it will be rechecked for
> > > > > > attestation.
> > > > > > You can add a UI to allow manual "refresh attestation".
> > > > > >
> > > > > 
> > > > > so the new thread will poll all for all hosts and update the db,
> > > > > then during the normal monitoring "detect" the issue?
> > > > > 
> > > > > > Ofri
> > > > > >
> > > > > > ----- Original Message -----
> > > > > >> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > > >> To: "Omer Frenkel" <ofrenkel at redhat.com>, "Doron Fediuck"
> > > > > >> <dfediuck at redhat.com>
> > > > > >> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > > >> Sent: Monday, April 15, 2013 5:53:34 PM
> > > > > >> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > > >> Attestation integration with oVirt in 4/9
> > > > > >>
> > > > > >> Good approach, thanks all. How long it needs to invoke periodic
> > > > > >> check in the class of VdsUpdateRunTimeInfo? Whether this time
> > > > > >> configurable?
> > > > > >> My concern is if the initial status of each host added into
> > > > > >> trusted cluster is non-operational, we need wait a long time for
> > > > > >> the invoking of this periodic check, so a trusted host's with
> > > > > >> initial status of non-operational will take a long time to flip to
> > > > > >> a
> > > > > >> trusted host.
> > > > > >>
> > > > > >> As I have not seen the configuration of periodic check in
> > > > > >> VdsUpdateRunTimeInfo, would you give me some tips if you has some
> > > > > >> clue.
> > > > > >> Thanks a lot.
> > > > > >>
> > > > > >> Best Regards,
> > > > > >> Dave Chen
> > > > > >>
> > > > > >> -----Original Message-----
> > > > > >> From: engine-devel-bounces at ovirt.org
> > > > > >> [mailto:engine-devel-bounces at ovirt.org]
> > > > > >> On Behalf Of Omer Frenkel
> > > > > >> Sent: Monday, April 15, 2013 7:01 PM
> > > > > >> To: Doron Fediuck
> > > > > >> Cc: Oved Ourfalli; engine-devel at ovirt.org
> > > > > >> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > > >> Attestation integration with oVirt in 4/9
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ----- Original Message -----
> > > > > >>> From: "Doron Fediuck" <dfediuck at redhat.com>
> > > > > >>> To: "Itamar Heim" <iheim at redhat.com>
> > > > > >>> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> > > > > >>> Sent: Monday, April 15, 2013 10:05:57 AM
> > > > > >>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > > >>> Attestation integration with oVirt in 4/9
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>> ----- Original Message -----
> > > > > >>>> From: "Itamar Heim" <iheim at redhat.com>
> > > > > >>>> To: "Oved Ourfalli" <ovedo at redhat.com>
> > > > > >>>> Cc: engine-devel at ovirt.org
> > > > > >>>> Sent: Monday, April 15, 2013 9:49:12 AM
> > > > > >>>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > > >>>> Attestation integration with oVirt in 4/9
> > > > > >>>>
> > > > > >>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote:
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> ----- Original Message -----
> > > > > >>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > > >>>>>> To: "Doron Fediuck" <dfediuck at redhat.com>, "Ofri Masad"
> > > > > >>>>>> <omasad at redhat.com>
> > > > > >>>>>> Cc: engine-devel at ovirt.org
> > > > > >>>>>> Sent: Monday, April 15, 2013 8:54:18 AM
> > > > > >>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open
> > > > > >>>>>> Attestation integration with oVirt in 4/9
> > > > > >>>>>>
> > > > > >>>>>> Hi Doron and Ofri,
> > > > > >>>>>>
> > > > > >>>>>> Thanks for your reply, here is some other question.
> > > > > >>>>>>
> > > > > >>>>>> ii.  When adding a host into the trusted cluster, the host
> > > > > >>>>>> will be attested via OAT service, only trusted hosted can be
> > > > > >>>>>> added.
> > > > > >>>>>>
> > > > > >>>>>> Would you also kindly tell me if there is any mandatory logic
> > > > > >>>>>> check when adding a host into a cluster, so we can also put
> > > > > >>>>>> attestation logic with these mandatory check together? Some
> > > > > >>>>>> example or code location is better.
> > > > > >>>>>> Thanks a lot.
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>> I think there are two approaches, depending on the use case.
> > > > > >>>>> #1:
> > > > > >>>>> If the host trusted property is static, then you can narrow
> > > > > >>>>> the tests to two locations:
> > > > > >>>>> * AddVdsCommand - (Vds = Host) - This command is triggered
> > > > > >>>>> when a new host is added to the system. You can use the
> > > > > >>>>> canDoAction method (which we have on all commands, and it
> > > > > >>>>> determines whether an action can be run or not), and there, if
> > > > > >>>>> cluster requires trusted hosts only, you can test whether this
> > > > > >>>>> host is trusted or not.
> > > > > >>>>> * ChangeVdsClusterCommand - this command is used when you
> > > > > >>>>> change the cluster of a host. So, if the target cluster
> > > > > >>>>> requires tursted hosts, you can do a similar test in its
> > > > > >>>>> canDoAction method.
> > > > > >>>>>
> > > > > >>>>> #2:
> > > > > >>>>> If the host trusted property can change, then I'd suggest
> > > > > >>>>> using the NonOperational property of a host.
> > > > > >>>>> We have a class called VdsUpdateRunTimeInfo that does periodic
> > > > > >>>>> tests of hosts, deciding what's their status, according to
> > > > > >>>>> specific flags.
> > > > > >>>>> The code there is quite complex, and would require refactoring
> > > > > >>>>> at some point, but in the meantime you can use the
> > > > > >>>>> MonitoringStrategy interface that is being used there, and
> > > > > >>>>> implement a new monitoring strategy for Open Attestation.
> > > > > >>>>>
> > > > > >>>>> There, in the processHardwareCapabilities, you can test that
> > > > > >>>>> the host is trusted.
> > > > > >>>>>
> > > > > >>>>> When creating the strategy, using the
> > > > > >>>>> MonitoringStrategyFactory, you'll need to create the
> > > > > >>>>> appropriate
> > > > > >>>>> strategy.
> > > > > >>>>> Currently we support either virt strategy or gluster strategy
> > > > > >>>>> or both. In your case you would need virt+attestation /
> > > > > >>>>> gluster+attestation /
> > > > > >>>>> virt+gluster+attestation, according to the services deployed
> > > > > >>>>> virt+gluster+on the
> > > > > >>>>> cluster.
> > > > > >>>>
> > > > > >>>> I'd go with the 2nd approach. i.e., not block the host from
> > > > > >>>> being added, only from being used, based on monitoring / non-op
> > > > > >>>> status
> > > > > >>>>
> > > > > >>> +1.
> > > > > >>> It means that the admin can add the host, but the host will not
> > > > > >>> be operational until we get a positive indication from the
> > > > > >>> attestation service.
> > > > > >>>
> > > > > >> +1
> > > > > >> if we want to test attestation only once, every time before host
> > > > > >> moves to up.
> > > > > >> the right place for it, imo, is InitVdsOnUpCommand if its
> > > > > >> periodic also after host is up, then it should be somewhere in
> > > > > >> the monitoring code
> > > > > >>
> > > > > >>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> Does that make sense?
> > > > > >>>>> Doron and Ofri, what did you have in mind for this feature?
> > > > > >>>>>
> > > > > >>>>> Regards,
> > > > > >>>>> Oved
> > > > > >>>>>
> > > > > >>>>>>
> > > > > >>>>>> Best Regards,
> > > > > >>>>>> Dave Chen
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> -----Original Message-----
> > > > > >>>>>> From: Doron Fediuck [mailto:dfediuck at redhat.com]
> > > > > >>>>>> Sent: Wednesday, April 10, 2013 8:03 PM
> > > > > >>>>>> To: Ofri Masad
> > > > > >>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan
> > > > > >>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > > >>>>>> integration with oVirt in 4/9
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> ----- Original Message -----
> > > > > >>>>>>> From: "Ofri Masad" <omasad at redhat.com>
> > > > > >>>>>>> To: "Gang Wei" <gang.wei at intel.com>
> > > > > >>>>>>> Cc: "Wei D Chen" <wei.d.chen at intel.com>, "Buddy Cao"
> > > > > >>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > > >>>>>>> <lijuan.zhang at intel.com>, "Doron Fediuck"
> > > > > >>>>>>> <dfediuck at redhat.com>
> > > > > >>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM
> > > > > >>>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > > >>>>>>> integration with oVirt in 4/9
> > > > > >>>>>>>
> > > > > >>>>>>> Hi,
> > > > > >>>>>>>
> > > > > >>>>>>> answers inside the mail body.
> > > > > >>>>>>>
> > > > > >>>>>>> ----- Original Message -----
> > > > > >>>>>>>> From: "Doron Fediuck" <dfediuck at redhat.com>
> > > > > >>>>>>>> To: "Wei D Chen" <wei.d.chen at intel.com>
> > > > > >>>>>>>> Cc: "Gang Wei" <gang.wei at intel.com>, "Buddy Cao"
> > > > > >>>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > > >>>>>>>> <lijuan.zhang at intel.com>, "Ofri Masad" <omasad at redhat.com>
> > > > > >>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM
> > > > > >>>>>>>> Subject: Re: minutes for sync up on Open Attestation
> > > > > >>>>>>>> integration with oVirt in 4/9
> > > > > >>>>>>>>
> > > > > >>>>>>>> Hi Dave,
> > > > > >>>>>>>> Adding Ofri to answer your questions.
> > > > > >>>>>>>>
> > > > > >>>>>>>> ----- Original Message -----
> > > > > >>>>>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> > > > > >>>>>>>>> To: "Gang Wei" <gang.wei at intel.com>, "Doron Fediuck"
> > > > > >>>>>>>>> <dfediuck at redhat.com>,
> > > > > >>>>>>>>> "Buddy Cao" <buddy.cao at intel.com>, "Lijuan Zhang"
> > > > > >>>>>>>>> <lijuan.zhang at intel.com>
> > > > > >>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM
> > > > > >>>>>>>>> Subject: RE: minutes for sync up on Open Attestation
> > > > > >>>>>>>>> integration with oVirt in 4/9
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Hi Doron,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> iii.    Then periodic trust check will be added into
> > > > > >>>>>>>>> background
> > > > > >>>>>>>>> process
> > > > > >>>>>>>>> for
> > > > > >>>>>>>>> all existing hosts, once becoming non-trusted, mark it as
> > > > > >>>>>>>>> non-operational.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> -          [Dave] Would you give me some details about the
> > > > > >>>>>>>>> “background
> > > > > >>>>>>>>> process” mentioned in our sync up meeting? What’s the
> > > > > >>>>>>>>> process and where is related code?
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> The use Quartz Job to scheduled background processes.
> > > > > >>>>>>> An example can be found in:
> > > > > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line:
> > > > > >>>>>>> 222)
> > > > > >>>>>>>
> > > > > >>>>>>> This code calls a method that is located in:
> > > > > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/o
> > > > > >>>>>>> virt /engin e/core/bll/quota/QuotaManager.java
> > > > > >>>>>>> (Line: 1000)
> > > > > >>>>>>> the @OnTimerMethodAnnotation annotation and the unique job
> > > > > >>>>>>> name connects the two in runtime.
> > > > > >>>>>>>
> > > > > >>>>>>> The job can query the attestation server for all the host
> > > > > >>>>>>> and then set untrusted hosts to Non-Operational.
> > > > > >>>>>>> the best way would be to call SetNonOperationalVdsCommand
> > > > > >>>>>>> with a new NonOperationalReason.
> > > > > >>>>>>> This command tries to migrate all VMs from the host and then
> > > > > >>>>>>> set it non operational.
> > > > > >>>>>>> (We need to think about non-migrational VMs - should we
> > > > > >>>>>>> close those VMs or keep the host running)
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>> One more thing you may want to consider; If we get a positive
> > > > > >>>>>> response for a host which is currently non-operational, issue
> > > > > >>>>>> an activate command to try and make it operational again.
> > > > > >>>>>> Alternatively, you may decide not to handle it, assuming
> > > > > >>>>>> untrusted hosts can only be activated manually (see below
> > > > > >>>>>> Ofri's response on activating a host).
> > > > > >>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> iv.     If admin fixed the non-operational node and try to
> > > > > >>>>>>>>> switch
> > > > > >>>>>>>>> it
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>> operational mode again, a trust check will happen to gate
> > > > > >>>>>>>>> the switching.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> -          [Dave] If there is a button provided from UI
> > > > > >>>>>>>>> for
> > > > > >>>>>>>>> checking
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>> logic? or we need add an extra-button for attestation check
> > > > > >>>>>>>>> only?
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> In the UI we already have the "Activate" option. What still
> > > > > >>>>>>> needs to be added is the check with the attestation provider
> > > > > >>>>>>> to make sure that if the cluster is defined 'Trusted'. If
> > > > > >>>>>>> so, only trusted host will succeed in the activate command
> > > > > >>>>>>> (and, of course, a proper Audit Log error in case the host
> > > > > >>>>>>> failed to activate due to trust issue.
> > > > > >>>>>>>
> > > > > >>>>>>> the correct place to add this check would be in:
> > > > > >>>>>>> ovirt-engine/backend/manager/modules/vdsbroker/src/main/java
> > > > > >>>>>>> /org
> > > > > >>>>>>> /ovirt
> > > > > >>>>>>> /engine/core/vdsbroker/VdsManager.java:activate()
> > > > > >>>>>>> this is where the re-activation process begins.
> > > > > >>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Thanks very much.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Best Regards,
> > > > > >>>>>>>>> Dave Chen
> > > > > >>>>>> _______________________________________________
> > > > > >>>>>> Engine-devel mailing list
> > > > > >>>>>> Engine-devel at ovirt.org
> > > > > >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >>>>>>
> > > > > >>>>> _______________________________________________
> > > > > >>>>> Engine-devel mailing list
> > > > > >>>>> Engine-devel at ovirt.org
> > > > > >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>> _______________________________________________
> > > > > >>>> Engine-devel mailing list
> > > > > >>>> Engine-devel at ovirt.org
> > > > > >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >>>>
> > > > > >>> _______________________________________________
> > > > > >>> Engine-devel mailing list
> > > > > >>> Engine-devel at ovirt.org
> > > > > >>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >>>
> > > > > >> _______________________________________________
> > > > > >> Engine-devel mailing list
> > > > > >> Engine-devel at ovirt.org
> > > > > >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >> _______________________________________________
> > > > > >> Engine-devel mailing list
> > > > > >> Engine-devel at ovirt.org
> > > > > >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >>
> > > > > > _______________________________________________
> > > > > > Engine-devel mailing list
> > > > > > Engine-devel at ovirt.org
> > > > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > > >
> > > > > 
> > > > > 
> > > > _______________________________________________
> > > > Engine-devel mailing list
> > > > Engine-devel at ovirt.org
> > > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > > 
> > > _______________________________________________
> > > Engine-devel mailing list
> > > Engine-devel at ovirt.org
> > > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > > 
> > 
> _______________________________________________
> Engine-devel mailing list
> Engine-devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
>

More information about the Devel mailing list