Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9

From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; To: "Itamar Heim" <iheim(a)redhat.com&gt;, "Ofri Masad" <omasad(a)redhat.com&gt; Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org Sent: Thursday, April 18, 2013 9:19:03 AM Subject: RE: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9 I think it's more sensible, the initial status should be the real status for this host (trusted / untrusted) only the untrusted host will be set to non-operational. we just need poll this host instead of all of the hosts in the cluster if this can be done in "InitVdsOnUpCommand", and we suppose this is the first time when this host add into the cluster. Follow these steps (conclusion based on your suggestion). - The first time when one host is added into a trust cluster then poll this host, the host will be in "up" status if get "trusted" result from attestation server, or else, set this host as non-operational status. - The periodic check will poll all of the hosts only once as this will cut down a lot of time needed instead of poll each host one by one, this conclusion is based on our experiments because most of time consumed is in parallel. - When host is down for a different reason and up again, we suppose "InitVdsOnUpCommand" will be triggered again (right?), so we will poll this host again to get the real status(trusted / untrusted). Then mark the host as up or non-operational. As a trusted host flip to untrusted and take effective only under the condition of this host has been hacked and rebooted, so we proposal no need to add periodic check if any host's reboot will invoke "InitVdsOnUpCommand" and we add attestation logic in "InitVdsOnUpCommand" is enough. My proposal would be like this (no periodic check is needed, will simply our integration) - The first time when one host is added into a trust cluster then poll this host, the host will be in "up" status if get "trusted" result from attestation server, or else, set this host as non-operational status. - The periodic check will poll all of the hosts only once as this will cut down a lot of time needed instead of poll each host one by one, this conclusion is based on our experiments because most of time consumed is in parallel. - (up -> nonoperationl / trusted -> untrusted)When host is down for a different reason and up again, we suppose "InitVdsOnUpCommand" will be triggered again (right?) and we also suppose the host will be non-operational if host is down for some reason (right), so we will poll this host again to get the real status(trusted / untrusted) after host's rebooting. - (nonoperationl -> up / untrusted -> trusted) Admin will activate this host manually after admin fix the issue of this host. Attestation logic will be added into "VdsManager:activate ()" as the clue you give me before. Any suggestion? Best Regards, Dave Chen -----Original Message----- From: Itamar Heim [mailto:iheim@redhat.com] Sent: Wednesday, April 17, 2013 8:31 PM To: Ofri Masad Cc: Chen, Wei D; Oved Ourfalli; engine-devel(a)ovirt.org Subject: Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9 On 04/17/2013 10:23 AM, Ofri Masad wrote: > Hi Dave, > The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a good place > to call the attestation host. > Instead, like we suggested earlier, create a new Quartz job (like the one > I've sent you in the QuotaManager class) which run every couple of minutes > and update the hosts state. > You can also add the check to InitVdsOnUpCommand, so that if a host was > down for a different reason, it will be rechecked for attestation. > You can add a UI to allow manual "refresh attestation". > so the new thread will poll all for all hosts and update the db, then during the normal monitoring "detect" the issue? > Ofri > > ----- Original Message ----- >> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; >> To: "Omer Frenkel" <ofrenkel(a)redhat.com&gt;, "Doron Fediuck" >> <dfediuck(a)redhat.com&gt; >> Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org >> Sent: Monday, April 15, 2013 5:53:34 PM >> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation >> integration with oVirt in 4/9 >> >> Good approach, thanks all. How long it needs to invoke periodic check >> in the class of VdsUpdateRunTimeInfo? Whether this time configurable? >> My concern is if the initial status of each host added into trusted >> cluster is non-operational, we need wait a long time for the invoking >> of this periodic check, so a trusted host's with initial status of >> non-operational will take a long time to flip to a trusted host. >> >> As I have not seen the configuration of periodic check in >> VdsUpdateRunTimeInfo, would you give me some tips if you has some clue. >> Thanks a lot. >> >> Best Regards, >> Dave Chen >> >> -----Original Message----- >> From: engine-devel-bounces(a)ovirt.org >> [mailto:engine-devel-bounces@ovirt.org] >> On Behalf Of Omer Frenkel >> Sent: Monday, April 15, 2013 7:01 PM >> To: Doron Fediuck >> Cc: Oved Ourfalli; engine-devel(a)ovirt.org >> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation >> integration with oVirt in 4/9 >> >> >> >> ----- Original Message ----- >>> From: "Doron Fediuck" <dfediuck(a)redhat.com&gt; >>> To: "Itamar Heim" <iheim(a)redhat.com&gt; >>> Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org >>> Sent: Monday, April 15, 2013 10:05:57 AM >>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation >>> integration with oVirt in 4/9 >>> >>> >>> >>> ----- Original Message ----- >>>> From: "Itamar Heim" <iheim(a)redhat.com&gt; >>>> To: "Oved Ourfalli" <ovedo(a)redhat.com&gt; >>>> Cc: engine-devel(a)ovirt.org >>>> Sent: Monday, April 15, 2013 9:49:12 AM >>>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation >>>> integration with oVirt in 4/9 >>>> >>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote: >>>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; >>>>>> To: "Doron Fediuck" <dfediuck(a)redhat.com&gt;, "Ofri Masad" >>>>>> <omasad(a)redhat.com&gt; >>>>>> Cc: engine-devel(a)ovirt.org >>>>>> Sent: Monday, April 15, 2013 8:54:18 AM >>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open >>>>>> Attestation integration with oVirt in 4/9 >>>>>> >>>>>> Hi Doron and Ofri, >>>>>> >>>>>> Thanks for your reply, here is some other question. >>>>>> >>>>>> ii. When adding a host into the trusted cluster, the host will >>>>>> be attested via OAT service, only trusted hosted can be added. >>>>>> >>>>>> Would you also kindly tell me if there is any mandatory logic >>>>>> check when adding a host into a cluster, so we can also put >>>>>> attestation logic with these mandatory check together? Some >>>>>> example or code location is better. >>>>>> Thanks a lot. >>>>>> >>>>> >>>>> I think there are two approaches, depending on the use case. >>>>> #1: >>>>> If the host trusted property is static, then you can narrow the >>>>> tests to two locations: >>>>> * AddVdsCommand - (Vds = Host) - This command is triggered when a >>>>> new host is added to the system. You can use the canDoAction >>>>> method (which we have on all commands, and it determines whether >>>>> an action can be run or not), and there, if cluster requires >>>>> trusted hosts only, you can test whether this host is trusted or >>>>> not. >>>>> * ChangeVdsClusterCommand - this command is used when you change >>>>> the cluster of a host. So, if the target cluster requires tursted >>>>> hosts, you can do a similar test in its canDoAction method. >>>>> >>>>> #2: >>>>> If the host trusted property can change, then I'd suggest using >>>>> the NonOperational property of a host. >>>>> We have a class called VdsUpdateRunTimeInfo that does periodic >>>>> tests of hosts, deciding what's their status, according to >>>>> specific flags. >>>>> The code there is quite complex, and would require refactoring at >>>>> some point, but in the meantime you can use the MonitoringStrategy >>>>> interface that is being used there, and implement a new monitoring >>>>> strategy for Open Attestation. >>>>> >>>>> There, in the processHardwareCapabilities, you can test that the >>>>> host is trusted. >>>>> >>>>> When creating the strategy, using the MonitoringStrategyFactory, >>>>> you'll need to create the appropriate strategy. >>>>> Currently we support either virt strategy or gluster strategy or >>>>> both. In your case you would need virt+attestation / >>>>> gluster+attestation / >>>>> virt+gluster+attestation, according to the services deployed on >>>>> virt+gluster+the >>>>> cluster. >>>> >>>> I'd go with the 2nd approach. i.e., not block the host from being >>>> added, only from being used, based on monitoring / non-op status >>>> >>> +1. >>> It means that the admin can add the host, but the host will not be >>> operational until we get a positive indication from the attestation >>> service. >>> >> +1 >> if we want to test attestation only once, every time before host moves to >> up. >> the right place for it, imo, is InitVdsOnUpCommand if its periodic >> also after host is up, then it should be somewhere in the monitoring >> code >> >>>> >>>>> >>>>> >>>>> Does that make sense? >>>>> Doron and Ofri, what did you have in mind for this feature? >>>>> >>>>> Regards, >>>>> Oved >>>>> >>>>>> >>>>>> Best Regards, >>>>>> Dave Chen >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Doron Fediuck [mailto:dfediuck@redhat.com] >>>>>> Sent: Wednesday, April 10, 2013 8:03 PM >>>>>> To: Ofri Masad >>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan >>>>>> Subject: Re: minutes for sync up on Open Attestation integration >>>>>> with oVirt in 4/9 >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Ofri Masad" <omasad(a)redhat.com&gt; >>>>>>> To: "Gang Wei" <gang.wei(a)intel.com&gt; >>>>>>> Cc: "Wei D Chen" <wei.d.chen(a)intel.com&gt;, "Buddy Cao" >>>>>>> <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" <lijuan.zhang(a)intel.com&gt;, >>>>>>> "Doron Fediuck" <dfediuck(a)redhat.com&gt; >>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM >>>>>>> Subject: Re: minutes for sync up on Open Attestation integration >>>>>>> with oVirt in 4/9 >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> answers inside the mail body. >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Doron Fediuck" <dfediuck(a)redhat.com&gt; >>>>>>>> To: "Wei D Chen" <wei.d.chen(a)intel.com&gt; >>>>>>>> Cc: "Gang Wei" <gang.wei(a)intel.com&gt;, "Buddy Cao" >>>>>>>> <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" <lijuan.zhang(a)intel.com&gt;, >>>>>>>> "Ofri Masad" <omasad(a)redhat.com&gt; >>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM >>>>>>>> Subject: Re: minutes for sync up on Open Attestation >>>>>>>> integration with oVirt in 4/9 >>>>>>>> >>>>>>>> Hi Dave, >>>>>>>> Adding Ofri to answer your questions. >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; >>>>>>>>> To: "Gang Wei" <gang.wei(a)intel.com&gt;, "Doron Fediuck" >>>>>>>>> <dfediuck(a)redhat.com&gt;, >>>>>>>>> "Buddy Cao" <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" >>>>>>>>> <lijuan.zhang(a)intel.com&gt; >>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM >>>>>>>>> Subject: RE: minutes for sync up on Open Attestation >>>>>>>>> integration with oVirt in 4/9 >>>>>>>>> >>>>>>>>> Hi Doron, >>>>>>>>> >>>>>>>>> >>>>>>>>> iii. Then periodic trust check will be added into background >>>>>>>>> process >>>>>>>>> for >>>>>>>>> all existing hosts, once becoming non-trusted, mark it as >>>>>>>>> non-operational. >>>>>>>>> >>>>>>>>> - [Dave] Would you give me some details about the >>>>>>>>> “background >>>>>>>>> process” mentioned in our sync up meeting? What’s the process >>>>>>>>> and where is related code? >>>>>>>>> >>>>>>> >>>>>>> The use Quartz Job to scheduled background processes. >>>>>>> An example can be found in: >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line: >>>>>>> 222) >>>>>>> >>>>>>> This code calls a method that is located in: >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt >>>>>>> /engin e/core/bll/quota/QuotaManager.java >>>>>>> (Line: 1000) >>>>>>> the @OnTimerMethodAnnotation annotation and the unique job name >>>>>>> connects the two in runtime. >>>>>>> >>>>>>> The job can query the attestation server for all the host and >>>>>>> then set untrusted hosts to Non-Operational. >>>>>>> the best way would be to call SetNonOperationalVdsCommand with a >>>>>>> new NonOperationalReason. >>>>>>> This command tries to migrate all VMs from the host and then set >>>>>>> it non operational. >>>>>>> (We need to think about non-migrational VMs - should we close >>>>>>> those VMs or keep the host running) >>>>>>> >>>>>> >>>>>> One more thing you may want to consider; If we get a positive >>>>>> response for a host which is currently non-operational, issue an >>>>>> activate command to try and make it operational again. >>>>>> Alternatively, you may decide not to handle it, assuming >>>>>> untrusted hosts can only be activated manually (see below Ofri's >>>>>> response on activating a host). >>>>>> >>>>>>>>> >>>>>>>>> iv. If admin fixed the non-operational node and try to switch >>>>>>>>> it >>>>>>>>> to >>>>>>>>> operational mode again, a trust check will happen to gate the >>>>>>>>> switching. >>>>>>>>> >>>>>>>>> - [Dave] If there is a button provided from UI for >>>>>>>>> checking >>>>>>>>> the >>>>>>>>> logic? or we need add an extra-button for attestation check only? >>>>>>>>> >>>>>>> >>>>>>> In the UI we already have the "Activate" option. What still >>>>>>> needs to be added is the check with the attestation provider to >>>>>>> make sure that if the cluster is defined 'Trusted'. If so, only >>>>>>> trusted host will succeed in the activate command (and, of >>>>>>> course, a proper Audit Log error in case the host failed to >>>>>>> activate due to trust issue. >>>>>>> >>>>>>> the correct place to add this check would be in: >>>>>>> ovirt-engine/backend/manager/modules/vdsbroker/src/main/java/org >>>>>>> /ovirt >>>>>>> /engine/core/vdsbroker/VdsManager.java:activate() >>>>>>> this is where the re-activation process begins. >>>>>>> >>>>>>>>> >>>>>>>>> Thanks very much. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Best Regards, >>>>>>>>> Dave Chen >>>>>> _______________________________________________ >>>>>> Engine-devel mailing list >>>>>> Engine-devel(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>>>> >>>>> _______________________________________________ >>>>> Engine-devel mailing list >>>>> Engine-devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>>> >>>> >>>> _______________________________________________ >>>> Engine-devel mailing list >>>> Engine-devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>>> >>> _______________________________________________ >>> Engine-devel mailing list >>> Engine-devel(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/engine-devel >>> >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> _______________________________________________ >> Engine-devel mailing list >> Engine-devel(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/engine-devel >> > _______________________________________________ > Engine-devel mailing list > Engine-devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/engine-devel >