Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9

From: "Ofri Masad" <omasad(a)redhat.com&gt; To: "Wei D Chen" <wei.d.chen(a)intel.com&gt; Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org Sent: Thursday, April 18, 2013 9:38:26 AM Subject: Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9 Hi Dave, Can't a host become untrusted without being rebooted? If that is really the case, there is no need for a periodic check - the trigger for the check would be the host rebooting (which is visible to the engine).

Thanks, Ofri ----- Original Message ----- > From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; > To: "Itamar Heim" <iheim(a)redhat.com&gt;, "Ofri Masad" > <omasad(a)redhat.com&gt; > Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org > Sent: Thursday, April 18, 2013 9:19:03 AM > Subject: RE: [Engine-devel] minutes for sync up on Open Attestation > integration with oVirt in 4/9 > > I think it's more sensible, the initial status should be the real > status for this host (trusted / untrusted) only the untrusted host > will be set to non-operational. > we just need poll this host instead of all of the hosts in the > cluster if this can be done in "InitVdsOnUpCommand", and we suppose > this is the first time when this host add into the cluster. > Follow these steps (conclusion based on your suggestion). > > - The first time when one host is added into a trust cluster then poll this > host, the host will be in "up" status if get "trusted" result from > attestation server, or else, set this host as non-operational status. > - The periodic check will poll all of the hosts only once as this will cut > down a lot of time needed instead of poll each host one by one, this > conclusion is based on our experiments because most of time consumed > is in parallel. > - When host is down for a different reason and up again, we suppose > "InitVdsOnUpCommand" will be triggered again (right?), so we will > poll this host again to get the real status(trusted / untrusted). > Then mark the host as up or non-operational.

> > As a trusted host flip to untrusted and take effective only under > the condition of this host has been hacked and rebooted, so we > proposal no need to add periodic check if any host's reboot will invoke "InitVdsOnUpCommand" > and we add attestation logic in "InitVdsOnUpCommand" is enough. > > My proposal would be like this (no periodic check is needed, will > simply our > integration) > > - The first time when one host is added into a trust cluster then poll this > host, the host will be in "up" status if get "trusted" result from > attestation server, or else, set this host as non-operational status. > - The periodic check will poll all of the hosts only once as this will cut > down a lot of time needed instead of poll each host one by one, this > conclusion is based on our experiments because most of time consumed > is in parallel. > - (up -> nonoperationl / trusted -> untrusted)When host is down for a > different reason and up again, we suppose "InitVdsOnUpCommand" will > be triggered again (right?) and we also suppose the host will be > non-operational if host is down for some reason (right), so we will > poll this host again to get the real status(trusted / untrusted) > after host's rebooting. > - (nonoperationl -> up / untrusted -> trusted) Admin will activate this > host > manually after admin fix the issue of this host. Attestation logic > will be added into "VdsManager:activate ()" as the clue you give me before. > > Any suggestion? > > Best Regards, > Dave Chen > > > -----Original Message----- > From: Itamar Heim [mailto:iheim@redhat.com] > Sent: Wednesday, April 17, 2013 8:31 PM > To: Ofri Masad > Cc: Chen, Wei D; Oved Ourfalli; engine-devel(a)ovirt.org > Subject: Re: [Engine-devel] minutes for sync up on Open Attestation > integration with oVirt in 4/9 > > On 04/17/2013 10:23 AM, Ofri Masad wrote: > > Hi Dave, > > The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a > > good place to call the attestation host. > > Instead, like we suggested earlier, create a new Quartz job (like > > the one I've sent you in the QuotaManager class) which run every > > couple of minutes and update the hosts state. > > You can also add the check to InitVdsOnUpCommand, so that if a > > host was down for a different reason, it will be rechecked for attestation. > > You can add a UI to allow manual "refresh attestation". > > > > so the new thread will poll all for all hosts and update the db, > then during the normal monitoring "detect" the issue? > > > Ofri > > > > ----- Original Message ----- > >> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; > >> To: "Omer Frenkel" <ofrenkel(a)redhat.com&gt;, "Doron Fediuck" > >> <dfediuck(a)redhat.com&gt; > >> Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org > >> Sent: Monday, April 15, 2013 5:53:34 PM > >> Subject: Re: [Engine-devel] minutes for sync up on Open > >> Attestation integration with oVirt in 4/9 > >> > >> Good approach, thanks all. How long it needs to invoke periodic > >> check in the class of VdsUpdateRunTimeInfo? Whether this time configurable? > >> My concern is if the initial status of each host added into > >> trusted cluster is non-operational, we need wait a long time for > >> the invoking of this periodic check, so a trusted host's with > >> initial status of non-operational will take a long time to flip to a trusted host. > >> > >> As I have not seen the configuration of periodic check in > >> VdsUpdateRunTimeInfo, would you give me some tips if you has some clue. > >> Thanks a lot. > >> > >> Best Regards, > >> Dave Chen > >> > >> -----Original Message----- > >> From: engine-devel-bounces(a)ovirt.org > >> [mailto:engine-devel-bounces@ovirt.org] > >> On Behalf Of Omer Frenkel > >> Sent: Monday, April 15, 2013 7:01 PM > >> To: Doron Fediuck > >> Cc: Oved Ourfalli; engine-devel(a)ovirt.org > >> Subject: Re: [Engine-devel] minutes for sync up on Open > >> Attestation integration with oVirt in 4/9 > >> > >> > >> > >> ----- Original Message ----- > >>> From: "Doron Fediuck" <dfediuck(a)redhat.com&gt; > >>> To: "Itamar Heim" <iheim(a)redhat.com&gt; > >>> Cc: "Oved Ourfalli" <ovedo(a)redhat.com&gt;, engine-devel(a)ovirt.org > >>> Sent: Monday, April 15, 2013 10:05:57 AM > >>> Subject: Re: [Engine-devel] minutes for sync up on Open > >>> Attestation integration with oVirt in 4/9 > >>> > >>> > >>> > >>> ----- Original Message ----- > >>>> From: "Itamar Heim" <iheim(a)redhat.com&gt; > >>>> To: "Oved Ourfalli" <ovedo(a)redhat.com&gt; > >>>> Cc: engine-devel(a)ovirt.org > >>>> Sent: Monday, April 15, 2013 9:49:12 AM > >>>> Subject: Re: [Engine-devel] minutes for sync up on Open > >>>> Attestation integration with oVirt in 4/9 > >>>> > >>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote: > >>>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; > >>>>>> To: "Doron Fediuck" <dfediuck(a)redhat.com&gt;, "Ofri Masad" > >>>>>> <omasad(a)redhat.com&gt; > >>>>>> Cc: engine-devel(a)ovirt.org > >>>>>> Sent: Monday, April 15, 2013 8:54:18 AM > >>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open > >>>>>> Attestation integration with oVirt in 4/9 > >>>>>> > >>>>>> Hi Doron and Ofri, > >>>>>> > >>>>>> Thanks for your reply, here is some other question. > >>>>>> > >>>>>> ii. When adding a host into the trusted cluster, the host > >>>>>> will be attested via OAT service, only trusted hosted can be added. > >>>>>> > >>>>>> Would you also kindly tell me if there is any mandatory logic > >>>>>> check when adding a host into a cluster, so we can also put > >>>>>> attestation logic with these mandatory check together? Some > >>>>>> example or code location is better. > >>>>>> Thanks a lot. > >>>>>> > >>>>> > >>>>> I think there are two approaches, depending on the use case. > >>>>> #1: > >>>>> If the host trusted property is static, then you can narrow > >>>>> the tests to two locations: > >>>>> * AddVdsCommand - (Vds = Host) - This command is triggered > >>>>> when a new host is added to the system. You can use the > >>>>> canDoAction method (which we have on all commands, and it > >>>>> determines whether an action can be run or not), and there, if > >>>>> cluster requires trusted hosts only, you can test whether this > >>>>> host is trusted or not. > >>>>> * ChangeVdsClusterCommand - this command is used when you > >>>>> change the cluster of a host. So, if the target cluster > >>>>> requires tursted hosts, you can do a similar test in its canDoAction method. > >>>>> > >>>>> #2: > >>>>> If the host trusted property can change, then I'd suggest > >>>>> using the NonOperational property of a host. > >>>>> We have a class called VdsUpdateRunTimeInfo that does periodic > >>>>> tests of hosts, deciding what's their status, according to > >>>>> specific flags. > >>>>> The code there is quite complex, and would require refactoring > >>>>> at some point, but in the meantime you can use the > >>>>> MonitoringStrategy interface that is being used there, and > >>>>> implement a new monitoring strategy for Open Attestation. > >>>>> > >>>>> There, in the processHardwareCapabilities, you can test that > >>>>> the host is trusted. > >>>>> > >>>>> When creating the strategy, using the > >>>>> MonitoringStrategyFactory, you'll need to create the appropriate strategy. > >>>>> Currently we support either virt strategy or gluster strategy > >>>>> or both. In your case you would need virt+attestation / > >>>>> gluster+attestation / > >>>>> virt+gluster+attestation, according to the services deployed > >>>>> virt+gluster+on the > >>>>> cluster. > >>>> > >>>> I'd go with the 2nd approach. i.e., not block the host from > >>>> being added, only from being used, based on monitoring / non-op > >>>> status > >>>> > >>> +1. > >>> It means that the admin can add the host, but the host will not > >>> be operational until we get a positive indication from the > >>> attestation service. > >>> > >> +1 > >> if we want to test attestation only once, every time before host > >> moves to up. > >> the right place for it, imo, is InitVdsOnUpCommand if its > >> periodic also after host is up, then it should be somewhere in > >> the monitoring code > >> > >>>> > >>>>> > >>>>> > >>>>> Does that make sense? > >>>>> Doron and Ofri, what did you have in mind for this feature? > >>>>> > >>>>> Regards, > >>>>> Oved > >>>>> > >>>>>> > >>>>>> Best Regards, > >>>>>> Dave Chen > >>>>>> > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: Doron Fediuck [mailto:dfediuck@redhat.com] > >>>>>> Sent: Wednesday, April 10, 2013 8:03 PM > >>>>>> To: Ofri Masad > >>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan > >>>>>> Subject: Re: minutes for sync up on Open Attestation > >>>>>> integration with oVirt in 4/9 > >>>>>> > >>>>>> > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Ofri Masad" <omasad(a)redhat.com&gt; > >>>>>>> To: "Gang Wei" <gang.wei(a)intel.com&gt; > >>>>>>> Cc: "Wei D Chen" <wei.d.chen(a)intel.com&gt;, "Buddy Cao" > >>>>>>> <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" > >>>>>>> <lijuan.zhang(a)intel.com&gt;, "Doron Fediuck" > >>>>>>> <dfediuck(a)redhat.com&gt; > >>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM > >>>>>>> Subject: Re: minutes for sync up on Open Attestation > >>>>>>> integration with oVirt in 4/9 > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> answers inside the mail body. > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>>> From: "Doron Fediuck" <dfediuck(a)redhat.com&gt; > >>>>>>>> To: "Wei D Chen" <wei.d.chen(a)intel.com&gt; > >>>>>>>> Cc: "Gang Wei" <gang.wei(a)intel.com&gt;, "Buddy Cao" > >>>>>>>> <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" > >>>>>>>> <lijuan.zhang(a)intel.com&gt;, "Ofri Masad" <omasad(a)redhat.com&gt; > >>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM > >>>>>>>> Subject: Re: minutes for sync up on Open Attestation > >>>>>>>> integration with oVirt in 4/9 > >>>>>>>> > >>>>>>>> Hi Dave, > >>>>>>>> Adding Ofri to answer your questions. > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>>> From: "Wei D Chen" <wei.d.chen(a)intel.com&gt; > >>>>>>>>> To: "Gang Wei" <gang.wei(a)intel.com&gt;, "Doron Fediuck" > >>>>>>>>> <dfediuck(a)redhat.com&gt;, > >>>>>>>>> "Buddy Cao" <buddy.cao(a)intel.com&gt;, "Lijuan Zhang" > >>>>>>>>> <lijuan.zhang(a)intel.com&gt; > >>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM > >>>>>>>>> Subject: RE: minutes for sync up on Open Attestation > >>>>>>>>> integration with oVirt in 4/9 > >>>>>>>>> > >>>>>>>>> Hi Doron, > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> iii. Then periodic trust check will be added into background > >>>>>>>>> process > >>>>>>>>> for > >>>>>>>>> all existing hosts, once becoming non-trusted, mark it as > >>>>>>>>> non-operational. > >>>>>>>>> > >>>>>>>>> - [Dave] Would you give me some details about the > >>>>>>>>> “background > >>>>>>>>> process” mentioned in our sync up meeting? What’s the > >>>>>>>>> process and where is related code? > >>>>>>>>> > >>>>>>> > >>>>>>> The use Quartz Job to scheduled background processes. > >>>>>>> An example can be found in: > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line: > >>>>>>> 222) > >>>>>>> > >>>>>>> This code calls a method that is located in: > >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/o > >>>>>>> virt /engin e/core/bll/quota/QuotaManager.java > >>>>>>> (Line: 1000) > >>>>>>> the @OnTimerMethodAnnotation annotation and the unique job > >>>>>>> name connects the two in runtime. > >>>>>>> > >>>>>>> The job can query the attestation server for all the host > >>>>>>> and then set untrusted hosts to Non-Operational. > >>>>>>> the best way would be to call SetNonOperationalVdsCommand > >>>>>>> with a new NonOperationalReason. > >>>>>>> This command tries to migrate all VMs from the host and then > >>>>>>> set it non operational. > >>>>>>> (We need to think about non-migrational VMs - should we > >>>>>>> close those VMs or keep the host running) > >>>>>>> > >>>>>> > >>>>>> One more thing you may want to consider; If we get a positive > >>>>>> response for a host which is currently non-operational, issue > >>>>>> an activate command to try and make it operational again. > >>>>>> Alternatively, you may decide not to handle it, assuming > >>>>>> untrusted hosts can only be activated manually (see below > >>>>>> Ofri's response on activating a host). > >>>>>> > >>>>>>>>> > >>>>>>>>> iv. If admin fixed the non-operational node and try to switch > >>>>>>>>> it > >>>>>>>>> to > >>>>>>>>> operational mode again, a trust check will happen to gate > >>>>>>>>> the switching. > >>>>>>>>> > >>>>>>>>> - [Dave] If there is a button provided from UI for > >>>>>>>>> checking > >>>>>>>>> the > >>>>>>>>> logic? or we need add an extra-button for attestation check only? > >>>>>>>>> > >>>>>>> > >>>>>>> In the UI we already have the "Activate" option. What still > >>>>>>> needs to be added is the check with the attestation provider > >>>>>>> to make sure that if the cluster is defined 'Trusted'. If > >>>>>>> so, only trusted host will succeed in the activate command > >>>>>>> (and, of course, a proper Audit Log error in case the host > >>>>>>> failed to activate due to trust issue. > >>>>>>> > >>>>>>> the correct place to add this check would be in: > >>>>>>> ovirt-engine/backend/manager/modules/vdsbroker/src/main/java > >>>>>>> /org > >>>>>>> /ovirt > >>>>>>> /engine/core/vdsbroker/VdsManager.java:activate() > >>>>>>> this is where the re-activation process begins. > >>>>>>> > >>>>>>>>> > >>>>>>>>> Thanks very much. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Best Regards, > >>>>>>>>> Dave Chen > >>>>>> _______________________________________________ > >>>>>> Engine-devel mailing list > >>>>>> Engine-devel(a)ovirt.org > >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>>>>> > >>>>> _______________________________________________ > >>>>> Engine-devel mailing list > >>>>> Engine-devel(a)ovirt.org > >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>>>> > >>>> > >>>> _______________________________________________ > >>>> Engine-devel mailing list > >>>> Engine-devel(a)ovirt.org > >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>>> > >>> _______________________________________________ > >>> Engine-devel mailing list > >>> Engine-devel(a)ovirt.org > >>> http://lists.ovirt.org/mailman/listinfo/engine-devel > >>> > >> _______________________________________________ > >> Engine-devel mailing list > >> Engine-devel(a)ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/engine-devel > >> _______________________________________________ > >> Engine-devel mailing list > >> Engine-devel(a)ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/engine-devel > >> > > _______________________________________________ > > Engine-devel mailing list > > Engine-devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/engine-devel > > > > _______________________________________________ Engine-devel mailing list Engine-devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/engine-devel