[Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9

Thu Apr 18 06:38:26 UTC 2013

Hi Dave,

Can't a host become untrusted  without being rebooted? 
If that is really the case, there is no need for a periodic check - the trigger for the check would be the host rebooting (which is visible to the engine).

Thanks,
Ofri 

----- Original Message -----
> From: "Wei D Chen" <wei.d.chen at intel.com>
> To: "Itamar Heim" <iheim at redhat.com>, "Ofri Masad" <omasad at redhat.com>
> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> Sent: Thursday, April 18, 2013 9:19:03 AM
> Subject: RE: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt in 4/9
> 
> I think it's more sensible, the initial status should be the real status for
> this host (trusted / untrusted) only the untrusted host will be set to
> non-operational.
> we just need poll this host instead of all of the hosts in the cluster if
> this can be done in "InitVdsOnUpCommand", and we suppose this is the first
> time when this host add into the cluster.
> Follow these steps (conclusion based on your suggestion).
> 
> -	The first time when one host is added into a trust cluster then poll this
> host, the host will be in "up" status if get "trusted" result from
> attestation server, or else, set this host as non-operational status.
> -	The periodic check will poll all of the hosts only once as this will cut
> down a lot of time needed instead of poll each host one by one, this
> conclusion is based on our experiments because most of time consumed is in
> parallel.
> -	When host is down for a different reason and up again, we suppose
> "InitVdsOnUpCommand" will be triggered again (right?), so we will poll this
> host again to get the real status(trusted / untrusted). Then mark the host
> as up or non-operational.
> 
> As a trusted host flip to untrusted and take effective only under the
> condition of this host has been hacked and rebooted, so we proposal no need
> to add periodic check if any host's reboot will invoke "InitVdsOnUpCommand"
> and we add attestation logic in "InitVdsOnUpCommand" is enough.
> 
> My proposal would be like this (no periodic check is needed, will simply our
> integration)
> 
> -	The first time when one host is added into a trust cluster then poll this
> host, the host will be in "up" status if get "trusted" result from
> attestation server, or else, set this host as non-operational status.
> -	The periodic check will poll all of the hosts only once as this will cut
> down a lot of time needed instead of poll each host one by one, this
> conclusion is based on our experiments because most of time consumed is in
> parallel.
> -	(up -> nonoperationl / trusted -> untrusted)When host is down for a
> different reason and up again, we suppose "InitVdsOnUpCommand" will be
> triggered again (right?) and we also suppose the host will be
> non-operational if host is down for some reason (right), so we will poll
> this host again to get the real status(trusted / untrusted) after host's
> rebooting.
> -	(nonoperationl -> up / untrusted -> trusted) Admin will activate this host
> manually after admin fix the issue of this host. Attestation logic will be
> added into "VdsManager:activate ()" as the clue you give me before.
> 
> Any suggestion?
> 
> Best Regards,
> Dave Chen
> 
> 
> -----Original Message-----
> From: Itamar Heim [mailto:iheim at redhat.com]
> Sent: Wednesday, April 17, 2013 8:31 PM
> To: Ofri Masad
> Cc: Chen, Wei D; Oved Ourfalli; engine-devel at ovirt.org
> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> integration with oVirt in 4/9
> 
> On 04/17/2013 10:23 AM, Ofri Masad wrote:
> > Hi Dave,
> > The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a good place
> > to call the attestation host.
> > Instead, like we suggested earlier, create a new Quartz job (like the one
> > I've sent you in the QuotaManager class) which run every couple of minutes
> > and update the hosts state.
> > You can also add the check to InitVdsOnUpCommand, so that if a host was
> > down for a different reason, it will be rechecked for attestation.
> > You can add a UI to allow manual "refresh attestation".
> >
> 
> so the new thread will poll all for all hosts and update the db, then during
> the normal monitoring "detect" the issue?
> 
> > Ofri
> >
> > ----- Original Message -----
> >> From: "Wei D Chen" <wei.d.chen at intel.com>
> >> To: "Omer Frenkel" <ofrenkel at redhat.com>, "Doron Fediuck"
> >> <dfediuck at redhat.com>
> >> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> >> Sent: Monday, April 15, 2013 5:53:34 PM
> >> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> >> integration with oVirt in 4/9
> >>
> >> Good approach, thanks all. How long it needs to invoke periodic check
> >> in the class of VdsUpdateRunTimeInfo? Whether this time configurable?
> >> My concern is if the initial status of each host added into trusted
> >> cluster is non-operational, we need wait a long time for the invoking
> >> of this periodic check, so a trusted host's with initial status of
> >> non-operational will take a long time to flip to a trusted host.
> >>
> >> As I have not seen the configuration of periodic check in
> >> VdsUpdateRunTimeInfo, would you give me some tips if you has some clue.
> >> Thanks a lot.
> >>
> >> Best Regards,
> >> Dave Chen
> >>
> >> -----Original Message-----
> >> From: engine-devel-bounces at ovirt.org
> >> [mailto:engine-devel-bounces at ovirt.org]
> >> On Behalf Of Omer Frenkel
> >> Sent: Monday, April 15, 2013 7:01 PM
> >> To: Doron Fediuck
> >> Cc: Oved Ourfalli; engine-devel at ovirt.org
> >> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> >> integration with oVirt in 4/9
> >>
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Doron Fediuck" <dfediuck at redhat.com>
> >>> To: "Itamar Heim" <iheim at redhat.com>
> >>> Cc: "Oved Ourfalli" <ovedo at redhat.com>, engine-devel at ovirt.org
> >>> Sent: Monday, April 15, 2013 10:05:57 AM
> >>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> >>> integration with oVirt in 4/9
> >>>
> >>>
> >>>
> >>> ----- Original Message -----
> >>>> From: "Itamar Heim" <iheim at redhat.com>
> >>>> To: "Oved Ourfalli" <ovedo at redhat.com>
> >>>> Cc: engine-devel at ovirt.org
> >>>> Sent: Monday, April 15, 2013 9:49:12 AM
> >>>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> >>>> integration with oVirt in 4/9
> >>>>
> >>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote:
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> >>>>>> To: "Doron Fediuck" <dfediuck at redhat.com>, "Ofri Masad"
> >>>>>> <omasad at redhat.com>
> >>>>>> Cc: engine-devel at ovirt.org
> >>>>>> Sent: Monday, April 15, 2013 8:54:18 AM
> >>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open
> >>>>>> Attestation integration with oVirt in 4/9
> >>>>>>
> >>>>>> Hi Doron and Ofri,
> >>>>>>
> >>>>>> Thanks for your reply, here is some other question.
> >>>>>>
> >>>>>> ii.  When adding a host into the trusted cluster, the host will
> >>>>>> be attested via OAT service, only trusted hosted can be added.
> >>>>>>
> >>>>>> Would you also kindly tell me if there is any mandatory logic
> >>>>>> check when adding a host into a cluster, so we can also put
> >>>>>> attestation logic with these mandatory check together? Some
> >>>>>> example or code location is better.
> >>>>>> Thanks a lot.
> >>>>>>
> >>>>>
> >>>>> I think there are two approaches, depending on the use case.
> >>>>> #1:
> >>>>> If the host trusted property is static, then you can narrow the
> >>>>> tests to two locations:
> >>>>> * AddVdsCommand - (Vds = Host) - This command is triggered when a
> >>>>> new host is added to the system. You can use the canDoAction
> >>>>> method (which we have on all commands, and it determines whether
> >>>>> an action can be run or not), and there, if cluster requires
> >>>>> trusted hosts only, you can test whether this host is trusted or
> >>>>> not.
> >>>>> * ChangeVdsClusterCommand - this command is used when you change
> >>>>> the cluster of a host. So, if the target cluster requires tursted
> >>>>> hosts, you can do a similar test in its canDoAction method.
> >>>>>
> >>>>> #2:
> >>>>> If the host trusted property can change, then I'd suggest using
> >>>>> the NonOperational property of a host.
> >>>>> We have a class called VdsUpdateRunTimeInfo that does periodic
> >>>>> tests of hosts, deciding what's their status, according to
> >>>>> specific flags.
> >>>>> The code there is quite complex, and would require refactoring at
> >>>>> some point, but in the meantime you can use the MonitoringStrategy
> >>>>> interface that is being used there, and implement a new monitoring
> >>>>> strategy for Open Attestation.
> >>>>>
> >>>>> There, in the processHardwareCapabilities, you can test that the
> >>>>> host is trusted.
> >>>>>
> >>>>> When creating the strategy, using the MonitoringStrategyFactory,
> >>>>> you'll need to create the appropriate strategy.
> >>>>> Currently we support either virt strategy or gluster strategy or
> >>>>> both. In your case you would need virt+attestation /
> >>>>> gluster+attestation /
> >>>>> virt+gluster+attestation, according to the services deployed on
> >>>>> virt+gluster+the
> >>>>> cluster.
> >>>>
> >>>> I'd go with the 2nd approach. i.e., not block the host from being
> >>>> added, only from being used, based on monitoring / non-op status
> >>>>
> >>> +1.
> >>> It means that the admin can add the host, but the host will not be
> >>> operational until we get a positive indication from the attestation
> >>> service.
> >>>
> >> +1
> >> if we want to test attestation only once, every time before host moves to
> >> up.
> >> the right place for it, imo, is InitVdsOnUpCommand if its periodic
> >> also after host is up, then it should be somewhere in the monitoring
> >> code
> >>
> >>>>
> >>>>>
> >>>>>
> >>>>> Does that make sense?
> >>>>> Doron and Ofri, what did you have in mind for this feature?
> >>>>>
> >>>>> Regards,
> >>>>> Oved
> >>>>>
> >>>>>>
> >>>>>> Best Regards,
> >>>>>> Dave Chen
> >>>>>>
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Doron Fediuck [mailto:dfediuck at redhat.com]
> >>>>>> Sent: Wednesday, April 10, 2013 8:03 PM
> >>>>>> To: Ofri Masad
> >>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan
> >>>>>> Subject: Re: minutes for sync up on Open Attestation integration
> >>>>>> with oVirt in 4/9
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Ofri Masad" <omasad at redhat.com>
> >>>>>>> To: "Gang Wei" <gang.wei at intel.com>
> >>>>>>> Cc: "Wei D Chen" <wei.d.chen at intel.com>, "Buddy Cao"
> >>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang" <lijuan.zhang at intel.com>,
> >>>>>>> "Doron Fediuck" <dfediuck at redhat.com>
> >>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM
> >>>>>>> Subject: Re: minutes for sync up on Open Attestation integration
> >>>>>>> with oVirt in 4/9
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> answers inside the mail body.
> >>>>>>>
> >>>>>>> ----- Original Message -----
> >>>>>>>> From: "Doron Fediuck" <dfediuck at redhat.com>
> >>>>>>>> To: "Wei D Chen" <wei.d.chen at intel.com>
> >>>>>>>> Cc: "Gang Wei" <gang.wei at intel.com>, "Buddy Cao"
> >>>>>>>> <buddy.cao at intel.com>, "Lijuan Zhang" <lijuan.zhang at intel.com>,
> >>>>>>>> "Ofri Masad" <omasad at redhat.com>
> >>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM
> >>>>>>>> Subject: Re: minutes for sync up on Open Attestation
> >>>>>>>> integration with oVirt in 4/9
> >>>>>>>>
> >>>>>>>> Hi Dave,
> >>>>>>>> Adding Ofri to answer your questions.
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>>> From: "Wei D Chen" <wei.d.chen at intel.com>
> >>>>>>>>> To: "Gang Wei" <gang.wei at intel.com>, "Doron Fediuck"
> >>>>>>>>> <dfediuck at redhat.com>,
> >>>>>>>>> "Buddy Cao" <buddy.cao at intel.com>, "Lijuan Zhang"
> >>>>>>>>> <lijuan.zhang at intel.com>
> >>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM
> >>>>>>>>> Subject: RE: minutes for sync up on Open Attestation
> >>>>>>>>> integration with oVirt in 4/9
> >>>>>>>>>
> >>>>>>>>> Hi Doron,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> iii.    Then periodic trust check will be added into background
> >>>>>>>>> process
> >>>>>>>>> for
> >>>>>>>>> all existing hosts, once becoming non-trusted, mark it as
> >>>>>>>>> non-operational.
> >>>>>>>>>
> >>>>>>>>> -          [Dave] Would you give me some details about the
> >>>>>>>>> “background
> >>>>>>>>> process” mentioned in our sync up meeting? What’s the process
> >>>>>>>>> and where is related code?
> >>>>>>>>>
> >>>>>>>
> >>>>>>> The use Quartz Job to scheduled background processes.
> >>>>>>> An example can be found in:
> >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line:
> >>>>>>> 222)
> >>>>>>>
> >>>>>>> This code calls a method that is located in:
> >>>>>>> ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt
> >>>>>>> /engin e/core/bll/quota/QuotaManager.java
> >>>>>>> (Line: 1000)
> >>>>>>> the @OnTimerMethodAnnotation annotation and the unique job name
> >>>>>>> connects the two in runtime.
> >>>>>>>
> >>>>>>> The job can query the attestation server for all the host and
> >>>>>>> then set untrusted hosts to Non-Operational.
> >>>>>>> the best way would be to call SetNonOperationalVdsCommand with a
> >>>>>>> new NonOperationalReason.
> >>>>>>> This command tries to migrate all VMs from the host and then set
> >>>>>>> it non operational.
> >>>>>>> (We need to think about non-migrational VMs - should we close
> >>>>>>> those VMs or keep the host running)
> >>>>>>>
> >>>>>>
> >>>>>> One more thing you may want to consider; If we get a positive
> >>>>>> response for a host which is currently non-operational, issue an
> >>>>>> activate command to try and make it operational again.
> >>>>>> Alternatively, you may decide not to handle it, assuming
> >>>>>> untrusted hosts can only be activated manually (see below Ofri's
> >>>>>> response on activating a host).
> >>>>>>
> >>>>>>>>>
> >>>>>>>>> iv.     If admin fixed the non-operational node and try to switch
> >>>>>>>>> it
> >>>>>>>>> to
> >>>>>>>>> operational mode again, a trust check will happen to gate the
> >>>>>>>>> switching.
> >>>>>>>>>
> >>>>>>>>> -          [Dave] If there is a button provided from UI  for
> >>>>>>>>> checking
> >>>>>>>>> the
> >>>>>>>>> logic? or we need add an extra-button for attestation check only?
> >>>>>>>>>
> >>>>>>>
> >>>>>>> In the UI we already have the "Activate" option. What still
> >>>>>>> needs to be added is the check with the attestation provider to
> >>>>>>> make sure that if the cluster is defined 'Trusted'. If so, only
> >>>>>>> trusted host will succeed in the activate command (and, of
> >>>>>>> course, a proper Audit Log error in case the host failed to
> >>>>>>> activate due to trust issue.
> >>>>>>>
> >>>>>>> the correct place to add this check would be in:
> >>>>>>> ovirt-engine/backend/manager/modules/vdsbroker/src/main/java/org
> >>>>>>> /ovirt
> >>>>>>> /engine/core/vdsbroker/VdsManager.java:activate()
> >>>>>>> this is where the re-activation process begins.
> >>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks very much.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Dave Chen
> >>>>>> _______________________________________________
> >>>>>> Engine-devel mailing list
> >>>>>> Engine-devel at ovirt.org
> >>>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>>>>>
> >>>>> _______________________________________________
> >>>>> Engine-devel mailing list
> >>>>> Engine-devel at ovirt.org
> >>>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>>>>
> >>>>
> >>>> _______________________________________________
> >>>> Engine-devel mailing list
> >>>> Engine-devel at ovirt.org
> >>>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>>>
> >>> _______________________________________________
> >>> Engine-devel mailing list
> >>> Engine-devel at ovirt.org
> >>> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>>
> >> _______________________________________________
> >> Engine-devel mailing list
> >> Engine-devel at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >> _______________________________________________
> >> Engine-devel mailing list
> >> Engine-devel at ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>
> > _______________________________________________
> > Engine-devel mailing list
> > Engine-devel at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 