I think it's more sensible, the initial status should be the real status for this host
(trusted / untrusted) only the untrusted host will be set to non-operational.
we just need poll this host instead of all of the hosts in the cluster if this can be done
in "InitVdsOnUpCommand", and we suppose this is the first time when this host
add into the cluster.
Follow these steps (conclusion based on your suggestion).
- The first time when one host is added into a trust cluster then poll this host, the host
will be in "up" status if get "trusted" result from attestation
server, or else, set this host as non-operational status.
- The periodic check will poll all of the hosts only once as this will cut down a lot of
time needed instead of poll each host one by one, this conclusion is based on our
experiments because most of time consumed is in parallel.
- When host is down for a different reason and up again, we suppose
"InitVdsOnUpCommand" will be triggered again (right?), so we will poll this host
again to get the real status(trusted / untrusted). Then mark the host as up or
non-operational.
As a trusted host flip to untrusted and take effective only under the condition of this
host has been hacked and rebooted, so we proposal no need to add periodic check if any
host's reboot will invoke "InitVdsOnUpCommand" and we add attestation logic
in "InitVdsOnUpCommand" is enough.
My proposal would be like this (no periodic check is needed, will simply our
integration)
- The first time when one host is added into a trust cluster then poll this host, the host
will be in "up" status if get "trusted" result from attestation
server, or else, set this host as non-operational status.
- The periodic check will poll all of the hosts only once as this will cut down a lot of
time needed instead of poll each host one by one, this conclusion is based on our
experiments because most of time consumed is in parallel.
- (up -> nonoperationl / trusted -> untrusted)When host is down for a different
reason and up again, we suppose "InitVdsOnUpCommand" will be triggered again
(right?) and we also suppose the host will be non-operational if host is down for some
reason (right), so we will poll this host again to get the real status(trusted /
untrusted) after host's rebooting.
- (nonoperationl -> up / untrusted -> trusted) Admin will activate this host
manually after admin fix the issue of this host. Attestation logic will be added into
"VdsManager:activate ()" as the clue you give me before.
Any suggestion?
Best Regards,
Dave Chen
-----Original Message-----
From: Itamar Heim [mailto:iheim@redhat.com]
Sent: Wednesday, April 17, 2013 8:31 PM
To: Ofri Masad
Cc: Chen, Wei D; Oved Ourfalli; engine-devel(a)ovirt.org
Subject: Re: [Engine-devel] minutes for sync up on Open Attestation integration with oVirt
in 4/9
On 04/17/2013 10:23 AM, Ofri Masad wrote:
Hi Dave,
The VdsUpdateRunTimeInfo runs every 3 seconds or so. so it not a good place to call the
attestation host.
Instead, like we suggested earlier, create a new Quartz job (like the one I've sent
you in the QuotaManager class) which run every couple of minutes and update the hosts
state.
You can also add the check to InitVdsOnUpCommand, so that if a host was down for a
different reason, it will be rechecked for attestation.
You can add a UI to allow manual "refresh attestation".
so the new thread will poll all for all hosts and update the db, then during the normal
monitoring "detect" the issue?
Ofri
----- Original Message -----
> From: "Wei D Chen" <wei.d.chen(a)intel.com>
> To: "Omer Frenkel" <ofrenkel(a)redhat.com>, "Doron Fediuck"
> <dfediuck(a)redhat.com>
> Cc: "Oved Ourfalli" <ovedo(a)redhat.com>, engine-devel(a)ovirt.org
> Sent: Monday, April 15, 2013 5:53:34 PM
> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> integration with oVirt in 4/9
>
> Good approach, thanks all. How long it needs to invoke periodic check
> in the class of VdsUpdateRunTimeInfo? Whether this time configurable?
> My concern is if the initial status of each host added into trusted
> cluster is non-operational, we need wait a long time for the invoking
> of this periodic check, so a trusted host's with initial status of
> non-operational will take a long time to flip to a trusted host.
>
> As I have not seen the configuration of periodic check in
> VdsUpdateRunTimeInfo, would you give me some tips if you has some clue.
> Thanks a lot.
>
> Best Regards,
> Dave Chen
>
> -----Original Message-----
> From: engine-devel-bounces(a)ovirt.org
> [mailto:engine-devel-bounces@ovirt.org]
> On Behalf Of Omer Frenkel
> Sent: Monday, April 15, 2013 7:01 PM
> To: Doron Fediuck
> Cc: Oved Ourfalli; engine-devel(a)ovirt.org
> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
> integration with oVirt in 4/9
>
>
>
> ----- Original Message -----
>> From: "Doron Fediuck" <dfediuck(a)redhat.com>
>> To: "Itamar Heim" <iheim(a)redhat.com>
>> Cc: "Oved Ourfalli" <ovedo(a)redhat.com>, engine-devel(a)ovirt.org
>> Sent: Monday, April 15, 2013 10:05:57 AM
>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
>> integration with oVirt in 4/9
>>
>>
>>
>> ----- Original Message -----
>>> From: "Itamar Heim" <iheim(a)redhat.com>
>>> To: "Oved Ourfalli" <ovedo(a)redhat.com>
>>> Cc: engine-devel(a)ovirt.org
>>> Sent: Monday, April 15, 2013 9:49:12 AM
>>> Subject: Re: [Engine-devel] minutes for sync up on Open Attestation
>>> integration with oVirt in 4/9
>>>
>>> On 04/15/2013 09:20 AM, Oved Ourfalli wrote:
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Wei D Chen" <wei.d.chen(a)intel.com>
>>>>> To: "Doron Fediuck" <dfediuck(a)redhat.com>, "Ofri
Masad"
>>>>> <omasad(a)redhat.com>
>>>>> Cc: engine-devel(a)ovirt.org
>>>>> Sent: Monday, April 15, 2013 8:54:18 AM
>>>>> Subject: Re: [Engine-devel] minutes for sync up on Open
>>>>> Attestation integration with oVirt in 4/9
>>>>>
>>>>> Hi Doron and Ofri,
>>>>>
>>>>> Thanks for your reply, here is some other question.
>>>>>
>>>>> ii. When adding a host into the trusted cluster, the host will
>>>>> be attested via OAT service, only trusted hosted can be added.
>>>>>
>>>>> Would you also kindly tell me if there is any mandatory logic
>>>>> check when adding a host into a cluster, so we can also put
>>>>> attestation logic with these mandatory check together? Some
>>>>> example or code location is better.
>>>>> Thanks a lot.
>>>>>
>>>>
>>>> I think there are two approaches, depending on the use case.
>>>> #1:
>>>> If the host trusted property is static, then you can narrow the
>>>> tests to two locations:
>>>> * AddVdsCommand - (Vds = Host) - This command is triggered when a
>>>> new host is added to the system. You can use the canDoAction
>>>> method (which we have on all commands, and it determines whether
>>>> an action can be run or not), and there, if cluster requires
>>>> trusted hosts only, you can test whether this host is trusted or
>>>> not.
>>>> * ChangeVdsClusterCommand - this command is used when you change
>>>> the cluster of a host. So, if the target cluster requires tursted
>>>> hosts, you can do a similar test in its canDoAction method.
>>>>
>>>> #2:
>>>> If the host trusted property can change, then I'd suggest using
>>>> the NonOperational property of a host.
>>>> We have a class called VdsUpdateRunTimeInfo that does periodic
>>>> tests of hosts, deciding what's their status, according to
>>>> specific flags.
>>>> The code there is quite complex, and would require refactoring at
>>>> some point, but in the meantime you can use the MonitoringStrategy
>>>> interface that is being used there, and implement a new monitoring
>>>> strategy for Open Attestation.
>>>>
>>>> There, in the processHardwareCapabilities, you can test that the
>>>> host is trusted.
>>>>
>>>> When creating the strategy, using the MonitoringStrategyFactory,
>>>> you'll need to create the appropriate strategy.
>>>> Currently we support either virt strategy or gluster strategy or
>>>> both. In your case you would need virt+attestation /
>>>> gluster+attestation /
>>>> virt+gluster+attestation, according to the services deployed on
>>>> virt+gluster+the
>>>> cluster.
>>>
>>> I'd go with the 2nd approach. i.e., not block the host from being
>>> added, only from being used, based on monitoring / non-op status
>>>
>> +1.
>> It means that the admin can add the host, but the host will not be
>> operational until we get a positive indication from the attestation
>> service.
>>
> +1
> if we want to test attestation only once, every time before host moves to up.
> the right place for it, imo, is InitVdsOnUpCommand if its periodic
> also after host is up, then it should be somewhere in the monitoring
> code
>
>>>
>>>>
>>>>
>>>> Does that make sense?
>>>> Doron and Ofri, what did you have in mind for this feature?
>>>>
>>>> Regards,
>>>> Oved
>>>>
>>>>>
>>>>> Best Regards,
>>>>> Dave Chen
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Doron Fediuck [mailto:dfediuck@redhat.com]
>>>>> Sent: Wednesday, April 10, 2013 8:03 PM
>>>>> To: Ofri Masad
>>>>> Cc: Wei, Gang; Chen, Wei D; Cao, Buddy; Zhang, Lijuan
>>>>> Subject: Re: minutes for sync up on Open Attestation integration
>>>>> with oVirt in 4/9
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Ofri Masad" <omasad(a)redhat.com>
>>>>>> To: "Gang Wei" <gang.wei(a)intel.com>
>>>>>> Cc: "Wei D Chen" <wei.d.chen(a)intel.com>,
"Buddy Cao"
>>>>>> <buddy.cao(a)intel.com>, "Lijuan Zhang"
<lijuan.zhang(a)intel.com>,
>>>>>> "Doron Fediuck" <dfediuck(a)redhat.com>
>>>>>> Sent: Wednesday, April 10, 2013 2:23:57 PM
>>>>>> Subject: Re: minutes for sync up on Open Attestation integration
>>>>>> with oVirt in 4/9
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> answers inside the mail body.
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Doron Fediuck" <dfediuck(a)redhat.com>
>>>>>>> To: "Wei D Chen" <wei.d.chen(a)intel.com>
>>>>>>> Cc: "Gang Wei" <gang.wei(a)intel.com>,
"Buddy Cao"
>>>>>>> <buddy.cao(a)intel.com>, "Lijuan Zhang"
<lijuan.zhang(a)intel.com>,
>>>>>>> "Ofri Masad" <omasad(a)redhat.com>
>>>>>>> Sent: Wednesday, April 10, 2013 12:12:26 PM
>>>>>>> Subject: Re: minutes for sync up on Open Attestation
>>>>>>> integration with oVirt in 4/9
>>>>>>>
>>>>>>> Hi Dave,
>>>>>>> Adding Ofri to answer your questions.
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>>> From: "Wei D Chen"
<wei.d.chen(a)intel.com>
>>>>>>>> To: "Gang Wei" <gang.wei(a)intel.com>,
"Doron Fediuck"
>>>>>>>> <dfediuck(a)redhat.com>,
>>>>>>>> "Buddy Cao" <buddy.cao(a)intel.com>,
"Lijuan Zhang"
>>>>>>>> <lijuan.zhang(a)intel.com>
>>>>>>>> Sent: Wednesday, April 10, 2013 6:31:05 AM
>>>>>>>> Subject: RE: minutes for sync up on Open Attestation
>>>>>>>> integration with oVirt in 4/9
>>>>>>>>
>>>>>>>> Hi Doron,
>>>>>>>>
>>>>>>>>
>>>>>>>> iii. Then periodic trust check will be added into
background
>>>>>>>> process
>>>>>>>> for
>>>>>>>> all existing hosts, once becoming non-trusted, mark it as
>>>>>>>> non-operational.
>>>>>>>>
>>>>>>>> - [Dave] Would you give me some details about
the
>>>>>>>> “background
>>>>>>>> process” mentioned in our sync up meeting? What’s the
process
>>>>>>>> and where is related code?
>>>>>>>>
>>>>>>
>>>>>> The use Quartz Job to scheduled background processes.
>>>>>> An example can be found in:
>>>>>>
ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt/engine/core/bll/Backend.java(Line:
>>>>>> 222)
>>>>>>
>>>>>> This code calls a method that is located in:
>>>>>>
ovirt-engine/backend/manager/modules/bll/src/main/java/org/ovirt
>>>>>> /engin e/core/bll/quota/QuotaManager.java
>>>>>> (Line: 1000)
>>>>>> the @OnTimerMethodAnnotation annotation and the unique job name
>>>>>> connects the two in runtime.
>>>>>>
>>>>>> The job can query the attestation server for all the host and
>>>>>> then set untrusted hosts to Non-Operational.
>>>>>> the best way would be to call SetNonOperationalVdsCommand with a
>>>>>> new NonOperationalReason.
>>>>>> This command tries to migrate all VMs from the host and then set
>>>>>> it non operational.
>>>>>> (We need to think about non-migrational VMs - should we close
>>>>>> those VMs or keep the host running)
>>>>>>
>>>>>
>>>>> One more thing you may want to consider; If we get a positive
>>>>> response for a host which is currently non-operational, issue an
>>>>> activate command to try and make it operational again.
>>>>> Alternatively, you may decide not to handle it, assuming
>>>>> untrusted hosts can only be activated manually (see below Ofri's
>>>>> response on activating a host).
>>>>>
>>>>>>>>
>>>>>>>> iv. If admin fixed the non-operational node and try
to switch
>>>>>>>> it
>>>>>>>> to
>>>>>>>> operational mode again, a trust check will happen to gate
the
>>>>>>>> switching.
>>>>>>>>
>>>>>>>> - [Dave] If there is a button provided from UI
for
>>>>>>>> checking
>>>>>>>> the
>>>>>>>> logic? or we need add an extra-button for attestation
check only?
>>>>>>>>
>>>>>>
>>>>>> In the UI we already have the "Activate" option. What
still
>>>>>> needs to be added is the check with the attestation provider to
>>>>>> make sure that if the cluster is defined 'Trusted'. If
so, only
>>>>>> trusted host will succeed in the activate command (and, of
>>>>>> course, a proper Audit Log error in case the host failed to
>>>>>> activate due to trust issue.
>>>>>>
>>>>>> the correct place to add this check would be in:
>>>>>>
ovirt-engine/backend/manager/modules/vdsbroker/src/main/java/org
>>>>>> /ovirt
>>>>>> /engine/core/vdsbroker/VdsManager.java:activate()
>>>>>> this is where the re-activation process begins.
>>>>>>
>>>>>>>>
>>>>>>>> Thanks very much.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Dave Chen
>>>>> _______________________________________________
>>>>> Engine-devel mailing list
>>>>> Engine-devel(a)ovirt.org
>>>>>
http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>>>
>>>> _______________________________________________
>>>> Engine-devel mailing list
>>>> Engine-devel(a)ovirt.org
>>>>
http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>>
>>>
>>> _______________________________________________
>>> Engine-devel mailing list
>>> Engine-devel(a)ovirt.org
>>>
http://lists.ovirt.org/mailman/listinfo/engine-devel
>>>
>> _______________________________________________
>> Engine-devel mailing list
>> Engine-devel(a)ovirt.org
>>
http://lists.ovirt.org/mailman/listinfo/engine-devel
>>
> _______________________________________________
> Engine-devel mailing list
> Engine-devel(a)ovirt.org
>
http://lists.ovirt.org/mailman/listinfo/engine-devel
> _______________________________________________
> Engine-devel mailing list
> Engine-devel(a)ovirt.org
>
http://lists.ovirt.org/mailman/listinfo/engine-devel
>
_______________________________________________
Engine-devel mailing list
Engine-devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel