[Users] oVirt/RHEV fencing; a single point of failure

[Andrew Beekhof]

On 4/03/12 7:16 AM, Perry Myers wrote:
> On 03/03/2012 11:52 AM, xrx wrote:
>> Hello,
>>
>> I was worried about the high availability approach taken by RHEV/oVirt.
>> I had read the thread titled "Some thoughts on enhancing High
>> Availability in oVirt" but couldn't help but feel that oVirt is missing
>> basic HA while it's developers are considering adding (and in my opinion
>> unneeded) complexity with service monitoring.
>
> Service monitoring is a highly desirable feature, but for the most part
> (today) people achieve it by running service monitoring in a layered
> fashion.
>
> For example, running the RHEL HA cluster stack on top of VMs on RHEV (or
> Fedora Clustering on top of oVirt VMs)
>
> So we could certainly skip providing service HA as an integral feature
> of oVirt and continue to leverage the Pacemaker style service HA as a
> layered option instead.
>
> In the past I've gotten the impression that tighter integration and a
> single UI/API for managing both VM and service HA was desirable.
>
>> It all comes down to fencing. Picture this: 3 HP hypervisors running
>> RHEV/oVirt with iLO fencing. Say hypervisor A runs 10 VMs, all of which
>> are set to be highly available. Now suppose that hypervisor A has a
>> power failure or an iLO failure (I've seen it happen more than once with
>> a batch of HP DL380 G6s). Because RHEV would not be able to fence the
>> hypervisor as it's iLO is unresponsive; those 10 HA VMs that were halted
>> are NOT moved to other hypervisors automatically.
>>
>> I suggest that oVirt concentrates on having support for multiple fencing
>> devices as a development priority. SCSI persistent reservation based
>> fencing would be an ideal secondary, if not primary, fencing device; it
>> would be easy to set up for users as SANs generally support it and is
>> proven to work well, as seen on Red Hat clusters.
>
> Completely agree here.  The Pacemaker/rgmanager cluster stacks already
> support an arbitrary number of fence devices per host, to provide
> support for both redundant power supplies and also for redundant fencing
> devices.  In order to provide resilient service HA, fixing this would be
> a prerequisite anyhow.  I've cc'd Andrew Beekhof from the
> Pacemaker/stonith_ng, since I think it might be useful to model the
> fencing for oVirt similarly to how Pacemaker/stonith_ng does it.
> Perhaps there's even some code that could be reused for this as well.

The idea is that fencing requests can be initiated from multiple sources 
and that clients can be notified regardless of where the request 
originates.  Even on non-local machines if the daemon (an independent 
part of pacemaker) is hooked up to corosync.

So the daemon takes care of the boring stuff, reading the configuration 
file, performing periodic health checks, provides a fencing history and 
notifications.

If you're interested, we can make it a sub-package to avoid pulling in 
all of pacemaker.  There is also the option of just using the library 
which knows how to invoke to the existing agents.

Happy to answer any questions people might have.

>
> As for SCSI III PR based fencing... the trouble here has been that the
> fence_scsi script provided in fence-agents is Perl based, and we were
> hesitant to drag Perl into the list of required things on oVirt Node
> (and in general)
>
> on the other hand, fence-scsi might not be the right level of
> granularity for oVirt based SCSI III PR based fencing anyhow.  Perhaps
> better would be to just have vdsm directly call sg_persist commands
> directly.
>
> I've cc'd Ryan O'Hara who wrote fence_scsi and knows a fair bit about
> SCSI III PR.  If oVirt is interested in pursuing this, perhaps he can be
> of assistance.
>
>> I have brought up this point about fencing being a single point of
>> failure in RHEV with a Red Hat employee (Mark Wagner) during the RHEV
>> virtual event; but he said that it is not. I don't see how it isn't; one
>> single loose iLO cable and the VMs are stuck until there is manual
>> intervention.
>
> Agreed.  This is something that should be easily fixed in order to
> provide greater HA.
>
> That being said, I still think more tightly integrated service HA is a
> good idea as well.
>
> Perry