<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Hi,<br><br>> ----- Original Message -----<br>> > From: "Ted Miller" <tmiller at hcjb.org><br>> > To: "users" <users at ovirt.org><br>> > Sent: Tuesday, May 20, 2014 11:31:42 PM<br>> > Subject: [ovirt-users] sanlock + gluster recovery -- RFE<br>> > <br>> > As you are aware, there is an ongoing split-brain problem with running<br>> > sanlock on replicated gluster storage. Personally, I believe that this is<br>> > the 5th time that I have been bitten by this sanlock+gluster problem.<br>> > <br>> > I believe that the following are true (if not, my entire request is probably<br>> > off base).<br>> > <br>> > <br>> > * ovirt uses sanlock in such a way that when the sanlock storage is on a<br>> > replicated gluster file system, very small storage disruptions can<br>> > result in a gluster split-brain on the sanlock space<br>> <br>> Although this is possible (at the moment) we are working hard to avoid it.<br>> The hardest part here is to ensure that the gluster volume is properly<br>> configured.<br>> <br>> The suggested configuration for a volume to be used with ovirt is:<br>> <br>> Volume Name: (...)<br>> Type: Replicate<br>> Volume ID: (...)<br>> Status: Started<br>> Number of Bricks: 1 x 3 = 3<br>> Transport-type: tcp<br>> Bricks:<br>> (...three bricks...)<br>> Options Reconfigured:<br>> network.ping-timeout: 10<br>> cluster.quorum-type: auto<br>> <br>> The two options ping-timeout and quorum-type are really important.<br>> <br>> You would also need a build where this bug is fixed in order to avoid any<br>> chance of a split-brain:<br>> <br>> https://bugzilla.redhat.com/show_bug.cgi?id=1066996<br><br>It seems that the aforementioned bug is peculiar to 3-bricks setups.<br><br>I understand that a 3-bricks setup can allow proper quorum formation without resorting to "first-configured-brick-has-more-weight" convention used with only 2 bricks and quorum "auto" (which makes one node "special", so not properly any-single-fault tolerant).<br><br>But, since we are on ovirt-users, is there a similar suggested configuration for a 2-hosts setup oVirt+GlusterFS with oVirt-side power management properly configured and tested-working?<br>I mean a configuration where "any" host can go south and oVirt (through the other one) fences it (forcibly powering it off with confirmation from IPMI or similar) then restarts HA-marked vms that were running there, all the while keeping the underlying GlusterFS-based storage domains responsive and readable/writeable (maybe apart from a lapse between detected other-node unresposiveness and confirmed fencing)?<br><br>Furthermore: is such a suggested configuration possible in a self-hosted-engine scenario?<br><br>Regards,<br>Giuseppe<br><br>> > How did I get into this mess?<br>> > <br>> > ...<br>> > <br>> > What I would like to see in ovirt to help me (and others like me). Alternates<br>> > listed in order from most desirable (automatic) to least desirable (set of<br>> > commands to type, with lots of variables to figure out).<br>> <br>> The real solution is to avoid the split-brain altogether. At the moment it<br>> seems that using the suggested configurations and the bug fix we shouldn't<br>> hit a split-brain.<br>> <br>> > 1. automagic recovery<br>> > <br>> > 2. recovery subcommand<br>> > <br>> > 3. script<br>> > <br>> > 4. commands<br>> <br>> I think that the commands to resolve a split-brain should be documented.<br>> I just started a page here:<br>> <br>> http://www.ovirt.org/Gluster_Storage_Domain_Reference<br>> <br>> Could you add your documentation there? Thanks!<br>> <br>> -- <br>> Federico<br><br> </div></body>
</html>