<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I resolved the issue.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">First I put the nodes in Maintenance mode (not stopping Gluster service).</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Then because the files that were in split-brain mode were not important I set the hex value to 0 on both nodes. I guess I could have used something like "getfattr -d -m . -e hex /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/ids" and find the correct hex and such, the following article explains it nicely: <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Managing_Split-brain.html">https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/sect-Managing_Split-brain.html</a> .</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">[root@oVirt-Node1 ~]# setfattr -n trusted.afr.vol1-client-1 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/ids</font><br></div><font face="arial, helvetica, sans-serif">[root@oVirt-Node1 ~]# setfattr -n trusted.afr.vol1-client-1 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div>/<div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"></div></font><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px">__DIRECT_IO_TEST__</span><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">[root@oVirt-Node1 ~]# setfattr -n trusted.afr.vol1-client-1 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/</font><span style="font-family:arial,helvetica,sans-serif">inbox</span><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">[root@oVirt-Node1 ~]# setfattr -n trusted.afr.vol1-client-1 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/leases<br></font></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[root@ovirt-node2 ~]# setfattr -n trusted.afr.vol1-client-0 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/ids<br></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif" style="font-family:arial,helvetica,sans-serif">[root@ovirt-node2 ~]# setfattr -n trusted.afr.vol1-client-0 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1<div class="gmail_default" style="display:inline"></div>/<div class="gmail_default" style="display:inline"></div></font><span style="font-family:arial,helvetica,sans-serif;font-size:12.8px">__DIRECT_IO_TEST__</span><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[root@ovirt-node2 ~]# setfattr -n trusted.afr.vol1-client-0 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/inbox</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[root@ovirt-node2 ~]# setfattr -n trusted.afr.vol1-client-0 -v 0x000000000000000000000000 /gluster-bricks/vol1/vol1/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/leases</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The heal command now gives no errors.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style=""><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">[root@oVirt-Node1 ~]# gluster volume heal vol1 info</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Brick 10.0.0.194:/gluster-bricks/vol1/vol1</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Status: Connected</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Number of entries: 2</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Brick 10.0.0.199:/gluster-bricks/vol1/vol1</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Status: Connected</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Number of entries: 2</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">I activated the nodes again in oVirt and activated the data center (going in Storage > master volume > Data Center > Activate). Now it goes to Locked and to Up. Now the SPM is automatically selected in the nodes.</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Thank you very much!!!</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Kind regards</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Robin Vanderveken</font></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 25 July 2016 at 16:10, Robin Vanderveken <span dir="ltr"><<a href="mailto:rvanderveken@dlttechnologies.com" target="_blank">rvanderveken@dlttechnologies.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Your command was very helpful! It seems to be a split-brain error! I should be able to fix this, I'll let you know when it's fixed.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">We are aware that 2 gluster volumes is not recommended, we didn't know a minimum of 3 is needed to counter a power loss.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The output of "gluster volume heal <volume_name> info" is:</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">[root@ovirt-node2 ~]# gluster volume heal vol1 info</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Brick 10.0.0.194:/gluster-bricks/vol1/vol1</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/ids - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__ - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/inbox - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">/9d1a204c-1b1a-450f-986d-a8c84babb4c1/dom_md/leases - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Status: Connected</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Number of entries: 4</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Brick 10.0.0.199:/gluster-bricks/vol1/vol1</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><gfid:b9d48873-846d-4964-93c7-0a662de04d22> - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">/__DIRECT_IO_TEST__ - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><gfid:cbc7b097-e600-43d3-8f20-9626f6d0239b> - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><gfid:cfa177c0-91fa-4649-b99b-73e03ddbc756> - Is in split-brain</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Status: Connected</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Number of entries: 4</font></div><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">I also added GlusterFS logs, I was unsure which you needed so I added as many as possible.</div></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Kind regards</div><span class="HOEnZb"><font color="#888888"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Robin Vanderveken</div><br></font></span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 25 July 2016 at 15:49, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">First off, a replica 2 gluster volume is not recommended if you want to survive power loss - you need to have a replica 3 gluster volume.<br>
<br>
>From engine logs, I see "Could not connect host to Data Center(Storage issue)" but there are no errors in the attached vdsm.log. Can you provide the relevant vdsm log and also the gluster mount logs?<br>
<br>
What's the output of "gluster volume heal <volname> info" ?<br>
<span><br>
----- Original Message -----<br>
> From: "Robin Vanderveken" <<a href="mailto:rvanderveken@dlttechnologies.com" target="_blank">rvanderveken@dlttechnologies.com</a>><br>
> To: <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
> Sent: Monday, July 25, 2016 2:01:24 PM<br>
> Subject: [ovirt-users] oVirt 4 master storage down - unable to activate after power loss<br>
><br>
> Dear oVirt users<br>
><br>
> I've been having several problems with my oVirt nodes utilising GlusterFS<br>
> after simulating a power loss.<br>
><br>
> Our testing setup for oVirt consists of 3 nodes, 2 with GlusterFS storage and<br>
> 1 just for computing. Everything seems to be setup correct and was working<br>
> correctly. Then after simulating a power loss the master storage goes down,<br>
> and therefore the VMs and data center go down as well.<br>
><br>
> I checked the GlusterFS configuration and it seems to be correct (see<br>
> attachment). I checked the oVirt configuration and it seems to be correct<br>
> (see attachment). I tried putting several Nodes in Maintenace several times,<br>
> even putting in maintenance and reinstalling them. Only when putting the<br>
> nodes in maintenance (and choosing to stop the GlusterFS service) triggers<br>
> Contending on the other Nodes. Only then there is a chance that a Node goes<br>
> from Contending to SPM, which does not always happen. After trying this<br>
> several times I got a main Node to become SPM, but the master storage<br>
> remains down. When I select the master storage, go in the Data center,<br>
> select the data center and click Activate, then both the master storage and<br>
> the data center go in the state Locked for a few seconds and then Inactive<br>
> again.<br>
><br>
</span>> Then I upgraded to oVirt 4 ( oVirt Engine Version: 4.0.0.6-1.el7.centos) and<br>
<span>> tried everything again, resulting in the same result.<br>
><br>
> I searched online and found this mailing list which is very similar:<br>
> <a href="http://lists.ovirt.org/pipermail/users/2014-August/026905.html" rel="noreferrer" target="_blank">http://lists.ovirt.org/pipermail/users/2014-August/026905.html</a> .<br>
> Unfortunately the solution was not posted/mailed. I also found this:<br>
> <a href="https://www.mail-archive.com/users@ovirt.org/msg08105.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/users@ovirt.org/msg08105.html</a> suggesting the<br>
> kernel version is not correct, but I am unsure how to check this.<br>
><br>
> In the attachment I added relevant logs:<br>
> - GlusterFS service on both main nodes<br>
> - /var/log/vdsm/vdsm.log , /var/log/vdsm/supervdsm.log , /var/log/messages ,<br>
</span>> /var/log/sanlock.log , /var/log/ovirt-engine/engine. log-20160725.gz ,<br>
<span>> /var/log/ovirt-engine/engine.log of the oVirt engine<br>
> as attachments (I clicked on Activate on the data center at 9:45:30 am GMT).<br>
> It can be possible that I need to send different engine logs, please tell me<br>
> if necessary. Any help would be highly appreciated.<br>
><br>
> Kind regards<br>
> Robin Vanderveken<br>
><br>
> --<br>
> Robin Vanderveken<br>
> DLT Technologies<br>
> Tel. <a href="tel:%2B352%20691%20412%20922" value="+352691412922" target="_blank">+352 691 412 922</a><br>
><br>
</span>> Disclaimer<br>
><br>
> Les informations contenues dans ce message sont destinées exclusivement à<br>
> l’usage de la personne à laquelle elles sont adressées. Elles peuvent<br>
> contenir des données confidentielles et/ou privilégiées et/ou protégées par<br>
> des droits de propriété intellectuelle ou d’autres lois. Si vous avez reçu<br>
> ce message par erreur, veuillez le détruire, ainsi que toutes ses annexes,<br>
> et notifier son expéditeur ; il vous est interdit de copier ou d’utiliser ce<br>
> messages ou ses annexes ou de divulguer son contenu à quiconque. La<br>
> transmission de données par e-mail ne peut être garantie comme un moyen sûr<br>
> et infaillible, ni comme étant exempt de tout virus. L’expéditeur décline<br>
> toute responsabilité en cas de perte ou de dommage résultant de ce message<br>
> ou de son utilisation, quelle qu’elle soit.<br>
><br>
> The information contained in this message is intended for the addressee only<br>
> and may contain confidential and/or privileged information and/or<br>
> information protected by intellectual property rights or other legal rules.<br>
> If you are not the intended recipient, please delete this message and any<br>
> attachment to it and notify the sender; you may not copy or use this message<br>
> or its attachments in any way nor disclose its contents to anyone. Emails<br>
> cannot be guaranteed to be secure or to be error or virus free. No liability<br>
> is accepted by the sender for any loss damage arising in any way from this<br>
> message or its use.<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">Robin Vanderveken</span><br></div><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">DLT Technologies</span></div><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">Tel. <a href="tel:%2B352%20691%20412%20922" value="+352691412922" target="_blank">+352 691 412 922</a></span></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">Robin Vanderveken</span><br></div><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">DLT Technologies</span></div><div style="color:rgb(80,0,80);font-size:12.8px;font-family:arial,helvetica,sans-serif"><span style="font-size:12.8px">Tel. +352 691 412 922</span></div></div></div>
</div>
<br>
<div><font face="Arial, Helvetica, sans-serif" size="1"><b><i>Disclaimer</i></b></font></div><div><font face="Arial, Helvetica, sans-serif" size="1"><b><i><br></i></b></font></div><div><font face="Arial, Helvetica, sans-serif" size="1"><i>Les informations contenues dans ce message sont destinées exclusivement à l’usage de la personne à laquelle elles sont adressées. Elles peuvent contenir des données confidentielles et/ou privilégiées et/ou protégées par des droits de propriété intellectuelle ou d’autres lois. Si vous avez reçu ce message par erreur, veuillez le détruire, ainsi que toutes ses annexes, et notifier son expéditeur ; il vous est interdit de copier ou d’utiliser ce messages ou ses annexes ou de divulguer son contenu à quiconque. La transmission de données par e-mail ne peut être garantie comme un moyen sûr et infaillible, ni comme étant exempt de tout virus. L’expéditeur décline toute responsabilité en cas de perte ou de dommage résultant de ce message ou de son utilisation, quelle qu’elle soit. </i></font></div><div><font face="Arial, Helvetica, sans-serif" size="1"><i><br></i></font></div><div><font face="Arial, Helvetica, sans-serif" size="1"><i>The information contained in this message is intended for the addressee only and may contain confidential and/or privileged information and/or information protected by intellectual property rights or other legal rules. If you are not the intended recipient, please delete this message and any attachment to it and notify the sender; you may not copy or use this message or its attachments in any way nor disclose its contents to anyone. Emails cannot be guaranteed to be secure or to be error or virus free. No liability is accepted by the sender for any loss damage arising in any way from this message or its use. </i></font></div>