<div dir="ltr"><div><div><div><div><div><div><div>Hi all, <br><br></div>For the records, I had to remove manually the conflicting directory and ts respective gfid from the arbiter volume: <br><br> getfattr -m . -d -e hex e1c80750-b880-495e-9609-b8bc7760d101/ha_agent<br><br></div>That gave me the gfid: 0x277c9caa9dce4a17a2a93775357befd5<br><br></div>Then cd .glusterfs/27/7c<br><br></div>rm -rf 277c9caa-9dce-4a17-a2a9-3775357befd5 (or move it out of there)<br><br></div>Triggerred heal: gluster volume heal engine<br><br></div>Then all ok: <br><br>gluster volume heal engine info<br>Brick gluster0:/gluster/engine/brick<br>Status: Connected<br>Number of entries: 0<br><br>Brick gluster1:/gluster/engine/brick<br>Status: Connected<br>Number of entries: 0<br><br>Brick gluster2:/gluster/engine/brick<br>Status: Connected<br>Number of entries: 0<br><br></div>Thanx. <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 23, 2017 at 7:21 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Denis, <br><br></div>I receive permission denied as below: <br><br><span style="font-family:monospace,monospace"><span class="">gluster volume heal engine split-brain latest-mtime /e1c80750-b880-495e-9609-<wbr>b8bc7760d101/ha_agent<br></span>Healing /e1c80750-b880-495e-9609-<wbr>b8bc7760d101/ha_agent failed:Operation not permitted.<br>Volume heal failed.</span><br><br><br></div>When I shutdown host3 then no split brain is reported from the remaining two hosts. When I power up host3 then I receive the mentioned split brain and host3 logs the following at ovirt-hosted-engine-ha/agent.<wbr>log <br><br><span style="font-family:monospace,monospace">MainThread::INFO::2017-06-23 16:18:06,067::hosted_engine::<wbr>594::ovirt_hosted_engine_ha.<wbr>agent.hosted_engine.<wbr>HostedEngine::(_initialize_<wbr>broker) Failed set the storage domain: 'Failed to set storage domain VdsmBackend, options {'hosted-engine.lockspace': '<wbr>7B22696D6167655F75756964223A20<wbr>2238323132626637382D663933332D<wbr>346465652D616333372D3462656337<wbr>34353035366235222C202270617468<wbr>223A206E756C6C2C2022766F6C756D<wbr>655F75756964223A20223632373930<wbr>3162652D666261332D346263342D39<wbr>3037632D3939313561383336326335<wbr>37227D', 'sp_uuid': '00000000-0000-0000-0000-<wbr>000000000000', 'dom_type': 'glusterfs', 'hosted-engine.metadata': '<wbr>7B22696D6167655F75756964223A20<wbr>2263353930633034372D613462322D<wbr>346539312D613832362D6434386239<wbr>61643537323330222C202270617468<wbr>223A206E756C6C2C2022766F6C756D<wbr>655F75756964223A20223035316665<wbr>3865612D333339632D346134302D38<wbr>3438382D3863353131386664383732<wbr>38227D', 'sd_uuid': 'e1c80750-b880-495e-9609-<wbr>b8bc7760d101'}: Request failed: <type 'exceptions.OSError'>'. Waiting '5's before the next attempt<br></span><br></div>and the following at /var/log/messages: <br><span style="font-family:monospace,monospace">Jun 23 16:19:43 v2 journal: vdsm root ERROR failed to retrieve Hosted Engine HA info#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-<wbr>packages/vdsm/host/api.py", line 231, in _getHaInfo#012 stats = instance.get_all_stats()#012 File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/client/client.py", line 105, in get_all_stats#012 stats = broker.get_stats_from_storage(<wbr>service)#012 File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/lib/brokerlink.py", line 233, in get_stats_from_storage#012 result = self._checked_communicate(<wbr>request)#012 File "/usr/lib/python2.7/site-<wbr>packages/ovirt_hosted_engine_<wbr>ha/lib/brokerlink.py", line 261, in _checked_communicate#012 .format(message or response))#012RequestError: Request failed: failed to read metadata: [Errno 5] Input/output error: '/rhev/data-center/mnt/<wbr>glusterSD/10.100.100.1:_<wbr>engine/e1c80750-b880-495e-<wbr>9609-b8bc7760d101/ha_agent/<wbr>hosted-engine.metadata'</span><br><br><div>Thanx<br></div><div><div><br></div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 23, 2017 at 6:05 PM, Denis Chaplygin <span dir="ltr"><<a href="mailto:dchaplyg@redhat.com" target="_blank">dchaplyg@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello Abi,<div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Jun 23, 2017 at 4:47 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>Hi All, <br><br></div>I have a 3 node ovirt 4.1 setup. I lost one node due to raid controller issues. Upon restoration I have the following split brain, although the hosts have mounted the storage domains: <br><br><span style="font-family:monospace,monospace">gluster volume heal engine info split-brain<br>Brick gluster0:/gluster/engine/brick<br>/e1c80750-b880-495e-9609-b8bc7<wbr>760d101/ha_agent<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>Brick gluster1:/gluster/engine/brick<br>/e1c80750-b880-495e-9609-b8bc7<wbr>760d101/ha_agent<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>Brick gluster2:/gluster/engine/brick<br>/e1c80750-b880-495e-9609-b8bc7<wbr>760d101/ha_agent<br>Status: Connected<br>Number of entries in split-brain: 1<br></span><br><br></div></div></div></blockquote><div><br></div></span><div>It is definitely on gluster side. You could try to use</div><div><br></div><div>gluster volume heal engine split-brain latest-mtime <span style="font-family:monospace,monospace">/e1c80750-b880-49<wbr>5e-9609-</span><span style="font-family:monospace,monospace">b8bc7760d101/ha_agent</span><br></div><div><span style="font-family:monospace,monospace"><br></span></div><div><br></div><div>I also added gluster developers to that thread, so they may provide you with better advices.</div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>