Hi All,

I have a 3 node ovirt 4.1 setup. I lost one node due to raid controller issues. Upon restoration I have the following split brain, although the hosts have mounted the storage domains:

gluster volume heal engine info split-brain
Brick gluster0:/gluster/engine/brick
/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
Status: Connected
Number of entries in split-brain: 1

Brick gluster1:/gluster/engine/brick
/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
Status: Connected
Number of entries in split-brain: 1

Brick gluster2:/gluster/engine/brick
/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent
Status: Connected
Number of entries in split-brain: 1

Hosted engine status gives the following:

hosted-engine --vm-status
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 173, in <module>
    if not status_checker.print_status():
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 103, in print_status
    all_host_stats = self._get_all_host_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 73, in _get_all_host_stats
    all_host_stats = ha_cli.get_all_host_stats()
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 160, in get_all_host_stats
    return self.get_all_stats(self.StatModes.HOST)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 105, in get_all_stats
    stats = broker.get_stats_from_storage(service)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 233, in get_stats_from_storage
    result = self._checked_communicate(request)
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 261, in _checked_communicate
    .format(message or response))
ovirt_hosted_engine_ha.lib.exceptions.RequestError: Request failed: failed to read metadata: [Errno 5] Input/output error: '/rhev/data-center/mnt/glusterSD/10.100.100.1:_engine/e1c80750-b880-495e-9609-b8bc7760d101/ha_agent/hosted-engine.metadata'

Any idea on how to resolve this split brain?

Thanx,
Alex