I took a HE VM down and stopped ovirt-ha-agents on both hosts. 
Tried  hosted-engine --reinitialize-lockspace  the command just silently executes and I'm not sure if it doing something at all.
I also tried to clean the metadata. On one host it went correct, on second host it always failing with following messages: 

INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain monitor status: PENDING
ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Failed to start monitoring domain (sd_uuid=4a7f8717-9bb0-4d80-8016-498fa4b88162, host_id=2): timeout during domain acquisition
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 67, in action_clean
    return he.clean(options.force_cleanup)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 345, in clean
    self._initialize_domain_monitor()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 829, in _initialize_domain_monitor
    raise Exception(msg)
Exception: Failed to start monitoring domain (sd_uuid=4a7f8717-9bb0-4d80-8016-498fa4b88162, host_id=2): timeout during domain acquisition

ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Trying to restart agent
WARNING:ovirt_hosted_engine_ha.agent.agent.Agent:Restarting agent, attempt '0'
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Too many errors occurred, giving up. Please review the log and consider filing a bug.
INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down

I'm not an expert when it comes to read the sanlock but the output looks a bit strange to me: 

from first host (host_id=2)

[root@ovirt1 ~]# sanlock client status 
daemon b1d7fea2-e8a9-4645-b449-97702fc3808e.ovirt1.tel
p -1 helper
p -1 listener
p -1 status
p 3763 
p 62861 quaggaVM
p 63111 powerDNS
p 107818 pjsip_freepbx_14
p 109092 revizorro_dev
p 109589 routerVM
s hosted-engine:2:/var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/093faa75-5e33-4559-84fa-1f1f8d48153b/911c7637-b49d-463e-b186-23b404e50769:0
s a40cc3a9-54d6-40fd-acee-525ef29c8ce3:2:/rhev/data-center/mnt/glusterSD/ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/ids:0
s 4a7f8717-9bb0-4d80-8016-498fa4b88162:1:/rhev/data-center/mnt/glusterSD/ovirt2.telia.ru\:_engine/4a7f8717-9bb0-4d80-8016-498fa4b88162/dom_md/ids:0
r a40cc3a9-54d6-40fd-acee-525ef29c8ce3:SDM:/rhev/data-center/mnt/glusterSD/ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/leases:1048576:49 p 3763

 
from second host (host_id=1)

[root@ovirt2 ~]# sanlock client status 
daemon 9263e081-e5ea-416b-866a-0a73fe32fe16.ovirt2.tel
p -1 helper
p -1 listener
p 150440 CentOS-Desk
p 151061 centos-dev-box
p 151288 revizorro_nfq
p 151954 gitlabVM
p -1 status
s hosted-engine:1:/var/run/vdsm/storage/4a7f8717-9bb0-4d80-8016-498fa4b88162/093faa75-5e33-4559-84fa-1f1f8d48153b/911c7637-b49d-463e-b186-23b404e50769:0
s a40cc3a9-54d6-40fd-acee-525ef29c8ce3:1:/rhev/data-center/mnt/glusterSD/ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/ids:0
s 4a7f8717-9bb0-4d80-8016-498fa4b88162:1:/rhev/data-center/mnt/glusterSD/ovirt2.telia.ru\:_engine/4a7f8717-9bb0-4d80-8016-498fa4b88162/dom_md/ids:0 ADD

Not sure if there is a problem with locspace 4a7f8717-9bb0-4d80-8016-498fa4b88162, but both hosts showing 1 as a host_id here. Is this correct? Should't they have different Id's here?

Once ha-agent's has been started hosted-engine --vm-status showing 'unknow-stale-data' for the second host. And HE just doesn't start on second host at all. 
Host redeployment haven't helped as well. 

Any advises on this?
Regards,
Artem


On Mon, Feb 19, 2018 at 9:32 PM, Artem Tambovskiy <artem.tambovskiy@gmail.com> wrote:
Thanks Martin.

As you suggested I updated hosted-engine.conf with correct host_id values and restarted ovirt-ha-agent services on both hosts and now I run into the problem with  status "unknown-stale-data" :(
And second host still doesn't looks as capable to run HE.

Should I stop HE VM, bring down ovirt-ha-agents and reinitialize-lockspace and start ovirt-ha-agents again?

Regards,
Artem



On Mon, Feb 19, 2018 at 6:45 PM, Martin Sivak <msivak@redhat.com> wrote:
Hi Artem,

just a restart of ovirt-ha-agent services should be enough.

Best regards

Martin Sivak

On Mon, Feb 19, 2018 at 4:40 PM, Artem Tambovskiy
<artem.tambovskiy@gmail.com> wrote:
> Ok, understood.
> Once I set correct host_id on both hosts how to take changes in force? With
> minimal downtime? Or i need reboot both hosts anyway?
>
> Regards,
> Artem
>
> 19 февр. 2018 г. 18:18 пользователь "Simone Tiraboschi"
> <stirabos@redhat.com> написал:
>
>>
>>
>> On Mon, Feb 19, 2018 at 4:12 PM, Artem Tambovskiy
>> <artem.tambovskiy@gmail.com> wrote:
>>>
>>>
>>> Thanks a lot, Simone!
>>>
>>> This is clearly shows a problem:
>>>
>>> [root@ov-eng ovirt-engine]# sudo -u postgres psql -d engine -c 'select
>>> vds_name, vds_spm_id from vds'
>>>     vds_name     | vds_spm_id
>>> -----------------+------------
>>>  ovirt1.local |          2
>>>  ovirt2.local |          1
>>> (2 rows)
>>>
>>> While hosted-engine.conf on ovirt1.local have host_id=1, and ovirt2.local
>>> host_id=2. So totally opposite values.
>>> So how to get this fixed in the simple way? Update the engine DB?
>>
>>
>> I'd suggest to manually fix /etc/ovirt-hosted-engine/hosted-engine.conf on
>> both the hosts
>>
>>>
>>>
>>> Regards,
>>> Artem
>>>
>>> On Mon, Feb 19, 2018 at 5:37 PM, Simone Tiraboschi <stirabos@redhat.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> On Mon, Feb 19, 2018 at 12:13 PM, Artem Tambovskiy
>>>> <artem.tambovskiy@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> Last weekend my cluster suffered form a massive power outage due to
>>>>> human mistake.
>>>>> I'm using SHE setup with Gluster, I managed to bring the cluster up
>>>>> quickly, but once again I have a problem with duplicated host_id
>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1543988) on second host and due
>>>>> to this second host is not capable to run HE.
>>>>>
>>>>> I manually updated file hosted_engine.conf with correct host_id and
>>>>> restarted agent & broker - no effect. Than I rebooted the host itself -
>>>>> still no changes. How to fix this issue?
>>>>
>>>>
>>>> I'd suggest to run this command on the engine VM:
>>>> sudo -u postgres scl enable rh-postgresql95 --  psql -d engine -c
>>>> 'select vds_name, vds_spm_id from vds'
>>>> (just  sudo -u postgres psql -d engine -c 'select vds_name, vds_spm_id
>>>> from vds'  if still on 4.1) and check
>>>> /etc/ovirt-hosted-engine/hosted-engine.conf on all the involved host.
>>>> Maybe you can also have a leftover configuration file on undeployed
>>>> host.
>>>>
>>>> When you find a conflict you should manually bring down sanlock
>>>> In doubt a reboot of both the hosts will solve for sure.
>>>>
>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Artem
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users@ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users@ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/users
>>>
>>
>
> _______________________________________________
> Users mailing list
> Users@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>