[ovirt-users] Fwd: why host is not capable to run HE?

Wed Feb 21 09:52:14 UTC 2018

I took a HE VM down and stopped ovirt-ha-agents on both hosts.
Tried  hosted-engine --reinitialize-lockspace  the command just silently
executes and I'm not sure if it doing something at all.
I also tried to clean the metadata. On one host it went correct, on second
host it always failing with following messages:

INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain
monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain
monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain
monitor status: PENDING
INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:VDSM domain
monitor status: PENDING
ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Failed to
start monitoring domain (sd_uuid=4a7f8717-9bb0-4d80-8016-498fa4b88162,
host_id=2): timeout during domain acquisition
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Traceback (most recent call
last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 67, in action_clean
    return he.clean(options.force_cleanup)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 345, in clean
    self._initialize_domain_monitor()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 829, in _initialize_domain_monitor
    raise Exception(msg)
Exception: Failed to start monitoring domain
(sd_uuid=4a7f8717-9bb0-4d80-8016-498fa4b88162,
host_id=2): timeout during domain acquisition

ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Trying to restart agent
WARNING:ovirt_hosted_engine_ha.agent.agent.Agent:Restarting agent, attempt
'0'
ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Too many errors occurred,
giving up. Please review the log and consider filing a bug.
INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down

I'm not an expert when it comes to read the sanlock but the output looks a
bit strange to me:

from first host (host_id=2)

[root at ovirt1 ~]# sanlock client status
daemon b1d7fea2-e8a9-4645-b449-97702fc3808e.ovirt1.tel
p -1 helper
p -1 listener
p -1 status
p 3763
p 62861 quaggaVM
p 63111 powerDNS
p 107818 pjsip_freepbx_14
p 109092 revizorro_dev
p 109589 routerVM
s hosted-engine:2:/var/run/vdsm/storage/4a7f8717-9bb0-4d80-
8016-498fa4b88162/093faa75-5e33-4559-84fa-1f1f8d48153b/
911c7637-b49d-463e-b186-23b404e50769:0
s a40cc3a9-54d6-40fd-acee-525ef29c8ce3:2:/rhev/data-center/mnt/glusterSD/
ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/ids:0
s 4a7f8717-9bb0-4d80-8016-498fa4b88162:1:/rhev/data-center/mnt/glusterSD/
ovirt2.telia.ru\:_engine/4a7f8717-9bb0-4d80-8016-498fa4b88162/dom_md/ids:0
r a40cc3a9-54d6-40fd-acee-525ef29c8ce3:SDM:/rhev/data-center/mnt/glusterSD/
ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/leases:1048576:49
p 3763

from second host (host_id=1)

[root at ovirt2 ~]# sanlock client status
daemon 9263e081-e5ea-416b-866a-0a73fe32fe16.ovirt2.tel
p -1 helper
p -1 listener
p 150440 CentOS-Desk
p 151061 centos-dev-box
p 151288 revizorro_nfq
p 151954 gitlabVM
p -1 status
s hosted-engine:1:/var/run/vdsm/storage/4a7f8717-9bb0-4d80-
8016-498fa4b88162/093faa75-5e33-4559-84fa-1f1f8d48153b/
911c7637-b49d-463e-b186-23b404e50769:0
s a40cc3a9-54d6-40fd-acee-525ef29c8ce3:1:/rhev/data-center/mnt/glusterSD/
ovirt2.telia.ru\:_data/a40cc3a9-54d6-40fd-acee-525ef29c8ce3/dom_md/ids:0
s 4a7f8717-9bb0-4d80-8016-498fa4b88162:1:/rhev/data-center/mnt/glusterSD/
ovirt2.telia.ru\:_engine/4a7f8717-9bb0-4d80-8016-498fa4b88162/dom_md/ids:0
ADD

Not sure if there is a problem with locspace
4a7f8717-9bb0-4d80-8016-498fa4b88162,
but both hosts showing 1 as a host_id here. Is this correct? Should't they
have different Id's here?

Once ha-agent's has been started hosted-engine --vm-status showing
'unknow-stale-data' for the second host. And HE just doesn't start on
second host at all.
Host redeployment haven't helped as well.

Any advises on this?
Regards,
Artem

On Mon, Feb 19, 2018 at 9:32 PM, Artem Tambovskiy <
artem.tambovskiy at gmail.com> wrote:

> Thanks Martin.
>
> As you suggested I updated hosted-engine.conf with correct host_id values
> and restarted ovirt-ha-agent services on both hosts and now I run into the
> problem with  status "unknown-stale-data" :(
> And second host still doesn't looks as capable to run HE.
>
> Should I stop HE VM, bring down ovirt-ha-agents and reinitialize-lockspace
> and start ovirt-ha-agents again?
>
> Regards,
> Artem
>
>
>
> On Mon, Feb 19, 2018 at 6:45 PM, Martin Sivak <msivak at redhat.com> wrote:
>
>> Hi Artem,
>>
>> just a restart of ovirt-ha-agent services should be enough.
>>
>> Best regards
>>
>> Martin Sivak
>>
>> On Mon, Feb 19, 2018 at 4:40 PM, Artem Tambovskiy
>> <artem.tambovskiy at gmail.com> wrote:
>> > Ok, understood.
>> > Once I set correct host_id on both hosts how to take changes in force?
>> With
>> > minimal downtime? Or i need reboot both hosts anyway?
>> >
>> > Regards,
>> > Artem
>> >
>> > 19 февр. 2018 г. 18:18 пользователь "Simone Tiraboschi"
>> > <stirabos at redhat.com> написал:
>> >
>> >>
>> >>
>> >> On Mon, Feb 19, 2018 at 4:12 PM, Artem Tambovskiy
>> >> <artem.tambovskiy at gmail.com> wrote:
>> >>>
>> >>>
>> >>> Thanks a lot, Simone!
>> >>>
>> >>> This is clearly shows a problem:
>> >>>
>> >>> [root at ov-eng ovirt-engine]# sudo -u postgres psql -d engine -c
>> 'select
>> >>> vds_name, vds_spm_id from vds'
>> >>>     vds_name     | vds_spm_id
>> >>> -----------------+------------
>> >>>  ovirt1.local |          2
>> >>>  ovirt2.local |          1
>> >>> (2 rows)
>> >>>
>> >>> While hosted-engine.conf on ovirt1.local have host_id=1, and
>> ovirt2.local
>> >>> host_id=2. So totally opposite values.
>> >>> So how to get this fixed in the simple way? Update the engine DB?
>> >>
>> >>
>> >> I'd suggest to manually fix /etc/ovirt-hosted-engine/hosted-engine.conf
>> on
>> >> both the hosts
>> >>
>> >>>
>> >>>
>> >>> Regards,
>> >>> Artem
>> >>>
>> >>> On Mon, Feb 19, 2018 at 5:37 PM, Simone Tiraboschi <
>> stirabos at redhat.com>
>> >>> wrote:
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Feb 19, 2018 at 12:13 PM, Artem Tambovskiy
>> >>>> <artem.tambovskiy at gmail.com> wrote:
>> >>>>>
>> >>>>> Hello,
>> >>>>>
>> >>>>> Last weekend my cluster suffered form a massive power outage due to
>> >>>>> human mistake.
>> >>>>> I'm using SHE setup with Gluster, I managed to bring the cluster up
>> >>>>> quickly, but once again I have a problem with duplicated host_id
>> >>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1543988) on second
>> host and due
>> >>>>> to this second host is not capable to run HE.
>> >>>>>
>> >>>>> I manually updated file hosted_engine.conf with correct host_id and
>> >>>>> restarted agent & broker - no effect. Than I rebooted the host
>> itself -
>> >>>>> still no changes. How to fix this issue?
>> >>>>
>> >>>>
>> >>>> I'd suggest to run this command on the engine VM:
>> >>>> sudo -u postgres scl enable rh-postgresql95 --  psql -d engine -c
>> >>>> 'select vds_name, vds_spm_id from vds'
>> >>>> (just  sudo -u postgres psql -d engine -c 'select vds_name,
>> vds_spm_id
>> >>>> from vds'  if still on 4.1) and check
>> >>>> /etc/ovirt-hosted-engine/hosted-engine.conf on all the involved
>> host.
>> >>>> Maybe you can also have a leftover configuration file on undeployed
>> >>>> host.
>> >>>>
>> >>>> When you find a conflict you should manually bring down sanlock
>> >>>> In doubt a reboot of both the hosts will solve for sure.
>> >>>>
>> >>>>
>> >>>>>
>> >>>>>
>> >>>>> Regards,
>> >>>>> Artem
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Users mailing list
>> >>>>> Users at ovirt.org
>> >>>>> http://lists.ovirt.org/mailman/listinfo/users
>> >>>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Users mailing list
>> >>> Users at ovirt.org
>> >>> http://lists.ovirt.org/mailman/listinfo/users
>> >>>
>> >>
>> >
>> > _______________________________________________
>> > Users mailing list
>> > Users at ovirt.org
>> > http://lists.ovirt.org/mailman/listinfo/users
>> >
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180221/a50bebb2/attachment.html>