On Thu, Mar 7, 2019 at 9:19 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
Hi Simone,

I think I found the problem - ovirt-ha cannot extract the file containing the needed data .
In my case it is completely empty:


[root@ovirt1 ~]# ll /rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/94ade632-6ecc-4901-8cec-8e39f3d69cb0
total 66561
-rw-rw----. 1 vdsm kvm       0 Mar  4 05:21 9460fc4b-54f3-48e3-b7b6-da962321ecf4
-rw-rw----. 1 vdsm kvm 1048576 Jan 31 13:24 9460fc4b-54f3-48e3-b7b6-da962321ecf4.lease
-rw-r--r--. 1 vdsm kvm     435 Mar  4 05:22 9460fc4b-54f3-48e3-b7b6-da962321ecf4.meta


Any hint how to recreate that ? Maybe wipe and restart the ovirt-ha-broker and agent ?

The OVF_STORE volume is going to get periodically recreated by the engine so at least you need a running engine.

In order to avoid this kind of issue we have two OVF_STORE disks, in your case:

MainThread::INFO::2019-03-06 06:50:02,391::ovf_store::120::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:441abdc8-6cb1-49a4-903f-a1ec0ed88429, volUUID:c3309fc0-8707-4de1-903d-8d4bbb024f81
MainThread::INFO::2019-03-06 06:50:02,748::ovf_store::120::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:94ade632-6ecc-4901-8cec-8e39f3d69cb0, volUUID:9460fc4b-54f3-48e3-b7b6-da962321ecf4

Can you please check if you have at lest the second copy?

And even in the case you lost both, we are storing on the shared storage the initial vm.conf:
MainThread::ERROR::2019-03-06 06:50:02,971::config_ovf::70::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm::(_get_vm_conf_content_from_ovf_store) Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf

Can you please check what do you have in /var/run/ovirt-hosted-engine-ha/vm.conf ?
 

Also, I think this happened when I was upgrading ovirt1 (last in the gluster cluster) from 4.3.0 to 4.3.1 . The engine got restarted , because I forgot to enable the global maintenance.

Sorry, I don't understand.
Can you please explain what happened?

 



Best Regards,
Strahil Nikolov

В сряда, 6 март 2019 г., 16:57:30 ч. Гринуич+2, Simone Tiraboschi <stirabos@redhat.com> написа:




On Wed, Mar 6, 2019 at 3:09 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
Hi Simone,

thanks for your reply.

>Are you really sure that the issue was on the ping?
>on storage errors the broker restart itself and while the broker is restarting >the agent cannot ask the broker to trigger the gateway monitor (the ping one) and >so that error message.

It seemed so in that moment, but I'm not so sure , right now :)

>Which kind of storage are you using?
>can you please attach /var/log/ovirt-hosted-engine-ha/broker.log ?

I'm using glustervs v5 from ovirt 4.3.1 with FUSE mount.
Please , have a look in the attached logs.

Nothing seems that strange there but that error.
Can you please try with ovirt-ha-agent and ovirt-ha-broker in debug mode?
you have to set level=DEBUG in [logger_root] section in /etc/ovirt-hosted-engine-ha/agent-log.conf and /etc/ovirt-hosted-engine-ha/broker-log.conf and restart the two services.
 

Best Regards,
Strahil Nikolov

В сряда, 6 март 2019 г., 9:53:20 ч. Гринуич+2, Simone Tiraboschi <stirabos@redhat.com> написа:




On Wed, Mar 6, 2019 at 6:13 AM Strahil <hunter86_bg@yahoo.com> wrote:

Hi guys,

After updating to 4.3.1 I had an issue where the ovirt-ha-broker was complaining that it couldn't ping the gateway.


Are you really sure that the issue was on the ping?
on storage errors the broker restart itself and while the broker is restarting the agent cannot ask the broker to trigger the gateway monitor (the ping one) and so that error message.
 

As I have seen that before - I stopped ovirt-ha-agent, ovirt-ha-broker, vdsmd, supervdsmd and sanlock on the nodes and reinitialized the lockspace.

I gues s I didn't do it properly as now I receive:

ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf

Any hints how to fix this ? Of course a redeploy is possible, but I prefer to recover from that.


Which kind of storage are you using?
can you please attach /var/log/ovirt-hosted-engine-ha/broker.log ?
 

Best Regards,
Strahil Nikolov

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/OU3FKLEPH7AHT2LO2IYZ47RJHRA72C3Z/

_______________________________________________
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BNV7AVUBLOV2UDVBTYN23ZEZ2Q4TJYHV/