Metadata error on hosted engine (fresh install)
Hi all, I've activated oVirt 3.5 with the hosted engine setup using CentOS 7 as host and CentOS 6 as engine OS. The storage for the hosted engine is iSCSI. The setup went smooth, I've some vm running on top of it without many issues. Right now I've some issue with hosted engine metadata: if I try to add an additional node, at the end of the deploy sequence it fails with error: Failed to execute stage 'Setup validation': Metadata version 9 from host 5 too new for this agent (highest compatible version: 1) On the first host, "hosted-engine --check-liveliness" confirms that the engine is up (Hosted Engine is up!) , but "hosted-engine --vm-status" fails with a python exception: Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 116, in <module> if not status_checker.print_status(): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 59, in print_status all_host_stats = ha_cli.get_all_host_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 155, in get_all_host_stats return self.get_all_stats(self.StatModes.HOST) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats stats = self._parse_stats(stats, mode) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 141, in _parse_stats md = metadata.parse_metadata_to_dict(host_id, data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/metadata.py", line 147, in parse_metadata_to_dict constants.METADATA_FEATURE_VERSION)) ovirt_hosted_engine_ha.lib.exceptions.FatalMetadataError: Metadata version 9 from host 5 too new for this agent (highest compatible version: 1) looking into metadata file, which is really a block device (via some symlinks), I see a lot of binary data after plain ascii text. Dumping this device results into a 134MB file. ( if someome is curious, the dump is here: https://cloud.brancaleoni.com/index.php/s/4Lx3M5PXkqhO80N ) Reading other posts where the setup was over nfs was suggested to truncate the file, but this is a device file and I'm not sure what to do. Any hint on where I can look at ? Thanks, Matteo
Well, I blindly tried the following: * check the name of the block device used for metadata * then shutdown the engine vm * then stop agent and broker on first host and finally zeroed the block device with dd if=/dev/zero of=/dev/dm-12 (dm-12 is the block device pointed by metadata file) started again broker and agent, after a while the engine was started by HA and the metadata was readable. now hosted_engine --vm-status works ok and I was able to add a 2nd node to the cluster. Maybe the metadata block device needs to be cleared when doing iSCSI setup? Don't know if this is correct, but seems to work ok now. Regards, Matteo
----- Original Message -----
From: "Matteo" <mbrancaleoni@voismart.it> To: users@ovirt.org Sent: Friday, July 3, 2015 5:06:13 PM Subject: Re: [ovirt-users] Metadata error on hosted engine (fresh install)
Well,
I blindly tried the following:
* check the name of the block device used for metadata * then shutdown the engine vm * then stop agent and broker on first host and finally zeroed the block device with dd if=/dev/zero of=/dev/dm-12
(dm-12 is the block device pointed by metadata file)
started again broker and agent, after a while the engine was started by HA and the metadata was readable.
now hosted_engine --vm-status works ok and I was able to add a 2nd node to the cluster.
Maybe the metadata block device needs to be cleared when doing iSCSI setup?
Shouldn't be needed.
Don't know if this is correct, but seems to work ok now.
Do you have a backup of the device with the bad data? Can you check/post logs of setup and ha? Thanks and best regards, -- Didi
Hi, ----- Il 5-lug-15, alle 7:53, Yedidyah Bar David didi@redhat.com ha scritto:
----- Original Message -----
From: "Matteo" <mbrancaleoni@voismart.it> To: users@ovirt.org Sent: Friday, July 3, 2015 5:06:13 PM Subject: Re: [ovirt-users] Metadata error on hosted engine (fresh install)
Well,
I blindly tried the following:
* check the name of the block device used for metadata * then shutdown the engine vm * then stop agent and broker on first host and finally zeroed the block device with dd if=/dev/zero of=/dev/dm-12
(dm-12 is the block device pointed by metadata file)
started again broker and agent, after a while the engine was started by HA and the metadata was readable.
now hosted_engine --vm-status works ok and I was able to add a 2nd node to the cluster.
Maybe the metadata block device needs to be cleared when doing iSCSI setup?
Shouldn't be needed.
Don't know if this is correct, but seems to work ok now.
Do you have a backup of the device with the bad data? Can you check/post logs of setup and ha?
The dump is here:
Thanks and best regards, -- Didi
Hi, sorry I sent an empty mail by mistake. ----- Il 5-lug-15, alle 7:53, Yedidyah Bar David didi@redhat.com ha scritto:
Do you have a backup of the device with the bad data? Can you check/post logs of setup and ha?
Sure, everything is here: https://cloud.brancaleoni.com/index.php/s/4Lx3M5PXkqhO80N dump.img.gz is the dump of the block device ovirt-hosted-engine-setup-1st-node.tgz are the logs of first node (the one with the initial engine vm deployed) ovirt-hosted-engine-setup-2nd-node.tgz are the logs of 2nd node Thanks! matteo
participants (2)
-
Matteo -
Yedidyah Bar David