[ovirt-users] Metadata error on hosted engine (fresh install)

[Matteo]

Hi all,

I've activated oVirt 3.5 with the hosted engine setup using CentOS 7 as
host and CentOS 6 as engine OS.

The storage for the hosted engine is iSCSI.

The setup went smooth, I've some vm running on top of it without many issues.

Right now I've some issue with hosted engine metadata:

if I try to add an additional node, at the end of the deploy sequence
it fails with error:

Failed to execute stage 'Setup validation': Metadata version 9 from host 5 too new for this agent (highest compatible version: 1)

On the first host, "hosted-engine --check-liveliness" confirms that
the engine is up (Hosted Engine is up!) , but "hosted-engine --vm-status"
fails with a python exception:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 116, in <module>
    if not status_checker.print_status():
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/vm_status.py", line 59, in print_status
    all_host_stats = ha_cli.get_all_host_stats()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 155, in get_all_host_stats
    return self.get_all_stats(self.StatModes.HOST)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 102, in get_all_stats
    stats = self._parse_stats(stats, mode)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 141, in _parse_stats
    md = metadata.parse_metadata_to_dict(host_id, data)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/metadata.py", line 147, in parse_metadata_to_dict
    constants.METADATA_FEATURE_VERSION))
ovirt_hosted_engine_ha.lib.exceptions.FatalMetadataError: Metadata version 9 from host 5 too new for this agent (highest compatible version: 1)

looking into metadata file, which is really a block device (via some symlinks), I see a lot
of binary data after plain ascii text. Dumping this device results into a 134MB file.

( if someome is curious, the dump is here:
  https://cloud.brancaleoni.com/index.php/s/4Lx3M5PXkqhO80N )

Reading other posts where the setup was over nfs was suggested to truncate the file,
but this is a device file and I'm not sure what to do.

Any hint on where I can look at ?

Thanks,
Matteo