Re: [ovirt-users] Dead hosts

Thanks Kasturi, I did try that but not getting anywhere, I get the following errors when trying to clean the meta data: # hosted-engine --clean-metadata --force-cleanup --host-id=1 INFO:ovirt_hosted_engine_ha.agent.agent.Agent:ovirt-hosted-engine-ha agent 2.2.0-0.0.master.20170914130805 started INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Found certificate common name: 192.168.0.14 INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing VDSM INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Connecting the storage INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Connecting storage server INFO:ovirt_hosted_engine_ha.lib.storage_server.StorageServer:Refreshing the storage domain INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing images INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Refreshing vm.conf INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Reloading vm.conf from the shared storage domain INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying to get a fresher copy of vm configuration from the OVF_STORE INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Found OVF_STORE: imgUUID:broken.4d32314f-8548-4d4c-9fb1-b575b41afe4b, volUUID:10dea31c-1c93-4662-b95b-90e4853fe84a INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Found OVF_STORE: imgUUID:88730268-59c6-4f5a-93a9-745f92a51025, volUUID:026b931a-ec4d-44a7-885a-4a7378011ad5 INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Extracting Engine VM OVF from the OVF_STORE INFO:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:OVF_STORE volume path: /rhev/data-center/mnt/nas.example.com:testlab-hosted/09f098b9-8c8b-4841-98d0-ec34c41d186a/images/88730268-59c6-4f5a-93a9-745f92a51025/026b931a-ec4d-44a7-885a-4a7378011ad5 INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Found an OVF for HE VM, trying to convert INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Got vm.conf from OVF_STORE INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Initializing ha-broker connection INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Starting monitor ping, options {'addr': '192.168.0.254'} INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Success, id ping INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Success, id mgmt-bridge INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Success, id mem-free INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '70c46fec-fd6d-4632-94cf-99bb4a1a8271', 'address': '0'} INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Success, id cpu-load-no-engine INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '70c46fec-fd6d-4632-94cf-99bb4a1a8271', 'address': '0'} INFO:ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink:Success, id engine-health INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Broker initialized, all submonitors started INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/09f098b9-8c8b-4841-98d0-ec34c41d186a/559a4adb-be9e-4786-9672-f7216fac7dcb/a9120e7a-7417-4303-b4c1-696faaef1051) ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:cannot get lock on host id 1: host already holds lock on a different host id WARNING:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Force requested, overriding sanlock failure. INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Cleaning the metadata block! ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent return action(he) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 58, in action_clean return he.clean(options.force_cleanup) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 389, in clean if self._broker and self._broker.is_connected(): AttributeError: 'BrokerLink' object has no attribute 'is_connected' ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Trying to restart agent INFO:ovirt_hosted_engine_ha.agent.agent.Agent:Agent shutting down Exception AttributeError: AttributeError("'NoneType' object has no attribute 'c_uint64'",) in <bound method _Server.__del__ of <vdsm.jsonrpcvdscli._Server object at 0x248e5d0>> ignored Not sure what to try next.... regards, Brett On 9 October 2017 at 12:55, Kasturi Narra <knarra(a)redhat.com&gt; wrote: