MainThread::INFO::2017-06-15 08:56:15,929::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.2-0.0.master.20170605141622 started MainThread::INFO::2017-06-15 08:56:15,953::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-hyp-01.reis.com MainThread::INFO::2017-06-15 08:56:15,954::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 08:56:18,218::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 08:56:18,219::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 08:56:20,459::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 08:56:20,673::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 08:56:20,838::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 08:56:20,839::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 08:56:23,313::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 08:56:23,314::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 08:56:23,314::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 08:56:25,558::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 08:56:25,606::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:56:25,795::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 08:56:25,796::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:56:25,804::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 08:56:25,807::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 08:56:25,807::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2017-06-15 08:56:25,807::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.170.1'} MainThread::INFO::2017-06-15 08:56:25,810::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41825424 MainThread::INFO::2017-06-15 08:56:25,811::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2017-06-15 08:56:25,812::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41933968 MainThread::INFO::2017-06-15 08:56:25,812::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2017-06-15 08:56:25,812::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41933776 MainThread::INFO::2017-06-15 08:56:25,812::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 08:56:25,813::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41960592 MainThread::INFO::2017-06-15 08:56:25,813::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 08:56:25,814::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41961360 MainThread::INFO::2017-06-15 08:56:29,053::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 41962064 MainThread::INFO::2017-06-15 08:56:29,053::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2017-06-15 08:56:29,118::hosted_engine::698::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/d022a79d-611a-4908-9211-9a81b55bcf76/7bbc8086-0d87-45e3-94a9-2f8e2fc8c9ec/fcf4f5dc-1a39-4094-81ae-8f0ff45197fb) MainThread::INFO::2017-06-15 08:59:10,133::hosted_engine::732::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Acquired lock on host id 1 MainThread::INFO::2017-06-15 08:59:12,297::upgrade::998::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Host configuration is already up-to-date MainThread::INFO::2017-06-15 08:59:12,297::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 08:59:12,297::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 08:59:14,530::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 08:59:14,561::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:14,686::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 08:59:14,687::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:14,691::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 08:59:14,694::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 08:59:14,726::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 08:59:14,726::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 08:59:16,961::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 08:59:16,992::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:17,149::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 08:59:17,150::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:17,154::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 08:59:17,157::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 08:59:17,157::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 08:59:17,157::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=20337 (Fri Jun 9 14:39:03 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=20351 (Fri Jun 9 14:39:17 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '936d4cf3', 'local_conf_timestamp': 20351, 'host-ts': 20337} MainThread::INFO::2017-06-15 08:59:17,157::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=20377 (Fri Jun 9 14:39:37 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=20391 (Fri Jun 9 14:39:51 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStop\nstopped=False\ntimeout=Thu Jan 1 00:43:08 1970\n', 'hostname': 'ovirt-hyp-03.reis.com', 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'powering down'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'f646334e', 'local_conf_timestamp': 20391, 'host-ts': 20377} MainThread::INFO::2017-06-15 08:59:17,157::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14999.0, 'maintenance': False, 'cpu-load': 0.0101, 'gateway': 1.0} MainThread::INFO::2017-06-15 08:59:17,158::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497531557.16 type=state_transition detail=StartState-ReinitializeFSM hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 08:59:17,200::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2017-06-15 08:59:17,200::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 08:59:19,401::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 08:59:19,402::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 08:59:23,782::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 08:59:23,814::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 08:59:23,929::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 08:59:23,930::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 08:59:26,407::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 08:59:26,408::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 08:59:26,408::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 08:59:28,655::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 08:59:28,687::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:28,860::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 08:59:28,861::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 08:59:28,865::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 08:59:28,868::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 08:59:31,063::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: NONE MainThread::INFO::2017-06-15 08:59:33,231::hosted_engine::805::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) Started VDSM domain monitor for d022a79d-611a-4908-9211-9a81b55bcf76 MainThread::INFO::2017-06-15 08:59:35,407::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 08:59:42,589::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 08:59:49,771::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 08:59:56,956::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:04,144::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:11,336::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:18,539::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:25,734::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:32,906::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:40,088::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:47,271::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:00:54,454::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:01,640::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:08,846::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:16,037::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:23,235::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:30,429::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:37,601::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:44,779::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:51,975::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:01:59,161::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:02:06,348::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:02:13,538::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:02:20,747::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:02:28,000::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2017-06-15 09:02:38,115::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:02:38,115::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:02:40,363::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:02:40,395::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:02:40,552::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:02:40,553::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:02:40,559::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:02:40,561::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:02:40,562::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497531760.56 type=state_transition detail=ReinitializeFSM-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:02:40,600::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-EngineDown) sent? sent MainThread::INFO::2017-06-15 09:02:40,600::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:02:42,786::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:02:42,787::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:02:47,142::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:02:47,173::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:02:47,288::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:02:47,289::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:02:49,758::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:02:49,759::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:02:49,759::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:02:51,995::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:02:52,026::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:02:52,167::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:02:52,168::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:02:52,174::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:02:52,177::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:02:56,562::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:02:56,562::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:03:06,616::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:03:06,616::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:03:08,869::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:03:08,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:09,041::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:03:09,041::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:09,047::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:03:09,049::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:03:09,050::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:03:09,050::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:03:11,256::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:03:11,256::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:03:15,607::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:03:15,615::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:03:15,730::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:03:15,730::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:03:18,206::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:03:18,206::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:03:18,207::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:03:20,441::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:03:20,488::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:20,645::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:03:20,646::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:20,652::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:03:20,654::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:03:25,040::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:03:25,040::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:03:35,100::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:03:35,101::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:03:37,343::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:03:37,391::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:37,532::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:03:37,532::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:37,538::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:03:37,540::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:03:37,542::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:03:39,751::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:03:39,752::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:03:44,120::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:03:44,128::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:03:44,242::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:03:44,243::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:03:46,708::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:03:46,709::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:03:46,709::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:03:48,940::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:03:48,987::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:49,128::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:03:49,129::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:03:49,134::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:03:49,137::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:03:53,519::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:03:53,519::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:04:03,578::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:04:03,578::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:04:05,818::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:04:05,866::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:05,991::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:04:05,991::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:05,997::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:04:06,000::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:04:06,001::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:04:08,203::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:04:08,203::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:04:12,597::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:04:12,605::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:04:12,720::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:04:12,720::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:04:15,210::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:04:15,210::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:04:15,210::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:04:17,435::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:04:17,482::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:17,607::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:04:17,608::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:17,613::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:04:17,616::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:04:21,992::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:04:21,992::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:04:32,050::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:04:32,050::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:04:34,288::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:04:34,319::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:34,461::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:04:34,461::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:34,467::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:04:34,469::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:04:34,469::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:04:34,470::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:04:36,673::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:04:36,674::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:04:41,065::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:04:41,072::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:04:41,187::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:04:41,188::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:04:43,641::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:04:43,641::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:04:43,641::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:04:45,865::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:04:45,896::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:46,037::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:04:46,038::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:04:46,043::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:04:46,046::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:04:50,417::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:04:50,417::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:05:00,477::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:05:00,477::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:05:02,711::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:05:02,742::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:02,899::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:05:02,899::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:02,905::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:05:02,908::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:05:02,909::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:05:05,107::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:05:05,108::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:05:09,489::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:05:09,497::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:05:09,611::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:05:09,612::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:05:12,043::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:05:12,043::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:05:12,043::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:05:14,291::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:05:14,322::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:14,496::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:05:14,496::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:14,502::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:05:14,505::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:05:18,892::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:05:18,892::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:05:28,947::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:05:28,947::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:05:31,180::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:05:31,227::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:31,368::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:05:31,369::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:31,375::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:05:31,378::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:05:31,379::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:05:33,575::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:05:33,576::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:05:37,959::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:05:37,966::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:05:38,081::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:05:38,081::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:05:40,541::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:05:40,541::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:05:40,541::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:05:42,783::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:05:42,814::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:42,987::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:05:42,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:42,993::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:05:42,996::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:05:47,407::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:05:47,407::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:05:57,468::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:05:57,468::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:05:59,693::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:05:59,740::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:59,897::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:05:59,898::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:05:59,904::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:05:59,906::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:05:59,906::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:05:59,908::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:06:02,102::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:06:02,102::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:06:06,481::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:06:06,488::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:06:06,603::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:06:06,603::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:06:09,065::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:06:09,065::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:06:09,065::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:06:11,305::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:06:11,336::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:11,494::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:06:11,494::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:11,500::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:06:11,503::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:06:15,896::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:06:15,896::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:06:25,962::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:06:25,962::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:06:28,229::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:06:28,260::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:28,434::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:06:28,434::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:28,440::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:06:28,443::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:06:28,444::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:06:30,629::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:06:30,629::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:06:34,987::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:06:34,995::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:06:35,109::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:06:35,110::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:06:37,582::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:06:37,582::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:06:37,582::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:06:39,820::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:06:39,852::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:39,993::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:06:39,993::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:39,999::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:06:40,002::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:06:44,391::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:06:44,391::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:06:54,458::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:06:54,458::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:06:56,706::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:06:56,737::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:56,894::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:06:56,895::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:06:56,901::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:06:56,904::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:06:56,905::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:06:59,119::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:06:59,119::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:07:03,470::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:07:03,477::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:07:03,592::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:07:03,592::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:07:06,029::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:07:06,030::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:07:06,030::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:07:08,272::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:07:08,319::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:08,492::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:07:08,493::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:08,499::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:07:08,502::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:07:12,890::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:07:12,890::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:07:22,958::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:07:22,958::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:07:25,205::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:07:25,237::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:25,362::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:07:25,362::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:25,368::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:07:25,371::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:07:25,371::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:07:25,373::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:07:27,579::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:07:27,579::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:07:31,953::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:07:31,961::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:07:32,075::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:07:32,076::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:07:34,580::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:07:34,580::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:07:34,580::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:07:36,815::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:07:36,847::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:36,988::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:07:36,988::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:36,994::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:07:36,996::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:07:41,393::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:07:41,394::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:07:51,459::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:07:51,459::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:07:53,702::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:07:53,733::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:53,890::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:07:53,891::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:07:53,897::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:07:53,899::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:07:53,901::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:07:56,106::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:07:56,107::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:00,510::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:00,518::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:08:00,632::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:08:00,633::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:08:03,063::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:08:03,063::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:08:03,063::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:08:05,298::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:08:05,329::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:05,454::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:08:05,455::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:05,461::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:08:05,463::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:08:09,845::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:08:09,845::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:08:19,905::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:08:19,905::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:08:22,146::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:08:22,194::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:22,367::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:08:22,368::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:22,374::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:08:22,376::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:08:22,378::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:08:24,582::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:08:24,582::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:28,976::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:28,983::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:08:29,098::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:08:29,098::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:08:31,591::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:08:31,591::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:08:31,591::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:08:33,816::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:08:33,847::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:33,972::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:08:33,973::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:33,979::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:08:33,981::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:08:38,358::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:08:38,358::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:08:48,452::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:08:48,452::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:08:50,687::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:08:50,734::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:50,907::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:08:50,908::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:08:50,914::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:08:50,917::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:08:50,917::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:08:50,919::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:08:53,118::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:08:53,118::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:57,493::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:08:57,500::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:08:57,615::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:08:57,616::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:09:00,083::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:09:00,083::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:09:00,083::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:09:02,338::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:09:02,385::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:02,511::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:09:02,511::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:02,517::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:09:02,520::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:09:06,889::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:09:06,889::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:09:16,955::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:09:16,955::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:09:19,224::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:09:19,255::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:19,412::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:09:19,413::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:19,419::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:09:19,422::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:09:19,422::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:09:19,422::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=886 (Thu Jun 15 09:08:44 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=900 (Thu Jun 15 09:08:58 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'c80eee37', 'local_conf_timestamp': 900, 'host-ts': 886} MainThread::INFO::2017-06-15 09:09:19,422::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=878 (Thu Jun 15 09:08:44 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=893 (Thu Jun 15 09:08:59 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '321e64bd', 'local_conf_timestamp': 893, 'host-ts': 878} MainThread::INFO::2017-06-15 09:09:19,422::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14962.0, 'maintenance': False, 'cpu-load': 0.0206, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:09:19,424::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:09:21,626::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:09:21,626::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:09:26,008::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:09:26,016::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:09:26,131::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:09:26,131::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:09:28,617::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:09:28,617::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:09:28,617::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:09:30,862::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:09:30,909::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:31,050::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:09:31,051::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:31,057::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:09:31,060::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:09:35,453::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:09:35,454::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:09:45,517::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:09:45,517::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:09:47,751::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:09:47,798::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:47,971::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:09:47,972::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:47,978::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:09:47,980::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:09:47,983::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:09:50,177::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:09:50,178::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:09:54,571::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:09:54,578::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:09:54,693::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:09:54,693::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:09:57,154::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:09:57,154::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:09:57,154::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:09:59,401::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:09:59,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:59,622::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:09:59,623::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:09:59,629::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:09:59,631::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:10:04,037::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:10:04,037::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:10:14,101::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:10:14,101::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:10:16,327::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:10:16,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:16,531::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:10:16,532::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:16,538::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:10:16,541::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:10:16,541::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:10:16,543::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:10:18,741::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:10:18,741::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:10:23,105::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:10:23,113::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:10:23,228::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:10:23,228::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:10:25,707::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:10:25,708::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:10:25,708::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:10:27,951::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:10:27,982::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:28,155::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:10:28,156::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:28,162::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:10:28,165::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:10:32,562::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:10:32,562::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:10:42,627::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:10:42,628::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:10:44,881::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:10:44,928::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:45,070::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:10:45,070::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:45,076::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:10:45,079::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:10:45,082::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:10:47,267::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:10:47,267::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:10:51,639::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:10:51,647::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:10:51,761::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:10:51,762::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:10:54,193::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:10:54,193::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:10:54,193::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:10:56,434::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:10:56,465::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:56,606::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:10:56,607::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:10:56,613::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:10:56,615::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:11:01,013::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:11:01,013::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:11:11,068::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:11:11,068::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:11:13,319::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:11:13,366::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:13,555::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:11:13,556::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:13,562::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:11:13,565::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:11:13,567::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:11:15,782::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:11:15,783::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:11:20,146::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:11:20,153::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:11:20,268::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:11:20,269::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:11:22,742::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:11:22,742::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:11:22,742::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:11:24,982::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:11:25,013::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:25,170::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:11:25,171::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:25,177::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:11:25,180::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:11:29,573::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:11:29,573::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:11:39,696::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:11:39,696::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:11:41,940::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:11:41,987::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:42,112::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:11:42,113::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:42,119::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:11:42,122::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:11:42,122::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:11:42,124::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:11:44,332::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:11:44,333::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:11:48,721::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:11:48,728::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:11:48,843::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:11:48,843::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:11:51,312::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:11:51,312::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:11:51,312::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:11:53,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:11:53,598::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:53,755::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:11:53,755::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:11:53,761::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:11:53,764::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:11:58,150::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:11:58,150::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:12:08,214::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:12:08,214::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:12:10,456::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:12:10,503::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:10,644::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:12:10,645::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:10,651::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:12:10,654::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:12:10,657::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:12:12,859::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:12:12,860::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:12:17,263::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:12:17,271::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:12:17,386::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:12:17,386::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:12:19,882::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:12:19,882::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:12:19,882::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:12:22,114::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:12:22,161::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:22,287::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:12:22,287::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:22,293::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:12:22,296::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:12:26,676::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:12:26,677::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:12:36,737::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:12:36,737::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:12:38,979::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:12:39,012::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:39,169::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:12:39,170::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:39,177::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:12:39,180::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:12:39,183::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:12:41,386::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:12:41,386::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:12:45,772::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:12:45,780::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:12:45,894::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:12:45,895::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:12:48,389::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:12:48,390::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:12:48,390::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:12:50,619::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:12:50,651::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:50,809::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:12:50,810::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:12:50,816::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:12:50,818::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:12:55,195::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:12:55,195::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:13:05,261::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:13:05,261::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:13:07,503::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:13:07,550::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:07,707::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:13:07,708::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:07,714::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:13:07,717::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:13:07,717::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:13:07,720::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:13:09,924::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:13:09,925::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:13:14,304::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:13:14,312::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:13:14,427::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:13:14,427::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:13:16,846::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:13:16,846::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:13:16,846::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:13:19,102::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:13:19,134::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:19,323::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:13:19,324::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:19,330::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:13:19,332::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:13:23,697::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:13:23,697::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:13:33,758::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:13:33,758::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:13:35,994::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:13:36,041::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:36,199::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:13:36,199::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:36,205::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:13:36,208::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:13:36,211::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:13:38,409::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:13:38,410::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:13:42,787::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:13:42,794::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:13:42,909::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:13:42,909::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:13:45,390::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:13:45,390::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:13:45,390::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:13:47,637::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:13:47,684::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:47,841::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:13:47,842::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:13:47,847::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:13:47,850::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:13:52,237::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:13:52,238::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:14:02,303::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:14:02,303::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:14:04,538::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:14:04,569::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:04,726::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:14:04,727::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:04,733::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:14:04,735::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:14:04,739::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:14:06,934::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:14:06,935::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:14:11,324::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:14:11,331::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:14:11,446::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:14:11,447::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:14:13,876::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:14:13,876::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:14:13,876::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:14:16,121::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:14:16,152::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:16,293::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:14:16,294::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:16,300::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:14:16,303::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:14:20,714::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:14:20,714::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:14:30,772::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:14:30,772::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:14:32,997::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:14:33,029::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:33,170::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:14:33,170::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:33,176::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:14:33,179::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:14:33,179::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 09:14:33,182::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:14:35,378::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:14:35,378::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:14:39,758::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:14:39,766::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:14:39,881::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:14:39,881::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:14:42,356::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:14:42,356::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:14:42,356::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:14:44,597::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:14:44,629::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:44,754::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:14:44,754::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:14:44,760::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:14:44,763::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:14:49,174::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:14:49,174::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:14:59,384::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:14:59,384::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:15:01,638::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:15:01,685::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:01,826::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:15:01,827::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:01,833::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:15:01,836::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:15:01,839::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:15:04,028::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:15:04,029::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:15:08,393::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:15:08,401::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:15:08,515::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:15:08,516::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:15:10,990::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:15:10,990::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:15:10,990::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:15:13,234::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:15:13,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:13,390::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:15:13,391::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:13,397::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:15:13,399::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:15:17,794::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:15:17,794::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:15:27,861::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:15:27,862::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:15:30,112::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:15:30,160::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:30,285::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:15:30,285::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:30,292::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:15:30,294::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:15:30,298::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:15:32,520::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:15:32,521::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:15:36,871::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:15:36,879::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:15:36,993::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:15:36,994::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:15:39,470::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:15:39,471::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:15:39,471::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:15:41,707::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:15:41,738::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:41,895::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:15:41,896::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:41,902::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:15:41,904::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:15:46,297::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:15:46,297::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:15:56,363::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:15:56,363::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:15:58,607::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:15:58,638::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:58,764::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:15:58,764::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:15:58,770::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:15:58,773::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:15:58,773::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 09:15:58,777::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532558.78 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:15:58,815::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 09:15:58,815::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:16:01,024::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:16:01,025::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:16:05,409::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:16:05,417::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:16:05,532::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:16:05,532::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:16:07,984::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:16:07,984::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:07,984::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:10,220::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:10,251::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:10,392::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:10,393::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:10,398::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:10,401::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:14,795::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 09:16:14,795::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:16:14,845::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:14,845::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:17,085::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:17,116::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:17,241::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:17,242::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:17,248::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:17,251::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:17,251::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:17,251::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:19,500::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:19,531::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:19,657::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:19,657::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:19,663::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:19,666::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:19,666::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 09:16:19,666::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:19,666::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:21,921::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:21,953::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:22,094::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:22,094::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:22,100::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:22,103::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:24,304::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 09:16:24,305::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 09:16:28,886::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 09:16:28,886::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner File "/usr/lib64/python2.7/threading.py", line 764, in run File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run File "/usr/lib/python2.7/site-packages/yajsonrpc/betterAsyncore.py", line 216, in process_requests File "/usr/lib64/python2.7/asyncore.py", line 220, in loop File "/usr/lib64/python2.7/asyncore.py", line 201, in poll2 : 'NoneType' object is not callable MainThread::INFO::2017-06-15 09:16:28,887::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 09:16:28,891::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532588.89 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:16:28,927::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 09:16:28,927::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:16:31,121::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:16:31,121::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:16:35,511::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:16:35,519::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:16:35,634::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:16:35,634::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:16:38,109::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:16:38,109::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:38,109::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:40,355::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:40,386::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:40,543::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:40,544::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:40,550::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:40,553::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:44,984::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:16:44,984::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:16:55,048::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:16:55,048::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:16:57,306::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:16:57,337::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:57,526::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:16:57,527::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:16:57,533::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:16:57,535::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:16:57,536::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 09:16:57,540::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 09:16:57,543::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532617.54 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:16:57,579::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 09:16:57,579::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:16:59,811::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:16:59,812::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:04,165::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:04,181::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:17:04,296::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:17:04,296::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:17:06,762::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:17:06,762::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:17:06,762::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:17:09,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:17:09,048::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:09,238::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:17:09,238::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:09,244::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:17:09,247::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:17:13,665::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 09:17:13,665::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:17:13,711::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:17:13,711::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:17:15,956::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:17:16,003::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:16,144::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:17:16,145::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:16,151::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:17:16,154::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:17:16,154::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 09:17:18,444::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 09:17:18,444::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 09:17:18,444::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 09:17:18,449::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532638.45 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:17:18,486::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 09:17:18,487::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:17:20,702::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:17:20,702::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:25,081::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:25,089::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:17:25,204::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:17:25,204::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:17:27,674::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:17:27,674::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:17:27,674::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:17:29,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:17:29,947::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:30,072::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:17:30,073::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:30,080::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:17:30,082::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:17:34,476::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:17:34,476::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:17:44,536::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:17:44,537::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:17:46,780::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:17:46,827::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:47,001::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:17:47,001::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:47,008::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:17:47,010::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:17:47,011::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:17:47,014::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:17:49,228::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:17:49,229::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:53,630::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:17:53,638::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:17:53,753::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:17:53,753::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:17:56,216::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:17:56,216::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:17:56,216::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:17:58,449::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:17:58,480::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:58,637::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:17:58,638::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:17:58,644::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:17:58,646::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:18:03,032::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:18:03,032::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:18:13,095::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:18:13,095::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:18:15,336::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:18:15,367::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:15,493::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:18:15,493::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:15,499::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:18:15,502::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:18:15,506::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:18:17,710::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:18:17,711::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:18:22,106::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:18:22,113::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:18:22,228::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:18:22,228::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:18:24,706::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:18:24,706::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:18:24,706::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:18:26,932::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:18:26,963::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:27,104::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:18:27,105::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:27,111::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:18:27,114::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:18:31,509::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:18:31,510::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:18:41,572::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:18:41,572::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:18:43,815::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:18:43,846::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:43,971::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:18:43,971::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:43,977::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:18:43,980::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:18:43,984::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:18:46,186::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:18:46,186::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:18:50,566::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:18:50,574::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:18:50,689::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:18:50,689::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:18:53,172::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:18:53,173::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:18:53,173::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:18:55,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:18:55,476::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:55,618::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:18:55,618::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:18:55,625::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:18:55,627::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:19:00,002::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:19:00,002::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:19:10,052::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:19:10,052::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:19:12,289::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:19:12,320::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:19:12,461::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:19:12,462::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:19:12,468::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:19:12,470::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:19:12,471::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:19:21,890::hosted_engine::779::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_domain_monitor) Stopped VDSM domain monitor for d022a79d-611a-4908-9211-9a81b55bcf76 MainThread::INFO::2017-06-15 09:19:21,892::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down MainThread::INFO::2017-06-15 09:19:22,023::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.2-0.0.master.20170605141622 started MainThread::INFO::2017-06-15 09:19:22,034::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-hyp-01.reis.com MainThread::INFO::2017-06-15 09:19:22,035::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:19:24,238::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:19:24,239::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:19:28,632::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:19:28,640::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:19:28,755::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:19:28,755::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:19:31,239::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:19:31,239::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:19:31,239::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:19:33,467::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:19:33,514::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:19:33,655::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:19:33,655::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:19:33,661::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:19:33,663::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:19:33,664::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2017-06-15 09:19:33,665::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.170.1'} MainThread::INFO::2017-06-15 09:19:33,666::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 41824976 MainThread::INFO::2017-06-15 09:19:33,666::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2017-06-15 09:19:33,670::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 139884939143952 MainThread::INFO::2017-06-15 09:19:33,670::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2017-06-15 09:19:33,671::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 139884939144080 MainThread::INFO::2017-06-15 09:19:33,671::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 09:19:33,671::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 139884939141776 MainThread::INFO::2017-06-15 09:19:33,671::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 09:19:33,672::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 139884939143376 MainThread::INFO::2017-06-15 09:19:36,935::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 139884939144528 MainThread::INFO::2017-06-15 09:19:36,936::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2017-06-15 09:19:36,957::hosted_engine::698::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/d022a79d-611a-4908-9211-9a81b55bcf76/7bbc8086-0d87-45e3-94a9-2f8e2fc8c9ec/fcf4f5dc-1a39-4094-81ae-8f0ff45197fb) MainThread::INFO::2017-06-15 09:19:57,960::hosted_engine::732::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Acquired lock on host id 1 MainThread::INFO::2017-06-15 09:20:00,135::upgrade::998::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Host configuration is already up-to-date MainThread::INFO::2017-06-15 09:20:00,135::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:20:00,135::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:20:02,379::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:20:02,411::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:02,552::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:20:02,552::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:02,558::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:20:02,560::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:20:02,593::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:20:02,593::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:20:04,833::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:20:04,864::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:04,989::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:20:04,990::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:04,995::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:20:04,998::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:20:04,998::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:20:04,998::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1544 (Thu Jun 15 09:19:42 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1558 (Thu Jun 15 09:19:56 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '4f39b4a0', 'local_conf_timestamp': 1558, 'host-ts': 1544} MainThread::INFO::2017-06-15 09:20:04,998::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1512 (Thu Jun 15 09:19:18 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=1526 (Thu Jun 15 09:19:32 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'cda6f0a2', 'local_conf_timestamp': 1526, 'host-ts': 1512} MainThread::INFO::2017-06-15 09:20:04,998::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14946.0, 'maintenance': False, 'cpu-load': 0.0129, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:20:04,999::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532805.0 type=state_transition detail=StartState-ReinitializeFSM hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:20:05,043::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2017-06-15 09:20:05,043::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:20:07,272::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:20:07,273::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:20:11,669::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:20:11,676::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:20:11,791::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:20:11,792::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:20:14,257::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:20:14,257::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:20:14,258::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:20:16,511::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:20:16,542::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:16,732::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:20:16,733::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:20:16,737::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:20:16,740::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:20:18,910::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: NONE MainThread::INFO::2017-06-15 09:20:21,086::hosted_engine::805::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_domain_monitor) Started VDSM domain monitor for d022a79d-611a-4908-9211-9a81b55bcf76 MainThread::INFO::2017-06-15 09:20:23,265::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:20:30,452::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:20:37,640::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:20:44,832::hosted_engine::848::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_domain_monitor_status) VDSM domain monitor status: PENDING MainThread::INFO::2017-06-15 09:20:52,054::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::ERROR::2017-06-15 09:20:52,054::brokerlink::280::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(_communicate) Connection closed: Connection closed MainThread::WARNING::2017-06-15 09:20:52,055::hosted_engine::469::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Error while monitoring engine: Connection closed MainThread::INFO::2017-06-15 09:20:52,055::hosted_engine::488::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Sleeping 60 seconds MainThread::ERROR::2017-06-15 09:21:52,099::agent::199::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Disconnected from broker 'Not connected to broker' - reinitializing MainThread::WARNING::2017-06-15 09:21:57,104::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0' MainThread::INFO::2017-06-15 09:21:57,116::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: ovirt-hyp-01.reis.com MainThread::INFO::2017-06-15 09:21:57,117::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:21:59,326::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:21:59,326::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:03,693::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:03,725::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:22:03,840::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:22:03,840::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:22:06,310::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:22:06,311::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:06,311::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:08,559::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:08,591::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:08,764::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:08,765::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:08,770::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:08,773::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:22:08,773::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection MainThread::INFO::2017-06-15 09:22:08,773::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.170.1'} MainThread::INFO::2017-06-15 09:22:08,774::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 36475536 MainThread::INFO::2017-06-15 09:22:08,774::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'} MainThread::INFO::2017-06-15 09:22:08,776::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 36477008 MainThread::INFO::2017-06-15 09:22:08,776::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'} MainThread::INFO::2017-06-15 09:22:08,777::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 36477136 MainThread::INFO::2017-06-15 09:22:08,777::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 09:22:08,777::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 36478864 MainThread::INFO::2017-06-15 09:22:08,778::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3d1f1ed0-8e8f-4d57-9bd7-eb0a50c6d36b', 'address': '0'} MainThread::INFO::2017-06-15 09:22:08,778::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 36492112 MainThread::INFO::2017-06-15 09:22:12,019::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 36492880 MainThread::INFO::2017-06-15 09:22:12,019::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started MainThread::INFO::2017-06-15 09:22:12,043::hosted_engine::698::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/d022a79d-611a-4908-9211-9a81b55bcf76/7bbc8086-0d87-45e3-94a9-2f8e2fc8c9ec/fcf4f5dc-1a39-4094-81ae-8f0ff45197fb) MainThread::INFO::2017-06-15 09:22:14,225::upgrade::998::ovirt_hosted_engine_ha.lib.upgrade.StorageServer::(upgrade_35_36) Host configuration is already up-to-date MainThread::INFO::2017-06-15 09:22:14,226::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:14,226::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:16,468::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:16,499::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:16,656::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:16,657::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:16,662::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:16,665::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:22:16,699::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:16,699::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:18,949::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:18,980::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:19,137::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:19,138::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:19,143::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:19,146::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:22:19,146::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:22:19,146::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1658 (Thu Jun 15 09:21:37 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=1672 (Thu Jun 15 09:21:51 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '16254b43', 'local_conf_timestamp': 1672, 'host-ts': 1658} MainThread::INFO::2017-06-15 09:22:19,146::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=1656 (Thu Jun 15 09:21:42 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=1670 (Thu Jun 15 09:21:56 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'b576f582', 'local_conf_timestamp': 1670, 'host-ts': 1656} MainThread::INFO::2017-06-15 09:22:19,146::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14966.0, 'maintenance': False, 'cpu-load': None, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:22:19,147::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532939.15 type=state_transition detail=StartState-ReinitializeFSM hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:22:19,180::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (StartState-ReinitializeFSM) sent? sent MainThread::INFO::2017-06-15 09:22:19,180::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:22:21,401::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:22:21,401::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:25,804::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:25,811::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:22:25,926::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:22:25,926::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:22:28,392::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:22:28,393::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:28,393::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:30,629::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:30,660::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:30,850::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:30,851::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:30,857::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:30,861::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:22:33,068::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state ReinitializeFSM (score: 0) MainThread::INFO::2017-06-15 09:22:43,129::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:43,129::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:45,378::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:45,425::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:45,566::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:45,567::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:45,571::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:45,574::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:22:45,575::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497532965.58 type=state_transition detail=ReinitializeFSM-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:22:45,609::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (ReinitializeFSM-EngineDown) sent? sent MainThread::INFO::2017-06-15 09:22:45,609::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:22:47,816::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:22:47,816::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:52,218::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:22:52,225::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:22:52,340::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:22:52,340::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:22:54,828::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:22:54,828::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:22:54,828::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:22:57,078::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:22:57,110::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:57,235::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:22:57,235::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:22:57,241::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:22:57,244::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:23:01,613::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:23:11,747::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:23:11,747::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:23:13,985::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:23:14,016::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:14,141::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:23:14,142::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:14,146::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:23:14,149::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:23:14,149::states::484::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) The engine is not running, but we do not have enough data to decide which hosts are alive MainThread::INFO::2017-06-15 09:23:14,150::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:23:16,352::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:23:16,352::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:23:20,743::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:23:20,750::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:23:20,866::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:23:20,866::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:23:23,348::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:23:23,348::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:23:23,348::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:23:25,599::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:23:25,631::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:25,772::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:23:25,773::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:25,779::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:23:25,781::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:23:30,173::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:23:40,240::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:23:40,240::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:23:42,476::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:23:42,508::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:42,649::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:23:42,649::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:42,654::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:23:42,657::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:23:42,657::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:23:42,658::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:23:44,857::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:23:44,858::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:23:49,233::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:23:49,240::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:23:49,355::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:23:49,355::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:23:51,841::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:23:51,842::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:23:51,842::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:23:54,087::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:23:54,134::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:54,308::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:23:54,308::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:23:54,314::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:23:54,317::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:23:58,727::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:23:58,728::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:24:08,794::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:24:08,794::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:24:11,031::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:24:11,094::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:11,220::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:24:11,221::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:11,226::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:24:11,228::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:24:11,229::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:24:13,425::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:24:13,425::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:24:17,810::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:24:17,818::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:24:17,932::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:24:17,933::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:24:20,429::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:24:20,429::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:24:20,430::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:24:22,674::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:24:22,721::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:22,894::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:24:22,895::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:22,900::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:24:22,902::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:24:27,307::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:24:27,307::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:24:37,375::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:24:37,375::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:24:39,627::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:24:39,658::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:39,784::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:24:39,784::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:39,790::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:24:39,793::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:24:39,794::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:24:42,031::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:24:42,031::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:24:46,395::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:24:46,402::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:24:46,517::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:24:46,517::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:24:48,994::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:24:48,995::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:24:48,995::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:24:51,244::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:24:51,291::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:51,449::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:24:51,449::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:24:51,454::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:24:51,457::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:24:55,856::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:24:55,856::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:25:05,923::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:25:05,923::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:25:08,175::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:25:08,206::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:08,331::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:25:08,332::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:08,338::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:25:08,340::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:25:08,340::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:25:08,341::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:25:10,555::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:25:10,555::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:25:14,949::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:25:14,957::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:25:15,071::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:25:15,072::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:25:17,548::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:25:17,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:25:17,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:25:19,787::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:25:19,819::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:19,960::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:25:19,960::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:19,965::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:25:19,968::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:25:24,370::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:25:24,370::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:25:34,436::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:25:34,436::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:25:36,689::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:25:36,721::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:36,846::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:25:36,846::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:36,851::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:25:36,854::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:25:36,855::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:25:39,067::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:25:39,067::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:25:43,476::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:25:43,483::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:25:43,598::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:25:43,598::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:25:46,046::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:25:46,046::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:25:46,046::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:25:48,283::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:25:48,315::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:48,439::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:25:48,440::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:25:48,446::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:25:48,449::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:25:52,839::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:25:52,839::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:26:02,896::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:26:02,896::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:26:05,143::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:26:05,174::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:05,315::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:26:05,315::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:05,321::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:26:05,324::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:26:05,325::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:26:07,538::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:26:07,538::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:26:11,942::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:26:11,958::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:26:12,072::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:26:12,073::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:26:14,539::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:26:14,539::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:26:14,539::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:26:16,778::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:26:16,825::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:16,998::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:26:16,998::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:17,004::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:26:17,007::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:26:21,406::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:26:21,406::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:26:31,472::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:26:31,472::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:26:33,720::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:26:33,767::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:33,924::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:26:33,925::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:33,930::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:26:33,932::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:26:33,933::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:26:33,934::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:26:36,150::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:26:36,151::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:26:40,548::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:26:40,556::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:26:40,671::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:26:40,671::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:26:43,147::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:26:43,147::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:26:43,147::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:26:45,408::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:26:45,455::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:45,644::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:26:45,645::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:26:45,651::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:26:45,653::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:26:50,027::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:26:50,027::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:27:00,087::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:27:00,087::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:27:02,351::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:27:02,398::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:02,523::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:27:02,524::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:02,529::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:27:02,532::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:27:02,533::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:27:04,737::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:27:04,737::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:27:09,127::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:27:09,135::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:27:09,250::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:27:09,250::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:27:11,739::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:27:11,739::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:27:11,739::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:27:13,990::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:27:14,038::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:14,179::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:27:14,179::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:14,185::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:27:14,188::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:27:18,587::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:27:18,587::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:27:28,652::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:27:28,652::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:27:30,897::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:27:30,944::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:31,086::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:27:31,086::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:31,092::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:27:31,094::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:27:31,097::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:27:33,299::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:27:33,300::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:27:37,689::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:27:37,697::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:27:37,812::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:27:37,812::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:27:40,296::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:27:40,296::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:27:40,296::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:27:42,544::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:27:42,592::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:42,749::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:27:42,749::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:42,755::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:27:42,758::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:27:47,180::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:27:47,180::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:27:57,250::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:27:57,250::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:27:59,484::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:27:59,515::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:59,641::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:27:59,641::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:27:59,647::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:27:59,650::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:27:59,650::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:27:59,652::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:28:01,858::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:28:01,858::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:28:06,236::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:28:06,244::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:28:06,359::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:28:06,359::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:28:08,826::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:28:08,826::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:28:08,827::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:28:11,078::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:28:11,126::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:11,283::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:28:11,284::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:11,289::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:28:11,292::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:28:15,703::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:28:15,703::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:28:25,770::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:28:25,770::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:28:28,030::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:28:28,077::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:28,202::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:28:28,203::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:28,208::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:28:28,211::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:28:28,213::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:28:30,405::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:28:30,405::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:28:34,784::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:28:34,791::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:28:34,906::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:28:34,906::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:28:37,384::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:28:37,384::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:28:37,384::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:28:39,628::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:28:39,659::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:39,832::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:28:39,833::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:39,839::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:28:39,842::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:28:44,249::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:28:44,249::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:28:54,313::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:28:54,314::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:28:56,573::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:28:56,604::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:56,761::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:28:56,762::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:28:56,767::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:28:56,769::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:28:56,772::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:28:58,997::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:28:58,997::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:29:03,362::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:29:03,370::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:29:03,485::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:29:03,485::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:29:05,961::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:29:05,961::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:29:05,961::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:29:08,202::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:29:08,250::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:08,423::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:29:08,424::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:08,430::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:29:08,432::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:29:12,837::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:29:12,837::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:29:22,900::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:29:22,900::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:29:25,153::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:29:25,184::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:25,309::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:29:25,310::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:25,316::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:29:25,318::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:29:25,319::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:29:25,321::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:29:27,537::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:29:27,538::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:29:31,930::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:29:31,946::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:29:32,060::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:29:32,061::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:29:34,505::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:29:34,506::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:29:34,506::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:29:36,748::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:29:36,795::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:36,921::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:29:36,921::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:36,927::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:29:36,929::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:29:41,327::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:29:41,327::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:29:51,392::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:29:51,392::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:29:53,644::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:29:53,676::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:53,849::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:29:53,849::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:29:53,855::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:29:53,857::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:29:53,860::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:29:56,073::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:29:56,073::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:00,479::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:00,486::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:30:00,601::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:30:00,601::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:30:03,124::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:30:03,124::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:03,125::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:05,364::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:05,395::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:05,552::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:05,553::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:05,558::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:05,561::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:09,953::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:30:09,953::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:30:20,021::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:20,022::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:22,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:22,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:22,515::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:22,516::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:22,521::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:22,524::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:22,524::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 09:30:22,526::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497533422.53 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:30:22,563::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 09:30:22,563::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:30:24,774::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:30:24,775::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:29,179::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:29,186::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:30:29,301::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:30:29,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:30:31,803::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:30:31,803::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:31,803::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:34,032::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:34,063::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:34,220::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:34,221::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:34,226::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:34,228::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:38,621::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 09:30:38,622::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:30:38,674::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:38,674::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:40,924::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:40,955::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:41,129::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:41,129::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:41,134::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:41,137::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:41,137::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:41,138::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:43,387::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:43,419::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:43,544::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:43,544::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:43,550::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:43,553::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:43,553::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 09:30:43,553::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:30:43,553::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:30:45,804::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:30:45,851::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:45,976::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:30:45,977::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:30:45,982::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:30:45,985::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:30:48,181::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 09:30:48,182::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 09:30:52,773::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 09:30:52,773::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner File "/usr/lib64/python2.7/threading.py", line 764, in run File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run File "/usr/lib/python2.7/site-packages/yajsonrpc/betterAsyncore.py", line 216, in process_requests File "/usr/lib64/python2.7/asyncore.py", line 220, in loop MainThread::INFO::2017-06-15 09:30:52,773::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 09:30:52,777::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497533452.78 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:30:52,809::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 09:30:52,809::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:30:55,066::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:30:55,067::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:59,503::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:30:59,510::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:30:59,625::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:30:59,626::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:31:02,103::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:31:02,103::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:31:02,103::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:31:04,403::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:31:04,452::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:04,694::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:31:04,695::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:04,702::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:31:04,705::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:31:09,200::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:31:09,200::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:31:19,268::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:31:19,268::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:31:21,678::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:31:21,726::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:21,949::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:31:21,949::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:21,956::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:31:21,959::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:31:21,960::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:31:21,962::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 09:41:19 2017 while transitioning -> MainThread::INFO::2017-06-15 09:31:21,965::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:31:24,191::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:31:24,192::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:31:28,613::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:31:28,621::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:31:28,736::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:31:28,736::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:31:31,221::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:31:31,221::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:31:31,222::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:31:33,464::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:31:33,512::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:33,637::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:31:33,638::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:33,643::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:31:33,645::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:31:38,067::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:31:38,067::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:31:48,146::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:31:48,146::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:31:50,396::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:31:50,427::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:50,568::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:31:50,569::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:31:50,573::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:31:50,576::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:31:50,576::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:31:50,579::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:31:52,798::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:31:52,799::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:31:57,208::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:31:57,216::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:31:57,331::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:31:57,331::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:31:59,846::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:31:59,846::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:31:59,846::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:32:02,118::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:32:02,165::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:02,354::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:32:02,355::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:02,360::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:32:02,362::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:32:06,752::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:32:06,753::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:32:16,810::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:32:16,810::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:32:19,075::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:32:19,106::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:19,231::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:32:19,232::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:19,237::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:32:19,240::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:32:19,240::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:32:19,240::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2259 (Thu Jun 15 09:31:37 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2273 (Thu Jun 15 09:31:51 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineForceStop\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'Storage of VM is locked. Is another host already starting the VM?', 'health': 'bad', 'vm': 'already_locked', 'detail': 'down'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '9535fffd', 'local_conf_timestamp': 2273, 'host-ts': 2259} MainThread::INFO::2017-06-15 09:32:19,241::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2259 (Thu Jun 15 09:31:45 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=2276 (Thu Jun 15 09:32:02 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'Storage of VM is locked. Is another host already starting the VM?', 'health': 'bad', 'vm': 'already_locked', 'detail': 'down'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '0e90c0c2', 'local_conf_timestamp': 2276, 'host-ts': 2259} MainThread::INFO::2017-06-15 09:32:19,241::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 12768.0, 'maintenance': False, 'cpu-load': 0.1711, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:32:19,241::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:32:19,243::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:32:21,468::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:32:21,469::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:32:25,868::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:32:25,876::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:32:25,990::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:32:25,991::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:32:28,488::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:32:28,488::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:32:28,488::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:32:30,754::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:32:30,802::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:30,991::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:32:30,992::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:30,998::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:32:31,000::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:32:35,378::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:32:35,378::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:32:45,445::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:32:45,445::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:32:47,695::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:32:47,726::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:47,850::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:32:47,851::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:47,856::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:32:47,859::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:32:47,859::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:32:47,862::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:32:50,067::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:32:50,068::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:32:54,472::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:32:54,480::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:32:54,595::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:32:54,595::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:32:57,070::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:32:57,070::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:32:57,071::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:32:59,332::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:32:59,363::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:59,553::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:32:59,553::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:32:59,558::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:32:59,561::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:33:03,961::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:33:03,962::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:33:14,027::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:33:14,027::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:33:16,269::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:33:16,301::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:16,490::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:33:16,491::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:16,496::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:33:16,498::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:33:16,499::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:33:16,502::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:33:18,703::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:33:18,704::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:33:23,088::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:33:23,096::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:33:23,211::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:33:23,211::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:33:25,681::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:33:25,682::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:33:25,682::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:33:27,949::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:33:27,997::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:28,186::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:33:28,187::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:28,192::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:33:28,194::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:33:32,621::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:33:32,621::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:33:42,673::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:33:42,673::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:33:44,922::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:33:44,953::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:45,143::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:33:45,143::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:45,148::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:33:45,151::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:33:45,151::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:33:45,154::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:33:47,354::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:33:47,355::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:33:51,734::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:33:51,742::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:33:51,857::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:33:51,857::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:33:54,341::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:33:54,341::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:33:54,341::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:33:56,593::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:33:56,624::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:56,830::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:33:56,830::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:33:56,835::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:33:56,838::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:34:01,256::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:34:01,256::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:34:11,314::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:34:11,314::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:34:13,579::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:34:13,626::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:13,816::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:34:13,817::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:13,822::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:34:13,824::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:34:13,825::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:34:13,828::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:34:16,020::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:34:16,020::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:34:20,394::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:34:20,402::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:34:20,517::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:34:20,517::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:34:23,007::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:34:23,007::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:34:23,007::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:34:25,256::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:34:25,287::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:25,477::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:34:25,477::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:25,482::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:34:25,485::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:34:29,896::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:34:29,896::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:34:39,956::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:34:39,956::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:34:42,218::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:34:42,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:42,455::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:34:42,456::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:42,461::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:34:42,464::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:34:42,464::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:34:42,468::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:34:44,693::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:34:44,693::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:34:49,055::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:34:49,062::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:34:49,177::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:34:49,177::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:34:51,656::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:34:51,656::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:34:51,656::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:34:53,902::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:34:53,949::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:54,123::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:34:54,124::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:34:54,129::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:34:54,131::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:34:58,536::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:34:58,536::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:35:08,604::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:35:08,604::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:35:10,855::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:35:10,902::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:11,076::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:35:11,076::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:11,081::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:35:11,084::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:35:11,084::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:35:11,088::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:35:13,308::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:35:13,308::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:35:17,700::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:35:17,708::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:35:17,823::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:35:17,823::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:35:20,283::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:35:20,283::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:35:20,283::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:35:22,537::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:35:22,569::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:22,758::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:35:22,759::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:22,766::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:35:22,768::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:35:27,194::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:35:27,194::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:35:37,260::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:35:37,260::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:35:39,506::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:35:39,570::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:39,760::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:35:39,761::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:39,767::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:35:39,770::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:35:39,771::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:35:39,774::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:35:41,996::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:35:41,996::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:35:46,413::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:35:46,421::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:35:46,536::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:35:46,536::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:35:49,003::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:35:49,003::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:35:49,004::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:35:51,263::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:35:51,310::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:51,500::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:35:51,501::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:35:51,506::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:35:51,509::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:35:55,907::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:35:55,907::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:36:05,968::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:36:05,968::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:36:08,215::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:36:08,247::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:08,436::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:36:08,437::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:08,442::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:36:08,445::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:36:08,445::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:36:08,449::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:36:10,664::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:36:10,664::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:36:15,072::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:36:15,080::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:36:15,194::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:36:15,195::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:36:17,678::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:36:17,679::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:36:17,679::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:36:19,912::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:36:19,943::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:20,116::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:36:20,117::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:20,123::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:36:20,125::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:36:24,520::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:36:24,520::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:36:34,578::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:36:34,579::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:36:36,822::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:36:36,853::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:37,043::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:36:37,044::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:37,050::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:36:37,053::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:36:37,053::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:36:37,057::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:36:39,266::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:36:39,266::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:36:43,657::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:36:43,665::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:36:43,780::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:36:43,780::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:36:46,268::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:36:46,269::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:36:46,269::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:36:48,529::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:36:48,576::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:48,766::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:36:48,766::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:36:48,772::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:36:48,775::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:36:53,160::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:36:53,161::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:37:03,221::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:37:03,221::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:37:05,462::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:37:05,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:05,651::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:37:05,652::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:05,657::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:37:05,660::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:37:05,660::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:37:05,665::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:37:07,872::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:37:07,872::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:37:12,262::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:37:12,270::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:37:12,385::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:37:12,385::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:37:14,859::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:37:14,860::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:37:14,860::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:37:17,112::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:37:17,143::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:17,284::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:37:17,285::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:17,291::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:37:17,294::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:37:21,707::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:37:21,707::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:37:31,790::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:37:31,790::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:37:34,031::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:37:34,063::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:34,252::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:37:34,253::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:34,259::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:37:34,262::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:37:34,262::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:37:34,266::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:37:36,471::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:37:36,471::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:37:40,856::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:37:40,864::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:37:40,979::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:37:40,979::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:37:43,439::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:37:43,440::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:37:43,440::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:37:45,689::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:37:45,721::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:45,894::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:37:45,895::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:37:45,901::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:37:45,903::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:37:50,328::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:37:50,329::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:38:00,397::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:38:00,397::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:38:02,628::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:38:02,675::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:02,801::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:38:02,802::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:02,807::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:38:02,810::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:38:02,810::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:38:02,814::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:38:05,016::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:38:05,016::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:38:09,401::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:38:09,408::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:38:09,523::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:38:09,524::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:38:12,015::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:38:12,016::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:38:12,016::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:38:14,265::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:38:14,312::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:14,438::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:38:14,438::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:14,444::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:38:14,447::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:38:18,864::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:38:18,865::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:38:28,930::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:38:28,930::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:38:31,198::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:38:31,245::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:31,418::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:38:31,419::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:31,424::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:38:31,427::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:38:31,427::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:38:31,431::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:38:33,623::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:38:33,623::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:38:38,000::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:38:38,008::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:38:38,123::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:38:38,123::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:38:40,609::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:38:40,609::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:38:40,609::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:38:42,857::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:38:42,904::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:43,077::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:38:43,078::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:43,084::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:38:43,087::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:38:47,498::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:38:47,498::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:38:57,566::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:38:57,566::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:38:59,819::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:38:59,850::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:59,975::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:38:59,976::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:38:59,982::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:38:59,984::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:38:59,985::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:38:59,989::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:39:02,224::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:39:02,225::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:39:06,595::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:39:06,602::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:39:06,717::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:39:06,718::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:39:09,196::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:39:09,197::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:39:09,197::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:39:11,443::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:39:11,491::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:11,648::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:39:11,648::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:11,653::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:39:11,656::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:39:16,061::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:39:16,062::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:39:26,127::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:39:26,127::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:39:28,377::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:39:28,425::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:28,582::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:39:28,582::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:28,588::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:39:28,590::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:39:28,591::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:39:28,594::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:39:30,811::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:39:30,812::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:39:35,191::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:39:35,199::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:39:35,314::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:39:35,314::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:39:37,792::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:39:37,793::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:39:37,793::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:39:40,033::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:39:40,064::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:40,206::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:39:40,207::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:40,212::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:39:40,216::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:39:44,619::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:39:44,619::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:39:54,674::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:39:54,674::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:39:56,923::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:39:56,971::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:57,112::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:39:57,113::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:39:57,119::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:39:57,122::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:39:57,122::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:39:57,126::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:39:59,339::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:39:59,340::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:40:03,757::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:40:03,764::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:40:03,879::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:40:03,879::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:40:06,345::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:40:06,345::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:40:06,345::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:40:08,585::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:40:08,616::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:08,741::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:40:08,742::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:08,748::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:40:08,751::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:40:13,142::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:40:13,143::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:40:23,202::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:40:23,202::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:40:25,450::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:40:25,497::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:25,687::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:40:25,687::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:25,692::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:40:25,695::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:40:25,695::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:40:25,699::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:40:27,910::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:40:27,910::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:40:32,311::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:40:32,319::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:40:32,433::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:40:32,434::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:40:34,931::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:40:34,931::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:40:34,931::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:40:37,160::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:40:37,207::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:37,381::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:40:37,381::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:37,387::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:40:37,390::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:40:41,789::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:40:41,789::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:40:51,887::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:40:51,887::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:40:54,133::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:40:54,180::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:54,354::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:40:54,355::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:40:54,361::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:40:54,363::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:40:54,364::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 09:40:54,367::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:40:56,578::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:40:56,578::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:41:00,991::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:41:00,999::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:41:01,114::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:41:01,114::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:41:03,602::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:41:03,603::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:41:03,603::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:41:05,864::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:41:05,896::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:06,020::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:41:06,021::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:06,027::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:41:06,030::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:41:10,408::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:41:10,408::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:41:20,472::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:41:20,472::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:41:22,719::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:41:22,766::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:22,955::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:41:22,956::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:22,961::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:41:22,963::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:41:22,967::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534082.97 type=state_transition detail=EngineStarting-EngineStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:41:22,997::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineStop) sent? sent MainThread::INFO::2017-06-15 09:41:22,997::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:41:25,204::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:41:25,205::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:41:29,598::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:41:29,606::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:41:29,721::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:41:29,721::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:41:32,216::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:41:32,216::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:41:32,216::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:41:34,474::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:41:34,506::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:34,663::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:41:34,664::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:34,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:41:34,672::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:41:39,075::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:41:39,076::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:41:49,142::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:41:49,142::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:41:51,384::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:41:51,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:51,621::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:41:51,622::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:41:51,627::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:41:51,630::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:41:51,630::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2017-06-15 09:41:53,917::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 09:41:53,918::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::ERROR::2017-06-15 09:41:53,918::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 09:41:53,924::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 09:46:49 2017 while transitioning -> MainThread::INFO::2017-06-15 09:41:53,928::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:41:56,132::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:41:56,132::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:00,524::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:00,532::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:42:00,647::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:42:00,647::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:42:03,137::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:42:03,137::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:42:03,137::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:42:05,391::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:42:05,422::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:05,580::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:42:05,580::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:05,586::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:42:05,589::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:42:10,014::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:42:10,015::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:42:20,080::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:42:20,080::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:42:22,336::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:42:22,367::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:22,541::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:42:22,541::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:22,546::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:42:22,549::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:42:22,549::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:42:22,549::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2878 (Thu Jun 15 09:41:57 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=2893 (Thu Jun 15 09:42:11 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'cfc5d391', 'local_conf_timestamp': 2893, 'host-ts': 2878} MainThread::INFO::2017-06-15 09:42:22,550::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=2864 (Thu Jun 15 09:41:50 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=2879 (Thu Jun 15 09:42:04 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'f63f8fde', 'local_conf_timestamp': 2879, 'host-ts': 2864} MainThread::INFO::2017-06-15 09:42:22,550::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'powering down'}, 'bridge': True, 'mem-free': 12494.0, 'maintenance': False, 'cpu-load': 0.0207, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:42:22,550::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (31 of 300 seconds) MainThread::INFO::2017-06-15 09:42:22,554::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:42:24,758::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:42:24,758::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:29,137::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:29,144::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:42:29,259::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:42:29,260::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:42:31,741::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:42:31,742::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:42:31,742::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:42:33,994::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:42:34,042::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:34,215::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:42:34,216::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:34,222::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:42:34,225::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:42:38,639::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:42:38,639::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:42:48,708::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:42:48,708::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:42:50,971::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:42:51,002::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:51,160::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:42:51,161::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:42:51,167::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:42:51,169::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:42:51,170::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (60 of 300 seconds) MainThread::INFO::2017-06-15 09:42:51,174::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:42:53,366::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:42:53,366::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:57,742::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:42:57,749::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:42:57,864::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:42:57,864::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:43:00,344::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:43:00,344::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:43:00,345::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:43:02,590::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:43:02,621::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:02,763::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:43:02,763::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:02,769::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:43:02,772::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:43:07,179::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:43:07,179::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:43:17,247::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:43:17,247::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:43:19,501::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:43:19,548::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:19,737::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:43:19,738::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:19,744::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:43:19,747::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:43:19,747::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (88 of 300 seconds) MainThread::INFO::2017-06-15 09:43:19,751::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:43:21,976::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:43:21,976::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:43:26,339::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:43:26,347::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:43:26,462::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:43:26,462::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:43:28,938::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:43:28,938::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:43:28,939::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:43:31,184::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:43:31,231::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:31,421::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:43:31,422::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:31,427::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:43:31,430::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:43:35,837::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:43:35,837::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:43:45,905::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:43:45,905::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:43:48,157::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:43:48,188::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:48,346::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:43:48,346::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:43:48,352::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:43:48,355::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:43:48,355::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (117 of 300 seconds) MainThread::INFO::2017-06-15 09:43:48,359::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:43:50,574::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:43:50,575::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:43:54,963::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:43:54,971::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:43:55,086::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:43:55,086::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:43:57,564::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:43:57,565::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:43:57,565::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:43:59,816::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:43:59,847::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:00,037::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:44:00,038::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:00,044::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:44:00,047::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:44:04,464::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 09:44:04,464::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:44:14,530::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:44:14,530::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:44:16,786::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:44:16,818::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:16,959::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:44:16,960::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:16,966::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:44:16,969::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:44:16,969::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm not running on local host MainThread::INFO::2017-06-15 09:44:16,973::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 09:44:16,977::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534256.98 type=state_transition detail=EngineStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:44:17,013::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 09:44:17,013::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:44:19,229::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:44:19,229::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:44:23,645::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:44:23,653::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:44:23,768::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:44:23,768::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:44:26,237::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:44:26,237::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:44:26,238::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:44:28,483::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:44:28,515::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:28,640::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:44:28,640::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:28,645::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:44:28,648::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:44:33,045::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:44:33,045::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:44:43,114::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:44:43,114::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:44:45,381::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:44:45,428::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:45,569::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:44:45,570::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:45,576::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:44:45,579::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:44:45,579::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 09:44:45,583::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534285.58 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:44:45,615::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 09:44:45,615::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:44:47,824::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:44:47,825::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:44:52,227::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:44:52,235::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:44:52,350::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:44:52,350::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:44:54,858::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:44:54,858::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:44:54,858::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:44:57,110::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:44:57,141::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:57,331::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:44:57,332::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:44:57,338::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:44:57,341::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:01,738::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 09:45:01,738::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:45:01,790::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:01,790::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:45:04,041::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:45:04,072::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:04,230::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:45:04,230::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:04,236::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:45:04,239::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:04,240::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:04,240::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:45:06,515::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:45:06,546::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:06,720::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:45:06,721::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:06,727::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:45:06,729::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:06,730::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 09:45:06,730::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:06,730::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:45:08,998::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:45:09,046::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:09,219::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:45:09,220::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:09,226::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:45:09,228::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:11,432::hosted_engine::1186::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Cleaning state for non-running VM MainThread::INFO::2017-06-15 09:45:16,843::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 09:45:16,844::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 09:45:21,446::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 09:45:21,446::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::INFO::2017-06-15 09:45:21,446::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 09:45:21,451::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534321.45 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:45:21,484::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 09:45:21,484::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:45:23,689::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:45:23,690::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:45:28,065::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:45:28,072::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:45:28,187::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:45:28,188::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:45:30,924::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:45:30,925::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:30,925::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:45:33,179::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:45:33,226::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:33,415::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:45:33,416::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:33,423::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:45:33,425::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:37,872::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 09:45:37,873::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:45:47,937::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:47,937::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:45:50,221::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:45:50,268::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:50,394::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:45:50,394::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:45:50,400::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:45:50,403::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:45:50,403::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 09:45:50,407::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 09:45:50,411::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534350.41 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:45:50,445::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 09:45:50,445::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:45:52,677::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:45:52,677::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:45:57,052::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:45:57,059::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:45:57,174::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:45:57,175::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:45:59,762::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:45:59,763::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:45:59,763::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:46:02,021::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:46:02,052::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:02,210::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:46:02,210::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:02,216::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:46:02,218::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:46:06,625::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 09:46:06,625::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:46:06,676::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:46:06,676::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:46:08,927::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:46:08,974::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:09,099::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:46:09,100::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:09,105::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:46:09,108::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:46:09,108::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 09:46:11,404::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 09:46:11,405::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 09:46:11,405::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 09:46:11,409::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497534371.41 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:46:11,443::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 09:46:11,444::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:46:13,669::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:46:13,669::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:46:18,054::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:46:18,062::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:46:18,177::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:46:18,177::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:46:20,573::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:46:20,573::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:46:20,573::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:46:22,815::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:46:22,847::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:22,972::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:46:22,972::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:22,978::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:46:22,981::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:46:27,382::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:46:27,382::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:46:37,447::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:46:37,447::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:46:39,695::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:46:39,726::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:39,851::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:46:39,852::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:39,858::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:46:39,861::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:46:39,861::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:46:39,865::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:46:42,103::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:46:42,103::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:46:46,512::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:46:46,520::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:46:46,635::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:46:46,635::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:46:49,068::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:46:49,068::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:46:49,068::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:46:51,334::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:46:51,365::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:51,522::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:46:51,523::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:46:51,528::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:46:51,531::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:46:55,924::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:46:55,924::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:47:05,980::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:47:05,981::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:47:08,227::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:47:08,274::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:08,399::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:47:08,400::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:08,406::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:47:08,408::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:47:08,413::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:47:10,621::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:47:10,621::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:47:15,030::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:47:15,038::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:47:15,153::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:47:15,153::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:47:17,634::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:47:17,634::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:47:17,634::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:47:19,865::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:47:19,896::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:20,022::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:47:20,022::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:20,028::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:47:20,030::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:47:24,419::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:47:24,420::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:47:34,489::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:47:34,489::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:47:36,732::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:47:36,779::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:36,936::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:47:36,937::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:36,943::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:47:36,946::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:47:36,950::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:47:39,159::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:47:39,160::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:47:43,561::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:47:43,568::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:47:43,683::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:47:43,684::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:47:46,171::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:47:46,172::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:47:46,172::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:47:48,433::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:47:48,480::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:48,621::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:47:48,622::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:47:48,628::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:47:48,630::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:47:53,009::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:47:53,009::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:48:03,066::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:48:03,066::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:48:05,307::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:48:05,354::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:05,528::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:48:05,529::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:05,534::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:48:05,536::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:48:05,536::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:48:05,540::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:48:07,746::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:48:07,747::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:48:12,143::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:48:12,151::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:48:12,266::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:48:12,266::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:48:14,755::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:48:14,756::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:48:14,756::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:48:17,009::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:48:17,040::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:17,197::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:48:17,198::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:17,204::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:48:17,207::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:48:21,613::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:48:21,613::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:48:31,685::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:48:31,685::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:48:33,922::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:48:33,970::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:34,095::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:48:34,095::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:34,101::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:48:34,104::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:48:34,108::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:48:36,310::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:48:36,310::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:48:40,701::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:48:40,708::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:48:40,823::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:48:40,824::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:48:43,299::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:48:43,299::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:48:43,299::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:48:45,554::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:48:45,601::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:45,726::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:48:45,727::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:48:45,733::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:48:45,735::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:48:50,166::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:48:50,166::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:49:00,226::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:49:00,226::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:49:02,468::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:49:02,516::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:02,640::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:49:02,641::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:02,647::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:49:02,650::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:49:02,654::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:49:04,854::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:49:04,854::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:49:09,242::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:49:09,250::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:49:09,364::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:49:09,365::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:49:11,849::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:49:11,850::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:49:11,850::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:49:14,098::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:49:14,145::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:14,287::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:49:14,287::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:14,293::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:49:14,295::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:49:18,728::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:49:18,728::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:49:28,823::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:49:28,824::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:49:31,083::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:49:31,146::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:31,288::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:49:31,289::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:31,294::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:49:31,297::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:49:31,297::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:49:31,301::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:49:33,494::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:49:33,494::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:49:37,868::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:49:37,876::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:49:37,990::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:49:37,991::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:49:40,455::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:49:40,455::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:49:40,455::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:49:42,704::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:49:42,751::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:42,892::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:49:42,893::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:42,899::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:49:42,901::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:49:47,313::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:49:47,314::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:49:57,381::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:49:57,381::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:49:59,635::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:49:59,667::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:59,824::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:49:59,824::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:49:59,829::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:49:59,832::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:49:59,836::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:50:02,058::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:50:02,059::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:50:06,423::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:50:06,430::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:50:06,545::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:50:06,545::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:50:09,024::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:50:09,024::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:50:09,024::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:50:11,270::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:50:11,302::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:11,459::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:50:11,460::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:11,466::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:50:11,469::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:50:15,876::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:50:15,876::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:50:25,938::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:50:25,938::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:50:28,190::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:50:28,237::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:28,442::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:50:28,443::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:28,449::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:50:28,452::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:50:28,456::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:50:30,672::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:50:30,672::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:50:35,056::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:50:35,064::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:50:35,178::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:50:35,179::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:50:37,654::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:50:37,654::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:50:37,654::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:50:39,896::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:50:39,944::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:40,069::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:50:40,069::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:40,076::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:50:40,079::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:50:44,485::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:50:44,485::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:50:54,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:50:54,549::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:50:56,796::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:50:56,843::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:57,033::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:50:57,033::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:50:57,040::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:50:57,042::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:50:57,043::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:50:57,047::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:50:59,261::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:50:59,261::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:51:03,680::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:51:03,688::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:51:03,803::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:51:03,803::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:51:06,238::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:51:06,239::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:51:06,239::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:51:08,477::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:51:08,524::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:08,649::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:51:08,649::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:08,654::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:51:08,657::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:51:13,051::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:51:13,051::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:51:23,118::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:51:23,118::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:51:25,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:51:25,403::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:25,529::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:51:25,529::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:25,534::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:51:25,537::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:51:25,541::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:51:27,751::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:51:27,751::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:51:32,162::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:51:32,226::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:51:32,341::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:51:32,341::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:51:34,807::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:51:34,807::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:51:34,807::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:51:37,039::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:51:37,070::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:37,195::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:51:37,196::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:37,202::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:51:37,204::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:51:41,596::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:51:41,596::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:51:51,657::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:51:51,658::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:51:53,901::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:51:53,948::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:54,073::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:51:54,074::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:51:54,080::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:51:54,083::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:51:54,087::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:51:56,298::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:51:56,298::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:00,688::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:00,696::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:52:00,810::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:52:00,811::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:52:03,286::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:52:03,286::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:52:03,286::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:52:05,545::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:52:05,593::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:05,718::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:52:05,718::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:05,724::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:52:05,727::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:52:10,106::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:52:10,106::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:52:20,169::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:52:20,169::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:52:22,418::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:52:22,514::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:22,671::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:52:22,672::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:22,678::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:52:22,681::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:52:22,681::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 09:52:22,681::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3473 (Thu Jun 15 09:51:51 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=3487 (Thu Jun 15 09:52:06 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'ca1c47bb', 'local_conf_timestamp': 3487, 'host-ts': 3473} MainThread::INFO::2017-06-15 09:52:22,681::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=3471 (Thu Jun 15 09:51:57 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=3485 (Thu Jun 15 09:52:11 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'd884bbe9', 'local_conf_timestamp': 3485, 'host-ts': 3471} MainThread::INFO::2017-06-15 09:52:22,681::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14940.0, 'maintenance': False, 'cpu-load': 0.02, 'gateway': 1.0} MainThread::INFO::2017-06-15 09:52:22,682::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:52:22,686::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:52:24,892::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:52:24,893::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:29,281::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:29,289::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:52:29,403::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:52:29,404::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:52:31,892::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:52:31,892::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:52:31,892::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:52:34,145::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:52:34,177::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:34,302::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:52:34,302::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:34,308::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:52:34,311::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:52:38,710::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:52:38,710::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:52:48,776::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:52:48,777::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:52:51,017::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:52:51,048::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:51,205::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:52:51,206::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:52:51,212::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:52:51,215::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:52:51,219::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:52:53,422::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:52:53,422::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:57,811::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:52:57,819::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:52:57,934::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:52:57,934::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:53:00,402::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:53:00,402::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:53:00,402::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:53:02,656::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:53:02,703::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:02,860::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:53:02,861::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:02,867::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:53:02,870::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:53:07,304::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:53:07,304::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:53:17,373::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:53:17,373::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:53:19,606::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:53:19,637::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:19,762::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:53:19,763::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:19,769::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:53:19,772::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:53:19,776::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:53:21,983::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:53:21,984::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:53:26,372::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:53:26,380::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:53:26,495::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:53:26,495::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:53:28,979::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:53:28,979::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:53:28,980::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:53:31,233::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:53:31,264::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:31,389::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:53:31,390::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:31,396::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:53:31,399::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:53:35,819::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:53:35,820::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:53:45,887::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:53:45,887::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:53:48,148::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:53:48,196::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:48,321::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:53:48,322::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:48,328::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:53:48,330::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:53:48,331::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:53:48,334::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:53:50,525::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:53:50,526::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:53:54,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:53:54,914::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:53:55,029::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:53:55,029::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:53:57,509::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:53:57,509::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:53:57,509::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:53:59,757::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:53:59,804::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:59,930::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:53:59,930::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:53:59,936::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:53:59,939::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:54:04,345::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:54:04,345::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:54:14,411::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:54:14,411::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:54:16,667::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:54:16,698::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:16,855::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:54:16,856::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:16,862::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:54:16,864::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:54:16,868::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:54:19,091::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:54:19,092::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:54:23,470::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:54:23,478::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:54:23,593::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:54:23,593::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:54:26,073::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:54:26,073::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:54:26,073::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:54:28,319::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:54:28,367::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:28,508::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:54:28,509::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:28,515::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:54:28,518::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:54:32,920::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:54:32,920::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:54:42,984::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:54:42,984::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:54:45,240::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:54:45,272::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:45,397::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:54:45,398::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:45,403::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:54:45,406::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:54:45,410::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:54:47,624::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:54:47,625::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:54:52,028::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:54:52,036::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:54:52,151::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:54:52,151::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:54:54,610::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:54:54,610::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:54:54,610::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:54:56,851::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:54:56,899::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:57,040::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:54:57,040::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:54:57,046::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:54:57,048::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:55:01,448::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:55:01,448::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:55:11,512::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:55:11,513::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:55:13,770::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:55:13,801::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:13,927::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:55:13,927::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:13,934::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:55:13,936::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:55:13,937::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:55:13,940::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:55:16,151::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:55:16,152::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:55:20,581::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:55:20,597::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:55:20,711::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:55:20,712::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:55:23,185::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:55:23,185::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:55:23,185::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:55:25,426::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:55:25,457::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:25,581::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:55:25,582::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:25,587::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:55:25,590::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:55:29,993::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:55:29,993::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:55:40,055::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:55:40,055::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:55:42,304::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:55:42,351::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:42,524::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:55:42,525::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:42,531::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:55:42,533::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:55:42,537::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:55:44,747::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:55:44,748::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:55:49,158::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:55:49,166::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:55:49,280::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:55:49,281::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:55:51,779::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:55:51,780::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:55:51,780::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:55:54,012::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:55:54,043::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:54,200::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:55:54,201::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:55:54,206::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:55:54,209::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:55:58,604::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:55:58,605::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:56:08,669::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:56:08,669::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:56:10,919::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:56:10,966::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:11,091::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:56:11,092::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:11,100::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:56:11,102::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:56:11,107::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:56:13,322::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:56:13,323::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:56:17,731::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:56:17,738::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:56:17,853::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:56:17,853::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:56:20,343::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:56:20,343::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:56:20,344::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:56:22,626::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:56:22,657::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:22,798::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:56:22,799::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:22,804::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:56:22,807::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:56:27,203::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:56:27,203::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:56:37,268::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:56:37,269::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:56:39,511::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:56:39,542::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:39,667::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:56:39,668::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:39,674::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:56:39,677::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:56:39,677::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:56:39,681::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:56:41,894::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:56:41,895::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:56:46,292::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:56:46,300::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:56:46,415::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:56:46,415::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:56:48,869::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:56:48,869::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:56:48,869::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:56:51,121::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:56:51,152::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:51,294::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:56:51,294::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:56:51,301::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:56:51,303::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:56:55,703::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:56:55,703::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:57:05,769::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:57:05,769::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:57:08,008::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:57:08,039::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:08,164::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:57:08,165::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:08,171::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:57:08,174::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:57:08,178::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:57:10,385::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:57:10,386::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:57:14,774::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:57:14,782::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:57:14,897::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:57:14,897::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:57:17,382::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:57:17,382::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:57:17,382::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:57:19,639::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:57:19,686::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:19,811::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:57:19,812::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:19,818::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:57:19,820::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:57:24,281::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:57:24,281::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:57:34,346::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:57:34,346::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:57:36,577::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:57:36,624::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:36,749::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:57:36,750::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:36,755::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:57:36,758::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:57:36,762::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:57:38,964::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:57:38,964::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:57:43,359::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:57:43,367::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:57:43,482::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:57:43,482::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:57:45,955::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:57:45,955::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:57:45,955::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:57:48,204::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:57:48,236::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:48,360::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:57:48,361::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:57:48,367::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:57:48,370::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:57:52,804::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:57:52,804::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:58:02,925::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:58:02,925::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:58:05,206::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:58:05,253::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:05,394::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:58:05,395::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:05,400::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:58:05,403::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:58:05,403::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 09:58:05,407::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:58:07,598::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:58:07,598::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:58:11,976::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:58:11,984::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:58:12,099::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:58:12,099::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:58:14,547::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:58:14,548::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:58:14,548::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:58:16,797::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:58:16,828::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:16,969::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:58:16,970::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:16,976::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:58:16,979::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:58:21,386::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:58:21,386::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:58:31,450::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:58:31,450::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:58:33,714::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:58:33,761::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:33,903::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:58:33,903::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:33,908::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:58:33,911::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:58:33,915::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:58:36,137::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:58:36,138::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:58:40,508::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:58:40,516::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:58:40,631::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:58:40,632::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:58:43,128::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:58:43,129::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:58:43,129::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:58:45,380::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:58:45,411::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:45,536::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:58:45,537::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:58:45,543::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:58:45,546::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:58:49,951::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:58:49,951::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:59:00,036::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:00,036::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:02,292::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:02,323::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:02,480::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:02,481::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:02,488::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:02,491::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:02,495::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:59:04,710::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:59:04,711::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:59:09,096::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:59:09,104::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:59:09,219::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:59:09,219::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:59:11,679::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:59:11,679::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:11,679::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:13,920::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:13,967::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:14,109::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:14,109::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:14,116::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:14,118::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:18,516::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 09:59:18,517::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:59:28,583::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:28,583::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:30,832::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:30,863::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:30,988::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:30,989::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:30,995::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:30,998::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:30,998::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 09:59:31,002::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497535171.0 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 09:59:32,949::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 09:59:32,949::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 09:59:35,166::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 09:59:35,167::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:59:39,579::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 09:59:39,587::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 09:59:39,702::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 09:59:39,702::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 09:59:42,155::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 09:59:42,155::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:42,156::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:44,395::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:44,427::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:44,568::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:44,568::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:44,574::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:44,577::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:48,972::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 09:59:48,972::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 09:59:49,021::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:49,021::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:51,271::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:51,302::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:51,427::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:51,428::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:51,434::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:51,437::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:51,437::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:51,437::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:53,699::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:53,730::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:53,872::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:53,873::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:53,879::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:53,882::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:53,882::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 09:59:53,882::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 09:59:53,882::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 09:59:56,147::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 09:59:56,211::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:56,417::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 09:59:56,417::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 09:59:56,424::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 09:59:56,426::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 09:59:58,626::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 09:59:58,627::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 10:00:03,232::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 10:00:03,233::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::INFO::2017-06-15 10:00:03,233::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 10:00:03,237::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497535203.24 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:00:04,546::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 10:00:04,546::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:00:06,783::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:00:06,784::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:00:11,164::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:00:11,180::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:00:11,294::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:00:11,295::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:00:13,801::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:00:13,801::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:00:13,801::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:00:16,052::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:00:16,083::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:16,241::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:00:16,242::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:16,249::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:00:16,251::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:00:20,684::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:00:20,685::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:00:30,744::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:00:30,744::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:00:33,010::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:00:33,042::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:33,167::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:00:33,167::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:33,172::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:00:33,175::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:00:33,176::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 10:00:33,179::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 10:00:33,183::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497535233.18 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:00:36,289::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 10:00:36,289::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:00:38,507::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:00:38,507::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:00:42,906::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:00:42,914::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:00:43,029::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:00:43,029::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:00:45,487::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:00:45,488::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:00:45,488::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:00:47,831::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:00:47,879::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:48,100::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:00:48,101::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:48,107::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:00:48,110::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:00:52,523::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 10:00:52,523::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:00:52,571::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:00:52,572::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:00:54,823::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:00:54,870::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:55,060::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:00:55,060::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:00:55,066::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:00:55,068::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:00:55,069::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 10:00:57,363::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 10:00:57,364::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 10:00:57,364::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 10:00:57,368::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497535257.37 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:00:57,706::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 10:00:57,706::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:00:59,920::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:00:59,920::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:01:04,349::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:01:04,357::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:01:04,472::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:01:04,472::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:01:06,908::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:01:06,908::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:01:06,908::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:01:09,148::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:01:09,195::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:09,368::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:01:09,369::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:09,375::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:01:09,378::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:01:13,779::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:01:13,779::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:01:23,842::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:01:23,842::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:01:26,094::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:01:26,125::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:26,283::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:01:26,284::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:26,290::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:01:26,292::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:01:26,293::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:01:26,296::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:01:28,511::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:01:28,511::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:01:32,931::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:01:32,938::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:01:33,053::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:01:33,054::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:01:35,562::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:01:35,562::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:01:35,563::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:01:37,798::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:01:37,845::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:38,035::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:01:38,036::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:38,042::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:01:38,044::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:01:42,439::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:01:42,439::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:01:52,505::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:01:52,505::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:01:54,752::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:01:54,800::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:54,973::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:01:54,974::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:01:54,980::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:01:54,983::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:01:54,987::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:01:57,198::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:01:57,199::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:01,611::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:01,619::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:02:01,734::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:02:01,734::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:02:04,227::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:02:04,227::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:02:04,227::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:02:06,488::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:02:06,535::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:06,677::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:02:06,678::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:06,684::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:02:06,686::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:02:11,065::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:02:11,066::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:02:21,124::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:02:21,124::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:02:23,367::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:02:23,398::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:23,524::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:02:23,524::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:23,531::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:02:23,533::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:02:23,533::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:02:23,534::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4073 (Thu Jun 15 10:01:52 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4087 (Thu Jun 15 10:02:06 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '40510c86', 'local_conf_timestamp': 4087, 'host-ts': 4073} MainThread::INFO::2017-06-15 10:02:23,534::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4061 (Thu Jun 15 10:01:47 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=4075 (Thu Jun 15 10:02:01 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'e54f8b29', 'local_conf_timestamp': 4075, 'host-ts': 4061} MainThread::INFO::2017-06-15 10:02:23,534::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14939.0, 'maintenance': False, 'cpu-load': 0.0192, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:02:23,538::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:02:25,744::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:02:25,744::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:30,148::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:30,156::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:02:30,271::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:02:30,271::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:02:32,760::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:02:32,761::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:02:32,761::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:02:35,016::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:02:35,063::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:35,237::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:02:35,238::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:35,243::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:02:35,246::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:02:39,651::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:02:39,651::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:02:49,708::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:02:49,708::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:02:51,964::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:02:52,011::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:52,168::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:02:52,169::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:02:52,174::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:02:52,177::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:02:52,177::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:02:52,181::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:02:54,389::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:02:54,389::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:58,782::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:02:58,789::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:02:58,904::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:02:58,905::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:03:01,363::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:03:01,363::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:03:01,363::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:03:03,615::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:03:03,662::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:03,851::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:03:03,852::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:03,858::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:03:03,860::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:03:08,284::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:03:08,284::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:03:18,350::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:03:18,350::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:03:20,580::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:03:20,611::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:20,737::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:03:20,737::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:20,742::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:03:20,745::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:03:20,749::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:03:22,949::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:03:22,950::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:03:27,338::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:03:27,346::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:03:27,461::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:03:27,461::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:03:29,946::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:03:29,947::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:03:29,947::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:03:32,195::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:03:32,227::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:32,464::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:03:32,465::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:32,470::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:03:32,473::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:03:36,913::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:03:36,913::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:03:46,977::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:03:46,977::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:03:49,237::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:03:49,269::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:49,458::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:03:49,459::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:03:49,465::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:03:49,468::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:03:49,472::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:03:51,666::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:03:51,666::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:03:56,043::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:03:56,051::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:03:56,166::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:03:56,166::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:03:58,649::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:03:58,650::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:03:58,650::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:04:00,896::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:04:00,927::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:01,069::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:04:01,069::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:01,075::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:04:01,078::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:04:05,487::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:04:05,488::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:04:15,552::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:04:15,552::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:04:17,804::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:04:17,836::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:18,009::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:04:18,010::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:18,016::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:04:18,019::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:04:18,019::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:04:18,023::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:04:20,248::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:04:20,248::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:04:24,636::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:04:24,643::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:04:24,758::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:04:24,759::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:04:27,235::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:04:27,236::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:04:27,236::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:04:29,479::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:04:29,526::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:29,684::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:04:29,684::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:29,691::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:04:29,693::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:04:34,110::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:04:34,111::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:04:44,206::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:04:44,206::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:04:46,479::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:04:46,511::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:46,652::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:04:46,652::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:46,659::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:04:46,661::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:04:46,665::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:04:48,881::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:04:48,881::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:04:53,262::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:04:53,269::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:04:53,384::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:04:53,384::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:04:55,858::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:04:55,858::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:04:55,859::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:04:58,099::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:04:58,130::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:58,271::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:04:58,272::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:04:58,278::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:04:58,281::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:05:02,692::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:05:02,692::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:05:12,752::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:05:12,752::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:05:15,000::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:05:15,032::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:15,221::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:05:15,222::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:15,228::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:05:15,231::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:05:15,235::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:05:17,451::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:05:17,451::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:05:21,864::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:05:21,871::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:05:21,986::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:05:21,986::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:05:24,454::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:05:24,454::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:05:24,454::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:05:26,695::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:05:26,743::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:26,900::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:05:26,901::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:26,906::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:05:26,909::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:05:31,304::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:05:31,305::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:05:41,368::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:05:41,368::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:05:43,617::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:05:43,665::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:43,822::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:05:43,823::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:43,829::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:05:43,832::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:05:43,832::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:05:43,836::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:05:46,053::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:05:46,053::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:05:50,471::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:05:50,479::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:05:50,594::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:05:50,594::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:05:53,078::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:05:53,078::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:05:53,079::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:05:55,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:05:55,362::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:55,504::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:05:55,504::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:05:55,510::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:05:55,513::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:05:59,904::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:05:59,904::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:06:09,977::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:06:09,977::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:06:12,224::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:06:12,272::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:12,477::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:06:12,478::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:12,484::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:06:12,486::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:06:12,491::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:06:14,700::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:06:14,701::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:06:19,094::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:06:19,101::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:06:19,216::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:06:19,216::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:06:21,707::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:06:21,707::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:06:21,707::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:06:23,972::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:06:24,019::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:24,161::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:06:24,162::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:24,167::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:06:24,172::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:06:28,550::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:06:28,550::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:06:38,615::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:06:38,615::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:06:40,859::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:06:40,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:41,080::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:06:41,080::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:41,086::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:06:41,089::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:06:41,093::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:06:43,300::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:06:43,301::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:06:47,697::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:06:47,705::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:06:47,820::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:06:47,820::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:06:50,306::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:06:50,306::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:06:50,306::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:06:52,568::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:06:52,615::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:52,772::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:06:52,773::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:06:52,778::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:06:52,781::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:06:57,181::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:06:57,182::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:07:07,238::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:07:07,238::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:07:09,480::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:07:09,511::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:09,636::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:07:09,637::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:09,643::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:07:09,646::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:07:09,646::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:07:09,650::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:07:11,857::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:07:11,858::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:07:16,250::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:07:16,258::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:07:16,373::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:07:16,373::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:07:18,827::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:07:18,827::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:07:18,828::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:07:21,080::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:07:21,127::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:21,317::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:07:21,318::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:21,324::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:07:21,326::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:07:25,753::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:07:25,753::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:07:35,819::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:07:35,819::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:07:38,050::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:07:38,081::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:38,254::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:07:38,255::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:38,260::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:07:38,263::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:07:38,267::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:07:40,472::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:07:40,473::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:07:44,871::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:07:44,879::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:07:44,994::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:07:44,994::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:07:47,481::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:07:47,481::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:07:47,481::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:07:49,728::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:07:49,775::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:49,965::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:07:49,966::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:07:49,972::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:07:49,975::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:07:54,393::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:07:54,393::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:08:04,485::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:08:04,485::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:08:06,750::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:08:06,798::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:06,923::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:08:06,924::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:06,930::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:08:06,932::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:08:06,936::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:08:09,128::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:08:09,129::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:08:13,509::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:08:13,517::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:08:13,632::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:08:13,632::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:08:16,086::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:08:16,087::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:08:16,087::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:08:18,333::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:08:18,381::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:18,554::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:08:18,555::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:18,561::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:08:18,564::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:08:22,975::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:08:22,975::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:08:33,035::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:08:33,035::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:08:35,297::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:08:35,329::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:35,486::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:08:35,487::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:35,494::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:08:35,496::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:08:35,497::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:08:35,500::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:08:37,725::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:08:37,726::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:08:42,093::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:08:42,101::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:08:42,216::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:08:42,216::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:08:44,703::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:08:44,704::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:08:44,704::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:08:46,948::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:08:46,996::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:47,185::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:08:47,186::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:08:47,192::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:08:47,195::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:08:51,602::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:08:51,603::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:09:01,666::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:09:01,666::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:09:03,919::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:09:03,966::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:04,124::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:09:04,125::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:04,131::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:09:04,134::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:09:04,138::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:09:06,354::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:09:06,354::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:09:10,745::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:09:10,753::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:09:10,868::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:09:10,868::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:09:13,337::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:09:13,337::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:09:13,338::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:09:15,579::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:09:15,611::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:15,784::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:09:15,785::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:15,791::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:09:15,794::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:09:20,204::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:09:20,204::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:09:30,271::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:09:30,271::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:09:32,559::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:09:32,606::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:32,748::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:09:32,748::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:32,754::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:09:32,756::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:09:32,760::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:09:34,973::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:09:34,974::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:09:39,396::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:09:39,404::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:09:39,518::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:09:39,519::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:09:42,030::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:09:42,030::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:09:42,030::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:09:44,270::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:09:44,318::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:44,443::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:09:44,443::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:09:44,449::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:09:44,452::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:09:48,846::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:09:48,846::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:09:58,914::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:09:58,914::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:10:01,164::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:10:01,195::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:01,353::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:10:01,354::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:01,360::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:10:01,363::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:10:01,363::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:10:01,367::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:10:03,580::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:10:03,581::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:10:08,005::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:10:08,012::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:10:08,127::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:10:08,128::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:10:10,615::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:10:10,615::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:10:10,615::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:10:12,846::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:10:12,878::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:13,051::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:10:13,052::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:13,057::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:10:13,060::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:10:17,453::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:10:17,453::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:10:27,518::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:10:27,518::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:10:29,764::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:10:29,795::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:29,968::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:10:29,969::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:29,975::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:10:29,978::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:10:29,982::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:10:32,199::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:10:32,200::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:10:36,608::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:10:36,616::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:10:36,731::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:10:36,731::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:10:39,206::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:10:39,206::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:10:39,206::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:10:41,468::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:10:41,515::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:41,641::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:10:41,642::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:41,648::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:10:41,650::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:10:46,027::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:10:46,027::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:10:56,093::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:10:56,093::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:10:58,333::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:10:58,381::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:58,570::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:10:58,571::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:10:58,577::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:10:58,580::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:10:58,584::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:11:00,792::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:11:00,792::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:11:05,185::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:11:05,193::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:11:05,307::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:11:05,308::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:11:07,779::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:11:07,779::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:11:07,779::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:11:10,033::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:11:10,081::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:10,254::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:11:10,255::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:10,261::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:11:10,264::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:11:14,670::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:11:14,670::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:11:24,734::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:11:24,734::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:11:26,976::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:11:27,007::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:27,181::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:11:27,182::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:27,188::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:11:27,191::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:11:27,191::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:11:27,195::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:11:29,400::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:11:29,401::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:11:33,788::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:11:33,796::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:11:33,911::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:11:33,911::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:11:36,367::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:11:36,367::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:11:36,367::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:11:38,618::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:11:38,650::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:38,775::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:11:38,775::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:38,782::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:11:38,784::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:11:43,215::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:11:43,216::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:11:53,277::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:11:53,277::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:11:55,508::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:11:55,555::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:55,681::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:11:55,681::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:11:55,686::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:11:55,689::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:11:55,693::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:11:57,894::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:11:57,894::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:02,298::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:02,306::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:12:02,421::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:12:02,421::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:12:04,930::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:12:04,930::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:12:04,930::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:12:07,178::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:12:07,225::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:07,350::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:12:07,351::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:07,357::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:12:07,360::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:12:11,777::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:12:11,777::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:12:21,864::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:12:21,864::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:12:24,134::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:12:24,181::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:24,355::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:12:24,356::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:24,362::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:12:24,364::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:12:24,365::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:12:24,365::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4672 (Thu Jun 15 10:11:51 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4687 (Thu Jun 15 10:12:05 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'f2deb684', 'local_conf_timestamp': 4687, 'host-ts': 4672} MainThread::INFO::2017-06-15 10:12:24,365::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4667 (Thu Jun 15 10:11:53 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=4682 (Thu Jun 15 10:12:07 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStop\nstopped=False\ntimeout=Wed Dec 31 20:21:47 1969\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'powering down'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '4ecb5899', 'local_conf_timestamp': 4682, 'host-ts': 4667} MainThread::INFO::2017-06-15 10:12:24,365::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14937.0, 'maintenance': False, 'cpu-load': 0.0211, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:12:24,369::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:12:26,559::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:12:26,559::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:30,940::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:30,956::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:12:31,071::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:12:31,071::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:12:33,538::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:12:33,538::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:12:33,538::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:12:35,785::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:12:35,832::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:36,006::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:12:36,007::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:36,013::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:12:36,015::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:12:40,426::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:12:40,426::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:12:50,493::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:12:50,494::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:12:52,753::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:12:52,801::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:52,926::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:12:52,926::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:12:52,932::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:12:52,935::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:12:52,936::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-03.reis.com (id 3) MainThread::INFO::2017-06-15 10:12:52,940::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:12:55,164::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:12:55,164::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:59,527::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:12:59,534::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:12:59,649::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:12:59,649::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:13:02,138::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:13:02,139::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:13:02,139::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:13:04,388::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:13:04,419::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:04,544::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:13:04,545::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:04,550::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:13:04,553::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:13:08,963::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:13:08,963::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:13:19,025::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:13:19,026::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:13:21,284::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:13:21,331::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:21,457::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:13:21,457::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:21,464::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:13:21,466::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:13:21,471::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:13:23,689::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:13:23,690::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:13:28,085::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:13:28,093::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:13:28,208::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:13:28,208::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:13:30,671::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:13:30,671::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:13:30,672::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:13:32,911::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:13:32,959::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:33,084::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:13:33,084::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:33,089::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:13:33,092::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:13:37,494::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:13:37,494::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:13:47,553::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:13:47,553::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:13:49,802::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:13:49,833::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:49,958::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:13:49,959::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:13:49,964::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:13:49,967::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:13:49,967::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 10:13:49,971::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536029.97 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:13:50,004::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 10:13:50,004::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:13:52,239::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:13:52,240::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:13:56,669::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:13:56,676::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:13:56,791::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:13:56,791::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:13:59,242::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:13:59,242::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:13:59,242::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:01,484::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:01,531::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:01,656::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:01,657::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:01,662::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:01,665::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:06,058::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 10:14:06,058::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:14:06,110::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:06,110::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:08,356::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:08,388::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:08,529::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:08,529::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:08,535::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:08,537::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:08,538::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:08,538::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:10,792::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:10,824::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:10,997::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:10,998::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:11,004::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:11,006::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:11,006::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 10:14:11,007::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:11,007::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:13,263::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:13,311::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:13,452::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:13,452::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:13,457::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:13,460::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:15,659::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 10:14:15,660::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 10:14:20,243::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 10:14:20,243::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist MainThread::INFO::2017-06-15 10:14:20,243::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 10:14:20,247::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536060.25 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:14:20,283::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 10:14:20,283::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:14:22,506::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:14:22,507::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:14:26,881::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:14:26,888::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:14:27,003::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:14:27,003::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:14:29,468::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:14:29,469::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:29,469::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:31,715::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:31,763::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:31,904::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:31,905::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:31,911::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:31,914::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:36,331::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:14:36,331::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:14:46,408::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:46,409::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:14:48,664::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:14:48,711::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:48,853::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:14:48,854::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:14:48,859::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:14:48,861::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:14:48,862::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 10:14:48,866::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 10:14:48,869::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536088.87 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:14:48,905::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 10:14:48,906::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:14:51,126::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:14:51,126::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:14:55,512::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:14:55,519::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:14:55,635::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:14:55,635::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:14:58,110::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:14:58,111::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:14:58,111::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:15:00,351::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:15:00,399::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:00,524::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:15:00,525::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:00,530::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:15:00,532::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:15:04,950::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 10:15:04,950::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:15:05,063::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:15:05,063::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:15:07,311::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:15:07,359::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:07,516::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:15:07,517::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:07,523::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:15:07,526::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:15:07,526::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 10:15:09,819::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 10:15:09,820::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner File "/usr/lib64/python2.7/threading.py", line 764, in run File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 368, in wrapper File "/usr/lib/python2.7/site-packages/vdsm/concurrent.py", line 180, in run File "/usr/lib/python2.7/site-packages/yajsonrpc/betterAsyncore.py", line 216, in process_requests File "/usr/lib64/python2.7/asyncore.py", line 220, in loop File "/usr/lib64/python2.7/asyncore.py", line 201, in poll2 : 'NoneType' object is not callable MainThread::ERROR::2017-06-15 10:15:09,820::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 10:15:09,824::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536109.82 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:15:09,862::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 10:15:09,862::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:15:12,087::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:15:12,104::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:15:16,521::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:15:16,529::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:15:16,644::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:15:16,644::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:15:19,107::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:15:19,107::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:15:19,108::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:15:21,349::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:15:21,380::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:21,506::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:15:21,506::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:21,513::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:15:21,516::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:15:25,914::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:15:25,914::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:15:35,983::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:15:35,983::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:15:38,230::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:15:38,261::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:38,418::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:15:38,419::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:38,426::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:15:38,428::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:15:38,429::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:15:38,432::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:15:40,644::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:15:40,645::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:15:45,042::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:15:45,050::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:15:45,165::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:15:45,166::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:15:47,664::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:15:47,665::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:15:47,665::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:15:49,894::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:15:49,942::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:50,131::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:15:50,132::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:15:50,138::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:15:50,141::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:15:54,530::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:15:54,530::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:16:04,594::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:16:04,595::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:16:06,839::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:16:06,886::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:07,011::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:16:07,012::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:07,018::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:16:07,021::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:16:07,025::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:16:09,233::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:16:09,234::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:16:13,625::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:16:13,633::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:16:13,748::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:16:13,749::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:16:16,239::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:16:16,240::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:16:16,240::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:16:18,501::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:16:18,549::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:18,722::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:16:18,723::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:18,729::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:16:18,732::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:16:23,112::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:16:23,112::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:16:33,178::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:16:33,178::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:16:35,425::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:16:35,456::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:35,614::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:16:35,615::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:35,621::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:16:35,624::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:16:35,628::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:16:37,836::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:16:37,836::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:16:42,242::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:16:42,250::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:16:42,365::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:16:42,365::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:16:44,863::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:16:44,864::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:16:44,864::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:16:47,119::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:16:47,150::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:47,275::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:16:47,276::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:16:47,282::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:16:47,285::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:16:51,688::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:16:51,688::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:17:01,872::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:17:01,872::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:17:04,120::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:17:04,151::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:04,325::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:17:04,325::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:04,330::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:17:04,333::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:17:04,334::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:17:04,337::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:17:06,540::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:17:06,540::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:17:10,934::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:17:10,941::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:17:11,056::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:17:11,057::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:17:13,543::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:17:13,544::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:17:13,544::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:17:15,797::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:17:15,844::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:15,986::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:17:15,987::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:15,993::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:17:15,995::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:17:20,424::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:17:20,425::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:17:30,488::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:17:30,489::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:17:32,764::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:17:32,796::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:32,937::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:17:32,938::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:32,943::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:17:32,946::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:17:32,950::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:17:35,156::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:17:35,157::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:17:39,537::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:17:39,545::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:17:39,660::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:17:39,660::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:17:42,182::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:17:42,182::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:17:42,182::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:17:44,434::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:17:44,465::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:44,591::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:17:44,591::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:17:44,597::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:17:44,599::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:17:49,012::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:17:49,013::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:17:59,079::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:17:59,079::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:18:01,340::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:18:01,372::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:01,497::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:18:01,497::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:01,503::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:18:01,506::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:18:01,510::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:18:03,702::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:18:03,702::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:18:08,081::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:18:08,088::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:18:08,203::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:18:08,204::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:18:10,652::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:18:10,653::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:18:10,653::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:18:12,904::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:18:12,952::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:13,109::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:18:13,110::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:13,115::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:18:13,118::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:18:17,532::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:18:17,532::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:18:27,600::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:18:27,601::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:18:29,860::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:18:29,907::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:30,049::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:18:30,050::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:30,056::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:18:30,058::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:18:30,059::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:18:30,063::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:18:32,293::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:18:32,294::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:18:36,667::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:18:36,675::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:18:36,790::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:18:36,790::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:18:39,266::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:18:39,266::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:18:39,266::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:18:41,510::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:18:41,558::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:41,683::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:18:41,683::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:41,690::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:18:41,692::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:18:46,099::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:18:46,099::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:18:56,165::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:18:56,165::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:18:58,416::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:18:58,447::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:58,621::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:18:58,621::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:18:58,628::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:18:58,630::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:18:58,634::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:19:00,851::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:19:00,851::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:19:05,236::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:19:05,244::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:19:05,359::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:19:05,359::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:19:07,836::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:19:07,836::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:19:07,837::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:19:10,078::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:19:10,125::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:10,251::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:19:10,251::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:10,257::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:19:10,259::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:19:14,677::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:19:14,677::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:19:24,766::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:19:24,767::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:19:27,015::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:19:27,062::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:27,204::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:19:27,204::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:27,211::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:19:27,213::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:19:27,218::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:19:29,433::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:19:29,433::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:19:33,850::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:19:33,858::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:19:33,973::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:19:33,973::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:19:36,440::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:19:36,440::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:19:36,440::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:19:38,683::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:19:38,731::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:38,872::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:19:38,872::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:38,878::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:19:38,881::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:19:43,283::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:19:43,283::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:19:53,352::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:19:53,352::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:19:55,616::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:19:55,664::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:55,789::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:19:55,790::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:19:55,795::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:19:55,797::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:19:55,798::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:19:55,802::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:19:58,017::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:19:58,018::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:02,432::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:02,440::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:20:02,555::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:20:02,555::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:20:05,112::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:20:05,113::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:20:05,113::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:20:07,342::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:20:07,373::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:07,514::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:20:07,515::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:07,521::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:20:07,524::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:20:11,935::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:20:11,935::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:20:22,009::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:20:22,009::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:20:24,253::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:20:24,301::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:24,490::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:20:24,491::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:24,497::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:20:24,499::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:20:24,504::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:20:26,717::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:20:26,718::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:31,122::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:31,129::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:20:31,244::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:20:31,245::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:20:33,736::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:20:33,736::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:20:33,736::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:20:35,997::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:20:36,045::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:36,202::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:20:36,203::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:36,209::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:20:36,212::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:20:40,598::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:20:40,598::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:20:50,660::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:20:50,660::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:20:52,919::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:20:52,951::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:53,076::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:20:53,077::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:20:53,082::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:20:53,085::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:20:53,089::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:20:55,295::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:20:55,296::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:59,697::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:20:59,705::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:20:59,820::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:20:59,820::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:21:02,364::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:21:02,364::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:21:02,365::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:21:04,651::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:21:04,683::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:04,840::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:21:04,841::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:04,846::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:21:04,849::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:21:09,250::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:21:09,250::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:21:19,306::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:21:19,306::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:21:21,549::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:21:21,580::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:21,754::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:21:21,754::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:21,760::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:21:21,763::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:21:21,763::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:21:21,767::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:21:23,974::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:21:23,975::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:21:28,365::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:21:28,373::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:21:28,488::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:21:28,488::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:21:30,979::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:21:30,979::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:21:30,979::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:21:33,233::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:21:33,280::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:33,421::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:21:33,422::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:33,429::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:21:33,431::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:21:37,864::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:21:37,864::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:21:47,933::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:21:47,933::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:21:50,197::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:21:50,244::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:50,402::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:21:50,403::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:21:50,408::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:21:50,410::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:21:50,415::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:21:52,616::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:21:52,616::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:21:57,001::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:21:57,008::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:21:57,123::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:21:57,123::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:21:59,614::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:21:59,615::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:21:59,615::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:22:01,864::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:22:01,911::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:02,101::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:22:02,102::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:02,108::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:22:02,111::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:22:06,534::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:22:06,534::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:22:16,603::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:22:16,603::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:22:18,865::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:22:18,897::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:19,021::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:22:19,022::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:19,027::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:22:19,030::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:22:19,034::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:22:21,227::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:22:21,227::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:22:25,616::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:22:25,624::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:22:25,739::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:22:25,739::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:22:28,223::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:22:28,223::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:22:28,223::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:22:30,472::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:22:30,504::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:30,645::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:22:30,646::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:30,652::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:22:30,655::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:22:35,061::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:22:35,061::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:22:45,136::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:22:45,136::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:22:47,392::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:22:47,423::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:47,548::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:22:47,549::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:47,554::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:22:47,557::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:22:47,557::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:22:47,557::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5303 (Thu Jun 15 10:22:21 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5317 (Thu Jun 15 10:22:35 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '0b4fc262', 'local_conf_timestamp': 5317, 'host-ts': 5303} MainThread::INFO::2017-06-15 10:22:47,557::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5299 (Thu Jun 15 10:22:25 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=5313 (Thu Jun 15 10:22:39 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '52d87400', 'local_conf_timestamp': 5313, 'host-ts': 5299} MainThread::INFO::2017-06-15 10:22:47,557::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14936.0, 'maintenance': False, 'cpu-load': 0.0212, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:22:47,558::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:22:47,561::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:22:49,798::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:22:49,799::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:22:54,164::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:22:54,172::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:22:54,287::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:22:54,287::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:22:56,766::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:22:56,766::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:22:56,766::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:22:59,013::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:22:59,045::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:59,170::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:22:59,171::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:22:59,177::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:22:59,180::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:23:03,588::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:23:03,588::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:23:13,651::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:23:13,651::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:23:15,902::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:23:15,949::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:16,122::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:23:16,123::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:16,128::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:23:16,131::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:23:16,135::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:23:18,352::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:23:18,352::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:23:22,743::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:23:22,750::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:23:22,865::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:23:22,866::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:23:25,280::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:23:25,280::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:23:25,281::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:23:27,522::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:23:27,554::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:27,679::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:23:27,680::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:27,685::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:23:27,688::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:23:32,094::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:23:32,094::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:23:42,160::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:23:42,160::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:23:44,411::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:23:44,459::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:44,632::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:23:44,633::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:44,638::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:23:44,641::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:23:44,645::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:23:46,859::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:23:46,860::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:23:51,274::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:23:51,282::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:23:51,397::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:23:51,397::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:23:53,872::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:23:53,872::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:23:53,872::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:23:56,113::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:23:56,145::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:56,287::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:23:56,288::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:23:56,293::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:23:56,295::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:24:00,690::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:24:00,690::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:24:10,758::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:24:10,758::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:24:13,013::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:24:13,092::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:13,218::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:24:13,218::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:13,224::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:24:13,226::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:24:13,226::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:24:13,230::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:24:15,445::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:24:15,446::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:24:19,848::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:24:19,856::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:24:19,971::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:24:19,971::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:24:22,468::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:24:22,468::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:24:22,469::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:24:24,699::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:24:24,746::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:24,888::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:24:24,888::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:24,894::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:24:24,897::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:24:29,285::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:24:29,286::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:24:39,351::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:24:39,351::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:24:41,594::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:24:41,642::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:41,799::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:24:41,800::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:41,805::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:24:41,808::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:24:41,812::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:24:44,022::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:24:44,022::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:24:48,436::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:24:48,444::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:24:48,559::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:24:48,559::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:24:51,050::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:24:51,050::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:24:51,051::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:24:53,316::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:24:53,363::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:53,505::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:24:53,506::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:24:53,512::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:24:53,514::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:24:57,893::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:24:57,893::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:25:07,962::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:25:07,962::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:25:10,204::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:25:10,236::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:10,393::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:25:10,394::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:10,400::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:25:10,403::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:25:10,407::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:25:12,615::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:25:12,615::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:25:17,018::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:25:17,025::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:25:17,140::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:25:17,140::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:25:19,630::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:25:19,630::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:25:19,630::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:25:21,886::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:25:21,917::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:22,059::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:25:22,059::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:22,065::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:25:22,068::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:25:26,477::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:25:26,477::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:25:36,541::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:25:36,541::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:25:38,783::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:25:38,814::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:38,956::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:25:38,956::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:38,961::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:25:38,964::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:25:38,964::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:25:38,968::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:25:41,179::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:25:41,179::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:25:45,576::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:25:45,583::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:25:45,698::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:25:45,699::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:25:48,168::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:25:48,168::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:25:48,169::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:25:50,419::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:25:50,451::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:50,592::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:25:50,593::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:25:50,598::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:25:50,601::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:25:55,033::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:25:55,033::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:26:05,101::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:26:05,101::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:26:07,333::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:26:07,397::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:07,538::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:26:07,539::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:07,544::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:26:07,547::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:26:07,551::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:26:09,757::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:26:09,758::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:26:14,135::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:26:14,143::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:26:14,258::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:26:14,258::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:26:16,741::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:26:16,742::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:26:16,742::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:26:18,992::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:26:19,039::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:19,196::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:26:19,197::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:19,203::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:26:19,206::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:26:23,623::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:26:23,623::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:26:33,679::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:26:33,679::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:26:35,942::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:26:35,974::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:36,164::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:26:36,164::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:36,171::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:26:36,174::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:26:36,178::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:26:38,368::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:26:38,369::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:26:42,748::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:26:42,755::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:26:42,870::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:26:42,871::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:26:45,322::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:26:45,323::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:26:45,323::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:26:47,571::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:26:47,618::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:47,760::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:26:47,761::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:26:47,767::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:26:47,770::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:26:52,187::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:26:52,187::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:27:02,255::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:27:02,255::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:27:04,511::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:27:04,543::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:04,668::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:27:04,669::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:04,675::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:27:04,678::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:27:04,678::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:27:04,682::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:27:06,906::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:27:06,907::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:27:11,271::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:27:11,279::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:27:11,394::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:27:11,394::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:27:13,863::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:27:13,863::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:27:13,863::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:27:16,111::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:27:16,158::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:16,284::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:27:16,284::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:16,291::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:27:16,293::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:27:20,699::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:27:20,699::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:27:30,766::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:27:30,766::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:27:33,055::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:27:33,087::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:33,228::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:27:33,229::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:33,235::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:27:33,238::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:27:33,242::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:27:35,465::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:27:35,465::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:27:39,849::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:27:39,857::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:27:39,972::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:27:39,972::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:27:42,417::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:27:42,418::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:27:42,418::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:27:44,661::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:27:44,693::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:44,851::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:27:44,852::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:27:44,858::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:27:44,860::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:27:49,261::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:27:49,262::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:27:59,333::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:27:59,333::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:01,583::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:01,615::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:01,805::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:01,805::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:01,811::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:01,814::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:01,818::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:28:04,033::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:28:04,034::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:28:08,450::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:28:08,457::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:28:08,572::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:28:08,573::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:28:11,039::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:28:11,039::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:11,039::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:13,282::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:13,329::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:13,519::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:13,520::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:13,526::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:13,529::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:17,924::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:28:17,925::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:28:27,993::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:27,993::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:30,243::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:30,290::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:30,448::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:30,449::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:30,454::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:30,456::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:30,457::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 10:28:30,461::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536910.46 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:28:30,497::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 10:28:30,497::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:28:32,710::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:28:32,710::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:28:37,119::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:28:37,127::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:28:37,242::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:28:37,242::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:28:39,740::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:28:39,740::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:39,740::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:41,972::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:42,004::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:42,147::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:42,148::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:42,154::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:42,157::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:46,556::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 10:28:46,556::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:28:46,609::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:46,609::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:48,854::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:48,901::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:49,059::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:49,060::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:49,065::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:49,067::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:49,068::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:49,068::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:51,321::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:51,368::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:51,509::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:51,510::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:51,515::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:51,518::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:51,518::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 10:28:51,518::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:28:51,518::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:28:53,777::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:28:53,808::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:53,934::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:28:53,934::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:28:53,939::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:28:53,942::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:28:56,149::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 10:28:56,150::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 10:29:00,736::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 10:29:00,736::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist MainThread::INFO::2017-06-15 10:29:00,736::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 10:29:00,741::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497536940.74 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:29:00,773::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 10:29:00,773::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:29:03,021::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:29:03,022::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:29:07,453::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:29:07,461::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:29:07,576::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:29:07,576::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:29:10,054::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:29:10,054::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:29:10,054::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:29:12,356::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:29:12,409::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:12,697::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:29:12,698::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:12,706::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:29:12,709::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:29:17,252::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:29:17,252::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:29:27,349::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:29:27,349::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:29:29,665::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:29:29,713::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:29,904::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:29:29,905::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:29,912::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:29:29,916::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:29:29,917::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:29:29,922::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 10:39:27 2017 while transitioning -> MainThread::INFO::2017-06-15 10:29:29,927::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:29:32,156::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:29:32,156::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:29:36,581::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:29:36,589::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:29:36,704::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:29:36,704::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:29:39,160::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:29:39,160::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:29:39,160::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:29:41,409::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:29:41,441::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:41,582::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:29:41,583::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:41,589::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:29:41,592::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:29:45,993::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:29:45,993::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:29:56,059::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:29:56,059::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:29:58,309::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:29:58,356::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:58,482::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:29:58,482::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:29:58,487::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:29:58,490::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:29:58,491::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:29:58,494::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:30:00,733::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:30:00,733::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:30:05,146::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:30:05,154::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:30:05,269::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:30:05,269::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:30:07,770::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:30:07,770::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:30:07,770::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:30:10,001::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:30:10,048::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:10,206::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:30:10,207::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:10,212::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:30:10,215::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:30:14,613::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:30:14,613::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:30:24,678::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:30:24,678::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:30:26,925::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:30:26,957::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:27,146::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:30:27,147::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:27,152::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:30:27,155::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:30:27,155::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:30:27,159::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:30:29,378::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:30:29,379::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:30:33,795::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:30:33,802::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:30:33,917::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:30:33,918::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:30:36,411::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:30:36,412::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:30:36,412::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:30:38,677::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:30:38,724::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:38,882::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:30:38,883::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:38,888::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:30:38,890::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:30:43,282::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:30:43,283::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:30:53,356::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:30:53,356::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:30:55,605::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:30:55,653::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:55,842::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:30:55,843::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:30:55,848::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:30:55,851::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:30:55,852::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:30:55,855::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:30:58,064::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:30:58,064::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:02,466::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:02,474::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:31:02,589::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:31:02,589::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:31:05,053::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:31:05,053::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:31:05,053::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:31:07,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:31:07,361::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:07,519::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:31:07,520::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:07,525::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:31:07,527::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:31:11,935::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:31:11,936::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:31:22,000::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:31:22,000::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:31:24,248::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:31:24,280::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:24,485::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:31:24,486::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:24,492::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:31:24,494::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:31:24,495::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:31:24,499::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:31:26,703::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:31:26,703::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:31,089::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:31,097::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:31:31,212::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:31:31,212::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:31:33,706::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:31:33,706::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:31:33,706::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:31:35,960::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:31:35,992::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:36,133::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:31:36,134::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:36,140::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:31:36,142::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:31:40,591::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:31:40,591::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:31:50,656::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:31:50,657::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:31:52,915::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:31:52,947::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:53,120::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:31:53,121::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:31:53,127::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:31:53,129::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:31:53,130::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:31:53,133::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:31:55,336::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:31:55,337::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:59,727::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:31:59,735::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:31:59,850::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:31:59,850::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:32:02,336::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:32:02,336::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:32:02,336::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:32:04,608::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:32:04,655::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:04,813::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:32:04,814::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:04,819::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:32:04,822::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:32:09,246::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:32:09,247::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:32:19,310::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:32:19,310::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:32:21,576::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:32:21,623::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:21,765::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:32:21,766::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:21,771::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:32:21,773::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:32:21,774::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:32:21,778::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:32:23,972::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:32:23,972::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:32:28,351::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:32:28,359::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:32:28,474::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:32:28,474::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:32:30,957::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:32:30,957::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:32:30,957::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:32:33,203::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:32:33,250::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:33,424::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:32:33,425::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:33,430::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:32:33,433::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:32:37,846::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:32:37,846::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:32:47,902::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:32:47,903::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:32:50,160::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:32:50,207::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:50,381::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:32:50,382::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:32:50,387::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:32:50,390::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:32:50,390::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:32:50,390::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5904 (Thu Jun 15 10:32:23 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=5918 (Thu Jun 15 10:32:37 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '0f931563', 'local_conf_timestamp': 5918, 'host-ts': 5904} MainThread::INFO::2017-06-15 10:32:50,390::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=5897 (Thu Jun 15 10:32:23 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=5911 (Thu Jun 15 10:32:37 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'e295860b', 'local_conf_timestamp': 5911, 'host-ts': 5897} MainThread::INFO::2017-06-15 10:32:50,391::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 12647.0, 'maintenance': False, 'cpu-load': 0.0201, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:32:50,391::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:32:50,395::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:32:52,620::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:32:52,621::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:32:56,999::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:32:57,006::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:32:57,121::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:32:57,121::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:32:59,604::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:32:59,604::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:32:59,604::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:33:01,853::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:33:01,900::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:02,058::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:33:02,059::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:02,065::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:33:02,068::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:33:06,496::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:33:06,497::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:33:16,649::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:33:16,649::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:33:18,903::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:33:18,934::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:19,060::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:33:19,061::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:19,066::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:33:19,069::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:33:19,069::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:33:19,073::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:33:21,295::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:33:21,295::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:33:25,687::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:33:25,695::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:33:25,810::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:33:25,811::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:33:28,290::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:33:28,291::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:33:28,291::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:33:30,541::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:33:30,589::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:30,778::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:33:30,779::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:30,786::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:33:30,788::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:33:35,209::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:33:35,209::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:33:45,276::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:33:45,277::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:33:47,527::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:33:47,574::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:47,764::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:33:47,765::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:47,770::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:33:47,773::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:33:47,773::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:33:47,777::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:33:49,991::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:33:49,992::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:33:54,436::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:33:54,444::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:33:54,559::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:33:54,559::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:33:57,017::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:33:57,017::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:33:57,018::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:33:59,258::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:33:59,290::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:59,479::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:33:59,480::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:33:59,486::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:33:59,488::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:34:03,901::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:34:03,901::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:34:14,032::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:34:14,032::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:34:16,290::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:34:16,322::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:16,463::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:34:16,464::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:16,470::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:34:16,473::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:34:16,474::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:34:16,477::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:34:18,691::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:34:18,692::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:34:23,131::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:34:23,138::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:34:23,253::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:34:23,254::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:34:25,757::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:34:25,757::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:34:25,757::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:34:27,989::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:34:28,037::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:28,227::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:34:28,228::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:28,233::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:34:28,235::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:34:32,635::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:34:32,635::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:34:42,707::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:34:42,707::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:34:44,954::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:34:45,002::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:45,192::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:34:45,192::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:45,199::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:34:45,201::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:34:45,202::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:34:45,205::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:34:47,418::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:34:47,418::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:34:51,822::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:34:51,830::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:34:51,945::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:34:51,945::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:34:54,440::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:34:54,440::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:34:54,440::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:34:56,706::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:34:56,753::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:56,895::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:34:56,896::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:34:56,901::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:34:56,904::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:35:01,287::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:35:01,287::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:35:11,351::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:35:11,351::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:35:13,593::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:35:13,625::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:13,782::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:35:13,783::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:13,789::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:35:13,791::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:35:13,792::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:35:13,796::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:35:16,004::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:35:16,004::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:35:20,409::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:35:20,417::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:35:20,532::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:35:20,532::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:35:23,025::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:35:23,025::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:35:23,026::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:35:25,282::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:35:25,330::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:25,519::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:35:25,520::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:25,526::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:35:25,529::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:35:29,946::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:35:29,947::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:35:40,017::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:35:40,017::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:35:42,258::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:35:42,306::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:42,479::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:35:42,480::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:42,485::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:35:42,488::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:35:42,488::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:35:42,492::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:35:44,708::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:35:44,708::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:35:49,118::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:35:49,126::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:35:49,241::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:35:49,242::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:35:51,733::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:35:51,733::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:35:51,733::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:35:53,989::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:35:54,021::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:54,194::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:35:54,195::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:35:54,200::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:35:54,203::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:35:58,663::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:35:58,663::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:36:08,728::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:36:08,728::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:36:10,960::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:36:11,008::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:11,165::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:36:11,166::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:11,171::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:36:11,174::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:36:11,174::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:36:11,178::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:36:13,385::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:36:13,386::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:36:17,776::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:36:17,784::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:36:17,899::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:36:17,899::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:36:20,386::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:36:20,386::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:36:20,386::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:36:22,638::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:36:22,670::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:22,811::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:36:22,812::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:22,818::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:36:22,820::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:36:27,267::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:36:27,267::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:36:37,374::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:36:37,374::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:36:39,639::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:36:39,687::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:39,845::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:36:39,846::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:39,851::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:36:39,853::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:36:39,854::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:36:39,858::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:36:42,053::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:36:42,053::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:36:46,444::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:36:46,452::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:36:46,567::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:36:46,567::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:36:49,051::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:36:49,051::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:36:49,051::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:36:51,303::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:36:51,335::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:51,460::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:36:51,461::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:36:51,467::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:36:51,470::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:36:55,924::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:36:55,924::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:37:05,994::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:37:05,994::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:37:08,252::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:37:08,284::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:08,441::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:37:08,442::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:08,447::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:37:08,450::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:37:08,450::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:37:08,454::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:37:10,681::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:37:10,681::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:37:15,057::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:37:15,064::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:37:15,179::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:37:15,180::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:37:17,664::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:37:17,664::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:37:17,664::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:37:19,922::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:37:19,953::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:20,111::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:37:20,112::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:20,118::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:37:20,120::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:37:24,542::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:37:24,543::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:37:34,605::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:37:34,605::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:37:36,866::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:37:36,914::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:37,104::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:37:37,105::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:37,110::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:37:37,113::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:37:37,113::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:37:37,117::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:37:39,336::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:37:39,337::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:37:43,736::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:37:43,744::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:37:43,859::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:37:43,859::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:37:46,337::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:37:46,337::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:37:46,337::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:37:48,582::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:37:48,613::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:48,771::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:37:48,772::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:37:48,777::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:37:48,780::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:37:53,197::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:37:53,197::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:38:03,306::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:38:03,306::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:38:05,561::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:38:05,608::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:05,766::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:38:05,766::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:05,772::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:38:05,774::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:38:05,775::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:38:05,778::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:38:07,991::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:38:07,992::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:38:12,418::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:38:12,425::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:38:12,540::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:38:12,541::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:38:15,008::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:38:15,008::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:38:15,008::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:38:17,248::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:38:17,296::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:17,437::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:38:17,438::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:17,443::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:38:17,446::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:38:21,851::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:38:21,852::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:38:31,913::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:38:31,913::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:38:34,162::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:38:34,210::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:34,335::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:38:34,336::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:34,342::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:38:34,344::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:38:34,345::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:38:34,348::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:38:36,560::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:38:36,561::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:38:40,988::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:38:40,996::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:38:41,111::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:38:41,111::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:38:43,610::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:38:43,610::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:38:43,610::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:38:45,844::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:38:45,876::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:46,001::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:38:46,002::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:38:46,007::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:38:46,010::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:38:50,417::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:38:50,417::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:39:00,484::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:39:00,484::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:39:02,732::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:39:02,779::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:02,953::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:39:02,954::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:02,959::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:39:02,962::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:39:02,962::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:39:02,966::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:39:05,180::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:39:05,181::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:39:09,586::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:39:09,594::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:39:09,709::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:39:09,710::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:39:12,202::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:39:12,202::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:39:12,202::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:39:14,465::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:39:14,497::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:14,656::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:39:14,657::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:14,662::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:39:14,664::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:39:19,060::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:39:19,061::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:39:29,126::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:39:29,126::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:39:31,368::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:39:31,416::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:31,605::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:39:31,606::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:31,612::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:39:31,614::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:39:31,619::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537571.62 type=state_transition detail=EngineStarting-EngineStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:39:31,652::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineStop) sent? sent MainThread::INFO::2017-06-15 10:39:31,653::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:39:33,861::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:39:33,862::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:39:38,262::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:39:38,270::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:39:38,385::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:39:38,385::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:39:40,884::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:39:40,884::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:39:40,884::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:39:43,147::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:39:43,227::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:43,369::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:39:43,370::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:39:43,376::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:39:43,378::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:39:47,792::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:39:47,792::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:39:57,853::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:39:57,853::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:40:00,095::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:40:00,142::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:00,267::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:40:00,268::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:00,273::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:40:00,276::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:40:00,276::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2017-06-15 10:40:02,568::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 10:40:02,568::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 10:40:02,568::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 10:40:02,573::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 10:44:57 2017 while transitioning -> MainThread::INFO::2017-06-15 10:40:02,577::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:40:04,781::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:40:04,781::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:40:09,168::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:40:09,176::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:40:09,291::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:40:09,291::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:40:11,763::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:40:11,764::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:40:11,764::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:40:14,020::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:40:14,068::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:14,241::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:40:14,242::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:14,247::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:40:14,250::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:40:18,679::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:40:18,679::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:40:28,745::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:40:28,746::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:40:30,975::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:40:31,007::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:31,164::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:40:31,165::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:31,170::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:40:31,173::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:40:31,173::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (31 of 300 seconds) MainThread::INFO::2017-06-15 10:40:31,177::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:40:33,389::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:40:33,389::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:40:37,771::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:40:37,779::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:40:37,894::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:40:37,894::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:40:40,379::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:40:40,379::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:40:40,379::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:40:42,630::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:40:42,662::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:42,803::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:40:42,804::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:42,809::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:40:42,812::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:40:47,232::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:40:47,233::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:40:57,317::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:40:57,318::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:40:59,584::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:40:59,631::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:59,821::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:40:59,822::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:40:59,827::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:40:59,830::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:40:59,830::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (59 of 300 seconds) MainThread::INFO::2017-06-15 10:40:59,834::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:41:02,025::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:41:02,026::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:41:06,413::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:41:06,420::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:41:06,535::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:41:06,536::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:41:09,017::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:41:09,018::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:41:09,018::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:41:11,269::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:41:11,317::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:11,507::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:41:11,508::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:11,513::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:41:11,516::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:41:15,930::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:41:15,931::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:41:25,999::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:41:25,999::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:41:28,256::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:41:28,287::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:28,412::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:41:28,413::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:28,418::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:41:28,421::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:41:28,421::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (88 of 300 seconds) MainThread::INFO::2017-06-15 10:41:28,425::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:41:30,652::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:41:30,653::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:41:35,023::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:41:35,031::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:41:35,146::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:41:35,146::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:41:37,626::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:41:37,626::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:41:37,627::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:41:39,878::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:41:39,909::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:40,050::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:41:40,051::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:40,056::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:41:40,059::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:41:44,480::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:41:44,481::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:41:54,692::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:41:54,692::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:41:56,945::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:41:56,993::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:57,183::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:41:57,184::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:41:57,189::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:41:57,192::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:41:57,192::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (117 of 300 seconds) MainThread::INFO::2017-06-15 10:41:57,196::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:41:59,418::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:41:59,418::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:42:03,816::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:42:03,824::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:42:03,939::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:42:03,939::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:42:06,416::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:42:06,416::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:42:06,416::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:42:08,670::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:42:08,718::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:08,908::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:42:08,910::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:08,915::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:42:08,918::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:42:13,325::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 10:42:13,325::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:42:23,397::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:42:23,397::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:42:25,646::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:42:25,693::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:25,851::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:42:25,852::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:25,857::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:42:25,860::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:42:25,860::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm not running on local host MainThread::INFO::2017-06-15 10:42:25,864::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 10:42:25,867::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537745.87 type=state_transition detail=EngineStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:42:25,918::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 10:42:25,918::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:42:28,137::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:42:28,138::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:42:32,562::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:42:32,570::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:42:32,685::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:42:32,685::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:42:35,155::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:42:35,155::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:42:35,156::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:42:37,393::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:42:37,441::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:37,630::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:42:37,631::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:37,636::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:42:37,639::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:42:42,038::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:42:42,038::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:42:52,101::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:42:52,101::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:42:54,349::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:42:54,396::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:54,586::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:42:54,587::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:42:54,592::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:42:54,595::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:42:54,595::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:42:54,595::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=6503 (Thu Jun 15 10:42:22 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=6517 (Thu Jun 15 10:42:36 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '2be8cf9b', 'local_conf_timestamp': 6517, 'host-ts': 6503} MainThread::INFO::2017-06-15 10:42:54,595::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=6500 (Thu Jun 15 10:42:26 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=6514 (Thu Jun 15 10:42:40 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'b2cfbef9', 'local_conf_timestamp': 6514, 'host-ts': 6500} MainThread::INFO::2017-06-15 10:42:54,595::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'bad vm status', 'health': 'bad', 'vm': 'down', 'detail': 'down'}, 'bridge': True, 'mem-free': 14933.0, 'maintenance': False, 'cpu-load': 0.0266, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:42:54,596::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 10:42:54,599::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537774.6 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:42:54,632::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 10:42:54,632::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:42:56,846::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:42:56,847::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:43:01,255::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:43:01,263::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:43:01,378::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:43:01,378::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:43:03,877::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:43:03,877::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:03,877::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:06,112::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:06,159::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:06,284::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:06,285::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:06,291::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:06,293::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:10,685::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 10:43:10,686::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:43:10,735::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:10,735::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:12,982::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:13,030::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:13,219::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:13,220::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:13,225::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:13,228::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:13,228::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:13,228::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:15,484::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:15,516::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:15,657::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:15,657::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:15,663::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:15,665::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:15,665::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 10:43:15,666::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:15,666::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:17,921::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:17,952::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:18,094::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:18,095::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:18,100::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:18,102::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:20,300::hosted_engine::1186::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Cleaning state for non-running VM MainThread::INFO::2017-06-15 10:43:25,707::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 10:43:25,708::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 10:43:30,289::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 10:43:30,289::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::INFO::2017-06-15 10:43:30,289::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 10:43:30,294::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537810.29 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:43:30,328::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 10:43:30,328::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:43:32,531::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:43:32,532::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:43:36,920::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:43:36,927::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:43:37,042::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:43:37,043::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:43:39,526::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:43:39,526::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:39,526::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:41,774::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:41,822::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:41,995::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:41,996::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:42,002::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:42,004::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:46,424::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:43:46,424::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:43:56,477::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:43:56,477::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:43:58,734::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:43:58,765::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:58,890::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:43:58,891::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:43:58,896::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:43:58,899::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:43:58,899::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 10:43:58,903::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 10:43:58,906::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537838.91 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:43:58,940::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 10:43:58,940::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:44:01,172::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:44:01,172::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:05,540::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:05,548::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:44:05,663::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:44:05,663::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:44:08,142::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:44:08,142::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:44:08,142::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:44:10,390::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:44:10,438::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:10,595::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:44:10,596::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:10,601::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:44:10,604::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:44:15,023::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 10:44:15,024::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:44:15,082::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:44:15,082::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:44:17,333::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:44:17,365::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:17,522::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:44:17,523::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:17,528::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:44:17,531::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:44:17,531::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 10:44:19,821::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 10:44:19,821::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 10:44:19,822::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 10:44:19,826::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497537859.83 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:44:19,863::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 10:44:19,863::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:44:22,082::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:44:22,083::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:26,484::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:26,492::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:44:26,607::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:44:26,608::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:44:29,084::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:44:29,084::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:44:29,084::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:44:31,329::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:44:31,377::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:31,566::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:44:31,567::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:31,573::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:44:31,575::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:44:35,980::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:44:35,981::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:44:46,044::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:44:46,044::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:44:48,294::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:44:48,341::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:48,515::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:44:48,516::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:44:48,521::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:44:48,524::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:44:48,524::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:44:48,528::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:44:50,743::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:44:50,743::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:55,168::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:44:55,175::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:44:55,290::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:44:55,291::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:44:57,759::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:44:57,759::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:44:57,760::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:45:00,007::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:45:00,054::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:00,212::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:45:00,212::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:00,218::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:45:00,220::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:45:04,619::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:45:04,620::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:45:14,687::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:45:14,687::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:45:16,935::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:45:16,982::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:17,156::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:45:17,157::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:17,162::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:45:17,165::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:45:17,169::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:45:19,381::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:45:19,382::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:45:23,834::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:45:23,850::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:45:23,965::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:45:23,965::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:45:26,465::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:45:26,465::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:45:26,465::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:45:28,697::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:45:28,728::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:28,902::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:45:28,903::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:28,908::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:45:28,911::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:45:33,337::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:45:33,337::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:45:43,407::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:45:43,407::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:45:45,652::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:45:45,699::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:45,857::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:45:45,858::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:45,863::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:45:45,866::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:45:45,870::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:45:48,078::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:45:48,079::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:45:52,474::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:45:52,482::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:45:52,596::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:45:52,597::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:45:55,089::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:45:55,089::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:45:55,089::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:45:57,353::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:45:57,385::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:57,510::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:45:57,510::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:45:57,516::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:45:57,518::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:46:01,895::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:46:01,895::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:46:11,960::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:46:11,960::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:46:14,208::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:46:14,255::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:14,413::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:46:14,414::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:14,419::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:46:14,422::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:46:14,422::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:46:14,426::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:46:16,634::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:46:16,635::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:46:21,024::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:46:21,032::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:46:21,147::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:46:21,147::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:46:23,628::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:46:23,628::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:46:23,628::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:46:25,885::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:46:25,916::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:26,058::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:46:26,059::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:26,064::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:46:26,067::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:46:30,468::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:46:30,468::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:46:40,532::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:46:40,532::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:46:42,774::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:46:42,806::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:42,979::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:46:42,980::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:42,985::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:46:42,988::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:46:42,992::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:46:45,199::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:46:45,199::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:46:49,591::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:46:49,599::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:46:49,714::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:46:49,715::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:46:52,169::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:46:52,170::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:46:52,170::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:46:54,429::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:46:54,477::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:54,651::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:46:54,652::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:46:54,657::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:46:54,660::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:46:59,090::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:46:59,091::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:47:09,161::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:47:09,161::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:47:11,391::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:47:11,438::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:11,564::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:47:11,565::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:11,570::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:47:11,572::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:47:11,577::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:47:13,778::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:47:13,778::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:47:18,162::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:47:18,170::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:47:18,285::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:47:18,285::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:47:20,771::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:47:20,771::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:47:20,771::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:47:23,025::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:47:23,072::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:23,278::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:47:23,279::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:23,284::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:47:23,287::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:47:27,711::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:47:27,712::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:47:37,774::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:47:37,775::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:47:40,060::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:47:40,108::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:40,297::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:47:40,298::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:40,304::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:47:40,306::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:47:40,307::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:47:40,310::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:47:42,500::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:47:42,501::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:47:46,877::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:47:46,885::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:47:47,000::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:47:47,000::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:47:49,450::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:47:49,450::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:47:49,450::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:47:51,698::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:47:51,730::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:51,855::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:47:51,856::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:47:51,861::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:47:51,864::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:47:56,291::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:47:56,291::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:48:06,352::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:48:06,352::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:48:08,608::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:48:08,656::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:08,797::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:48:08,798::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:08,803::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:48:08,806::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:48:08,810::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:48:11,038::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:48:11,038::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:48:15,405::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:48:15,413::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:48:15,528::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:48:15,528::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:48:18,016::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:48:18,017::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:48:18,017::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:48:20,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:48:20,313::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:20,455::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:48:20,455::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:20,461::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:48:20,463::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:48:24,874::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:48:24,875::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:48:34,938::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:48:34,938::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:48:37,193::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:48:37,240::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:37,398::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:48:37,399::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:37,404::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:48:37,407::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:48:37,411::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:48:39,630::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:48:39,630::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:48:44,014::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:48:44,022::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:48:44,136::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:48:44,137::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:48:46,615::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:48:46,616::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:48:46,616::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:48:48,866::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:48:48,897::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:49,055::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:48:49,056::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:48:49,061::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:48:49,064::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:48:53,480::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:48:53,480::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:49:03,600::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:49:03,600::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:49:05,852::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:49:05,884::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:06,073::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:49:06,074::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:06,079::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:49:06,082::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:49:06,082::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:49:06,086::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:49:08,299::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:49:08,300::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:49:12,724::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:49:12,732::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:49:12,847::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:49:12,847::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:49:15,283::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:49:15,283::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:49:15,283::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:49:17,529::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:49:17,576::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:17,750::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:49:17,751::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:17,757::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:49:17,759::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:49:22,160::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:49:22,161::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:49:32,224::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:49:32,224::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:49:34,483::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:49:34,530::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:34,672::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:49:34,673::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:34,678::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:49:34,681::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:49:34,685::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:49:36,896::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:49:36,896::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:49:41,301::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:49:41,309::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:49:41,423::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:49:41,424::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:49:43,940::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:49:43,940::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:49:43,940::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:49:46,183::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:49:46,230::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:46,372::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:49:46,373::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:49:46,378::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:49:46,381::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:49:50,774::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:49:50,774::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:50:00,837::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:50:00,837::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:50:03,108::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:50:03,155::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:03,296::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:50:03,297::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:03,303::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:50:03,305::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:50:03,309::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:50:05,530::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:50:05,530::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:50:09,948::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:50:09,956::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:50:10,071::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:50:10,071::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:50:12,569::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:50:12,569::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:50:12,569::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:50:14,853::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:50:14,885::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:15,027::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:50:15,027::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:15,033::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:50:15,035::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:50:19,413::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:50:19,413::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:50:29,477::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:50:29,478::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:50:31,738::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:50:31,769::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:31,911::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:50:31,911::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:31,917::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:50:31,919::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:50:31,920::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:50:31,924::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:50:34,132::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:50:34,132::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:50:38,529::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:50:38,537::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:50:38,652::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:50:38,652::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:50:41,165::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:50:41,165::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:50:41,165::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:50:43,446::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:50:43,493::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:43,651::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:50:43,652::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:50:43,657::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:50:43,660::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:50:48,064::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:50:48,065::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:50:58,129::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:50:58,129::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:51:00,385::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:51:00,432::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:00,574::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:51:00,575::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:00,580::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:51:00,582::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:51:00,587::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:51:02,789::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:51:02,790::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:51:07,211::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:51:07,219::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:51:07,334::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:51:07,334::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:51:09,825::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:51:09,826::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:51:09,826::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:51:12,072::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:51:12,120::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:12,261::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:51:12,262::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:12,267::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:51:12,270::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:51:16,701::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:51:16,701::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:51:26,763::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:51:26,763::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:51:28,999::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:51:29,047::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:29,188::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:51:29,189::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:29,194::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:51:29,197::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:51:29,201::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:51:31,404::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:51:31,404::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:51:35,793::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:51:35,800::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:51:35,915::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:51:35,916::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:51:38,387::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:51:38,387::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:51:38,387::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:51:40,658::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:51:40,706::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:40,880::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:51:40,881::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:40,886::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:51:40,889::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:51:45,312::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:51:45,313::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:51:55,436::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:51:55,436::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:51:57,704::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:51:57,735::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:57,909::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:51:57,910::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:51:57,915::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:51:57,918::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:51:57,918::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:51:57,922::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:52:00,111::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:52:00,111::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:52:04,485::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:52:04,492::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:52:04,607::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:52:04,608::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:52:07,082::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:52:07,083::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:52:07,083::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:52:09,329::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:52:09,377::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:09,551::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:52:09,552::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:09,558::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:52:09,561::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:52:13,973::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:52:13,973::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:52:24,030::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:52:24,030::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:52:26,284::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:52:26,316::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:26,490::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:52:26,491::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:26,497::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:52:26,499::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:52:26,504::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:52:28,730::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:52:28,730::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:52:33,093::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:52:33,101::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:52:33,216::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:52:33,216::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:52:35,672::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:52:35,672::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:52:35,672::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:52:37,920::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:52:37,967::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:38,109::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:52:38,109::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:38,115::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:52:38,117::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:52:42,525::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:52:42,526::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:52:52,588::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:52:52,588::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:52:54,851::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:52:54,882::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:55,072::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:52:55,073::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:52:55,079::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:52:55,082::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:52:55,082::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 10:52:55,083::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=7105 (Thu Jun 15 10:52:24 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=7119 (Thu Jun 15 10:52:38 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '4469d43d', 'local_conf_timestamp': 7119, 'host-ts': 7105} MainThread::INFO::2017-06-15 10:52:55,083::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=7098 (Thu Jun 15 10:52:24 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=7112 (Thu Jun 15 10:52:38 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '1d2e29c7', 'local_conf_timestamp': 7112, 'host-ts': 7098} MainThread::INFO::2017-06-15 10:52:55,083::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'bridge': True, 'mem-free': 14933.0, 'maintenance': False, 'cpu-load': 0.0211, 'gateway': 1.0} MainThread::INFO::2017-06-15 10:52:55,087::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:52:57,312::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:52:57,313::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:01,707::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:01,715::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:53:01,829::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:53:01,830::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:53:04,256::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:53:04,257::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:53:04,257::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:53:06,504::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:53:06,535::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:06,709::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:53:06,710::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:06,715::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:53:06,718::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:53:11,123::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:53:11,123::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:53:21,185::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:53:21,185::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:53:23,437::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:53:23,468::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:23,594::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:53:23,595::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:23,600::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:53:23,602::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:53:23,603::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:53:23,607::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:53:25,819::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:53:25,820::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:30,235::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:30,243::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:53:30,358::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:53:30,358::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:53:32,761::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:53:32,762::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:53:32,762::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:53:35,008::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:53:35,056::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:35,230::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:53:35,231::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:35,236::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:53:35,239::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:53:39,633::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:53:39,633::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:53:49,696::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:53:49,696::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:53:51,947::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:53:51,978::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:52,104::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:53:52,106::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:53:52,111::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:53:52,114::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:53:52,118::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:53:54,330::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:53:54,330::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:58,740::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:53:58,748::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:53:58,863::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:53:58,863::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:54:01,365::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:54:01,366::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:54:01,366::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:54:03,596::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:54:03,644::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:03,785::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:54:03,786::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:03,791::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:54:03,794::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:54:08,185::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:54:08,185::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:54:18,250::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:54:18,250::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:54:20,496::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:54:20,528::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:20,669::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:54:20,670::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:20,675::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:54:20,678::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:54:20,682::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:54:22,892::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:54:22,893::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:54:27,297::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:54:27,305::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:54:27,420::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:54:27,420::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:54:29,911::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:54:29,911::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:54:29,911::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:54:32,175::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:54:32,223::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:32,380::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:54:32,381::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:32,386::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:54:32,389::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:54:36,769::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:54:36,770::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:54:46,834::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:54:46,834::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:54:49,078::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:54:49,109::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:49,251::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:54:49,252::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:54:49,257::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:54:49,260::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:54:49,260::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:54:49,264::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:54:51,471::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:54:51,471::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:54:55,863::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:54:55,871::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:54:55,986::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:54:55,986::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:54:58,475::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:54:58,476::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:54:58,476::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:55:00,731::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:55:00,762::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:00,888::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:55:00,889::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:00,895::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:55:00,897::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:55:05,299::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:55:05,299::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:55:15,428::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:55:15,428::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:55:17,671::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:55:17,702::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:17,860::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:55:17,861::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:17,866::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:55:17,869::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:55:17,873::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:55:20,077::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:55:20,077::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:55:24,473::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:55:24,481::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:55:24,596::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:55:24,596::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:55:27,085::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:55:27,085::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:55:27,085::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:55:29,337::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:55:29,368::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:29,510::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:55:29,510::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:29,516::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:55:29,518::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:55:33,943::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:55:33,943::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:55:44,003::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:55:44,003::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:55:46,232::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:55:46,264::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:46,389::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:55:46,390::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:46,395::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:55:46,398::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:55:46,402::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:55:48,603::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:55:48,603::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:55:53,020::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:55:53,028::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:55:53,143::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:55:53,143::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:55:55,613::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:55:55,613::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:55:55,613::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:55:57,871::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:55:57,902::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:58,044::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:55:58,045::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:55:58,050::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:55:58,053::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:56:02,474::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:56:02,474::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:56:12,538::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:56:12,538::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:56:14,799::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:56:14,831::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:14,972::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:56:14,973::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:14,978::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:56:14,980::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:56:14,981::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 10:56:14,985::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:56:17,182::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:56:17,182::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:56:21,561::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:56:21,569::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:56:21,684::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:56:21,684::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:56:24,153::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:56:24,153::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:56:24,153::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:56:26,403::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:56:26,434::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:26,559::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:56:26,560::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:26,566::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:56:26,569::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:56:30,979::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:56:30,980::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:56:41,043::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:56:41,043::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:56:43,348::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:56:43,379::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:43,505::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:56:43,506::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:43,511::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:56:43,514::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:56:43,518::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:56:45,743::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:56:45,743::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:56:50,107::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:56:50,115::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:56:50,230::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:56:50,230::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:56:52,710::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:56:52,710::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:56:52,710::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:56:54,967::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:56:55,014::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:55,140::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:56:55,141::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:56:55,147::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:56:55,149::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:56:59,564::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:56:59,564::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:57:09,637::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:09,637::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:57:11,891::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:57:11,938::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:12,079::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:57:12,080::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:12,086::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:57:12,088::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:57:12,092::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:57:14,308::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:57:14,309::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:57:18,695::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:57:18,702::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:57:18,817::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:57:18,818::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:57:21,278::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:57:21,278::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:21,278::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:57:23,527::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:57:23,559::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:23,684::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:57:23,685::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:23,691::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:57:23,693::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:57:28,104::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 10:57:28,104::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:57:38,171::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:38,171::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:57:40,420::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:57:40,451::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:40,577::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:57:40,577::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:40,584::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:57:40,586::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:57:40,587::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 10:57:40,591::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497538660.59 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:57:40,627::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 10:57:40,627::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:57:42,869::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:57:42,869::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:57:47,304::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:57:47,312::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:57:47,426::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:57:47,427::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:57:49,845::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:57:49,846::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:49,846::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:57:52,089::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:57:52,136::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:52,278::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:57:52,279::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:52,284::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:57:52,286::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:57:56,701::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 10:57:56,701::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:57:56,753::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:56,753::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:57:59,004::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:57:59,035::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:59,193::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:57:59,194::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:57:59,199::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:57:59,202::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:57:59,202::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:57:59,202::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:58:01,484::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:58:01,515::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:01,640::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:58:01,641::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:01,646::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:58:01,649::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:58:01,649::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 10:58:01,649::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:58:01,649::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:58:03,914::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:58:03,946::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:04,087::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:58:04,088::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:04,094::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:58:04,097::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:58:06,304::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 10:58:06,305::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 10:58:10,890::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 10:58:10,890::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist MainThread::INFO::2017-06-15 10:58:10,890::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 10:58:10,895::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497538690.9 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 10:58:10,928::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 10:58:10,928::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:58:13,173::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:58:13,174::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:58:17,577::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:58:17,585::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:58:17,700::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:58:17,700::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:58:20,179::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:58:20,180::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:58:20,180::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:58:22,490::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:58:22,540::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:22,797::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:58:22,800::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:22,810::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:58:22,814::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:58:27,540::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:58:27,540::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:58:37,742::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:58:37,742::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:58:40,028::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:58:40,092::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:40,330::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:58:40,331::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:40,337::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:58:40,339::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:58:40,340::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:58:40,344::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 11:08:37 2017 while transitioning -> MainThread::INFO::2017-06-15 10:58:40,347::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:58:42,566::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:58:42,566::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:58:46,974::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:58:46,982::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:58:47,097::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:58:47,098::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:58:49,514::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:58:49,515::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:58:49,515::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:58:51,767::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:58:51,814::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:51,940::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:58:51,941::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:58:51,947::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:58:51,950::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:58:56,358::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:58:56,358::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:59:06,425::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:59:06,425::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:59:08,677::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:59:08,708::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:08,834::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:59:08,835::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:08,841::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:59:08,843::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:59:08,844::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:59:08,848::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:59:11,063::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:59:11,064::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:59:15,495::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:59:15,503::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:59:15,618::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:59:15,618::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:59:18,085::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:59:18,085::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:59:18,085::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:59:20,326::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:59:20,358::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:20,515::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:59:20,516::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:20,523::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:59:20,526::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:59:24,927::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:59:24,928::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 10:59:35,026::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:59:35,026::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:59:37,278::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:59:37,309::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:37,499::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:59:37,500::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:37,506::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:59:37,508::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:59:37,509::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 10:59:37,513::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 10:59:39,726::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 10:59:39,726::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:59:44,131::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 10:59:44,138::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 10:59:44,253::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 10:59:44,254::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 10:59:46,754::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 10:59:46,754::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 10:59:46,754::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 10:59:48,986::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 10:59:49,033::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:49,159::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 10:59:49,160::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 10:59:49,165::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 10:59:49,168::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 10:59:53,561::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 10:59:53,562::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:00:03,610::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:00:03,610::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:00:05,856::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:00:05,903::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:06,094::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:00:06,095::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:06,100::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:00:06,103::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:00:06,103::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:00:06,107::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:00:08,320::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:00:08,320::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:00:12,738::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:00:12,746::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:00:12,861::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:00:12,861::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:00:15,358::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:00:15,358::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:00:15,359::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:00:17,634::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:00:17,681::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:17,871::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:00:17,872::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:17,877::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:00:17,880::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:00:22,263::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:00:22,263::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:00:32,335::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:00:32,335::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:00:34,581::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:00:34,613::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:34,802::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:00:34,803::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:34,808::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:00:34,811::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:00:34,811::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:00:34,815::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:00:37,040::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:00:37,040::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:00:41,439::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:00:41,447::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:00:41,562::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:00:41,563::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:00:44,085::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:00:44,086::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:00:44,086::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:00:46,341::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:00:46,389::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:46,546::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:00:46,547::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:00:46,554::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:00:46,556::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:00:50,959::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:00:50,960::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:01:01,024::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:01:01,024::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:01:03,294::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:01:03,342::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:03,467::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:01:03,468::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:03,474::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:01:03,476::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:01:03,477::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:01:03,481::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:01:05,689::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:01:05,690::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:01:10,094::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:01:10,102::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:01:10,217::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:01:10,218::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:01:12,705::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:01:12,706::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:01:12,706::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:01:14,963::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:01:15,010::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:15,184::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:01:15,185::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:15,191::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:01:15,194::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:01:19,633::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:01:19,633::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:01:29,685::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:01:29,685::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:01:31,917::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:01:31,965::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:32,154::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:01:32,155::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:32,161::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:01:32,163::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:01:32,164::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:01:32,168::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:01:34,373::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:01:34,374::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:01:38,763::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:01:38,771::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:01:38,886::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:01:38,886::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:01:41,373::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:01:41,373::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:01:41,374::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:01:43,625::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:01:43,656::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:43,830::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:01:43,831::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:01:43,836::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:01:43,839::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:01:48,261::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:01:48,262::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:01:58,326::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:01:58,326::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:02:00,593::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:02:00,641::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:00,814::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:02:00,815::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:00,821::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:02:00,823::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:02:00,824::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:02:00,828::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:02:03,019::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:02:03,020::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:02:07,406::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:02:07,414::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:02:07,529::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:02:07,530::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:02:10,014::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:02:10,014::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:02:10,014::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:02:12,274::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:02:12,322::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:12,495::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:02:12,496::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:12,502::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:02:12,505::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:02:16,918::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:02:16,918::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:02:26,986::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:02:26,986::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:02:29,273::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:02:29,321::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:29,479::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:02:29,480::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:29,486::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:02:29,488::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:02:29,489::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:02:29,493::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:02:31,722::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:02:31,722::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:02:36,102::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:02:36,110::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:02:36,225::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:02:36,226::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:02:38,705::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:02:38,705::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:02:38,705::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:02:40,953::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:02:40,984::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:41,174::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:02:41,175::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:41,180::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:02:41,183::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:02:45,594::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:02:45,594::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:02:55,659::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:02:55,659::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:02:57,912::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:02:57,959::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:58,150::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:02:58,151::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:02:58,157::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:02:58,160::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:02:58,160::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 11:02:58,160::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=7707 (Thu Jun 15 11:02:25 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=7721 (Thu Jun 15 11:02:40 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'dbfde110', 'local_conf_timestamp': 7721, 'host-ts': 7707} MainThread::INFO::2017-06-15 11:02:58,160::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=7696 (Thu Jun 15 11:02:22 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=7710 (Thu Jun 15 11:02:36 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineDown\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '5e63b1f3', 'local_conf_timestamp': 7710, 'host-ts': 7696} MainThread::INFO::2017-06-15 11:02:58,160::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'failed liveliness check', 'health': 'bad', 'vm': 'up', 'detail': 'up'}, 'bridge': True, 'mem-free': 12487.0, 'maintenance': False, 'cpu-load': 0.02, 'gateway': 1.0} MainThread::INFO::2017-06-15 11:02:58,161::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:02:58,164::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:03:00,385::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:03:00,385::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:03:04,791::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:03:04,799::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:03:04,914::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:03:04,915::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:03:07,392::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:03:07,392::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:03:07,393::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:03:09,635::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:03:09,682::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:09,840::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:03:09,841::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:09,846::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:03:09,849::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:03:14,255::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:03:14,255::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:03:24,314::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:03:24,314::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:03:26,568::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:03:26,615::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:26,757::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:03:26,758::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:26,764::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:03:26,767::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:03:26,767::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:03:26,771::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:03:28,989::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:03:28,990::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:03:33,420::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:03:33,428::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:03:33,543::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:03:33,543::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:03:36,011::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:03:36,011::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:03:36,012::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:03:38,267::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:03:38,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:38,472::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:03:38,473::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:38,478::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:03:38,480::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:03:42,890::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:03:42,890::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:03:52,946::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:03:52,946::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:03:55,197::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:03:55,228::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:55,370::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:03:55,371::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:03:55,376::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:03:55,379::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:03:55,379::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:03:55,383::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:03:57,601::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:03:57,601::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:02,007::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:02,015::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:04:02,130::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:04:02,130::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:04:04,631::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:04:04,631::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:04:04,631::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:04:06,864::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:04:06,911::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:07,087::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:04:07,089::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:07,096::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:04:07,099::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:04:11,497::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:04:11,497::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:04:21,558::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:04:21,558::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:04:23,806::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:04:23,854::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:23,996::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:04:23,997::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:24,003::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:04:24,006::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:04:24,006::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:04:24,010::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:04:26,220::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:04:26,220::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:30,620::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:30,628::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:04:30,743::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:04:30,744::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:04:33,239::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:04:33,239::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:04:33,240::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:04:35,509::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:04:35,557::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:35,715::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:04:35,716::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:35,721::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:04:35,724::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:04:40,111::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:04:40,111::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:04:50,180::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:04:50,181::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:04:52,425::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:04:52,473::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:52,662::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:04:52,663::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:04:52,669::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:04:52,671::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:04:52,672::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:04:52,676::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:04:54,888::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:04:54,889::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:59,286::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:04:59,294::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:04:59,409::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:04:59,410::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:05:01,901::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:05:01,901::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:05:01,901::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:05:04,159::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:05:04,207::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:04,364::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:05:04,365::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:04,371::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:05:04,373::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:05:08,776::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:05:08,777::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:05:18,842::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:05:18,843::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:05:21,084::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:05:21,132::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:21,289::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:05:21,290::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:21,296::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:05:21,298::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:05:21,299::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:05:21,303::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:05:23,515::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:05:23,515::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:05:27,909::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:05:27,916::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:05:28,031::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:05:28,032::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:05:30,520::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:05:30,520::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:05:30,520::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:05:32,778::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:05:32,826::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:33,048::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:05:33,049::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:33,055::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:05:33,057::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:05:37,497::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:05:37,498::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:05:47,567::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:05:47,567::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:05:49,801::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:05:49,832::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:49,958::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:05:49,959::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:05:49,964::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:05:49,966::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:05:49,967::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:05:49,971::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:05:52,173::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:05:52,174::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:05:56,560::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:05:56,567::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:05:56,683::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:05:56,683::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:05:59,170::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:05:59,170::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:05:59,170::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:06:01,424::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:06:01,455::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:01,629::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:06:01,630::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:01,635::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:06:01,638::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:06:06,062::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:06:06,062::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:06:16,125::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:06:16,125::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:06:18,400::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:06:18,448::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:18,573::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:06:18,574::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:18,579::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:06:18,582::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:06:18,582::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:06:18,586::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:06:20,782::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:06:20,783::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:06:25,161::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:06:25,169::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:06:25,284::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:06:25,284::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:06:27,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:06:27,768::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:06:27,768::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:06:30,031::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:06:30,063::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:30,220::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:06:30,221::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:30,227::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:06:30,229::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:06:34,646::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:06:34,646::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:06:44,715::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:06:44,715::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:06:46,973::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:06:47,005::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:47,131::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:06:47,133::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:47,139::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:06:47,141::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:06:47,142::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:06:47,146::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:06:49,373::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:06:49,374::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:06:53,745::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:06:53,753::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:06:53,867::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:06:53,868::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:06:56,335::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:06:56,335::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:06:56,335::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:06:58,580::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:06:58,612::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:58,754::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:06:58,755::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:06:58,761::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:06:58,764::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:07:03,182::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:07:03,183::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:07:13,310::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:07:13,310::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:07:15,567::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:07:15,598::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:15,740::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:07:15,741::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:15,747::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:07:15,750::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:07:15,750::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:07:15,754::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:07:17,973::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:07:17,974::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:07:22,377::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:07:22,385::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:07:22,500::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:07:22,500::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:07:24,977::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:07:24,978::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:07:24,978::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:07:27,221::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:07:27,268::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:27,426::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:07:27,427::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:27,433::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:07:27,436::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:07:31,842::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:07:31,842::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:07:41,902::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:07:41,902::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:07:44,157::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:07:44,188::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:44,362::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:07:44,363::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:44,369::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:07:44,372::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:07:44,372::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:07:44,376::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:07:46,591::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:07:46,592::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:07:51,003::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:07:51,011::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:07:51,126::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:07:51,126::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:07:53,640::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:07:53,640::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:07:53,640::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:07:55,882::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:07:55,930::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:56,071::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:07:56,072::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:07:56,077::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:07:56,080::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:08:00,480::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:08:00,481::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:08:10,540::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:08:10,540::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:08:12,789::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:08:12,836::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:13,058::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:08:13,059::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:13,065::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:08:13,067::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:08:13,068::states::760::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) VM is powering up.. MainThread::INFO::2017-06-15 11:08:13,072::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:08:15,284::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:08:15,284::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:08:19,700::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:08:19,707::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:08:19,822::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:08:19,823::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:08:22,292::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:08:22,293::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:08:22,293::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:08:24,526::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:08:24,574::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:24,731::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:08:24,732::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:24,737::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:08:24,740::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:08:29,143::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:08:29,143::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:08:39,199::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:08:39,199::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:08:41,446::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:08:41,477::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:41,603::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:08:41,603::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:41,610::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:08:41,612::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:08:41,617::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539321.62 type=state_transition detail=EngineStarting-EngineStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:08:41,653::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineStop) sent? sent MainThread::INFO::2017-06-15 11:08:41,653::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:08:43,868::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:08:43,868::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:08:48,266::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:08:48,274::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:08:48,389::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:08:48,390::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:08:50,890::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:08:50,891::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:08:50,891::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:08:53,161::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:08:53,209::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:53,399::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:08:53,400::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:08:53,406::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:08:53,409::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:08:57,794::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:08:57,794::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:09:07,862::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:09:07,862::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:09:10,123::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:09:10,154::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:10,312::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:09:10,313::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:10,319::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:09:10,322::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:09:10,322::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-shutdown` MainThread::INFO::2017-06-15 11:09:12,614::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 11:09:12,614::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::ERROR::2017-06-15 11:09:12,614::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 11:09:12,619::state_decorators::95::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout set to Thu Jun 15 11:14:07 2017 while transitioning -> MainThread::INFO::2017-06-15 11:09:12,623::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:09:14,836::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:09:14,836::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:09:19,241::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:09:19,249::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:09:19,364::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:09:19,365::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:09:21,857::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:09:21,858::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:09:21,858::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:09:24,118::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:09:24,166::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:24,291::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:09:24,292::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:24,299::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:09:24,302::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:09:28,708::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:09:28,709::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:09:38,775::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:09:38,775::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:09:41,020::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:09:41,067::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:41,257::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:09:41,258::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:41,263::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:09:41,266::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:09:41,267::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (31 of 300 seconds) MainThread::INFO::2017-06-15 11:09:41,270::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:09:43,477::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:09:43,478::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:09:47,887::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:09:47,894::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:09:48,009::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:09:48,010::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:09:50,497::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:09:50,497::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:09:50,497::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:09:52,753::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:09:52,784::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:52,942::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:09:52,943::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:09:52,949::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:09:52,952::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:09:57,387::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:09:57,387::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:10:07,464::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:10:07,464::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:10:09,699::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:10:09,746::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:09,920::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:10:09,921::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:09,926::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:10:09,929::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:10:09,929::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (59 of 300 seconds) MainThread::INFO::2017-06-15 11:10:09,933::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:10:12,137::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:10:12,137::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:10:16,532::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:10:16,539::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:10:16,655::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:10:16,655::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:10:19,140::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:10:19,140::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:10:19,140::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:10:21,393::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:10:21,440::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:21,614::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:10:21,615::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:21,620::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:10:21,623::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:10:26,043::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:10:26,043::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:10:36,110::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:10:36,110::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:10:38,377::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:10:38,409::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:38,583::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:10:38,584::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:38,590::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:10:38,592::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:10:38,593::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (88 of 300 seconds) MainThread::INFO::2017-06-15 11:10:38,597::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:10:40,790::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:10:40,791::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:10:45,185::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:10:45,193::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:10:45,308::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:10:45,308::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:10:47,794::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:10:47,794::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:10:47,794::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:10:50,045::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:10:50,076::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:50,218::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:10:50,219::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:10:50,225::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:10:50,228::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:10:54,643::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:10:54,643::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:11:04,699::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:11:04,699::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:11:06,957::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:11:06,988::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:07,146::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:11:07,147::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:07,153::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:11:07,156::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:11:07,156::states::574::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Waiting on shutdown to complete (117 of 300 seconds) MainThread::INFO::2017-06-15 11:11:07,160::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:11:09,384::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:11:09,385::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:11:13,763::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:11:13,771::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:11:13,886::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:11:13,886::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:11:16,367::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:11:16,368::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:11:16,368::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:11:18,650::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:11:18,698::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:18,872::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:11:18,874::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:18,880::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:11:18,882::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:11:23,297::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStop (score: 3400) MainThread::INFO::2017-06-15 11:11:23,298::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:11:33,364::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:11:33,364::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:11:35,619::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:11:35,666::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:35,840::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:11:35,841::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:35,848::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:11:35,850::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:11:35,851::states::562::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm not running on local host MainThread::INFO::2017-06-15 11:11:35,854::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 11:11:35,858::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539495.86 type=state_transition detail=EngineStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:11:35,896::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 11:11:35,896::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:11:38,118::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:11:38,118::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:11:42,505::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:11:42,513::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:11:42,628::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:11:42,628::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:11:45,104::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:11:45,104::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:11:45,105::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:11:47,348::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:11:47,379::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:47,537::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:11:47,538::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:11:47,545::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:11:47,548::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:11:51,953::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:11:51,953::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:12:02,017::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:02,017::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:04,266::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:04,313::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:04,471::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:04,472::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:04,478::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:04,481::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:04,482::states::497::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine down and local host has best score (3400), attempting to start engine VM MainThread::INFO::2017-06-15 11:12:04,485::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539524.49 type=state_transition detail=EngineDown-EngineStart hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:12:04,515::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineDown-EngineStart) sent? sent MainThread::INFO::2017-06-15 11:12:04,515::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:12:06,732::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:12:06,733::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:12:11,170::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:12:11,186::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:12:11,301::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:12:11,301::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:12:13,767::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:12:13,767::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:13,767::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:16,010::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:16,042::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:16,199::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:16,200::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:16,206::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:16,209::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:20,626::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStart (score: 3400) MainThread::INFO::2017-06-15 11:12:20,626::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:12:20,678::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:20,678::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:22,934::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:22,966::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:23,140::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:23,141::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:23,146::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:23,149::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:23,149::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:23,149::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:25,410::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:25,474::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:25,664::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:25,665::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:25,670::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:25,673::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:25,673::hosted_engine::1158::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Ensuring VDSM state is clear for engine VM MainThread::INFO::2017-06-15 11:12:25,673::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:25,673::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:27,936::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:27,984::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:28,141::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:28,142::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:28,147::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:28,150::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:30,353::hosted_engine::1186::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Cleaning state for non-running VM MainThread::INFO::2017-06-15 11:12:35,747::hosted_engine::1170::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_clean_vdsm_state) Vdsm state for VM clean MainThread::INFO::2017-06-15 11:12:35,748::hosted_engine::1119::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Starting vm using `/usr/sbin/hosted-engine --vm-start` MainThread::INFO::2017-06-15 11:12:40,347::hosted_engine::1125::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stdout: MainThread::INFO::2017-06-15 11:12:40,348::hosted_engine::1126::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) stderr: Virtual machine does not exist Exception in thread Client localhost:54321 (most likely raised during interpreter shutdown): MainThread::INFO::2017-06-15 11:12:40,348::hosted_engine::1138::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_start_engine_vm) Engine VM started on localhost MainThread::INFO::2017-06-15 11:12:40,357::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539560.36 type=state_transition detail=EngineStart-EngineStarting hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:12:40,390::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStart-EngineStarting) sent? sent MainThread::INFO::2017-06-15 11:12:40,390::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:12:42,594::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:12:42,595::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:12:46,982::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:12:46,990::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:12:47,105::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:12:47,105::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:12:49,607::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:12:49,607::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:12:49,607::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:12:51,859::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:12:51,907::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:52,080::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:12:52,082::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:12:52,087::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:12:52,090::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:12:56,516::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineStarting (score: 3400) MainThread::INFO::2017-06-15 11:12:56,517::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:13:06,611::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:13:06,612::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:13:08,874::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:13:08,906::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:09,047::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:13:09,049::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:09,055::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:13:09,058::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:13:09,058::state_machine::169::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Global metadata: {'maintenance': False} MainThread::INFO::2017-06-15 11:13:09,058::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-02.reis.com (id 2): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=8296 (Thu Jun 15 11:12:15 2017)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=8322 (Thu Jun 15 11:12:41 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-02.reis.com', 'alive': True, 'host-id': 2, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': 'a1e33524', 'local_conf_timestamp': 8322, 'host-ts': 8296} MainThread::INFO::2017-06-15 11:13:09,058::state_machine::174::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Host ovirt-hyp-03.reis.com (id 3): {'conf_on_shared_storage': True, 'extra': 'metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=8289 (Thu Jun 15 11:12:15 2017)\nhost-id=3\nscore=3400\nvm_conf_refresh_time=8315 (Thu Jun 15 11:12:41 2017)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineStarting\nstopped=False\n', 'hostname': 'ovirt-hyp-03.reis.com', 'alive': True, 'host-id': 3, 'engine-status': {'reason': 'vm not running on this host', 'health': 'bad', 'vm': 'down', 'detail': 'unknown'}, 'score': 3400, 'stopped': False, 'maintenance': False, 'crc32': '38df7fdc', 'local_conf_timestamp': 8315, 'host-ts': 8289} MainThread::INFO::2017-06-15 11:13:09,058::state_machine::177::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(refresh) Local (id 1): {'engine-health': {'reason': 'Storage of VM is locked. Is another host already starting the VM?', 'health': 'bad', 'vm': 'already_locked', 'detail': 'down'}, 'bridge': True, 'mem-free': 14930.0, 'maintenance': False, 'cpu-load': 0.0235, 'gateway': 1.0} MainThread::INFO::2017-06-15 11:13:09,059::states::763::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Another host already took over.. MainThread::INFO::2017-06-15 11:13:09,062::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning -> MainThread::INFO::2017-06-15 11:13:09,066::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539589.07 type=state_transition detail=EngineStarting-EngineForceStop hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:13:09,105::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineStarting-EngineForceStop) sent? sent MainThread::INFO::2017-06-15 11:13:09,105::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:13:11,304::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:13:11,304::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:13:15,706::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:13:15,714::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:13:15,829::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:13:15,829::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:13:18,318::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:13:18,318::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:13:18,318::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:13:20,568::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:13:20,616::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:20,790::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:13:20,791::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:20,797::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:13:20,800::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:13:25,228::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineForceStop (score: 3400) MainThread::INFO::2017-06-15 11:13:25,228::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:13:25,281::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:13:25,281::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:13:27,539::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:13:27,586::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:27,728::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:13:27,729::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:27,735::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:13:27,738::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:13:27,739::hosted_engine::1206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Shutting down vm using `/usr/sbin/hosted-engine --vm-poweroff` MainThread::INFO::2017-06-15 11:13:30,036::hosted_engine::1211::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stdout: MainThread::INFO::2017-06-15 11:13:30,037::hosted_engine::1212::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) stderr: MainThread::ERROR::2017-06-15 11:13:30,037::hosted_engine::1220::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_stop_engine_vm) Engine VM stopped on localhost MainThread::INFO::2017-06-15 11:13:30,041::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1497539610.04 type=state_transition detail=EngineForceStop-EngineDown hostname='ovirt-hyp-01.reis.com' MainThread::INFO::2017-06-15 11:13:30,077::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineForceStop-EngineDown) sent? sent MainThread::INFO::2017-06-15 11:13:30,077::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:13:32,307::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:13:32,307::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:13:36,679::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:13:36,687::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:13:36,802::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:13:36,802::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:13:39,284::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:13:39,284::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:13:39,284::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:13:41,530::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:13:41,562::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:41,719::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:13:41,720::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:41,727::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:13:41,730::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:13:46,138::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:13:46,138::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:13:56,200::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:13:56,200::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:13:58,453::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:13:58,501::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:58,690::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:13:58,691::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:13:58,697::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:13:58,699::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:13:58,700::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 11:13:58,704::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:14:00,922::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:14:00,922::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:14:05,308::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:14:05,316::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:14:05,431::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:14:05,431::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:14:07,911::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:14:07,911::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:14:07,911::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:14:10,157::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:14:10,204::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:10,378::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:14:10,379::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:10,386::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:14:10,388::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:14:14,797::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:14:14,797::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:14:24,871::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:14:24,871::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:14:27,122::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:14:27,170::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:27,312::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:14:27,313::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:27,319::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:14:27,322::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:14:27,326::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:14:29,542::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:14:29,542::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:14:33,983::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:14:33,991::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:14:34,106::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:14:34,106::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:14:36,582::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:14:36,582::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:14:36,582::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:14:38,822::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:14:38,854::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:38,995::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:14:38,996::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:39,002::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:14:39,004::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:14:43,406::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:14:43,406::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:14:53,470::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:14:53,470::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:14:55,726::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:14:55,773::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:55,963::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:14:55,964::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:14:55,970::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:14:55,972::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:14:55,977::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:14:58,190::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:14:58,190::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:02,601::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:02,609::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:15:02,724::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:15:02,724::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:15:05,225::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:15:05,225::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:15:05,225::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:15:07,457::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:15:07,489::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:07,614::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:15:07,615::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:07,620::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:15:07,623::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:15:12,018::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:15:12,019::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:15:22,085::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:15:22,085::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:15:24,331::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:15:24,363::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:24,504::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:15:24,506::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:24,511::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:15:24,514::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:15:24,514::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 11:15:24,518::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:15:26,744::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:15:26,745::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:31,151::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:31,159::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:15:31,274::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:15:31,274::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:15:33,768::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:15:33,768::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:15:33,768::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:15:36,042::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:15:36,091::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:36,281::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:15:36,282::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:36,288::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:15:36,290::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:15:40,678::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:15:40,678::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:15:50,748::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:15:50,748::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:15:52,995::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:15:53,026::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:53,152::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:15:53,153::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:15:53,159::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:15:53,162::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:15:53,166::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:15:55,380::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:15:55,381::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:59,777::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:15:59,785::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:15:59,900::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:15:59,900::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:16:02,418::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:16:02,418::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:16:02,418::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:16:04,674::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:16:04,722::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:04,911::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:16:04,913::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:04,919::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:16:04,922::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:16:09,323::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:16:09,323::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:16:19,391::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:16:19,391::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:16:21,641::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:16:21,689::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:21,847::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:16:21,848::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:21,854::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:16:21,857::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:16:21,861::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:16:24,084::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:16:24,084::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:16:28,487::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:16:28,495::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:16:28,610::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:16:28,610::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:16:31,099::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:16:31,099::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:16:31,099::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:16:33,355::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:16:33,403::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:33,609::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:16:33,610::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:33,616::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:16:33,618::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:16:38,047::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:16:38,047::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:16:48,113::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:16:48,113::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:16:50,350::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:16:50,398::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:50,572::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:16:50,573::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:16:50,580::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:16:50,582::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:16:50,583::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 11:16:50,587::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:16:52,799::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:16:52,799::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:16:57,190::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:16:57,198::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:16:57,313::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:16:57,313::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:16:59,797::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:16:59,797::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:16:59,797::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:17:02,049::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:17:02,080::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:02,302::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:17:02,303::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:02,310::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:17:02,313::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:17:06,732::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:17:06,732::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:17:16,798::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:17:16,798::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:17:19,072::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:17:19,103::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:19,229::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:17:19,230::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:19,236::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:17:19,239::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:17:19,243::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:17:21,443::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:17:21,444::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:17:25,842::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:17:25,858::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:17:25,973::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:17:25,973::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:17:28,456::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:17:28,457::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:17:28,457::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:17:30,708::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:17:30,755::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:30,945::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:17:30,946::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:30,953::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:17:30,955::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:17:35,371::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:17:35,372::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:17:45,437::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:17:45,437::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:17:47,710::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:17:47,741::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:47,883::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:17:47,884::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:47,890::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:17:47,893::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:17:47,897::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:17:50,131::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:17:50,132::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:17:54,507::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:17:54,514::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:17:54,629::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:17:54,630::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:17:57,110::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:17:57,110::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:17:57,111::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:17:59,360::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:17:59,392::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:59,517::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:17:59,518::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:17:59,524::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:17:59,527::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:18:03,937::hosted_engine::453::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Current state EngineDown (score: 3400) MainThread::INFO::2017-06-15 11:18:03,938::hosted_engine::458::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Best remote host ovirt-hyp-02.reis.com (id: 2, score: 3400) MainThread::INFO::2017-06-15 11:18:14,001::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:18:14,001::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:18:16,264::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:18:16,312::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:18:16,502::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:18:16,503::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:18:16,510::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:18:16,513::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::INFO::2017-06-15 11:18:16,513::states::473::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm is running on host ovirt-hyp-02.reis.com (id 2) MainThread::INFO::2017-06-15 11:18:16,517::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM MainThread::INFO::2017-06-15 11:18:18,756::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage MainThread::INFO::2017-06-15 11:18:18,757::storage_server::219::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:18:23,165::storage_server::226::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server MainThread::INFO::2017-06-15 11:18:23,181::storage_server::233::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain MainThread::INFO::2017-06-15 11:18:23,296::hosted_engine::657::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images MainThread::INFO::2017-06-15 11:18:23,296::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images MainThread::INFO::2017-06-15 11:18:25,774::hosted_engine::660::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf MainThread::INFO::2017-06-15 11:18:25,774::config::485::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain MainThread::INFO::2017-06-15 11:18:25,774::config::412::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE MainThread::INFO::2017-06-15 11:18:28,017::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:5c40a1c0-f0a6-45cf-9c97-21b7966ac9cb, volUUID:71d3d23c-cf69-4179-8f4c-83504e9ed14e MainThread::INFO::2017-06-15 11:18:28,049::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:2f1455db-50b0-4911-a763-918d447b59ab, volUUID:621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:18:28,190::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-06-15 11:18:28,191::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/ovirt-hyp-01.reis.com:engine/d022a79d-611a-4908-9211-9a81b55bcf76/images/2f1455db-50b0-4911-a763-918d447b59ab/621367b0-9a1f-4da9-b671-0f9fec99a0aa MainThread::INFO::2017-06-15 11:18:28,198::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-06-15 11:18:28,201::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE