MainThread::INFO::2017-12-20 12:40:02,709::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.8 started
MainThread::INFO::2017-12-20 12:40:02,778::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 12:40:02,780::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 12:40:05,324::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 12:40:05,325::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 12:40:07,670::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:40:12,362::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:40:12,370::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 12:40:12,637::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 12:40:12,637::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 12:40:15,379::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 12:40:15,379::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 12:40:15,380::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 12:40:17,736::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:0401aa22-a467-4a31-b4cf-27c4e395cfd3, volUUID:8c0534b8-3647-4c75-9da5-88828d8d2faf
MainThread::INFO::2017-12-20 12:40:17,968::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:773e16f1-14be-419e-af5f-d4562ba415ca, volUUID:24065970-25e0-4942-8397-5b3387c75e5b
MainThread::INFO::2017-12-20 12:40:18,041::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 12:40:18,041::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 12:40:18,049::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 12:40:18,053::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 12:40:18,054::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 12:40:18,054::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 12:40:18,056::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 52150864
MainThread::INFO::2017-12-20 12:40:18,056::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 12:40:18,058::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 52105808
MainThread::INFO::2017-12-20 12:40:18,059::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 12:40:18,060::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 52105872
MainThread::INFO::2017-12-20 12:40:18,060::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:40:18,062::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 52251088
MainThread::INFO::2017-12-20 12:40:18,062::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:40:18,063::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 52251856
MainThread::INFO::2017-12-20 12:40:22,335::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 52252432
MainThread::INFO::2017-12-20 12:40:22,335::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 12:40:22,471::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 12:42:43,500::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:45:09,535::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:47:35,569::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:50:01,595::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:52:27,631::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 12:52:32,647::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 12:52:32,648::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 12:52:37,653::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0'
MainThread::INFO::2017-12-20 12:52:37,720::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 12:52:37,721::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 12:52:40,170::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 12:52:40,171::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 12:52:42,488::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:52:47,158::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:52:47,174::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 12:52:47,441::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 12:52:47,441::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 12:52:50,033::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 12:52:50,033::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 12:52:50,033::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 12:52:50,033::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 12:52:50,034::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 12:52:50,042::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 12:52:50,046::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 12:52:50,046::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 12:52:50,047::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 12:52:50,052::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140271824914192
MainThread::INFO::2017-12-20 12:52:50,052::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 12:52:50,054::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140271824915536
MainThread::INFO::2017-12-20 12:52:50,054::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 12:52:50,055::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140271824915408
MainThread::INFO::2017-12-20 12:52:50,055::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:52:50,056::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140271824914576
MainThread::INFO::2017-12-20 12:52:50,056::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:52:50,058::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140271824913168
MainThread::INFO::2017-12-20 12:52:54,376::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140271824912976
MainThread::INFO::2017-12-20 12:52:54,376::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 12:52:54,508::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 12:55:15,527::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:55:21,536::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:55:27,542::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:55:33,548::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 12:55:39,554::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 12:55:44,560::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 12:55:44,560::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 12:55:49,565::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'
MainThread::INFO::2017-12-20 12:55:49,632::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 12:55:49,634::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 12:55:52,101::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 12:55:52,102::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 12:55:54,438::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 12:55:54,440::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:55:56,835::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:55:57,151::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 12:55:57,418::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 12:55:57,418::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 12:57:26,748::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.8 started
MainThread::INFO::2017-12-20 12:57:26,817::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 12:57:26,818::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 12:57:29,323::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 12:57:29,324::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 12:57:31,680::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:57:36,364::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 12:57:36,373::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 12:57:36,640::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 12:57:36,640::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 12:57:39,251::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 12:57:39,252::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 12:57:39,252::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 12:57:41,639::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:0401aa22-a467-4a31-b4cf-27c4e395cfd3, volUUID:8c0534b8-3647-4c75-9da5-88828d8d2faf
MainThread::INFO::2017-12-20 12:57:41,960::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:773e16f1-14be-419e-af5f-d4562ba415ca, volUUID:24065970-25e0-4942-8397-5b3387c75e5b
MainThread::INFO::2017-12-20 12:57:42,065::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 12:57:42,066::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 12:57:42,073::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 12:57:42,078::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 12:57:42,078::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 12:57:42,079::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 12:57:42,081::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 51720336
MainThread::INFO::2017-12-20 12:57:42,081::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 12:57:42,083::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 51472720
MainThread::INFO::2017-12-20 12:57:42,083::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 12:57:42,084::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 51472784
MainThread::INFO::2017-12-20 12:57:42,084::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:57:42,085::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 51766544
MainThread::INFO::2017-12-20 12:57:42,086::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 12:57:42,087::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 51767376
MainThread::INFO::2017-12-20 12:57:46,471::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 51767952
MainThread::INFO::2017-12-20 12:57:46,472::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 12:57:46,604::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 13:00:07,625::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:02:33,651::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:04:59,674::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:22:06,724::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.8 started
MainThread::INFO::2017-12-20 13:22:06,825::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 13:22:06,827::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 13:22:09,823::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 13:22:09,824::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 13:22:12,151::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 13:22:12,153::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:22:14,647::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:22:15,716::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 13:22:15,983::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 13:22:15,984::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 13:22:18,576::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 13:22:18,576::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 13:22:18,576::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 13:22:21,004::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:0401aa22-a467-4a31-b4cf-27c4e395cfd3, volUUID:8c0534b8-3647-4c75-9da5-88828d8d2faf
MainThread::INFO::2017-12-20 13:22:21,196::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:773e16f1-14be-419e-af5f-d4562ba415ca, volUUID:24065970-25e0-4942-8397-5b3387c75e5b
MainThread::INFO::2017-12-20 13:22:21,270::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 13:22:21,270::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 13:22:21,434::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 13:22:21,452::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 13:22:21,452::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 13:22:21,453::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 13:22:21,457::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 33456784
MainThread::INFO::2017-12-20 13:22:21,457::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 13:22:21,459::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 33411536
MainThread::INFO::2017-12-20 13:22:21,459::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 13:22:21,460::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 33411728
MainThread::INFO::2017-12-20 13:22:21,460::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:22:21,462::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 33556880
MainThread::INFO::2017-12-20 13:22:21,462::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:22:21,463::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 33557648
MainThread::INFO::2017-12-20 13:22:26,362::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 33558288
MainThread::INFO::2017-12-20 13:22:26,362::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 13:22:26,498::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 13:24:47,526::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:27:13,558::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:29:39,584::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:32:05,614::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:34:31,087::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 13:34:36,124::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 13:34:36,125::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 13:34:41,130::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0'
MainThread::ERROR::2017-12-20 13:34:41,131::config::163::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_load_single_conf_file) Configuration file '/etc/ovirt-hosted-engine/hosted-engine.conf' not available [[Errno 2] No such file or directory: '/etc/ovirt-hosted-engine/hosted-engine.conf']
MainThread::INFO::2017-12-20 13:34:41,215::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::ERROR::2017-12-20 13:34:41,215::hosted_engine::397::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring) Hosted Engine is not configured. Shutting down.
MainThread::INFO::2017-12-20 13:34:41,216::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(run) Agent shutting down
MainThread::INFO::2017-12-20 13:38:48,789::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.8 started
MainThread::INFO::2017-12-20 13:38:48,854::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 13:38:48,855::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 13:38:51,344::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 13:38:51,346::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 13:38:53,686::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:38:58,371::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:38:58,379::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 13:38:58,646::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 13:38:58,646::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 13:39:01,256::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 13:39:01,257::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 13:39:01,257::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 13:39:03,605::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:0401aa22-a467-4a31-b4cf-27c4e395cfd3, volUUID:8c0534b8-3647-4c75-9da5-88828d8d2faf
MainThread::INFO::2017-12-20 13:39:03,796::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:773e16f1-14be-419e-af5f-d4562ba415ca, volUUID:24065970-25e0-4942-8397-5b3387c75e5b
MainThread::INFO::2017-12-20 13:39:03,870::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 13:39:03,870::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 13:39:03,879::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 13:39:03,883::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 13:39:03,884::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 13:39:03,884::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 13:39:03,886::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50811152
MainThread::INFO::2017-12-20 13:39:03,886::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 13:39:03,888::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50766480
MainThread::INFO::2017-12-20 13:39:03,889::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 13:39:03,890::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50767248
MainThread::INFO::2017-12-20 13:39:03,890::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:39:03,891::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50911632
MainThread::INFO::2017-12-20 13:39:03,891::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:39:03,892::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50912336
MainThread::INFO::2017-12-20 13:39:08,043::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 50913040
MainThread::INFO::2017-12-20 13:39:08,043::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 13:39:08,200::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 13:41:29,225::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:43:55,256::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:46:21,288::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:48:47,316::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:51:13,348::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 13:51:18,354::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 13:51:18,355::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 13:51:23,360::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0'
MainThread::INFO::2017-12-20 13:51:23,427::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 13:51:23,428::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 13:51:25,899::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 13:51:25,900::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 13:51:28,211::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 13:51:28,213::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:51:30,741::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 13:51:31,458::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 13:51:31,725::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 13:51:31,726::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 13:51:34,307::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 13:51:34,307::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 13:51:34,307::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 13:51:34,308::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 13:51:34,308::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 13:51:34,320::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 13:51:34,325::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 13:51:34,325::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 13:51:34,325::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 13:51:34,328::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597029958288
MainThread::INFO::2017-12-20 13:51:34,328::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 13:51:34,333::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597029958992
MainThread::INFO::2017-12-20 13:51:34,334::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 13:51:34,335::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597029958800
MainThread::INFO::2017-12-20 13:51:34,335::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:51:34,336::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597029960656
MainThread::INFO::2017-12-20 13:51:34,336::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 13:51:34,338::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597029961424
MainThread::INFO::2017-12-20 13:51:39,432::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140597029959504
MainThread::INFO::2017-12-20 13:51:39,432::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 13:51:39,567::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 13:54:00,591::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:56:26,620::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 13:58:52,648::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:01:18,672::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:03:44,702::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 14:03:49,708::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 14:03:49,709::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 14:03:54,714::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'
MainThread::INFO::2017-12-20 14:03:54,781::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:03:54,782::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:03:57,264::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:03:57,264::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 14:03:59,591::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 14:03:59,592::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:04:04,280::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:04:04,288::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:04:04,554::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:04:04,555::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:04:07,128::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:04:07,128::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:04:07,129::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:04:07,129::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:04:07,129::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:04:07,136::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:04:07,140::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:04:07,141::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:04:07,141::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:04:07,161::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596896469392
MainThread::INFO::2017-12-20 14:04:07,161::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:04:07,218::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 50766672
MainThread::INFO::2017-12-20 14:04:07,218::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:04:07,273::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596432917904
MainThread::INFO::2017-12-20 14:04:07,273::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:04:07,488::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437440464
MainThread::INFO::2017-12-20 14:04:07,489::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:04:07,564::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597030999376
MainThread::INFO::2017-12-20 14:04:11,513::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140597232271056
MainThread::INFO::2017-12-20 14:04:11,513::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:04:11,639::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 14:06:32,663::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:08:58,692::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:11:24,721::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:13:50,752::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:16:16,781::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 14:16:21,787::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 14:16:21,788::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 14:16:26,793::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '2'
MainThread::INFO::2017-12-20 14:16:26,873::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:16:26,875::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:16:29,343::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:16:29,344::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 14:16:31,678::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 14:16:31,680::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:16:36,350::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:16:36,358::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:16:36,625::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:16:36,626::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:16:39,206::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:16:39,207::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:16:39,207::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:16:39,207::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:16:39,207::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:16:39,214::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:16:39,219::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:16:39,219::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:16:39,219::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:16:39,222::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437376080
MainThread::INFO::2017-12-20 14:16:39,222::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:16:39,228::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437373328
MainThread::INFO::2017-12-20 14:16:39,229::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:16:39,230::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437373200
MainThread::INFO::2017-12-20 14:16:39,230::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:16:39,231::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437376016
MainThread::INFO::2017-12-20 14:16:39,231::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:16:39,232::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437374096
MainThread::INFO::2017-12-20 14:16:43,680::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140596437374800
MainThread::INFO::2017-12-20 14:16:43,680::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:16:43,809::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 14:19:04,835::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:21:30,866::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:23:56,898::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:26:22,930::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:28:48,960::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 14:28:53,965::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 14:28:53,965::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 14:28:58,971::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '3'
MainThread::INFO::2017-12-20 14:28:59,038::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:28:59,039::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:29:01,515::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:29:01,516::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::WARNING::2017-12-20 14:29:03,858::storage_server::232::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Hosted-engine storage domain is in invalid state
MainThread::INFO::2017-12-20 14:29:03,859::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:29:08,590::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:29:08,598::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:29:08,966::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:29:08,966::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:29:11,523::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:29:11,524::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:29:11,524::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:29:11,525::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:29:11,525::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:29:11,533::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:29:11,538::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:29:11,538::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:29:11,539::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:29:11,541::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437313680
MainThread::INFO::2017-12-20 14:29:11,541::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:29:11,548::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437314128
MainThread::INFO::2017-12-20 14:29:11,549::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:29:11,550::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437314000
MainThread::INFO::2017-12-20 14:29:11,550::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:29:11,551::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437313296
MainThread::INFO::2017-12-20 14:29:11,551::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:29:11,552::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596437311568
MainThread::INFO::2017-12-20 14:29:16,746::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140596437313424
MainThread::INFO::2017-12-20 14:29:16,746::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:29:16,877::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 14:31:37,903::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:34:03,936::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:34:04,964::agent::79::ovirt_hosted_engine_ha.agent.agent.Agent::(run) ovirt-hosted-engine-ha agent 2.1.8 started
MainThread::INFO::2017-12-20 14:34:05,034::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:34:05,035::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:34:07,495::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:34:07,496::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 14:34:09,825::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:34:14,532::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:34:14,564::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:34:14,881::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:34:14,882::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:34:17,489::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:34:17,489::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:34:17,489::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:34:19,877::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:0401aa22-a467-4a31-b4cf-27c4e395cfd3, volUUID:8c0534b8-3647-4c75-9da5-88828d8d2faf
MainThread::INFO::2017-12-20 14:34:20,069::ovf_store::109::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:773e16f1-14be-419e-af5f-d4562ba415ca, volUUID:24065970-25e0-4942-8397-5b3387c75e5b
MainThread::INFO::2017-12-20 14:34:20,142::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:34:20,142::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:34:20,150::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:34:20,154::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:34:20,155::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:34:20,155::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:34:20,157::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596834419408
MainThread::INFO::2017-12-20 14:34:20,157::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:34:20,165::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596834420112
MainThread::INFO::2017-12-20 14:34:20,165::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:34:20,167::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596834419600
MainThread::INFO::2017-12-20 14:34:20,167::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:34:20,168::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596834421328
MainThread::INFO::2017-12-20 14:34:20,168::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:34:20,169::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596834422544
MainThread::INFO::2017-12-20 14:34:24,729::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140596834422672
MainThread::INFO::2017-12-20 14:34:24,729::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:34:24,860::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 14:36:45,886::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:39:11,918::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:41:37,951::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:44:03,983::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:46:30,013::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 14:46:35,020::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 14:46:35,020::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 14:46:40,027::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '0'
MainThread::INFO::2017-12-20 14:46:40,093::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:46:40,094::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:46:42,558::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:46:42,559::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 14:46:44,876::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:46:49,553::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:46:49,585::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:46:49,902::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:46:49,903::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:46:52,482::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:46:52,482::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:46:52,483::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:46:52,483::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:46:52,483::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:46:52,491::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:46:52,495::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:46:52,496::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:46:52,496::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:46:52,602::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596031602000
MainThread::INFO::2017-12-20 14:46:52,602::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:46:52,612::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596031602384
MainThread::INFO::2017-12-20 14:46:52,612::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:46:52,614::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596031601488
MainThread::INFO::2017-12-20 14:46:52,614::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:46:52,616::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597685237776
MainThread::INFO::2017-12-20 14:46:52,617::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:46:52,648::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140597036501712
MainThread::INFO::2017-12-20 14:46:57,073::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140595958659536
MainThread::INFO::2017-12-20 14:46:57,074::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:46:57,202::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 14:49:18,227::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:51:44,258::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:54:10,289::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:56:36,320::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 14:59:02,350::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::ERROR::2017-12-20 14:59:07,356::agent::205::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 191, in _run_agent
    return action(he)
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 64, in action_proper
    return he.start_monitoring()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 411, in start_monitoring
    self._initialize_sanlock()
  File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 749, in _initialize_sanlock
    "Failed to initialize sanlock, the number of errors has"
SanlockInitializationError: Failed to initialize sanlock, the number of errors has exceeded the limit

MainThread::ERROR::2017-12-20 14:59:07,357::agent::206::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Trying to restart agent
MainThread::WARNING::2017-12-20 14:59:12,362::agent::209::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Restarting agent, attempt '1'
MainThread::INFO::2017-12-20 14:59:12,430::hosted_engine::242::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) Found certificate common name: node-05.eurooil.ru
MainThread::INFO::2017-12-20 14:59:12,431::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM
MainThread::INFO::2017-12-20 14:59:14,908::hosted_engine::630::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Connecting the storage
MainThread::INFO::2017-12-20 14:59:14,908::storage_server::220::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(validate_storage_server) Validating storage server
MainThread::INFO::2017-12-20 14:59:17,254::storage_server::239::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:59:21,921::storage_server::246::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Connecting storage server
MainThread::INFO::2017-12-20 14:59:21,954::storage_server::253::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server) Refreshing the storage domain
MainThread::INFO::2017-12-20 14:59:22,271::hosted_engine::663::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Preparing images
MainThread::INFO::2017-12-20 14:59:22,272::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images) Preparing images
MainThread::INFO::2017-12-20 14:59:24,921::hosted_engine::666::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images) Refreshing vm.conf
MainThread::INFO::2017-12-20 14:59:24,922::config::493::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_vm_conf) Reloading vm.conf from the shared storage domain
MainThread::INFO::2017-12-20 14:59:24,922::config::416::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Trying to get a fresher copy of vm configuration from the OVF_STORE
MainThread::INFO::2017-12-20 14:59:24,922::ovf_store::132::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE
MainThread::INFO::2017-12-20 14:59:24,922::ovf_store::134::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/773e16f1-14be-419e-af5f-d4562ba415ca/24065970-25e0-4942-8397-5b3387c75e5b 
MainThread::INFO::2017-12-20 14:59:24,931::config::435::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert
MainThread::INFO::2017-12-20 14:59:24,935::config::440::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE
MainThread::INFO::2017-12-20 14:59:24,935::hosted_engine::509::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Initializing ha-broker connection
MainThread::INFO::2017-12-20 14:59:24,936::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor ping, options {'addr': '192.168.3.9'}
MainThread::INFO::2017-12-20 14:59:25,060::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140595958657872
MainThread::INFO::2017-12-20 14:59:25,061::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mgmt-bridge, options {'use_ssl': 'true', 'bridge_name': 'ovirtmgmt', 'address': '0'}
MainThread::INFO::2017-12-20 14:59:25,067::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140596837069328
MainThread::INFO::2017-12-20 14:59:25,067::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor mem-free, options {'use_ssl': 'true', 'address': '0'}
MainThread::INFO::2017-12-20 14:59:25,068::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140595958657424
MainThread::INFO::2017-12-20 14:59:25,068::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor cpu-load-no-engine, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:59:25,069::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140595958657552
MainThread::INFO::2017-12-20 14:59:25,069::brokerlink::130::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Starting monitor engine-health, options {'use_ssl': 'true', 'vm_uuid': '3ab8b1c6-3f75-4fb8-85cb-58ce247b2c16', 'address': '0'}
MainThread::INFO::2017-12-20 14:59:25,071::brokerlink::141::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(start_monitor) Success, id 140595958660048
MainThread::INFO::2017-12-20 14:59:29,644::brokerlink::179::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(set_storage_domain) Success, id 140595958657360
MainThread::INFO::2017-12-20 14:59:29,644::hosted_engine::601::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Broker initialized, all submonitors started
MainThread::INFO::2017-12-20 14:59:29,812::hosted_engine::704::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Ensuring lease for lockspace hosted-engine, host id 1 is acquired (file: /var/run/vdsm/storage/f0179535-b218-4d15-8324-57bb87300e59/1a778d60-51ae-44f7-9bd4-8261dd05cfab/23448296-5432-4449-bcb3-b3df00c286b4)
MainThread::INFO::2017-12-20 15:01:50,838::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 15:04:16,868::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt
MainThread::INFO::2017-12-20 15:06:42,899::hosted_engine::745::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_sanlock) Failed to acquire the lock. Waiting '5's before the next attempt