Continuing with this message. I have found that the NFS server stops responding. What logs from the NFS server (Linux Centos 7.6 with NFS exporting v4.2) do i need? Aug 30 10:42:01 hvm002 libvirtd: 2019-08-30 15:42:01.239+0000: 20196: warning : qemuDomainObjBeginJobInternal:6724 : Cannot start job (modify, none, none) for domain CentosPrueba; current job is (query, none, none) owned by (20200 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (4072s, 0s, 0s) Aug 30 10:42:01 hvm002 libvirtd: 2019-08-30 15:42:01.239+0000: 20196: error : qemuDomainObjBeginJobInternal:6746 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) Aug 30 10:42:01 hvm002 vdsm[25926]: ERROR Error running VM callback#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 663, in dispatchLibvirtEvents#012 v.onIOError(devAlias, reason, action)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5074, in onIOError#012 self._update_metadata()#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5129, in _update_metadata#012 self._sync_metadata()#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5142, in _sync_metadata#012 self._md_desc.dump(self._dom)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 515, in dump#012 0)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 108, in f#012 raise toe#012TimeoutError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) Aug 30 10:46:23 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:46:23 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:48:32 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:48:32 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:50:01 hvm002 systemd: Created slice User Slice of root. Aug 30 10:50:01 hvm002 systemd: Started Session 719 of user root. Aug 30 10:50:01 hvm002 systemd: Removed slice User Slice of root. Aug 30 10:53:24 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:53:24 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:55:09 hvm002 systemd-logind: New session 720 of user root. Aug 30 10:55:09 hvm002 systemd: Created slice User Slice of root. Aug 30 10:55:09 hvm002 systemd: Started Session 720 of user root. Aug 30 10:55:33 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:55:33 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 11:00:01 hvm002 systemd: Started Session 721 of user root. Aug 30 11:01:01 hvm002 systemd: Started Session 722 of user root. [root@hvm002 ~]# --------------------- Erick Perez Soluciones Tacticas Pasivas/Activas de Inteligencia y Analitica de Datos para Gobiernos Quadrian Enterprises S.A. - Panama, Republica de Panama Skype chat: eaperezh WhatsApp IM: +507-6675-5083 --------------------- On Mon, Aug 26, 2019 at 4:44 AM Erick Perez - Quadrian Enterprises <eperez@quadrianweb.com> wrote:

Hi, I was just creating a VM on a freshly installed 2-node selfhosted ovirt setup with a NFS backend. NFS connections are v4.2 for VM data domain

What other log is needed?

Thanks, in advance.

Here is the output of the last 300 lines of /var/log/messages [root@hvm002 ~]# [root@hvm002 ~]# tail /var/log/messages -n 300 Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:34:56 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:06 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:06 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:06 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:06 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:07 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:15 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:15 hvm002 systemd: Starting Cockpit Web Service... Aug 26 04:35:15 hvm002 systemd: Started Cockpit Web Service. Aug 26 04:35:15 hvm002 cockpit-ws: Using certificate: /etc/cockpit/ws-certs.d/0-self-signed.cert Aug 26 04:35:15 hvm002 cockpit-ws: couldn't read from connection: Peer sent fatal TLS alert: Certificate is bad Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:17 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:17 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:17 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:17 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:27 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:27 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:27 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:27 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:28 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/5 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> timeout=60, duration=60.00 at 0x7f091c0aadd0> task#=4 at 0x7f091c0c6e10>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:35:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:38 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:38 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:38 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:38 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:48 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:48 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:48 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:48 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:49 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:35:52 hvm002 systemd: Created slice User Slice of root. Aug 26 04:35:52 hvm002 systemd-logind: New session 2 of user root. Aug 26 04:35:52 hvm002 systemd: Started Session 2 of user root. Aug 26 04:35:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:35:59 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:35:59 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:59 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:35:59 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:09 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:09 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:09 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:09 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:10 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:20 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:20 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:20 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:20 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:30 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:30 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:30 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:30 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:31 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/5 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> timeout=60, duration=120.00 at 0x7f091c0aadd0> task#=4 at 0x7f091c0c6e10>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:36:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:36:41 hvm002 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 26 04:36:41 hvm002 abrt-server: Duplicate: core backtrace Aug 26 04:36:41 hvm002 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 Aug 26 04:36:41 hvm002 abrt-server: Deleting problem directory Python-2019-08-26-04:36:41-16985 (dup of Python-2019-08-26-04:32:28-3432) Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:41 hvm002 dbus[3444]: [system] Activating service name='org.freedesktop.problems' (using servicehelper) Aug 26 04:36:41 hvm002 dbus[3444]: [system] Successfully activated service 'org.freedesktop.problems' Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-broker.service entered failed state. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service failed. Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:36:42 hvm002 systemd: Started Session c15 of user root. Aug 26 04:36:42 hvm002 systemd: Started Session c16 of user root. Aug 26 04:36:51 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:36:51 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:36:51 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:51 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:36:52 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:36:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:02 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:02 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:02 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:02 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:12 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:12 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:12 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:12 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:13 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:23 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:23 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:23 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:23 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:33 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:33 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:33 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:33 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:34 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> timeout=60, duration=60.00 at 0x7f08f8259750> task#=8 at 0x7f091c110c50>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:44 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:44 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:44 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:44 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:37:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:37:54 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:37:54 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:37:54 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:54 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:37:55 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:05 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:05 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:05 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:05 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:15 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:15 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:15 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:15 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:16 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:26 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:26 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:26 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:26 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:36 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:36 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:36 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:36 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:37 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker name=jsonrpc/3 running <Task <JsonRpcTask {'params': {u'connectionParams': [{u'protocol_version': u'auto', u'connection': u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': '2.0', 'method': u'StoragePool.connectStorageServer', 'id': u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> timeout=60, duration=120.00 at 0x7f08f8259750> task#=8 at 0x7f091c110c50>, traceback:#012File: "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner#012 self.run()#012File: "/usr/lib64/python2.7/threading.py", line 765, in run#012 self.__target(*self.__args, **self.__kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 195, in run#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run#012 self._execute_task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in _execute_task#012 task()#012File: "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__#012 self._callable()#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in __call__#012 self._handler(self._ctx, self._req)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in _serveRequest#012 response = self._handle_request(req, ctx)#012File: "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in _handle_request#012 res = method(**params)#012File: "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in _dynamicMethod#012 result = fn(*methodArgs)#012File: "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in connectStorageServer#012 connectionParams)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line 74, in wrapper#012 result = ctask.prepare(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in wrapper#012 return m(self, *a, **kw)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in prepare#012 result = self._run(func, *args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in connectStorageServer#012File: "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in method#012 ret = func(*args, **kwargs)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in connectStorageServer#012 conObj.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 431, in connect#012 return self._mountCon.connect()#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 172, in connect#012 self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP)#012File: "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount#012 cgroup=cgroup)#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, in __call__#012 return callMethod()#012File: "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, in <lambda>#012 **kwargs)#012File: "<string>", line 2, in mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", line 759, in _callmethod#012 kind, result = conn.recv() Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:47 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:47 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:47 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:47 hvm002 python: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Aug 26 04:38:47 hvm002 abrt-server: Duplicate: core backtrace Aug 26 04:38:47 hvm002 abrt-server: DUP_OF_DIR: /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 Aug 26 04:38:47 hvm002 abrt-server: Deleting problem directory Python-2019-08-26-04:38:47-17322 (dup of Python-2019-08-26-04:32:28-3432) Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service: main process exited, code=exited, status=1/FAILURE Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-broker.service entered failed state. Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service failed. Aug 26 04:38:48 hvm002 systemd: ovirt-ha-broker.service holdoff time over, scheduling restart. Aug 26 04:38:48 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:48 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:38:48 hvm002 systemd: Started oVirt Hosted Engine High Availability Communications Broker. Aug 26 04:38:48 hvm002 systemd: Started Session c17 of user root. Aug 26 04:38:48 hvm002 systemd: Started Session c18 of user root. Aug 26 04:38:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:38:57 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:38:57 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:38:57 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:57 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:38:58 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service failed. Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service holdoff time over, scheduling restart. Aug 26 04:39:08 hvm002 systemd: Cannot add dependency job for unit lvm2-lvmetad.socket, ignoring: Unit is masked. Aug 26 04:39:08 hvm002 systemd: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:39:08 hvm002 systemd: Started oVirt Hosted Engine High Availability Monitoring Agent. Aug 26 04:39:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted Engine HA score '[Errno 2] No such file or directory'Is the Hosted Engine setup finished? Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 432, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 556, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 86, in start_monitor#012 ).format(t=type, o=options, e=e)#012RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'tcp_t_address': '', 'network_test': 'dns', 'tcp_t_port': '', 'addr': '172.21.48.1'}] Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service: main process exited, code=exited, status=157/n/a Aug 26 04:39:08 hvm002 systemd: Unit ovirt-ha-agent.service entered failed state. Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service failed. [root@hvm002 ~]#

--------------------- Erick Perez ---------------------