Continuing with this message. I have found that the NFS server stops responding. What logs from the NFS server (Linux Centos 7.6 with NFS exporting v4.2) do i need? Aug 30 10:42:01 hvm002 libvirtd: 2019-08-30 15:42:01.239+0000: 20196: warning : qemuDomainObjBeginJobInternal:6724 : Cannot start job (modify, none, none) for domain CentosPrueba; current job is (query, none, none) owned by (20200 remoteDispatchConnectGetAllDomainStats, 0 <null>, 0 <null> (flags=0x0)) for (4072s, 0s, 0s) Aug 30 10:42:01 hvm002 libvirtd: 2019-08-30 15:42:01.239+0000: 20196: error : qemuDomainObjBeginJobInternal:6746 : Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) Aug 30 10:42:01 hvm002 vdsm[25926]: ERROR Error running VM callback#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 663, in dispatchLibvirtEvents#012 v.onIOError(devAlias, reason, action)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5074, in onIOError#012 self._update_metadata()#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5129, in _update_metadata#012 self._sync_metadata()#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5142, in _sync_metadata#012 self._md_desc.dump(self._dom)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/metadata.py", line 515, in dump#012 0)#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 108, in f#012 raise toe#012TimeoutError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchConnectGetAllDomainStats) Aug 30 10:46:23 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:46:23 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:48:32 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:48:32 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:50:01 hvm002 systemd: Created slice User Slice of root. Aug 30 10:50:01 hvm002 systemd: Started Session 719 of user root. Aug 30 10:50:01 hvm002 systemd: Removed slice User Slice of root. Aug 30 10:53:24 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:53:24 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:55:09 hvm002 systemd-logind: New session 720 of user root. Aug 30 10:55:09 hvm002 systemd: Created slice User Slice of root. Aug 30 10:55:09 hvm002 systemd: Started Session 720 of user root. Aug 30 10:55:33 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:55:33 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 10:57:36 hvm002 kernel: nfs: server 10.10.10.2 not responding, timed out Aug 30 11:00:01 hvm002 systemd: Started Session 721 of user root. Aug 30 11:01:01 hvm002 systemd: Started Session 722 of user root. [root@hvm002 ~]# --------------------- Erick Perez Soluciones Tacticas Pasivas/Activas de Inteligencia y Analitica de Datos para Gobiernos Quadrian Enterprises S.A. - Panama, Republica de Panama Skype chat: eaperezh WhatsApp IM: +507-6675-5083 --------------------- On Mon, Aug 26, 2019 at 4:44 AM Erick Perez - Quadrian Enterprises <eperez(a)quadrianweb.com&gt; wrote: > > Hi, > I was just creating a VM on a freshly installed 2-node selfhosted > ovirt setup with a NFS backend. > NFS connections are v4.2 for VM data domain > > What other log is needed? > > Thanks, in advance. > > > Here is the output of the last 300 lines of /var/log/messages > [root@hvm002 ~]# > [root@hvm002 ~]# tail /var/log/messages -n 300 > Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:34:56 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:06 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:06 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:06 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:06 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:07 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:35:15 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:15 hvm002 systemd: Starting Cockpit Web Service... > Aug 26 04:35:15 hvm002 systemd: Started Cockpit Web Service. > Aug 26 04:35:15 hvm002 cockpit-ws: Using certificate: > /etc/cockpit/ws-certs.d/0-self-signed.cert > Aug 26 04:35:15 hvm002 cockpit-ws: couldn't read from connection: Peer > sent fatal TLS alert: Certificate is bad > Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:17 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:17 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:17 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:17 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:35:27 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:27 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:27 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:27 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:28 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker > name=jsonrpc/5 running <Task <JsonRpcTask {'params': > {u'connectionParams': [{u'protocol_version': u'auto', u'connection': > u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': > u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': > u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': > '2.0', 'method': u'StoragePool.connectStorageServer', 'id': > u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> > timeout=60, duration=60.00 at 0x7f091c0aadd0> task#=4 at > 0x7f091c0c6e10>, traceback:#012File: > "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 > self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", > line 812, in __bootstrap_inner#012 self.run()#012File: > "/usr/lib64/python2.7/threading.py", line 765, in run#012 > self.__target(*self.__args, **self.__kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line > 195, in run#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in > _run#012 self._execute_task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in > _execute_task#012 task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in > __call__#012 self._callable()#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in > __call__#012 self._handler(self._ctx, self._req)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in > _serveRequest#012 response = self._handle_request(req, ctx)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in > _handle_request#012 res = method(**params)#012File: > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in > _dynamicMethod#012 result = fn(*methodArgs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in > connectStorageServer#012 connectionParams)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line > 74, in wrapper#012 result = ctask.prepare(func, *args, > **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in > wrapper#012 return m(self, *a, **kw)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in > prepare#012 result = self._run(func, *args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in > connectStorageServer#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in > method#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in > connectStorageServer#012 conObj.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 431, in connect#012 return self._mountCon.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 172, in connect#012 self._mount.mount(self.options, self._vfsType, > cgroup=self.CGROUP)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in > mount#012 cgroup=cgroup)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, > in __call__#012 return callMethod()#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, > in <lambda>#012 **kwargs)#012File: "<string>", line 2, in > mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", > line 759, in _callmethod#012 kind, result = conn.recv() > Aug 26 04:35:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:38 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:38 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:38 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:38 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:48 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:48 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:48 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:48 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:49 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:35:52 hvm002 systemd: Created slice User Slice of root. > Aug 26 04:35:52 hvm002 systemd-logind: New session 2 of user root. > Aug 26 04:35:52 hvm002 systemd: Started Session 2 of user root. > Aug 26 04:35:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:35:59 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:35:59 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:59 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:35:59 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:36:09 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:36:09 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:09 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:09 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:36:10 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:36:20 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:20 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:20 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:36:20 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:36:30 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:36:30 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:30 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:30 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:36:31 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker > name=jsonrpc/5 running <Task <JsonRpcTask {'params': > {u'connectionParams': [{u'protocol_version': u'auto', u'connection': > u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': > u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': > u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': > '2.0', 'method': u'StoragePool.connectStorageServer', 'id': > u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0> > timeout=60, duration=120.00 at 0x7f091c0aadd0> task#=4 at > 0x7f091c0c6e10>, traceback:#012File: > "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 > self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", > line 812, in __bootstrap_inner#012 self.run()#012File: > "/usr/lib64/python2.7/threading.py", line 765, in run#012 > self.__target(*self.__args, **self.__kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line > 195, in run#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in > _run#012 self._execute_task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in > _execute_task#012 task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in > __call__#012 self._callable()#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in > __call__#012 self._handler(self._ctx, self._req)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in > _serveRequest#012 response = self._handle_request(req, ctx)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in > _handle_request#012 res = method(**params)#012File: > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in > _dynamicMethod#012 result = fn(*methodArgs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in > connectStorageServer#012 connectionParams)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line > 74, in wrapper#012 result = ctask.prepare(func, *args, > **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in > wrapper#012 return m(self, *a, **kw)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in > prepare#012 result = self._run(func, *args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in > connectStorageServer#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in > method#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in > connectStorageServer#012 conObj.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 431, in connect#012 return self._mountCon.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 172, in connect#012 self._mount.mount(self.options, self._vfsType, > cgroup=self.CGROUP)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in > mount#012 cgroup=cgroup)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, > in __call__#012 return callMethod()#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, > in <lambda>#012 **kwargs)#012File: "<string>", line 2, in > mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", > line 759, in _callmethod#012 kind, result = conn.recv() > Aug 26 04:36:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:36:41 hvm002 python: detected unhandled Python exception in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Aug 26 04:36:41 hvm002 abrt-server: Duplicate: core backtrace > Aug 26 04:36:41 hvm002 abrt-server: DUP_OF_DIR: > /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 > Aug 26 04:36:41 hvm002 abrt-server: Deleting problem directory > Python-2019-08-26-04:36:41-16985 (dup of > Python-2019-08-26-04:32:28-3432) > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:41 hvm002 dbus[3444]: [system] Activating service > name='org.freedesktop.problems' (using servicehelper) > Aug 26 04:36:41 hvm002 dbus[3444]: [system] Successfully activated > service 'org.freedesktop.problems' > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service: main process > exited, code=exited, status=1/FAILURE > Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-broker.service entered > failed state. > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service failed. > Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service holdoff time > over, scheduling restart. > Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Communications Broker. > Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High > Availability Communications Broker. > Aug 26 04:36:42 hvm002 systemd: Started Session c15 of user root. > Aug 26 04:36:42 hvm002 systemd: Started Session c16 of user root. > Aug 26 04:36:51 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:36:51 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:36:51 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:51 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:36:52 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:36:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:02 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:02 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:02 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:02 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:37:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:37:12 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:12 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:12 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:12 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:13 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:23 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:23 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:23 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:23 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:37:33 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:33 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:33 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:33 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:34 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:37:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:37:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker > name=jsonrpc/3 running <Task <JsonRpcTask {'params': > {u'connectionParams': [{u'protocol_version': u'auto', u'connection': > u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': > u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': > u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': > '2.0', 'method': u'StoragePool.connectStorageServer', 'id': > u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> > timeout=60, duration=60.00 at 0x7f08f8259750> task#=8 at > 0x7f091c110c50>, traceback:#012File: > "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 > self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", > line 812, in __bootstrap_inner#012 self.run()#012File: > "/usr/lib64/python2.7/threading.py", line 765, in run#012 > self.__target(*self.__args, **self.__kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line > 195, in run#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in > _run#012 self._execute_task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in > _execute_task#012 task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in > __call__#012 self._callable()#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in > __call__#012 self._handler(self._ctx, self._req)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in > _serveRequest#012 response = self._handle_request(req, ctx)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in > _handle_request#012 res = method(**params)#012File: > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in > _dynamicMethod#012 result = fn(*methodArgs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in > connectStorageServer#012 connectionParams)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line > 74, in wrapper#012 result = ctask.prepare(func, *args, > **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in > wrapper#012 return m(self, *a, **kw)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in > prepare#012 result = self._run(func, *args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in > connectStorageServer#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in > method#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in > connectStorageServer#012 conObj.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 431, in connect#012 return self._mountCon.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 172, in connect#012 self._mount.mount(self.options, self._vfsType, > cgroup=self.CGROUP)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in > mount#012 cgroup=cgroup)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, > in __call__#012 return callMethod()#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, > in <lambda>#012 **kwargs)#012File: "<string>", line 2, in > mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", > line 759, in _callmethod#012 kind, result = conn.recv() > Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:44 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:44 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:44 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:44 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:37:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:37:54 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:37:54 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:37:54 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:54 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:37:55 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:05 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:05 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:05 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:05 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:38:15 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:15 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:15 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:15 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:16 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:26 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:26 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:26 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:26 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:36 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:36 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:36 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:36 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:37 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:38:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker > name=jsonrpc/3 running <Task <JsonRpcTask {'params': > {u'connectionParams': [{u'protocol_version': u'auto', u'connection': > u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id': > u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID': > u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc': > '2.0', 'method': u'StoragePool.connectStorageServer', 'id': > u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650> > timeout=60, duration=120.00 at 0x7f08f8259750> task#=8 at > 0x7f091c110c50>, traceback:#012File: > "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012 > self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py", > line 812, in __bootstrap_inner#012 self.run()#012File: > "/usr/lib64/python2.7/threading.py", line 765, in run#012 > self.__target(*self.__args, **self.__kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line > 195, in run#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in > _run#012 self._execute_task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in > _execute_task#012 task()#012File: > "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in > __call__#012 self._callable()#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in > __call__#012 self._handler(self._ctx, self._req)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in > _serveRequest#012 response = self._handle_request(req, ctx)#012File: > "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in > _handle_request#012 res = method(**params)#012File: > "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in > _dynamicMethod#012 result = fn(*methodArgs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in > connectStorageServer#012 connectionParams)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line > 74, in wrapper#012 result = ctask.prepare(func, *args, > **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in > wrapper#012 return m(self, *a, **kw)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in > prepare#012 result = self._run(func, *args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in > _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in > connectStorageServer#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in > method#012 ret = func(*args, **kwargs)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in > connectStorageServer#012 conObj.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 431, in connect#012 return self._mountCon.connect()#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line > 172, in connect#012 self._mount.mount(self.options, self._vfsType, > cgroup=self.CGROUP)#012File: > "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in > mount#012 cgroup=cgroup)#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56, > in __call__#012 return callMethod()#012File: > "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54, > in <lambda>#012 **kwargs)#012File: "<string>", line 2, in > mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py", > line 759, in _callmethod#012 kind, result = conn.recv() > Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:47 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:47 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:47 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:47 hvm002 python: detected unhandled Python exception in > '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' > Aug 26 04:38:47 hvm002 abrt-server: Duplicate: core backtrace > Aug 26 04:38:47 hvm002 abrt-server: DUP_OF_DIR: > /var/tmp/abrt/Python-2019-08-26-04:32:28-3432 > Aug 26 04:38:47 hvm002 abrt-server: Deleting problem directory > Python-2019-08-26-04:38:47-17322 (dup of > Python-2019-08-26-04:32:28-3432) > Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service: main process > exited, code=exited, status=1/FAILURE > Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-broker.service entered > failed state. > Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service failed. > Aug 26 04:38:48 hvm002 systemd: ovirt-ha-broker.service holdoff time > over, scheduling restart. > Aug 26 04:38:48 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:48 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Communications Broker. > Aug 26 04:38:48 hvm002 systemd: Started oVirt Hosted Engine High > Availability Communications Broker. > Aug 26 04:38:48 hvm002 systemd: Started Session c17 of user root. > Aug 26 04:38:48 hvm002 systemd: Started Session c18 of user root. > Aug 26 04:38:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:38:57 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:38:57 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:38:57 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:57 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:38:58 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service failed. > Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service holdoff time > over, scheduling restart. > Aug 26 04:39:08 hvm002 systemd: Cannot add dependency job for unit > lvm2-lvmetad.socket, ignoring: Unit is masked. > Aug 26 04:39:08 hvm002 systemd: Stopped oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:39:08 hvm002 systemd: Started oVirt Hosted Engine High > Availability Monitoring Agent. > Aug 26 04:39:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted > Engine HA score '[Errno 2] No such file or directory'Is the Hosted > Engine setup finished? > Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed > to start necessary monitors > Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent > call last):#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 131, in _run_agent#012 return action(he)#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > line 55, in action_proper#012 return he.start_monitoring()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 432, in start_monitoring#012 self._initialize_broker()#012 > File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > line 556, in _initialize_broker#012 m.get('options', {}))#012 File > "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", > line 86, in start_monitor#012 ).format(t=type, o=options, > e=e)#012RequestError: brokerlink - failed to start monitor via > ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: > 'network', options: {'tcp_t_address': '', 'network_test': 'dns', > 'tcp_t_port': '', 'addr': '172.21.48.1'}] > Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent > ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent > Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service: main process > exited, code=exited, status=157/n/a > Aug 26 04:39:08 hvm002 systemd: Unit ovirt-ha-agent.service entered > failed state. > Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service failed. > [root@hvm002 ~]# > > > > --------------------- > Erick Perez > ---------------------