4:02 a.m.
THis post can be ignored. Because the main error was the NFS server
dying due to excess read/write on the SATA based storage.
Timeouts were huge making the NFS-based storage domain unusable.
---------------------
Erick Perez
On Mon, Aug 26, 2019 at 4:44 AM Erick Perez - Quadrian Enterprises
<eperez(a)quadrianweb.com> wrote:
>
> Hi,
> I was just creating a VM on a freshly installed 2-node selfhosted
> ovirt setup with a NFS backend.
> NFS connections are v4.2 for VM data domain
>
> What other log is needed?
>
> Thanks, in advance.
>
>
> Here is the output of the last 300 lines of /var/log/messages
> [root@hvm002 ~]#
> [root@hvm002 ~]# tail /var/log/messages -n 300
> Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:34:56 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:06 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:06 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:06 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:06 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:07 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:35:15 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:15 hvm002 systemd: Starting Cockpit Web Service...
> Aug 26 04:35:15 hvm002 systemd: Started Cockpit Web Service.
> Aug 26 04:35:15 hvm002 cockpit-ws: Using certificate:
> /etc/cockpit/ws-certs.d/0-self-signed.cert
> Aug 26 04:35:15 hvm002 cockpit-ws: couldn't read from connection: Peer
> sent fatal TLS alert: Certificate is bad
> Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:17 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:17 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:17 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:17 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:35:27 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:27 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:27 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:27 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:28 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
> name=jsonrpc/5 running <Task <JsonRpcTask {'params':
> {u'connectionParams': [{u'protocol_version': u'auto',
u'connection':
> u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
> u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
> u'00000000-0000-0000-0000-000000000000', u'domainType': 1},
'jsonrpc':
> '2.0', 'method': u'StoragePool.connectStorageServer',
'id':
> u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0>
> timeout=60, duration=60.00 at 0x7f091c0aadd0> task#=4 at
> 0x7f091c0c6e10>, traceback:#012File:
> "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
> self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
> line 812, in __bootstrap_inner#012 self.run()#012File:
> "/usr/lib64/python2.7/threading.py", line 765, in run#012
> self.__target(*self.__args, **self.__kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
> 195, in run#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
> _run#012 self._execute_task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
> _execute_task#012 task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
> __call__#012 self._callable()#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
> __call__#012 self._handler(self._ctx, self._req)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
> _serveRequest#012 response = self._handle_request(req, ctx)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
> _handle_request#012 res = method(**params)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod#012 result = fn(*methodArgs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
> connectStorageServer#012 connectionParams)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
> 74, in wrapper#012 result = ctask.prepare(func, *args,
> **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> wrapper#012 return m(self, *a, **kw)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
> prepare#012 result = self._run(func, *args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
> connectStorageServer#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
> connectStorageServer#012 conObj.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 431, in connect#012 return self._mountCon.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 172, in connect#012 self._mount.mount(self.options, self._vfsType,
> cgroup=self.CGROUP)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
> mount#012 cgroup=cgroup)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
> in __call__#012 return callMethod()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
> in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
> mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
> line 759, in _callmethod#012 kind, result = conn.recv()
> Aug 26 04:35:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:38 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:38 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:38 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:38 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:48 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:48 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:48 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:48 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:49 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:35:52 hvm002 systemd: Created slice User Slice of root.
> Aug 26 04:35:52 hvm002 systemd-logind: New session 2 of user root.
> Aug 26 04:35:52 hvm002 systemd: Started Session 2 of user root.
> Aug 26 04:35:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:35:59 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:35:59 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:59 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:35:59 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:36:09 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:09 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:09 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:09 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:36:10 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:20 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:20 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:20 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:36:20 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:36:30 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:30 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:30 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:30 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:36:31 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
> name=jsonrpc/5 running <Task <JsonRpcTask {'params':
> {u'connectionParams': [{u'protocol_version': u'auto',
u'connection':
> u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
> u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
> u'00000000-0000-0000-0000-000000000000', u'domainType': 1},
'jsonrpc':
> '2.0', 'method': u'StoragePool.connectStorageServer',
'id':
> u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0>
> timeout=60, duration=120.00 at 0x7f091c0aadd0> task#=4 at
> 0x7f091c0c6e10>, traceback:#012File:
> "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
> self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
> line 812, in __bootstrap_inner#012 self.run()#012File:
> "/usr/lib64/python2.7/threading.py", line 765, in run#012
> self.__target(*self.__args, **self.__kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
> 195, in run#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
> _run#012 self._execute_task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
> _execute_task#012 task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
> __call__#012 self._callable()#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
> __call__#012 self._handler(self._ctx, self._req)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
> _serveRequest#012 response = self._handle_request(req, ctx)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
> _handle_request#012 res = method(**params)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod#012 result = fn(*methodArgs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
> connectStorageServer#012 connectionParams)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
> 74, in wrapper#012 result = ctask.prepare(func, *args,
> **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> wrapper#012 return m(self, *a, **kw)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
> prepare#012 result = self._run(func, *args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
> connectStorageServer#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
> connectStorageServer#012 conObj.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 431, in connect#012 return self._mountCon.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 172, in connect#012 self._mount.mount(self.options, self._vfsType,
> cgroup=self.CGROUP)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
> mount#012 cgroup=cgroup)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
> in __call__#012 return callMethod()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
> in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
> mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
> line 759, in _callmethod#012 kind, result = conn.recv()
> Aug 26 04:36:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:36:41 hvm002 python: detected unhandled Python exception in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Aug 26 04:36:41 hvm002 abrt-server: Duplicate: core backtrace
> Aug 26 04:36:41 hvm002 abrt-server: DUP_OF_DIR:
> /var/tmp/abrt/Python-2019-08-26-04:32:28-3432
> Aug 26 04:36:41 hvm002 abrt-server: Deleting problem directory
> Python-2019-08-26-04:36:41-16985 (dup of
> Python-2019-08-26-04:32:28-3432)
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:41 hvm002 dbus[3444]: [system] Activating service
> name='org.freedesktop.problems' (using servicehelper)
> Aug 26 04:36:41 hvm002 dbus[3444]: [system] Successfully activated
> service 'org.freedesktop.problems'
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service: main process
> exited, code=exited, status=1/FAILURE
> Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-broker.service entered
> failed state.
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service failed.
> Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Communications Broker.
> Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Communications Broker.
> Aug 26 04:36:42 hvm002 systemd: Started Session c15 of user root.
> Aug 26 04:36:42 hvm002 systemd: Started Session c16 of user root.
> Aug 26 04:36:51 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:36:51 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:36:51 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:51 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:36:52 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:36:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:02 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:02 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:02 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:02 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:37:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:37:12 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:12 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:12 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:12 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:13 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:23 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:23 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:23 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:23 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:37:33 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:33 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:33 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:33 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:34 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:37:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:37:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
> name=jsonrpc/3 running <Task <JsonRpcTask {'params':
> {u'connectionParams': [{u'protocol_version': u'auto',
u'connection':
> u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
> u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
> u'00000000-0000-0000-0000-000000000000', u'domainType': 1},
'jsonrpc':
> '2.0', 'method': u'StoragePool.connectStorageServer',
'id':
> u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650>
> timeout=60, duration=60.00 at 0x7f08f8259750> task#=8 at
> 0x7f091c110c50>, traceback:#012File:
> "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
> self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
> line 812, in __bootstrap_inner#012 self.run()#012File:
> "/usr/lib64/python2.7/threading.py", line 765, in run#012
> self.__target(*self.__args, **self.__kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
> 195, in run#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
> _run#012 self._execute_task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
> _execute_task#012 task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
> __call__#012 self._callable()#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
> __call__#012 self._handler(self._ctx, self._req)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
> _serveRequest#012 response = self._handle_request(req, ctx)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
> _handle_request#012 res = method(**params)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod#012 result = fn(*methodArgs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
> connectStorageServer#012 connectionParams)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
> 74, in wrapper#012 result = ctask.prepare(func, *args,
> **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> wrapper#012 return m(self, *a, **kw)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
> prepare#012 result = self._run(func, *args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
> connectStorageServer#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
> connectStorageServer#012 conObj.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 431, in connect#012 return self._mountCon.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 172, in connect#012 self._mount.mount(self.options, self._vfsType,
> cgroup=self.CGROUP)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
> mount#012 cgroup=cgroup)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
> in __call__#012 return callMethod()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
> in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
> mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
> line 759, in _callmethod#012 kind, result = conn.recv()
> Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:44 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:44 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:44 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:44 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:37:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:37:54 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:37:54 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:37:54 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:54 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:37:55 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:05 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:05 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:05 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:05 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:38:15 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:15 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:15 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:15 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:16 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:26 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:26 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:26 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:26 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:36 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:36 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:36 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:36 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:37 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:38:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
> name=jsonrpc/3 running <Task <JsonRpcTask {'params':
> {u'connectionParams': [{u'protocol_version': u'auto',
u'connection':
> u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
> u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
> u'00000000-0000-0000-0000-000000000000', u'domainType': 1},
'jsonrpc':
> '2.0', 'method': u'StoragePool.connectStorageServer',
'id':
> u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650>
> timeout=60, duration=120.00 at 0x7f08f8259750> task#=8 at
> 0x7f091c110c50>, traceback:#012File:
> "/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
> self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
> line 812, in __bootstrap_inner#012 self.run()#012File:
> "/usr/lib64/python2.7/threading.py", line 765, in run#012
> self.__target(*self.__args, **self.__kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
> 195, in run#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
> _run#012 self._execute_task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
> _execute_task#012 task()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
> __call__#012 self._callable()#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
> __call__#012 self._handler(self._ctx, self._req)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
> _serveRequest#012 response = self._handle_request(req, ctx)#012File:
> "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
> _handle_request#012 res = method(**params)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
> _dynamicMethod#012 result = fn(*methodArgs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
> connectStorageServer#012 connectionParams)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
> 74, in wrapper#012 result = ctask.prepare(func, *args,
> **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
> wrapper#012 return m(self, *a, **kw)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
> prepare#012 result = self._run(func, *args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
> _run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
> connectStorageServer#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
> method#012 ret = func(*args, **kwargs)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
> connectStorageServer#012 conObj.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 431, in connect#012 return self._mountCon.connect()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
> 172, in connect#012 self._mount.mount(self.options, self._vfsType,
> cgroup=self.CGROUP)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
> mount#012 cgroup=cgroup)#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
> in __call__#012 return callMethod()#012File:
> "/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
> in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
> mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
> line 759, in _callmethod#012 kind, result = conn.recv()
> Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:47 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:47 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:47 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:47 hvm002 python: detected unhandled Python exception in
> '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
> Aug 26 04:38:47 hvm002 abrt-server: Duplicate: core backtrace
> Aug 26 04:38:47 hvm002 abrt-server: DUP_OF_DIR:
> /var/tmp/abrt/Python-2019-08-26-04:32:28-3432
> Aug 26 04:38:47 hvm002 abrt-server: Deleting problem directory
> Python-2019-08-26-04:38:47-17322 (dup of
> Python-2019-08-26-04:32:28-3432)
> Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service: main process
> exited, code=exited, status=1/FAILURE
> Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-broker.service entered
> failed state.
> Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service failed.
> Aug 26 04:38:48 hvm002 systemd: ovirt-ha-broker.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:48 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:48 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Communications Broker.
> Aug 26 04:38:48 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Communications Broker.
> Aug 26 04:38:48 hvm002 systemd: Started Session c17 of user root.
> Aug 26 04:38:48 hvm002 systemd: Started Session c18 of user root.
> Aug 26 04:38:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:38:57 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:38:57 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:38:57 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:57 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:38:58 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service failed.
> Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service holdoff time
> over, scheduling restart.
> Aug 26 04:39:08 hvm002 systemd: Cannot add dependency job for unit
> lvm2-lvmetad.socket, ignoring: Unit is masked.
> Aug 26 04:39:08 hvm002 systemd: Stopped oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:39:08 hvm002 systemd: Started oVirt Hosted Engine High
> Availability Monitoring Agent.
> Aug 26 04:39:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
> Engine HA score '[Errno 2] No such file or directory'Is the Hosted
> Engine setup finished?
> Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
> to start necessary monitors
> Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
> call last):#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 131, in _run_agent#012 return action(he)#012 File
> "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
> line 55, in action_proper#012 return he.start_monitoring()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 432, in start_monitoring#012 self._initialize_broker()#012
> File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
> line 556, in _initialize_broker#012 m.get('options', {}))#012 File
>
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
> line 86, in start_monitor#012 ).format(t=type, o=options,
> e=e)#012RequestError: brokerlink - failed to start monitor via
> ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
> 'network', options: {'tcp_t_address': '',
'network_test': 'dns',
> 'tcp_t_port': '', 'addr': '172.21.48.1'}]
> Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
> Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service: main process
> exited, code=exited, status=157/n/a
> Aug 26 04:39:08 hvm002 systemd: Unit ovirt-ha-agent.service entered
> failed state.
> Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service failed.
> [root@hvm002 ~]#
>
>
>
> ---------------------
> Erick Perez
> ---------------------