Engine died while creating VM
by Erick Perez - Quadrian Enterprises
Hi,
I was just creating a VM on a freshly installed 2-node selfhosted
ovirt setup with a NFS backend.
NFS connections are v4.2 for VM data domain
What other log is needed?
Thanks, in advance.
Here is the output of the last 300 lines of /var/log/messages
[root@hvm002 ~]#
[root@hvm002 ~]# tail /var/log/messages -n 300
Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:34:56 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:34:56 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:06 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:06 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:06 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:06 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:07 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:07 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:07 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:35:15 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:15 hvm002 systemd: Starting Cockpit Web Service...
Aug 26 04:35:15 hvm002 systemd: Started Cockpit Web Service.
Aug 26 04:35:15 hvm002 cockpit-ws: Using certificate:
/etc/cockpit/ws-certs.d/0-self-signed.cert
Aug 26 04:35:15 hvm002 cockpit-ws: couldn't read from connection: Peer
sent fatal TLS alert: Certificate is bad
Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:17 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:17 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:17 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:17 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:17 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:17 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:35:27 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:27 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:27 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:27 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:28 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:28 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:28 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
name=jsonrpc/5 running <Task <JsonRpcTask {'params':
{u'connectionParams': [{u'protocol_version': u'auto', u'connection':
u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc':
'2.0', 'method': u'StoragePool.connectStorageServer', 'id':
u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0>
timeout=60, duration=60.00 at 0x7f091c0aadd0> task#=4 at
0x7f091c0c6e10>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
line 812, in __bootstrap_inner#012 self.run()#012File:
"/usr/lib64/python2.7/threading.py", line 765, in run#012
self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
195, in run#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
_run#012 self._execute_task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task#012 task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File:
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod#012 result = fn(*methodArgs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
connectStorageServer#012 connectionParams)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
74, in wrapper#012 result = ctask.prepare(func, *args,
**kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper#012 return m(self, *a, **kw)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
prepare#012 result = self._run(func, *args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
connectStorageServer#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
connectStorageServer#012 conObj.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
431, in connect#012 return self._mountCon.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
172, in connect#012 self._mount.mount(self.options, self._vfsType,
cgroup=self.CGROUP)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
mount#012 cgroup=cgroup)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
in __call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
line 759, in _callmethod#012 kind, result = conn.recv()
Aug 26 04:35:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:38 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:38 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:38 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:38 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:38 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:38 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:48 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:48 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:48 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:48 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:49 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:49 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:49 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:35:52 hvm002 systemd: Created slice User Slice of root.
Aug 26 04:35:52 hvm002 systemd-logind: New session 2 of user root.
Aug 26 04:35:52 hvm002 systemd: Started Session 2 of user root.
Aug 26 04:35:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:35:59 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:35:59 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:59 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:35:59 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:35:59 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:35:59 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:36:09 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:36:09 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:09 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:09 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:36:10 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:36:10 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:36:10 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:36:20 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:20 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:20 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:36:20 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:36:20 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:36:20 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:36:30 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:36:30 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:30 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:30 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:36:31 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:36:31 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:36:31 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:36 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
name=jsonrpc/5 running <Task <JsonRpcTask {'params':
{u'connectionParams': [{u'protocol_version': u'auto', u'connection':
u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc':
'2.0', 'method': u'StoragePool.connectStorageServer', 'id':
u'767bed1a-dc31-44f6-98a8-94faebc82345'} at 0x7f08f83cadd0>
timeout=60, duration=120.00 at 0x7f091c0aadd0> task#=4 at
0x7f091c0c6e10>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
line 812, in __bootstrap_inner#012 self.run()#012File:
"/usr/lib64/python2.7/threading.py", line 765, in run#012
self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
195, in run#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
_run#012 self._execute_task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task#012 task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File:
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod#012 result = fn(*methodArgs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
connectStorageServer#012 connectionParams)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
74, in wrapper#012 result = ctask.prepare(func, *args,
**kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper#012 return m(self, *a, **kw)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
prepare#012 result = self._run(func, *args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
connectStorageServer#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
connectStorageServer#012 conObj.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
431, in connect#012 return self._mountCon.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
172, in connect#012 self._mount.mount(self.options, self._vfsType,
cgroup=self.CGROUP)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
mount#012 cgroup=cgroup)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
in __call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
line 759, in _callmethod#012 kind, result = conn.recv()
Aug 26 04:36:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:36:41 hvm002 python: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Aug 26 04:36:41 hvm002 abrt-server: Duplicate: core backtrace
Aug 26 04:36:41 hvm002 abrt-server: DUP_OF_DIR:
/var/tmp/abrt/Python-2019-08-26-04:32:28-3432
Aug 26 04:36:41 hvm002 abrt-server: Deleting problem directory
Python-2019-08-26-04:36:41-16985 (dup of
Python-2019-08-26-04:32:28-3432)
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:41 hvm002 dbus[3444]: [system] Activating service
name='org.freedesktop.problems' (using servicehelper)
Aug 26 04:36:41 hvm002 dbus[3444]: [system] Successfully activated
service 'org.freedesktop.problems'
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service: main process
exited, code=exited, status=1/FAILURE
Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-broker.service entered
failed state.
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service failed.
Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:36:41 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:36:41 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:41 hvm002 systemd: ovirt-ha-broker.service holdoff time
over, scheduling restart.
Aug 26 04:36:41 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:41 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Communications Broker.
Aug 26 04:36:41 hvm002 systemd: Started oVirt Hosted Engine High
Availability Communications Broker.
Aug 26 04:36:42 hvm002 systemd: Started Session c15 of user root.
Aug 26 04:36:42 hvm002 systemd: Started Session c16 of user root.
Aug 26 04:36:51 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:36:51 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:36:51 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:51 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:36:52 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:36:52 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:36:52 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:36:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:02 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:02 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:02 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:02 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:02 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:02 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:37:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:37:12 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:12 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:12 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:12 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:13 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:13 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:13 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:23 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:23 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:23 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:23 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:23 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:23 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:37:33 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:33 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:33 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:33 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:34 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:34 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:34 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:37:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:37:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
name=jsonrpc/3 running <Task <JsonRpcTask {'params':
{u'connectionParams': [{u'protocol_version': u'auto', u'connection':
u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc':
'2.0', 'method': u'StoragePool.connectStorageServer', 'id':
u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650>
timeout=60, duration=60.00 at 0x7f08f8259750> task#=8 at
0x7f091c110c50>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
line 812, in __bootstrap_inner#012 self.run()#012File:
"/usr/lib64/python2.7/threading.py", line 765, in run#012
self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
195, in run#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
_run#012 self._execute_task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task#012 task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File:
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod#012 result = fn(*methodArgs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
connectStorageServer#012 connectionParams)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
74, in wrapper#012 result = ctask.prepare(func, *args,
**kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper#012 return m(self, *a, **kw)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
prepare#012 result = self._run(func, *args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
connectStorageServer#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
connectStorageServer#012 conObj.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
431, in connect#012 return self._mountCon.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
172, in connect#012 self._mount.mount(self.options, self._vfsType,
cgroup=self.CGROUP)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
mount#012 cgroup=cgroup)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
in __call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
line 759, in _callmethod#012 kind, result = conn.recv()
Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:44 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:44 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:44 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:44 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:44 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:44 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:37:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:37:54 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:37:54 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:37:54 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:54 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:37:55 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:37:55 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:37:55 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:05 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:05 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:05 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:05 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:05 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:05 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:38:15 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:15 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:15 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:15 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:16 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:16 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:16 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:23 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:26 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:26 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:26 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:26 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:26 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:26 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:36 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:36 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:36 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:36 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:37 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:37 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:37 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:38 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:38:42 hvm002 vdsm[7369]: WARN Worker blocked: <Worker
name=jsonrpc/3 running <Task <JsonRpcTask {'params':
{u'connectionParams': [{u'protocol_version': u'auto', u'connection':
u'10.10.10.2:/mnt/lv_vdo01', u'user': u'kvm', u'id':
u'e29cf818-5ee5-46e1-85c1-8aeefa33e95d'}], u'storagepoolID':
u'00000000-0000-0000-0000-000000000000', u'domainType': 1}, 'jsonrpc':
'2.0', 'method': u'StoragePool.connectStorageServer', 'id':
u'c8887c86-fb5e-4be1-9552-45620144b74e'} at 0x7f08f8259650>
timeout=60, duration=120.00 at 0x7f08f8259750> task#=8 at
0x7f091c110c50>, traceback:#012File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap#012
self.__bootstrap_inner()#012File: "/usr/lib64/python2.7/threading.py",
line 812, in __bootstrap_inner#012 self.run()#012File:
"/usr/lib64/python2.7/threading.py", line 765, in run#012
self.__target(*self.__args, **self.__kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line
195, in run#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in
_run#012 self._execute_task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task#012 task()#012File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in
__call__#012 self._callable()#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 262, in
__call__#012 self._handler(self._ctx, self._req)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 305, in
_serveRequest#012 response = self._handle_request(req, ctx)#012File:
"/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line 345, in
_handle_request#012 res = method(**params)#012File:
"/usr/lib/python2.7/site-packages/vdsm/rpc/Bridge.py", line 198, in
_dynamicMethod#012 result = fn(*methodArgs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/API.py", line 1102, in
connectStorageServer#012 connectionParams)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/dispatcher.py", line
74, in wrapper#012 result = ctask.prepare(func, *args,
**kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 108, in
wrapper#012 return m(self, *a, **kw)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1179, in
prepare#012 result = self._run(func, *args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in
_run#012 return fn(*args, **kargs)#012File: "<string>", line 2, in
connectStorageServer#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, in
method#012 ret = func(*args, **kwargs)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2416, in
connectStorageServer#012 conObj.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
431, in connect#012 return self._mountCon.connect()#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line
172, in connect#012 self._mount.mount(self.options, self._vfsType,
cgroup=self.CGROUP)#012File:
"/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in
mount#012 cgroup=cgroup)#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 56,
in __call__#012 return callMethod()#012File:
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 54,
in <lambda>#012 **kwargs)#012File: "<string>", line 2, in
mount#012File: "/usr/lib64/python2.7/multiprocessing/managers.py",
line 759, in _callmethod#012 kind, result = conn.recv()
Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:47 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:47 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:47 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:47 hvm002 python: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Aug 26 04:38:47 hvm002 abrt-server: Duplicate: core backtrace
Aug 26 04:38:47 hvm002 abrt-server: DUP_OF_DIR:
/var/tmp/abrt/Python-2019-08-26-04:32:28-3432
Aug 26 04:38:47 hvm002 abrt-server: Deleting problem directory
Python-2019-08-26-04:38:47-17322 (dup of
Python-2019-08-26-04:32:28-3432)
Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:47 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:47 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service: main process
exited, code=exited, status=1/FAILURE
Aug 26 04:38:47 hvm002 systemd: Unit ovirt-ha-broker.service entered
failed state.
Aug 26 04:38:47 hvm002 systemd: ovirt-ha-broker.service failed.
Aug 26 04:38:48 hvm002 systemd: ovirt-ha-broker.service holdoff time
over, scheduling restart.
Aug 26 04:38:48 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:48 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Communications Broker.
Aug 26 04:38:48 hvm002 systemd: Started oVirt Hosted Engine High
Availability Communications Broker.
Aug 26 04:38:48 hvm002 systemd: Started Session c17 of user root.
Aug 26 04:38:48 hvm002 systemd: Started Session c18 of user root.
Aug 26 04:38:53 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:38:57 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:38:57 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:38:57 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:57 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:38:58 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:38:58 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:38:58 hvm002 systemd: ovirt-ha-agent.service failed.
Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service holdoff time
over, scheduling restart.
Aug 26 04:39:08 hvm002 systemd: Cannot add dependency job for unit
lvm2-lvmetad.socket, ignoring: Unit is masked.
Aug 26 04:39:08 hvm002 systemd: Stopped oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:39:08 hvm002 systemd: Started oVirt Hosted Engine High
Availability Monitoring Agent.
Aug 26 04:39:08 hvm002 vdsm[7369]: ERROR failed to retrieve Hosted
Engine HA score '[Errno 2] No such file or directory'Is the Hosted
Engine setup finished?
Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed
to start necessary monitors
Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent
call last):#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/agent.py",
line 55, in action_proper#012 return he.start_monitoring()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 432, in start_monitoring#012 self._initialize_broker()#012
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 556, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 86, in start_monitor#012 ).format(t=type, o=options,
e=e)#012RequestError: brokerlink - failed to start monitor via
ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'tcp_t_address': '', 'network_test': 'dns',
'tcp_t_port': '', 'addr': '172.21.48.1'}]
Aug 26 04:39:08 hvm002 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service: main process
exited, code=exited, status=157/n/a
Aug 26 04:39:08 hvm002 systemd: Unit ovirt-ha-agent.service entered
failed state.
Aug 26 04:39:08 hvm002 systemd: ovirt-ha-agent.service failed.
[root@hvm002 ~]#
---------------------
Erick Perez
---------------------
