2:30 p.m.
Hi all,
We have had a bit of a strange issue with the starting of the self-hosted engine due
to the vdsm and ovirt-ha processes having errors and not able to be started.
Running Ovirt version 4.2 with Gluster mounts (not hyper converged) gluster 3.12.13
replicated volumes with arbiter brick.
The mounts for the storage are mounted on all the Ovirt hosts and able to read/touch
files on the mounts.
Unfortunately we can not pin a specific timeframe when this started happening. Meaning we
don’t know if it started after a specific update or any other event.
The vdsmd is throwing errors (bottom – with version numbers) when trying to check
the status of the self -hosted engine getting :
# hosted-engine --vm-status
The hosted engine configuration has not been retrieved from shared storage. Please ensure
that ovirt-ha-agent is running and the storage server is reachable.
Thanks for any assistance.
systemctl status ovirt-ha-agent -l
● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent
Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset:
disabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2018-08-30 13:24:40
CDT; 2s ago
Process: 17135 ExecStart=/usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent (code=exited,
status=157)
Main PID: 17135 (code=exited, status=157)
systemctl status ovirt-ha-broker -l
● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker
Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor
preset: disabled)
Active: active (running) since Thu 2018-08-30 13:10:35 CDT; 14min ago
Main PID: 14863 (ovirt-ha-broker)
Tasks: 4
CGroup: /system.slice/ovirt-ha-broker.service
└─14863 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker
Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Started oVirt Hosted Engine High
Availability Communications Broker.
Aug 30 13:10:35 rrb-vmhost1.rrc.local systemd[1]: Starting oVirt Hosted Engine High
Availability Communications Broker...
Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start
monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout
during domain acquisition
Aug 30 13:14:44 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC
call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2,
host_id=1): timeout during domain acquisition
Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker ERROR Failed to start
monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2, host_id=1): timeout
during domain acquisition
Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.listener.Action.start_domain_monitor ERROR Error in RPC
call: Failed to start monitoring domain (sd_uuid=d0bfc335-ab6c-4378-9bcb-2f5f833431c2,
host_id=1): timeout during domain acquisition
Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Failed to update
state.
Traceback (most recent call
last):
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 82, in run
if
(self._status_broker._inquire_whiteboard_lock() or
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 190, in _inquire_whiteboard_lock
self.host_id,
self._lease_file)
File
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 128, in host_id
raise
ex.HostIdNotLockedError("Host id is not set")
HostIdNotLockedError: Host
id is not set
Aug 30 13:19:42 rrb-vmhost1.rrc.local ovirt-ha-broker[14863]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update ERROR Trying to restart
the broker
Aug 30 13:19:42 rrb-vmhost1.rrc.local python[14863]: detected unhandled Python exception
in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Aug 30 13:19:48 rrb-vmhost1.rrc.local python[14863]: communication with ABRT daemon
failed: timed out
glusterfs-api-3.12.13-1.el7.x86_64
glusterfs-server-3.12.13-1.el7.x86_64
glusterfs-fuse-3.12.13-1.el7.x86_64
vdsm-gluster-4.20.35-1.el7.x86_64
centos-release-gluster312-1.0-2.el7.centos.noarch
glusterfs-client-xlators-3.12.13-1.el7.x86_64
glusterfs-geo-replication-3.12.13-1.el7.x86_64
glusterfs-libs-3.12.13-1.el7.x86_64
glusterfs-cli-3.12.13-1.el7.x86_64
glusterfs-rdma-3.12.13-1.el7.x86_64
glusterfs-events-3.12.13-1.el7.x86_64
glusterfs-3.12.13-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.7.x86_64
python2-gluster-3.12.13-1.el7.x86_64
glusterfs-gnfs-3.12.13-1.el7.x86_64
ovirt-setup-lib-1.1.4-1.el7.centos.noarch
ovirt-iso-uploader-4.2.0-1.el7.centos.noarch
ovirt-engine-setup-base-4.2.5.3-1.el7.noarch
ovirt-host-dependencies-4.2.3-1.el7.x86_64
ovirt-guest-agent-common-1.0.14-1.el7.noarch
python-ovirt-engine-sdk4-4.2.7-2.el7.x86_64
ovirt-engine-setup-plugin-ovirt-engine-common-4.2.5.3-1.el7.noarch
ovirt-vmconsole-1.0.5-4.el7.centos.noarch
ovirt-engine-tools-backup-4.2.5.3-1.el7.noarch
ovirt-guest-agent-windows-1.0.14-1.el7.centos.noarch
cockpit-ovirt-dashboard-0.11.31-1.el7.noarch
ovirt-host-deploy-1.7.4-1.el7.noarch
ovirt-engine-lib-4.2.5.3-1.el7.noarch
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-hosted-engine-ha-2.2.16-1.el7.noarch
ovirt-guest-tools-iso-4.2-1.el7.centos.noarch
ovirt-imageio-common-1.4.2-0.el7.noarch
ovirt-hosted-engine-setup-2.2.25-1.el7.noarch
ovirt-imageio-daemon-1.4.2-0.el7.noarch
ovirt-vmconsole-host-1.0.5-4.el7.centos.noarch
ovirt-provider-ovn-driver-1.2.13-1.el7.noarch
ovirt-host-4.2.3-1.el7.x86_64
vdsm-jsonrpc-4.20.35-1.el7.noarch
vdsm-hook-ethtool-options-4.20.35-1.el7.noarch
vdsm-gluster-4.20.35-1.el7.x86_64
vdsm-api-4.20.35-1.el7.noarch
vdsm-yajsonrpc-4.20.35-1.el7.noarch
vdsm-client-4.20.35-1.el7.noarch
vdsm-hook-fcoe-4.20.35-1.el7.noarch
vdsm-network-4.20.35-1.el7.x86_64
vdsm-hook-vmfex-dev-4.20.35-1.el7.noarch
vdsm-hook-vhostmd-4.20.35-1.el7.noarch
vdsm-http-4.20.35-1.el7.noarch
vdsm-hook-openstacknet-4.20.35-1.el7.noarch
vdsm-python-4.20.35-1.el7.noarch
vdsm-hook-vfio-mdev-4.20.35-1.el7.noarch
vdsm-common-4.20.35-1.el7.noarch
vdsm-4.20.35-1.el7.x86_64
Aug 30 13:02:42 rrb-vmhost1.rrc.local vdsm[6792]: WARN Worker blocked: <Worker
name=periodic/1 running <Task <Operation action=<vdsm.virt.sampling.HostMonitor
object at 0x7f713c0b0fd0> at 0x7f713c063050> timeout=15, duration=75 at
0x7f7170698c90> task#=354 at 0x7f713c10ec90>, traceback:
File:
"/usr/lib64/python2.7/threading.py", line 785, in __bootstrap
self.__bootstrap_inner()
File:
"/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File:
"/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args,
**self.__kwargs)
File:
"/usr/lib/python2.7/site-packages/vdsm/common/concurrent.py", line 194, in run
ret = func(*args, **kwargs)
File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 301, in _run
self._execute_task()
File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 315, in
_execute_task
task()
File:
"/usr/lib/python2.7/site-packages/vdsm/executor.py", line 391, in __call__
self._callable()
File:
"/usr/lib/python2.7/site-packages/vdsm/virt/periodic.py", line 220, in
__call__
self._func()
File:
"/usr/lib/python2.7/site-packages/vdsm/virt/sampling.py", line 580, in
__call__
stats = hostapi.get_stats(self._cif,
self._samples.stats())
File:
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 77, in get_stats
ret['haStats'] =
_getHaInfo()
File:
"/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 182, in _getHaInfo
stats = instance.get_all_stats()
File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line
94, in get_all_stats
stats =
broker.get_stats_from_storage()
File:
"/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 135, in get_stats_from_storage
result = self._proxy.get_stats()
File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name,
args)
File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request
verbose=self.__verbose
File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host,
handler, request_body, verbose)
File:
"/usr/lib64/python2.7/xmlrpclib.py", line 1303, in single_request
response =
h.getresponse(buffering=True)
File:
"/usr/lib64/python2.7/httplib.py", line 1113, in getresponse
response.begin()
File:
"/usr/lib64/python2.7/httplib.py", line 444, in begin
version, status, reason =
self._read_status()
File:
"/usr/lib64/python2.7/httplib.py", line 400, in _read_status
line = self.fp.readline(_MAXLINE +
1)
File:
"/usr/lib64/python2.7/socket.py", line 476, in readline
data =
self._sock.recv(self._rbufsize)