6:47 a.m.
3 node cluster. Gluster for shared storage.
CentOS8
Updated to CentOS 8 Streams :P -> https://bugzilla.redhat.com/show_bug.cgi?id=1911910
After several weeks .. I am really in need of direction to get this fixed.
I saw several postings about oVirt package issues but not found a fix.
[root@thor ~]# dnf update
Last metadata expiration check: 2:54:29 ago on Fri 15 Jan 2021 06:49:16 AM EST.
Error:
Problem 1: package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of
the providers can be installed
- package cockpit-bridge-234-1.el8.x86_64 conflicts with cockpit-dashboard < 233
provided by cockpit-dashboard-217-1.el8.noarch
- cannot install the best update candidate for package ovirt-host-4.4.1-4.el8.x86_64
- cannot install the best update candidate for package cockpit-bridge-217-1.el8.x86_64
Problem 2: problem with installed package ovirt-host-4.4.1-4.el8.x86_64
- package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by
cockpit-dashboard-217-1.el8.noarch
- cannot install the best update candidate for package
cockpit-dashboard-217-1.el8.noarch
Problem 3: package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch requires ovirt-host >=
4.4.0, but none of the providers can be installed
- package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by
cockpit-dashboard-217-1.el8.noarch
- cannot install the best update candidate for package
ovirt-hosted-engine-setup-2.4.9-1.el8.noarch
- cannot install the best update candidate for package cockpit-system-217-1.el8.noarch
(try to add '--allowerasing' to command line to replace conflicting packages or
'--skip-broken' to skip uninstallable packages or '--nobest' to use not
only best candidate packages)
[root@thor ~]# yum install cockpit-dashboard --nobest
Last metadata expiration check: 2:54:52 ago on Fri 15 Jan 2021 06:49:16 AM EST.
Package cockpit-dashboard-217-1.el8.noarch is already installed.
Dependencies resolved.
Problem: problem with installed package ovirt-host-4.4.1-4.el8.x86_64
- package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the
providers can be installed
- package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by
cockpit-dashboard-217-1.el8.noarch
- cannot install the best candidate for the job
=========================================================================================================================================================================================================================================
Package Architecture
Version Repository
Size
=========================================================================================================================================================================================================================================
Skipping packages with broken dependencies:
ovirt-host x86_64
4.4.1-1.el8 ovirt-4.4
13 k
ovirt-host x86_64
4.4.1-2.el8 ovirt-4.4
13 k
ovirt-host x86_64
4.4.1-3.el8 ovirt-4.4
13 k
Transaction Summary
=========================================================================================================================================================================================================================================
Skip 3 Packages
Nothing to do.
Complete!
[root@thor ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE
MOUNTPOINT
sda 8:0 0 931.5G 0 disk
└─WDC_WDS100T2B0B-00YS70_19106A802926 253:3 0 931.5G 0 mpath
└─vdo_2926 253:5 0 4T 0 vdo
/gluster_bricks/gv0
sdb 8:16 0 931.5G 0 disk
└─WDC_WDS100T2B0B-00YS70_192490801828 253:4 0 931.5G 0 mpath
sdc 8:32 0 477G 0 disk
└─vdo_sdc 253:6 0 2.1T 0 vdo
├─gluster_vg_sdc-gluster_lv_engine 253:7 0 100G 0 lvm
/gluster_bricks/engine
├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tmeta 253:8 0 1G 0 lvm
│ └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm
│ ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm
│ ├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm
/gluster_bricks/data
│ └─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm
/gluster_bricks/vmstore
└─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tdata 253:9 0 2T 0 lvm
└─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm
├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm
├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm
/gluster_bricks/data
└─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm
/gluster_bricks/vmstore
sdd 8:48 1 58.8G 0 disk
├─sdd1 8:49 1 1G 0 part
/boot
└─sdd2 8:50 1 57.8G 0 part
├─cl-root 253:0 0 36.1G 0 lvm /
├─cl-swap 253:1 0 4G 0 lvm
[SWAP]
└─cl-home 253:2 0 17.6G 0 lvm
/home
[root@thor ~]# mount |grep engine
/dev/mapper/gluster_vg_sdc-gluster_lv_engine on /gluster_bricks/engine type xfs
(rw,noatime,nodiratime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota,_netdev,x-systemd.requires=vdo.service)
thorst.penguinpages.local:/engine on /media/engine type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev)
thorst.penguinpages.local:/engine on
/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev,x-systemd.device-timeout=0)
[root@thor ~]#
[root@thor ~]# tail -50 /var/log/messages
Jan 15 09:46:43 thor platform-python[28088]: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 15 09:46:43 thor abrt-server[28116]: Not saving repeating crash in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Main process exited,
code=exited, status=1/FAILURE
Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Failed with result
'exit-code'.
Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms
expired, scheduling restart.
Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart
counter is at 241.
Jan 15 09:46:43 thor systemd[1]: Stopped oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:43 thor systemd[1]: Started oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:45 thor systemd[1]: Started Session c448 of user root.
Jan 15 09:46:45 thor systemd[1]: session-c448.scope: Succeeded.
Jan 15 09:46:45 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor]
does not exist on server localhost
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Service RestartSec=10s expired,
scheduling restart.
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Scheduled restart job, restart
counter is at 211.
Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Monitoring
Agent.
Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability Monitoring
Agent.
Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno
107] Transport endpoint is not connected:
'/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata'
Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 97, in __init__#012 self._backend.connect()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path,
service_link)#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107]
Transport endpoint is not connected:
'/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata'
Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker
Jan 15 09:46:48 thor platform-python[28118]: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Main process exited,
code=exited, status=1/FAILURE
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Failed with result
'exit-code'.
Jan 15 09:46:48 thor abrt-server[28144]: Deleting problem directory
Python3-2021-01-15-09:46:48-28118 (dup of Python3-2020-09-18-14:25:13-1363)
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms
expired, scheduling restart.
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart
counter is at 242.
Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary
monitors
Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012
File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 85, in start_monitor#012 response = self._proxy.start_monitor(type, options)#012
File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__#012
return self.__send(self.__name, args)#012 File
"/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request#012
verbose=self.__verbose#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line
1154, in request#012 return self.single_request(host, handler, request_body,
verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in
single_request#012 http_conn = self.send_request(host, handler, request_body,
verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in
send_request#012 self.send_content(connection, request_body)#012 File
"/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_content#012
connection.endheaders(request_body)#012 File
"/usr/lib64/python3.6/http/client.py", line 1264, in endheaders#012
self._send_output(message_body, encode_chunked=encode_chunked)#012 File
"/usr/lib64/python3.6/http/client.py", line 1040, in _send_output#012
self.send(msg)#012 File "/usr/lib64/python3.6/http/client.py", line 978, in
send#012 self.connect()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line
74, in connect#012 self.sock.connect(base64.b16decode(self.host))#012FileNotFoundError:
[Errno 2] No such file or directory#012#012During handling of the above exception, another
exception occurred:#012#012Traceback (most recent call last):#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
131, in _run_agent#012 return action(he)#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line
55, in action_proper#012 return he.start_monitoring()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 437, in start_monitoring#012 self._initialize_broker()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py",
line 561, in _initialize_broker#012 m.get('options', {}))#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py",
line 91, in start_monitor#012 ).format(t=type, o=options,
e=e)#012ovirt_hosted_engine_ha.lib.exceptions.RequestError: brokerlink - failed to start
monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor:
'network', options: {'addr': '172.16.100.1',
'network_test': 'dns', 'tcp_t_address': '',
'tcp_t_port': ''}]
Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent
Jan 15 09:46:48 thor abrt-server[28144]: /bin/sh: reporter-systemd-journal: command not
found
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Main process exited, code=exited,
status=157/n/a
Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Failed with result
'exit-code'.
Jan 15 09:46:49 thor vdsm[8421]: WARN Failed to retrieve Hosted Engine HA info, is Hosted
Engine setup finished?
Jan 15 09:46:50 thor systemd[1]: Started Session c449 of user root.
Jan 15 09:46:50 thor systemd[1]: session-c449.scope: Succeeded.
Jan 15 09:46:50 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor]
does not exist on server localhost
Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno
107] Transport endpoint is not connected:
'/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata'
Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012
File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py",
line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py",
line 97, in __init__#012 self._backend.connect()#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path,
service_link)#012 File
"/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py",
line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107]
Transport endpoint is not connected:
'/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata'
Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker
ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker
Jan 15 09:46:53 thor platform-python[28165]: detected unhandled Python exception in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 15 09:46:53 thor abrt-server[28199]: Not saving repeating crash in
'/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker'
Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Main process exited,
code=exited, status=1/FAILURE
Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Failed with result
'exit-code'.
Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms
expired, scheduling restart.
Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart
counter is at 243.
Jan 15 09:46:53 thor systemd[1]: Stopped oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:53 thor systemd[1]: Started oVirt Hosted Engine High Availability
Communications Broker.
Jan 15 09:46:55 thor systemd[1]: Started Session c450 of user root.
Jan 15 09:46:55 thor systemd[1]: session-c450.scope: Succeeded.
Jan 15 09:46:55 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor]
does not exist on server localhost
Questions:
1) I have two important VMs that have snapshots that I need to boot up. Is their a means
with an HCI configuration to manually start the VMs without oVirt engine being up?
2) Is their a means to debug what is going on with the engine failing to start to repair
(I hate reloading as the only fix for systems)
3) Is their a means to re-deploy HCI setup wizard, but use the "engine" volume
and so retain the VMs and templates?