3 node cluster. Gluster for shared storage. CentOS8 Updated to CentOS 8 Streams :P -> https://bugzilla.redhat.com/show_bug.cgi?id=1911910 After several weeks .. I am really in need of direction to get this fixed. I saw several postings about oVirt package issues but not found a fix. [root@thor ~]# dnf update Last metadata expiration check: 2:54:29 ago on Fri 15 Jan 2021 06:49:16 AM EST. Error: Problem 1: package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-bridge-234-1.el8.x86_64 conflicts with cockpit-dashboard < 233 provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-host-4.4.1-4.el8.x86_64 - cannot install the best update candidate for package cockpit-bridge-217-1.el8.x86_64 Problem 2: problem with installed package ovirt-host-4.4.1-4.el8.x86_64 - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package cockpit-dashboard-217-1.el8.noarch Problem 3: package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch requires ovirt-host >= 4.4.0, but none of the providers can be installed - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best update candidate for package ovirt-hosted-engine-setup-2.4.9-1.el8.noarch - cannot install the best update candidate for package cockpit-system-217-1.el8.noarch (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) [root@thor ~]# yum install cockpit-dashboard --nobest Last metadata expiration check: 2:54:52 ago on Fri 15 Jan 2021 06:49:16 AM EST. Package cockpit-dashboard-217-1.el8.noarch is already installed. Dependencies resolved. Problem: problem with installed package ovirt-host-4.4.1-4.el8.x86_64 - package ovirt-host-4.4.1-4.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-1.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-2.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package ovirt-host-4.4.1-3.el8.x86_64 requires cockpit-dashboard, but none of the providers can be installed - package cockpit-system-234-1.el8.noarch obsoletes cockpit-dashboard provided by cockpit-dashboard-217-1.el8.noarch - cannot install the best candidate for the job ========================================================================================================================================================================================================================================= Package Architecture Version Repository Size ========================================================================================================================================================================================================================================= Skipping packages with broken dependencies: ovirt-host x86_64 4.4.1-1.el8 ovirt-4.4 13 k ovirt-host x86_64 4.4.1-2.el8 ovirt-4.4 13 k ovirt-host x86_64 4.4.1-3.el8 ovirt-4.4 13 k Transaction Summary ========================================================================================================================================================================================================================================= Skip 3 Packages Nothing to do. Complete! [root@thor ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk └─WDC_WDS100T2B0B-00YS70_19106A802926 253:3 0 931.5G 0 mpath └─vdo_2926 253:5 0 4T 0 vdo /gluster_bricks/gv0 sdb 8:16 0 931.5G 0 disk └─WDC_WDS100T2B0B-00YS70_192490801828 253:4 0 931.5G 0 mpath sdc 8:32 0 477G 0 disk └─vdo_sdc 253:6 0 2.1T 0 vdo ├─gluster_vg_sdc-gluster_lv_engine 253:7 0 100G 0 lvm /gluster_bricks/engine ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tmeta 253:8 0 1G 0 lvm │ └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm │ ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm │ ├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm /gluster_bricks/data │ └─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm /gluster_bricks/vmstore └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc_tdata 253:9 0 2T 0 lvm └─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc-tpool 253:10 0 2T 0 lvm ├─gluster_vg_sdc-gluster_thinpool_gluster_vg_sdc 253:11 0 2T 1 lvm ├─gluster_vg_sdc-gluster_lv_data 253:12 0 1000G 0 lvm /gluster_bricks/data └─gluster_vg_sdc-gluster_lv_vmstore 253:13 0 1000G 0 lvm /gluster_bricks/vmstore sdd 8:48 1 58.8G 0 disk ├─sdd1 8:49 1 1G 0 part /boot └─sdd2 8:50 1 57.8G 0 part ├─cl-root 253:0 0 36.1G 0 lvm / ├─cl-swap 253:1 0 4G 0 lvm [SWAP] └─cl-home 253:2 0 17.6G 0 lvm /home [root@thor ~]# mount |grep engine /dev/mapper/gluster_vg_sdc-gluster_lv_engine on /gluster_bricks/engine type xfs (rw,noatime,nodiratime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota,_netdev,x-systemd.requires=vdo.service) thorst.penguinpages.local:/engine on /media/engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev) thorst.penguinpages.local:/engine on /rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072,_netdev,x-systemd.device-timeout=0) [root@thor ~]# [root@thor ~]# tail -50 /var/log/messages Jan 15 09:46:43 thor platform-python[28088]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:43 thor abrt-server[28116]: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:43 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 241. Jan 15 09:46:43 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:43 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:45 thor systemd[1]: Started Session c448 of user root. Jan 15 09:46:45 thor systemd[1]: session-c448.scope: Succeeded. Jan 15 09:46:45 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Service RestartSec=10s expired, scheduling restart. Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Scheduled restart job, restart counter is at 211. Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Monitoring Agent. Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability Monitoring Agent. Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 97, in __init__#012 self._backend.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path, service_link)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:48 thor journal[28118]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker Jan 15 09:46:48 thor platform-python[28118]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:48 thor abrt-server[28144]: Deleting problem directory Python3-2021-01-15-09:46:48-28118 (dup of Python3-2020-09-18-14:25:13-1363) Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:48 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 242. Jan 15 09:46:48 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:48 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Failed to start necessary monitors Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 85, in start_monitor#012 response = self._proxy.start_monitor(type, options)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in __call__#012 return self.__send(self.__name, args)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in __request#012 verbose=self.__verbose#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1154, in request#012 return self.single_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1166, in single_request#012 http_conn = self.send_request(host, handler, request_body, verbose)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1279, in send_request#012 self.send_content(connection, request_body)#012 File "/usr/lib64/python3.6/xmlrpc/client.py", line 1309, in send_content#012 connection.endheaders(request_body)#012 File "/usr/lib64/python3.6/http/client.py", line 1264, in endheaders#012 self._send_output(message_body, encode_chunked=encode_chunked)#012 File "/usr/lib64/python3.6/http/client.py", line 1040, in _send_output#012 self.send(msg)#012 File "/usr/lib64/python3.6/http/client.py", line 978, in send#012 self.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/unixrpc.py", line 74, in connect#012 self.sock.connect(base64.b16decode(self.host))#012FileNotFoundError: [Errno 2] No such file or directory#012#012During handling of the above exception, another exception occurred:#012#012Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 131, in _run_agent#012 return action(he)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", line 55, in action_proper#012 return he.start_monitoring()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 437, in start_monitoring#012 self._initialize_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", line 561, in _initialize_broker#012 m.get('options', {}))#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 91, in start_monitor#012 ).format(t=type, o=options, e=e)#012ovirt_hosted_engine_ha.lib.exceptions.RequestError: brokerlink - failed to start monitor via ovirt-ha-broker: [Errno 2] No such file or directory, [monitor: 'network', options: {'addr': '172.16.100.1', 'network_test': 'dns', 'tcp_t_address': '', 'tcp_t_port': ''}] Jan 15 09:46:48 thor journal[28140]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Trying to restart agent Jan 15 09:46:48 thor abrt-server[28144]: /bin/sh: reporter-systemd-journal: command not found Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Main process exited, code=exited, status=157/n/a Jan 15 09:46:48 thor systemd[1]: ovirt-ha-agent.service: Failed with result 'exit-code'. Jan 15 09:46:49 thor vdsm[8421]: WARN Failed to retrieve Hosted Engine HA info, is Hosted Engine setup finished? Jan 15 09:46:50 thor systemd[1]: Started Session c449 of user root. Jan 15 09:46:50 thor systemd[1]: session-c449.scope: Succeeded. Jan 15 09:46:50 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Failed initializing the broker: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Traceback (most recent call last):#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 64, in run#012 self._storage_broker_instance = self._get_storage_broker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/broker.py", line 143, in _get_storage_broker#012 return storage_broker.StorageBroker()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 97, in __init__#012 self._backend.connect()#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 408, in connect#012 self._check_symlinks(self._storage_path, volume.path, service_link)#012 File "/usr/lib/python3.6/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 105, in _check_symlinks#012 os.unlink(service_link)#012OSError: [Errno 107] Transport endpoint is not connected: '/rhev/data-center/mnt/glusterSD/thorst.penguinpages.local:_engine/3afc47ba-afb9-413f-8de5-8d9a2f45ecde/ha_agent/hosted-engine.metadata' Jan 15 09:46:53 thor journal[28165]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.broker.Broker ERROR Trying to restart the broker Jan 15 09:46:53 thor platform-python[28165]: detected unhandled Python exception in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:53 thor abrt-server[28199]: Not saving repeating crash in '/usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker' Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Main process exited, code=exited, status=1/FAILURE Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Failed with result 'exit-code'. Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Service RestartSec=100ms expired, scheduling restart. Jan 15 09:46:53 thor systemd[1]: ovirt-ha-broker.service: Scheduled restart job, restart counter is at 243. Jan 15 09:46:53 thor systemd[1]: Stopped oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:53 thor systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Jan 15 09:46:55 thor systemd[1]: Started Session c450 of user root. Jan 15 09:46:55 thor systemd[1]: session-c450.scope: Succeeded. Jan 15 09:46:55 thor upsmon[2232]: Poll UPS [nutmonitor@localhost] failed - [nutmonitor] does not exist on server localhost Questions: 1) I have two important VMs that have snapshots that I need to boot up. Is their a means with an HCI configuration to manually start the VMs without oVirt engine being up? 2) Is their a means to debug what is going on with the engine failing to start to repair (I hate reloading as the only fix for systems) 3) Is their a means to re-deploy HCI setup wizard, but use the "engine" volume and so retain the VMs and templates?