[ovirt-users] Cannot get Ovirt 4.5 to work, how ever I try. Virgin install: no pki ca-cert gen, restoring: no OVN connection

Hi there, over the last months I've hunkered down to update my companies antiquated Ovirt 4.3. To manage this in an orderly fashion we replicated the setup. In the update process I always arrive at the same problem. Once I managed to solve it by chance, but I cannot reproduce the solution. The setup is Ovirt Engine running on a dedicated Centos-Stream-8 virtual machine managed in VirtManager. The nodes are either OvirtNode 4.4 or 4.5. The problem exists on both. Issue1: Updating to 4.4 works without issue. Then, regardless whether I update by restoring to Ovirt 4.5 or by updating the engine through the update path networks stop functioning and, very peculiarly I get a very strange keymap in the vm console. It's no real keymap. It's quertz, but # resolves as 3 and all kind of strange stuff. However, this can be resolved on individual basis by setting the vm-console keymap to de (german). Connected hosts and new hosts always dispaly "OVN connected: No". The error log hints at some kind of ssl error. I either get dropping connections, or protocol miss-matches in the node log. I deactivated Ovirt4.4-repositories on the engine and did a distro-sync, because I found an old bug-report that implicated protocol mismatched may result from unclean python-library versioning. I reenrolled certificates, I reinstalled the host and still cannot get a connection: Logs on host: /var/log/ovn-controller.log: 2023-12-19T11:27:14.245Z|00018|memory|INFO|6604 kB peak resident set size after 15.1 seconds 2023-12-19T11:27:14.245Z|00019|memory|INFO|idl-cells:100 2023-12-19T11:29:34.483Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log 2023-12-19T11:29:34.512Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting... 2023-12-19T11:29:34.513Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected 2023-12-19T11:29:34.517Z|00004|main|INFO|OVN internal version is : [21.12.3-20.21.0-61.4] 2023-12-19T11:29:34.517Z|00005|main|INFO|OVS IDL reconnected, force recompute. 2023-12-19T11:29:34.573Z|00006|reconnect|INFO|ssl:127.0.0.1:6642: connecting... 2023-12-19T11:29:34.573Z|00007|main|INFO|OVNSB IDL reconnected, force recompute. 2023-12-19T11:29:34.573Z|00008|reconnect|INFO|ssl:127.0.0.1:6642: connection attempt failed (Connection refused) 2023-12-19T11:29:35.575Z|00009|reconnect|INFO|ssl:127.0.0.1:6642: connecting... 2023-12-19T11:29:35.589Z|00010|reconnect|INFO|ssl:127.0.0.1:6642: connection attempt failed (Connection refused) 2023-12-19T11:29:35.589Z|00011|reconnect|INFO|ssl:127.0.0.1:6642: waiting 2 seconds before reconnect 2023-12-19T11:29:37.592Z|00012|reconnect|INFO|ssl:127.0.0.1:6642: connecting... 2023-12-19T11:29:37.592Z|00013|reconnect|INFO|ssl:127.0.0.1:6642: connection attempt failed (Connection refused) 2023-12-19T11:29:37.592Z|00014|reconnect|INFO|ssl:127.0.0.1:6642: waiting 4 seconds before reconnect 2023-12-19T11:29:41.596Z|00015|reconnect|INFO|ssl:127.0.0.1:6642: connecting... 2023-12-19T11:29:41.596Z|00016|reconnect|INFO|ssl:127.0.0.1:6642: connection attempt failed (Connection refused) 2023-12-19T11:29:41.596Z|00017|reconnect|INFO|ssl:127.0.0.1:6642: continuing to reconnect in the background but suppressing further logging /var/log/openvswitch/ovsdb-server.log: 2023-12-19T11:26:56.889Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2023-12-19T11:26:56.915Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.15.8 2023-12-19T11:27:06.922Z|00003|memory|INFO|20624 kB peak resident set size after 10.0 seconds 2023-12-19T11:27:06.922Z|00004|memory|INFO|cells:128 monitors:5 sessions:3 2023-12-19T11:29:30.771Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2023-12-19T11:29:30.813Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.15.8 2023-12-19T11:29:31.047Z|00003|jsonrpc|WARN|unix#0: receive error: Connection reset by peer 2023-12-19T11:29:31.047Z|00004|reconnect|WARN|unix#0: connection dropped (Connection reset by peer) 2023-12-19T11:29:32.821Z|00005|jsonrpc|WARN|unix#2: receive error: Connection reset by peer 2023-12-19T11:29:32.821Z|00006|reconnect|WARN|unix#2: connection dropped (Connection reset by peer) 2023-12-19T11:29:33.139Z|00007|jsonrpc|WARN|unix#4: receive error: Connection reset by peer 2023-12-19T11:29:33.139Z|00008|reconnect|WARN|unix#4: connection dropped (Connection reset by peer) 2023-12-19T11:29:40.864Z|00009|memory|INFO|23108 kB peak resident set size after 10.1 seconds 2023-12-19T11:29:40.864Z|00010|memory|INFO|cells:128 monitors:4 sessions:3 Logs on engine: /var/log/ovn/ovsdb-server-nb.log: 2023-12-18T19:36:23.056Z|00001|vlog|INFO|opened log file /var/log/ovn/ovsdb-server-nb.log 2023-12-18T19:36:23.784Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.15.8 2023-12-18T19:36:24.275Z|00003|jsonrpc|WARN|unix#0: receive error: Connection reset by peer 2023-12-18T19:36:24.276Z|00004|reconnect|WARN|unix#0: connection dropped (Connection reset by peer) 2023-12-18T19:36:33.808Z|00005|memory|INFO|22528 kB peak resident set size after 10.8 seconds 2023-12-18T19:36:33.808Z|00006|memory|INFO|cells:99 monitors:2 sessions:1 /var/log/ovirt-engine/engine.log (currently unable to start vms. normally not the case in my tests but error message seems related) 2023-12-19 06:49:17,982-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [43d1e22d] EVENT_ID: PROVIDER_SYNCHRONIZATION_STARTED(223), Provider ovirt-provider-ovn synchronization started. 2023-12-19 06:49:18,122-05 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [43d1e22d] EVENT_ID: PROVIDER_SYNCHRONIZATION_ENDED(224), Provider ovirt-provider-ovn synchronization ended. 2023-12-19 06:49:18,122-05 ERROR [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-34) [43d1e22d] Command 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' failed: EngineException: (Failed with error Unsupported or unrecognized SSL message and code 5050) Issue2: When installing ovirt4.5 engine-setup always fails in pki-phase because no new root cert is generated. I believe it ultimately say apache.ca is missing. This is also on a fresh Centos-Stream-8 machine following official install instructions. Please help. :)