[ovirt-users] Q: Non-Operational Node Hosts - Ghost Network Problem

Hi, I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2 non-operational node hosts, which seem relate to ghost network glitch. Everything worked fine for a very long time until I did the following. 1) Made a backup of several VMs which have network “CloudLink-ISP2”. 2) Moved VM disks into another node hosts, which don’t have link “CloudLink-ISP2”, and forgot (preliminary to move) to re-configure network interface (remove link to “CloudLink-ISP2”) as I did before.. 3) Now I have 2 hosts in non-operational mode which is a really big problem - there are only 3 nodes total in a cluster. I tried to fix this by adding 3rd network “CloudLink-ISP2” to each non-operational hosts, and connected ethernet interfaces to a switch. Unfortunately, it didn’t helped either because for whatever reason link did not go up. How I can remove any reference to “CloudLink-ISP2” from node11 and node14? They don’t have any VMs needed that “CloudLink-ISP2”. I edited even inactive VMs and removed “CloudLink-ISP2”. May be there is an option to ignore missing or downed links whn activating node hosts? Thanks in advance for any help. —————————— Logs: 2022-04-27 11:16:59,769+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328] Host 'node14' is set to Non-Operational, it is missing the following networks: 'CloudLink-ISP2' 2022-04-27 11:16:59,790+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node14 does not comply with the cluster ClusterRiga11 networks, the following networks are missing on host: 'CloudLink-ISP2' 2022-04-27 11:16:59,884+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [55610e1e] EVENT_ID: VDS_DETECTED(13), Status of host node14 was set to NonOperational. 2022-04-27 11:16:59,907+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5d6ef791] Host 'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is already in NonOperational status for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped. 2022-04-27 11:12:26,050+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e] Host 'node11' is set to Non-Operational, it is missing the following networks: 'CloudLink-ISP2' 2022-04-27 11:12:26,070+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node11 does not comply with the cluster ClusterRiga11 networks, the following networks are missing on host: 'CloudLink-ISP2' 2022-04-27 11:12:26,192+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [57d79e54] EVENT_ID: VDS_DETECTED(13), Status of host node11 was set to NonOperational. 2022-04-27 11:12:26,214+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [151821e3] Host 'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is already in NonOperational status for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped. 2022-04-27 11:51:10,539+03 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Clearing domains data for host node14 2022-04-27 11:51:10,539+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Host 'node14' moved to Non-Operational state because interface/s which are down are needed by required network/s in the current cluster: 'eno3 (CloudLink-ISP2)' 2022-04-27 11:51:10,569+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] EVENT_ID: VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host node14 moved to Non-Operational state because interfaces which are down are needed by required networks in the current cluster: 'eno3 (CloudLink-ISP2)'.