[ovirt-users] Re: Q: Non-Operational Node Hosts - Ghost Network Problem

On 28 Apr 2022, at 07:45, Patrick Hibbs <hibbsncc1701(a)gmail.com&gt; wrote: Your logs suggest that VDSM hasn't detected the removal of the network yet. Did you try clicking on the "Sync Host Networks" button on the non-operational hosts? If you've changed the network config, VDSM may have failed to set up the host properly. What does the output of "ip addr show" on these hosts? Manually removing the network from the hosts is only a temporary fix. VDSM will attempt to recreate the network on restart if VDSM's config isn't sync'd with the engine. -Patrick Hibbs On Wed, 2022-04-27 at 12:36 +0300, Andrei Verovski wrote: > Hi, > > > I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2 > non-operational node hosts, which seem relate to ghost network > glitch. > Everything worked fine for a very long time until I did the > following. > > 1) Made a backup of several VMs which have network “CloudLink-ISP2”. > 2) Moved VM disks into another node hosts, which don’t have link > “CloudLink-ISP2”, and forgot (preliminary to move) to re-configure > network interface (remove link to “CloudLink-ISP2”) as I did before.. > 3) Now I have 2 hosts in non-operational mode which is a really big > problem - there are only 3 nodes total in a cluster. > > I tried to fix this by adding 3rd network “CloudLink-ISP2” to each > non-operational hosts, and connected ethernet interfaces to a switch. > Unfortunately, it didn’t helped either because for whatever reason > link did not go up. > > How I can remove any reference to “CloudLink-ISP2” from node11 and > node14? They don’t have any VMs needed that “CloudLink-ISP2”. I > edited even inactive VMs and removed “CloudLink-ISP2”. > May be there is an option to ignore missing or downed links whn > activating node hosts? > > Thanks in advance for any help. > > > —————————— > > Logs: > > 2022-04-27 11:16:59,769+03 ERROR > [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) > [5ab20328] Host 'node14' is set to Non-Operational, it is missing the > following networks: 'CloudLink-ISP2' > 2022-04-27 11:16:59,790+03 WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector > ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- > Thread-51) [5ab20328] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), > Host node14 does not comply with the cluster ClusterRiga11 networks, > the following networks are missing on host: 'CloudLink-ISP2' > 2022-04-27 11:16:59,884+03 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector > ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- > Thread-51) [55610e1e] EVENT_ID: VDS_DETECTED(13), Status of host > node14 was set to NonOperational. > 2022-04-27 11:16:59,907+03 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) > [5d6ef791] Host 'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is > already in NonOperational status for reason 'NETWORK_UNREACHABLE'. > SetNonOperationalVds command is skipped. > > 2022-04-27 11:12:26,050+03 ERROR > [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) > [5400352e] Host 'node11' is set to Non-Operational, it is missing the > following networks: 'CloudLink-ISP2' > 2022-04-27 11:12:26,070+03 WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector > ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- > Thread-19) [5400352e] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), > Host node11 does not comply with the cluster ClusterRiga11 networks, > the following networks are missing on host: 'CloudLink-ISP2' > 2022-04-27 11:12:26,192+03 INFO > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector > ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- > Thread-19) [57d79e54] EVENT_ID: VDS_DETECTED(13), Status of host > node11 was set to NonOperational. > 2022-04-27 11:12:26,214+03 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) > [151821e3] Host 'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is > already in NonOperational status for reason 'NETWORK_UNREACHABLE'. > SetNonOperationalVds command is skipped. > > 2022-04-27 11:51:10,539+03 INFO > [org.ovirt.engine.core.vdsbroker.VdsManager] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) > [] Clearing domains data for host node14 > 2022-04-27 11:51:10,539+03 INFO > [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE- > ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) > [] Host 'node14' moved to Non-Operational state because interface/s > which are down are needed by required network/s in the current > cluster: 'eno3 (CloudLink-ISP2)' > 2022-04-27 11:51:10,569+03 WARN > [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector > ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool- > Thread-62) [] EVENT_ID: VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host > node14 moved to Non-Operational state because interfaces which are > down are needed by required networks in the current cluster: 'eno3 > (CloudLink-ISP2)'. > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/EGENEHBHJYP... _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z34DAZFEBAO...