4:36 a.m.
Hi,
I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2 non-operational node
hosts, which seem relate to ghost network glitch.
Everything worked fine for a very long time until I did the following.
1) Made a backup of several VMs which have network “CloudLink-ISP2”.
2) Moved VM disks into another node hosts, which don’t have link “CloudLink-ISP2”, and
forgot (preliminary to move) to re-configure network interface (remove link to
“CloudLink-ISP2”) as I did before..
3) Now I have 2 hosts in non-operational mode which is a really big problem - there are
only 3 nodes total in a cluster.
I tried to fix this by adding 3rd network “CloudLink-ISP2” to each non-operational hosts,
and connected ethernet interfaces to a switch.
Unfortunately, it didn’t helped either because for whatever reason link did not go up.
How I can remove any reference to “CloudLink-ISP2” from node11 and node14? They don’t have
any VMs needed that “CloudLink-ISP2”. I edited even inactive VMs and removed
“CloudLink-ISP2”.
May be there is an option to ignore missing or downed links whn activating node hosts?
Thanks in advance for any help.
——————————
Logs:
2022-04-27 11:16:59,769+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328] Host
'node14' is set to Non-Operational, it is missing the following networks:
'CloudLink-ISP2'
2022-04-27 11:16:59,790+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328]
EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node14 does not comply with the
cluster ClusterRiga11 networks, the following networks are missing on host:
'CloudLink-ISP2'
2022-04-27 11:16:59,884+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [55610e1e]
EVENT_ID: VDS_DETECTED(13), Status of host node14 was set to NonOperational.
2022-04-27 11:16:59,907+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5d6ef791] Host
'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is already in NonOperational status
for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped.
2022-04-27 11:12:26,050+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e] Host
'node11' is set to Non-Operational, it is missing the following networks:
'CloudLink-ISP2'
2022-04-27 11:12:26,070+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e]
EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node11 does not comply with the
cluster ClusterRiga11 networks, the following networks are missing on host:
'CloudLink-ISP2'
2022-04-27 11:12:26,192+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [57d79e54]
EVENT_ID: VDS_DETECTED(13), Status of host node11 was set to NonOperational.
2022-04-27 11:12:26,214+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [151821e3] Host
'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is already in NonOperational status
for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped.
2022-04-27 11:51:10,539+03 INFO [org.ovirt.engine.core.vdsbroker.VdsManager]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Clearing
domains data for host node14
2022-04-27 11:51:10,539+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Host
'node14' moved to Non-Operational state because interface/s which are down are
needed by required network/s in the current cluster: 'eno3 (CloudLink-ISP2)'
2022-04-27 11:51:10,569+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] EVENT_ID:
VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host node14 moved to Non-Operational state because
interfaces which are down are needed by required networks in the current cluster:
'eno3 (CloudLink-ISP2)'.