12:36 p.m.
Hi,
I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2 non-operational node
hosts, which seem relate to ghost network glitch.
Everything worked fine for a very long time until I did the following.
1) Made a backup of several VMs which have network “CloudLink-ISP2”.
2) Moved VM disks into another node hosts, which don’t have link “CloudLink-ISP2”, and
forgot (preliminary to move) to re-configure network interface (remove link to
“CloudLink-ISP2”) as I did before..
3) Now I have 2 hosts in non-operational mode which is a really big problem - there are
only 3 nodes total in a cluster.
I tried to fix this by adding 3rd network “CloudLink-ISP2” to each non-operational hosts,
and connected ethernet interfaces to a switch.
Unfortunately, it didn’t helped either because for whatever reason link did not go up.
How I can remove any reference to “CloudLink-ISP2” from node11 and node14? They don’t have
any VMs needed that “CloudLink-ISP2”. I edited even inactive VMs and removed
“CloudLink-ISP2”.
May be there is an option to ignore missing or downed links whn activating node hosts?
Thanks in advance for any help.
——————————
Logs:
2022-04-27 11:16:59,769+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328] Host
'node14' is set to Non-Operational, it is missing the following networks:
'CloudLink-ISP2'
2022-04-27 11:16:59,790+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5ab20328]
EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node14 does not comply with the
cluster ClusterRiga11 networks, the following networks are missing on host:
'CloudLink-ISP2'
2022-04-27 11:16:59,884+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [55610e1e]
EVENT_ID: VDS_DETECTED(13), Status of host node14 was set to NonOperational.
2022-04-27 11:16:59,907+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51) [5d6ef791] Host
'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is already in NonOperational status
for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped.
2022-04-27 11:12:26,050+03 ERROR [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e] Host
'node11' is set to Non-Operational, it is missing the following networks:
'CloudLink-ISP2'
2022-04-27 11:12:26,070+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [5400352e]
EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519), Host node11 does not comply with the
cluster ClusterRiga11 networks, the following networks are missing on host:
'CloudLink-ISP2'
2022-04-27 11:12:26,192+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [57d79e54]
EVENT_ID: VDS_DETECTED(13), Status of host node11 was set to NonOperational.
2022-04-27 11:12:26,214+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19) [151821e3] Host
'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is already in NonOperational status
for reason 'NETWORK_UNREACHABLE'. SetNonOperationalVds command is skipped.
2022-04-27 11:51:10,539+03 INFO [org.ovirt.engine.core.vdsbroker.VdsManager]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Clearing
domains data for host node14
2022-04-27 11:51:10,539+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] Host
'node14' moved to Non-Operational state because interface/s which are down are
needed by required network/s in the current cluster: 'eno3 (CloudLink-ISP2)'
2022-04-27 11:51:10,569+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62) [] EVENT_ID:
VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host node14 moved to Non-Operational state because
interfaces which are down are needed by required networks in the current cluster:
'eno3 (CloudLink-ISP2)'.
7:45 a.m.
New subject: Q: Non-Operational Node Hosts - Ghost Network Problem
Your logs suggest that VDSM hasn't detected the removal of the network
yet. Did you try clicking on the "Sync Host Networks" button on the
non-operational hosts?
If you've changed the network config, VDSM may have failed to set up
the host properly. What does the output of "ip addr show" on these
hosts?
Manually removing the network from the hosts is only a temporary fix.
VDSM will attempt to recreate the network on restart if VDSM's config
isn't sync'd with the engine.
-Patrick Hibbs
On Wed, 2022-04-27 at 12:36 +0300, Andrei Verovski wrote:
Hi,
I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2
non-operational node hosts, which seem relate to ghost network
glitch.
Everything worked fine for a very long time until I did the
following.
1) Made a backup of several VMs which have network “CloudLink-ISP2”.
2) Moved VM disks into another node hosts, which don’t have link
“CloudLink-ISP2”, and forgot (preliminary to move) to re-configure
network interface (remove link to “CloudLink-ISP2”) as I did before..
3) Now I have 2 hosts in non-operational mode which is a really big
problem - there are only 3 nodes total in a cluster.
I tried to fix this by adding 3rd network “CloudLink-ISP2” to each
non-operational hosts, and connected ethernet interfaces to a switch.
Unfortunately, it didn’t helped either because for whatever reason
link did not go up.
How I can remove any reference to “CloudLink-ISP2” from node11 and
node14? They don’t have any VMs needed that “CloudLink-ISP2”. I
edited even inactive VMs and removed “CloudLink-ISP2”.
May be there is an option to ignore missing or downed links whn
activating node hosts?
Thanks in advance for any help.
——————————
Logs:
2022-04-27 11:16:59,769+03 ERROR
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51)
[5ab20328] Host 'node14' is set to Non-Operational, it is missing the
following networks: 'CloudLink-ISP2'
2022-04-27 11:16:59,790+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
Thread-51) [5ab20328] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519),
Host node14 does not comply with the cluster ClusterRiga11 networks,
the following networks are missing on host: 'CloudLink-ISP2'
2022-04-27 11:16:59,884+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
Thread-51) [55610e1e] EVENT_ID: VDS_DETECTED(13), Status of host
node14 was set to NonOperational.
2022-04-27 11:16:59,907+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51)
[5d6ef791] Host 'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is
already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
SetNonOperationalVds command is skipped.
2022-04-27 11:12:26,050+03 ERROR
[org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19)
[5400352e] Host 'node11' is set to Non-Operational, it is missing the
following networks: 'CloudLink-ISP2'
2022-04-27 11:12:26,070+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
Thread-19) [5400352e] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519),
Host node11 does not comply with the cluster ClusterRiga11 networks,
the following networks are missing on host: 'CloudLink-ISP2'
2022-04-27 11:12:26,192+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
Thread-19) [57d79e54] EVENT_ID: VDS_DETECTED(13), Status of host
node11 was set to NonOperational.
2022-04-27 11:12:26,214+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19)
[151821e3] Host 'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is
already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
SetNonOperationalVds command is skipped.
2022-04-27 11:51:10,539+03 INFO
[org.ovirt.engine.core.vdsbroker.VdsManager] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62)
[] Clearing domains data for host node14
2022-04-27 11:51:10,539+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62)
[] Host 'node14' moved to Non-Operational state because interface/s
which are down are needed by required network/s in the current
cluster: 'eno3 (CloudLink-ISP2)'
2022-04-27 11:51:10,569+03 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
Thread-62) [] EVENT_ID: VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host
node14 moved to Non-Operational state because interfaces which are
down are needed by required networks in the current cluster: 'eno3
(CloudLink-ISP2)'.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EGENEHBHJYP...
4:02 p.m.
New subject: Q: Non-Operational Node Hosts - Ghost Network Problem
Hi,
I solved this problem, added network “CloudLink-ISP2” to 2 problematic nodes and connected
respective Ethernet interfaces to a separate switch bringing bringing up.
On 28 Apr 2022, at 07:45, Patrick Hibbs
<hibbsncc1701(a)gmail.com> wrote:
Your logs suggest that VDSM hasn't detected the removal of the network
yet. Did you try clicking on the "Sync Host Networks" button on the
non-operational hosts?
If you've changed the network config, VDSM may have failed to set up
the host properly. What does the output of "ip addr show" on these
hosts?
Manually removing the network from the hosts is only a temporary fix.
VDSM will attempt to recreate the network on restart if VDSM's config
isn't sync'd with the engine.
-Patrick Hibbs
On Wed, 2022-04-27 at 12:36 +0300, Andrei Verovski wrote:
> Hi,
>
>
> I run into nasty and unexpected problem (oVirt 4.4.7.6-1.el8 ) and 2
> non-operational node hosts, which seem relate to ghost network
> glitch.
> Everything worked fine for a very long time until I did the
> following.
>
> 1) Made a backup of several VMs which have network “CloudLink-ISP2”.
> 2) Moved VM disks into another node hosts, which don’t have link
> “CloudLink-ISP2”, and forgot (preliminary to move) to re-configure
> network interface (remove link to “CloudLink-ISP2”) as I did before..
> 3) Now I have 2 hosts in non-operational mode which is a really big
> problem - there are only 3 nodes total in a cluster.
>
> I tried to fix this by adding 3rd network “CloudLink-ISP2” to each
> non-operational hosts, and connected ethernet interfaces to a switch.
> Unfortunately, it didn’t helped either because for whatever reason
> link did not go up.
>
> How I can remove any reference to “CloudLink-ISP2” from node11 and
> node14? They don’t have any VMs needed that “CloudLink-ISP2”. I
> edited even inactive VMs and removed “CloudLink-ISP2”.
> May be there is an option to ignore missing or downed links whn
> activating node hosts?
>
> Thanks in advance for any help.
>
>
> ——————————
>
> Logs:
>
> 2022-04-27 11:16:59,769+03 ERROR
> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51)
> [5ab20328] Host 'node14' is set to Non-Operational, it is missing the
> following networks: 'CloudLink-ISP2'
> 2022-04-27 11:16:59,790+03 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
> Thread-51) [5ab20328] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519),
> Host node14 does not comply with the cluster ClusterRiga11 networks,
> the following networks are missing on host: 'CloudLink-ISP2'
> 2022-04-27 11:16:59,884+03 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
> Thread-51) [55610e1e] EVENT_ID: VDS_DETECTED(13), Status of host
> node14 was set to NonOperational.
> 2022-04-27 11:16:59,907+03 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-51)
> [5d6ef791] Host 'node14'(aa871d44-94e2-4fdb-aeb3-ca0ae8dc568f) is
> already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
> SetNonOperationalVds command is skipped.
>
> 2022-04-27 11:12:26,050+03 ERROR
> [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19)
> [5400352e] Host 'node11' is set to Non-Operational, it is missing the
> following networks: 'CloudLink-ISP2'
> 2022-04-27 11:12:26,070+03 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
> Thread-19) [5400352e] EVENT_ID: VDS_SET_NONOPERATIONAL_NETWORK(519),
> Host node11 does not comply with the cluster ClusterRiga11 networks,
> the following networks are missing on host: 'CloudLink-ISP2'
> 2022-04-27 11:12:26,192+03 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
> Thread-19) [57d79e54] EVENT_ID: VDS_DETECTED(13), Status of host
> node11 was set to NonOperational.
> 2022-04-27 11:12:26,214+03 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-19)
> [151821e3] Host 'node11'(3c854f9c-2cdd-423e-bca0-37964ba76702) is
> already in NonOperational status for reason 'NETWORK_UNREACHABLE'.
> SetNonOperationalVds command is skipped.
>
> 2022-04-27 11:51:10,539+03 INFO
> [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62)
> [] Clearing domains data for host node14
> 2022-04-27 11:51:10,539+03 INFO
> [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-
> ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-62)
> [] Host 'node14' moved to Non-Operational state because interface/s
> which are down are needed by required network/s in the current
> cluster: 'eno3 (CloudLink-ISP2)'
> 2022-04-27 11:51:10,569+03 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector
> ] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-
> Thread-62) [] EVENT_ID: VDS_SET_NONOPERATIONAL_IFACE_DOWN(603), Host
> node14 moved to Non-Operational state because interfaces which are
> down are needed by required networks in the current cluster: 'eno3
> (CloudLink-ISP2)'.
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EGENEHBHJYP...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/Z34DAZFEBAO...
939
days inactive
940
days old
2 comments
2 participants
participants (2)
-
Andrei Verovski -
Patrick Hibbs