Out-of-sync networks can only be detached
Hi, I have a 3-node oVirt cluster. I have configured 2 logical networks: ovirtmgmt and public. Public logical network is attached in only 2 nodes and failing to attach on the 3rd node with the below error Invalid operation, out-of-sync network 'public' can only be detached. Please have been stuck on this for almost the whole day now. How do I fix this error? -- Regards, Sakhi Hadebe
On Tue, 9 Oct 2018 13:24:51 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
Hi,
I have a 3-node oVirt cluster. I have configured 2 logical networks: ovirtmgmt and public. Public logical network is attached in only 2 nodes and failing to attach on the 3rd node with the below error Invalid operation, out-of-sync network 'public' can only be detached.
Please have been stuck on this for almost the whole day now. How do I fix this error?
The error message in the UI might be include the wrong network name. I guess the network ovirtmgmt is out-of-sync on the 3rd node. Why the network is out of sync, is shown as a tooltip if you hover the mouse pointer over ovirtmgmt in "Compute > Hosts > 3rd host > Network Interfaces > Setup Host Networks" If the shown information does not help you, please share a screenshot of this dialog. If there is a line like: The following Network definitions on the Network Interface are different than those on the Logical Network. Please synchronize the Network Interface before editing network ${NETWORK_NOT_IN_SYNC}. The non-synchronized values are\: ${OUT_OF_SYNC_VALUES}. in engine.log, please share this line, too.
Hi, The main issue here is the Logical that cannot be attached to the 3rd node (please check attached AttachErr.png file), while it has attached to the other two nodes. The Logical Network configurations are the same in all 3 networks., please check the logical configurations in attached scrnshot2.png file Here are the logs grepped from engine.log file: [root@engine ~]# tail -f /var/log/ovirt-engine/engine.log | egrep "${OUT_OF_SYNC_VALUES} | goku.sanren.ac.za" 2018-10-11 13:52:20,779+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:52:20,779+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:52:20,780+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 722583f2-b11a-11e8-9a47-00163e5858df Type: StoragePool 2018-10-11 13:52:20,784+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update template OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated templates OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to remove unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully removed unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,791+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock free to object 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:55:38,215+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]', sharedLocks=''}' 2018-10-11 13:55:38,226+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Running command: SyncNetworkProviderCommand internal: true. 2018-10-11 13:55:38,327+02 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-342) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2018-10-11 13:55:38,483+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock freed to object 'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]', sharedLocks=''}' On Tue, Oct 9, 2018 at 5:01 PM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 9 Oct 2018 13:24:51 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
Hi,
I have a 3-node oVirt cluster. I have configured 2 logical networks: ovirtmgmt and public. Public logical network is attached in only 2 nodes and failing to attach on the 3rd node with the below error Invalid operation, out-of-sync network 'public' can only be detached.
Please have been stuck on this for almost the whole day now. How do I fix this error?
The error message in the UI might be include the wrong network name. I guess the network ovirtmgmt is out-of-sync on the 3rd node. Why the network is out of sync, is shown as a tooltip if you hover the mouse pointer over ovirtmgmt in "Compute > Hosts > 3rd host > Network Interfaces > Setup Host Networks" If the shown information does not help you, please share a screenshot of this dialog.
If there is a line like: The following Network definitions on the Network Interface are different than those on the Logical Network. Please synchronize the Network Interface before editing network ${NETWORK_NOT_IN_SYNC}. The non-synchronized values are\: ${OUT_OF_SYNC_VALUES}. in engine.log, please share this line, too.
-- Regards, Sakhi Hadebe Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR Tel: +27 12 841 2308 <+27128414213> Fax: +27 12 841 4223 <+27128414223> Cell: +27 71 331 9622 <+27823034657> Email: sakhi@sanren.ac.za <shadebe@csir.co.za>
On Thu, 11 Oct 2018 14:00:50 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
Hi,
The main issue here is the Logical that cannot be attached to the 3rd node (please check attached AttachErr.png file), while it has attached to the other two nodes.
In the first step the configuration of ovirtmgmt on the 3rd node has to be synchronized. If you hover ovirtmgmt in the "Setup Host 3rd Networks", you see a list of settings which has to be adjusted. Each setting can be configured manually or the "Sync All Networks" button in "Compute >> Hosts >> 3rd >> Network Interfaces" can be used. But if the hostname (FQDN or IP address) used to add the host to oVirt does not match the desired configuration, this might not work. For this reason I recommend to: 1. Put the host in maintenance state 2. Remove the host from oVirt 3. Configure manually em1 (or em2 or bond0) to use the desired IP address and VLAN for ovirtmgmt network 4. Add the host to oVirt using this IP address as hostname 5. Configure all attributes of ovirtmgmt, which are marked as "Out-of-sync" via oVirt 6. ovirtmgmt should be synchronized now, so public can now be added to the same interface (em1, em2 or bond0 if bonded), if public has another VLAN tag
The Logical Network configurations are the same in all 3 networks., please check the logical configurations in attached scrnshot2.png file
Not really, the configuration shown in scrnshot2.png is not valid, because it is not allowed to attach two logical networks with the same VLAN id (VLAN 20) to the same NIC/bond. Maybe ovirtmgmt configuration on the host is configured like public should be?
Here are the logs grepped from engine.log file:
[root@engine ~]# tail -f /var/log/ovirt-engine/engine.log | egrep "${OUT_OF_SYNC_VALUES} | goku.sanren.ac.za" 2018-10-11 13:52:20,779+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:52:20,779+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:52:20,780+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 722583f2-b11a-11e8-9a47-00163e5858df Type: StoragePool 2018-10-11 13:52:20,784+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update template OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated templates OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to remove unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully removed unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,791+02 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock free to object 'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]', sharedLocks=''}' 2018-10-11 13:55:38,215+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock Acquired to object 'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]', sharedLocks=''}' 2018-10-11 13:55:38,226+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Running command: SyncNetworkProviderCommand internal: true. 2018-10-11 13:55:38,327+02 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-342) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2018-10-11 13:55:38,483+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock freed to object 'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]', sharedLocks=''}'
On Tue, Oct 9, 2018 at 5:01 PM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 9 Oct 2018 13:24:51 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
[...]
fix [...]
The error message in the UI might be include the wrong network name. I guess the network ovirtmgmt is out-of-sync on the 3rd node. Why the network is out of sync, is shown as a tooltip if you hover the mouse pointer over ovirtmgmt in "Compute > Hosts > 3rd host > Network Interfaces > Setup Host Networks" If the shown information does not help you, please share a screenshot of this dialog.
If there is a line like: The following Network definitions on the Network Interface are different than those on the Logical Network. Please synchronize the Network Interface before editing network ${NETWORK_NOT_IN_SYNC}. The non-synchronized values are\: ${OUT_OF_SYNC_VALUES}. in engine.log, please share this line, too.
Hi Dominik, Thank you for your help. No i can attach all logical networks The problem was the configurations on the Data Center and the hosts. They were not the same. After changing everything to be the same, it worked fine. On Thu, Oct 11, 2018 at 3:52 PM Dominik Holler <dholler@redhat.com> wrote:
On Thu, 11 Oct 2018 14:00:50 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
Hi,
The main issue here is the Logical that cannot be attached to the 3rd node (please check attached AttachErr.png file), while it has attached to the other two nodes.
In the first step the configuration of ovirtmgmt on the 3rd node has to be synchronized. If you hover ovirtmgmt in the "Setup Host 3rd Networks", you see a list of settings which has to be adjusted. Each setting can be configured manually or the "Sync All Networks" button in "Compute >> Hosts >> 3rd >> Network Interfaces" can be used. But if the hostname (FQDN or IP address) used to add the host to oVirt does not match the desired configuration, this might not work. For this reason I recommend to: 1. Put the host in maintenance state 2. Remove the host from oVirt 3. Configure manually em1 (or em2 or bond0) to use the desired IP address and VLAN for ovirtmgmt network 4. Add the host to oVirt using this IP address as hostname 5. Configure all attributes of ovirtmgmt, which are marked as "Out-of-sync" via oVirt 6. ovirtmgmt should be synchronized now, so public can now be added to the same interface (em1, em2 or bond0 if bonded), if public has another VLAN tag
The Logical Network configurations are the same in all 3 networks., please check the logical configurations in attached scrnshot2.png file
Not really, the configuration shown in scrnshot2.png is not valid, because it is not allowed to attach two logical networks with the same VLAN id (VLAN 20) to the same NIC/bond. Maybe ovirtmgmt configuration on the host is configured like public should be?
Here are the logs grepped from engine.log file:
[root@engine ~]# tail -f /var/log/ovirt-engine/engine.log | egrep "${OUT_OF_SYNC_VALUES} | goku.sanren.ac.za" 2018-10-11 13:52:20,779+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Before acquiring and wait lock
'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]',
sharedLocks=''}' 2018-10-11 13:52:20,779+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock-wait acquired to object
'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]',
sharedLocks=''}' 2018-10-11 13:52:20,780+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 722583f2-b11a-11e8-9a47-00163e5858df Type: StoragePool 2018-10-11 13:52:20,784+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated VM OVFs in Data Center 'Default' 2018-10-11 13:52:20,789+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to update template OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully updated templates OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Attemptin to remove unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,790+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Successfully removed unneeded template/vm OVFs in Data Center 'Default' 2018-10-11 13:52:20,791+02 INFO
[org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [555cc9ce] Lock free to object
'EngineLock:{exclusiveLocks='[722583f2-b11a-11e8-9a47-00163e5858df=OVF_UPDATE]',
sharedLocks=''}' 2018-10-11 13:55:38,215+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock Acquired to object
'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]',
sharedLocks=''}' 2018-10-11 13:55:38,226+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Running command: SyncNetworkProviderCommand internal: true. 2018-10-11 13:55:38,327+02 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-342) [] User admin@internal successfully logged in with scopes: ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search ovirt-ext=token-info:validate ovirt-ext=token:password-access 2018-10-11 13:55:38,483+02 INFO [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-65) [550b0278] Lock freed to object
'EngineLock:{exclusiveLocks='[1a7e6539-4a99-4051-9998-803883d118f2=PROVIDER]',
sharedLocks=''}'
On Tue, Oct 9, 2018 at 5:01 PM Dominik Holler <dholler@redhat.com> wrote:
On Tue, 9 Oct 2018 13:24:51 +0200 Sakhi Hadebe <sakhi@sanren.ac.za> wrote:
[...]
fix [...]
The error message in the UI might be include the wrong network name. I guess the network ovirtmgmt is out-of-sync on the 3rd node. Why the network is out of sync, is shown as a tooltip if you hover the mouse pointer over ovirtmgmt in "Compute > Hosts > 3rd host > Network Interfaces > Setup Host Networks" If the shown information does not help you, please share a screenshot of this dialog.
If there is a line like: The following Network definitions on the Network Interface are different than those on the Logical Network. Please synchronize the Network Interface before editing network ${NETWORK_NOT_IN_SYNC}. The non-synchronized values are\: ${OUT_OF_SYNC_VALUES}. in engine.log, please share this line, too.
-- Regards, Sakhi Hadebe Engineer: South African National Research Network (SANReN)Competency Area, Meraka, CSIR Tel: +27 12 841 2308 <+27128414213> Fax: +27 12 841 4223 <+27128414223> Cell: +27 71 331 9622 <+27823034657> Email: sakhi@sanren.ac.za <shadebe@csir.co.za>
Hi Sakhi, Can you please help here.Facing same issue as Cannot setup Networks. The following Network definitions on the Network Interface are different than those on the Logical Network. Please synchronize the Network Interface before editing network ovirtmgmt. Thanks, Srini
participants (3)
-
Dominik Holler -
Sakhi Hadebe -
Srini vasan