oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
Hello, I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky 8.6 system and am having and issue where "ovirt-hosted-engine-setup" is failing when it tries to create the ovirtmgmt network with the error "error: Must be number, not str"}]". When this happens, the engine setup pauses and if I can login to the bootstrap engine UI and when I attempt to manually assign the ovirtmgmt network to the correct nic on the host, I get the same error message. This server has (2) active network interfaces - a gigabit NIC that will be a VM network for all networks except gluster and migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for gluster and migration (I previously had these servers in the same hardware configuration running oVirt 4.3 on CentOS 7 and would like to have the same setup again - just with latest versions of EL and oVirt). I don't believe it's related, however for transparency I should note that the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64) because both native EL and elrepo support for my Infiniband HBA was dropped in the standard EL8 kernel due to known bugs with that version of the kernel. Thanks in advance for any assistance. Here is the specific error from engine.log on the bootstrap engine. I see similar messages in vdsm.log on the host. 2022-09-04 18:01:10,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START, HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 6bc2c376 2022-09-04 18:01:10,726-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH, HostSetupNetworksVDSCommand, return: , log id: 6bc2c376 2022-09-04 18:01:11,251-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in 'HostSetupNetworksVDS' method 2022-09-04 18:01:11,252-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"} 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,262-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com' from data-center 'Default' 2022-09-04 18:01:11,262-04 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 (Failed with error unexpected and code 16) 2022-09-04 18:01:11,263-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]', sharedLocks=''}' 2022-09-04 18:01:11,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to configure management network on host vmh1.my.domain.com due to setup networks failure.
It appears that this error is somehow related to the infiniband network. I found that if I did not assign an IP to the ib0 adapter then ovirt-hosted-engine-setup would complete successfully. Once the cluster was up, I could go in and create a new non-vm network, assign gluster and migration roles, configure the interface and assign the new network to the ib0 interface. I could then add my other (2) hosts via the UI, however they would initially come up as NonOperational and the "error: must be number, not str" would be logged. I could then go into "Setup Host Networks" and manually assign both networks and successfully activate the hosts. I'm not sure if issues will arise during future oVirt host updates, however everything is working fine at this point. On Mon, Sep 5, 2022 at 7:18 AM Thomas Simmons <twsnnva@gmail.com> wrote:
Hello, I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky 8.6 system and am having and issue where "ovirt-hosted-engine-setup" is failing when it tries to create the ovirtmgmt network with the error "error: Must be number, not str"}]". When this happens, the engine setup pauses and if I can login to the bootstrap engine UI and when I attempt to manually assign the ovirtmgmt network to the correct nic on the host, I get the same error message. This server has (2) active network interfaces - a gigabit NIC that will be a VM network for all networks except gluster and migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for gluster and migration (I previously had these servers in the same hardware configuration running oVirt 4.3 on CentOS 7 and would like to have the same setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64) because both native EL and elrepo support for my Infiniband HBA was dropped in the standard EL8 kernel due to known bugs with that version of the kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START, HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 6bc2c376 2022-09-04 18:01:10,726-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH, HostSetupNetworksVDSCommand, return: , log id: 6bc2c376 2022-09-04 18:01:11,251-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in 'HostSetupNetworksVDS' method 2022-09-04 18:01:11,252-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"} 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,262-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks finished. Lock released. Monitoring can run now for host ' vmh1.my.domain.com' from data-center 'Default' 2022-09-04 18:01:11,262-04 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 (Failed with error unexpected and code 16) 2022-09-04 18:01:11,263-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]', sharedLocks=''}' 2022-09-04 18:01:11,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to configure management network on host vmh1.my.domain.com due to setup networks failure.
participants (1)
-
Thomas Simmons