I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky 8.6 system and am having and issue where "ovirt-hosted-engine-setup" is failing when it tries to create the ovirtmgmt network with the error "error: Must be number, not str"}]". When this happens, the engine setup pauses and if I can login to the bootstrap engine UI and when I attempt to manually assign the ovirtmgmt network to the correct nic on the host, I get the same error message. This server has (2) active network interfaces - a gigabit NIC that will be a VM network for all networks except gluster and migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for gluster and migration (I previously had these servers in the same hardware configuration running oVirt 4.3 on CentOS 7 and would like to have the same setup again - just with latest versions of EL and oVirt).

I don't believe it's related, however for transparency I should note that the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64) because both native EL and elrepo support for my Infiniband HBA was dropped in the standard EL8 kernel due to known bugs with that version of the kernel. Thanks in advance for any assistance.

Here is the specific error from engine.log on the bootstrap engine. I see similar messages in vdsm.log on the host.

2022-09-04 18:01:10,725-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START, HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 6bc2c376
2022-09-04 18:01:10,726-04 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH, HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in 'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN  [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com' from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 (Failed with error unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO  [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]', sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to configure management network on host vmh1.my.domain.com due to setup networks failure.