6:18 a.m.
Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic':
'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic':
'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'})'
execution
failed: VDSGenericException: VDSErrorException: Failed to
HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks
finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com'
from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603 (Failed with error
unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]',
sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to
configure management network on host vmh1.my.domain.com due to setup
networks failure.