oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
2:18 p.m.
Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic':
'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic':
'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'})'
execution
failed: VDSGenericException: VDSErrorException: Failed to
HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks
finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com'
from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603 (Failed with error
unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]',
sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to
configure management network on host vmh1.my.domain.com due to setup
networks failure.
6:01 p.m.
New subject: oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
It appears that this error is somehow related to the infiniband network. I
found that if I did not assign an IP to the ib0 adapter then
ovirt-hosted-engine-setup would complete successfully. Once the cluster was
up, I could go in and create a new non-vm network, assign gluster and
migration roles, configure the interface and assign the new network to the
ib0 interface. I could then add my other (2) hosts via the UI, however they
would initially come up as NonOperational and the "error: must be number,
not str" would be logged. I could then go into "Setup Host Networks" and
manually assign both networks and successfully activate the hosts.
I'm not sure if issues will arise during future oVirt host updates,
however everything is working fine at this point.
On Mon, Sep 5, 2022 at 7:18 AM Thomas Simmons <twsnnva(a)gmail.com> wrote:
Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null',
nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False,
'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments:
({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False,
'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101',
'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway':
'10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120,
'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not
str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120',
networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true',
stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE',
ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null',
nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'})'
execution
failed: VDSGenericException: VDSErrorException: Failed to
HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks
finished. Lock released. Monitoring can run now for host '
vmh1.my.domain.com' from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6':
False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True,
'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603 (Failed with error
unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]',
sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to
configure management network on host vmh1.my.domain.com due to setup
networks failure.
808
days inactive
809
days old
1 comments
1 participants
participants (1)
-
Thomas Simmons