oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
by Thomas Simmons
Hello,
I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky
8.6 system and am having and issue where "ovirt-hosted-engine-setup" is
failing when it tries to create the ovirtmgmt network with the error
"error: Must be number, not str"}]". When this happens, the engine setup
pauses and if I can login to the bootstrap engine UI and when I attempt to
manually assign the ovirtmgmt network to the correct nic on the host, I get
the same error message. This server has (2) active network interfaces - a
gigabit NIC that will be a VM network for all networks except gluster and
migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for
gluster and migration (I previously had these servers in the same hardware
configuration running oVirt 4.3 on CentOS 7 and would like to have the same
setup again - just with latest versions of EL and oVirt).
I don't believe it's related, however for transparency I should note that
the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64)
because both native EL and elrepo support for my Infiniband HBA was dropped
in the standard EL8 kernel due to known bugs with that version of the
kernel. Thanks in advance for any assistance.
Here is the specific error from engine.log on the bootstrap engine. I see
similar messages in vdsm.log on the host.
2022-09-04 18:01:10,725-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START,
HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id:
6bc2c376
2022-09-04 18:01:10,726-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH,
HostSetupNetworksVDSCommand, return: , log id: 6bc2c376
2022-09-04 18:01:11,251-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,252-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in
'HostSetupNetworksVDS' method
2022-09-04 18:01:11,252-04 WARN
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return
value: Status [code=-32603, message=Internal JSON-RPC error: {'reason':
"Attempt to call function: <bound method Global.setupNetworks of
<vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt':
{'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0',
'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True,
'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500,
'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess':
True, 'connectivityCheck': 'true'}) error: Must be number, not str"}]
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command
HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,261-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com,
HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10',
vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]',
rollbackOnFailure='true', commitOnSuccess='true',
connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true',
bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt',
nicName='enp3s0', vlan='null', vmNetwork='true', stp='false',
properties='null', ipv4BootProtocol='STATIC_IP',
ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0',
ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null',
ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]',
removedNetworks='[]', bonds='[]', removedBonds='[]',
clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution
failed: VDSGenericException: VDSErrorException: Failed to
HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt
to call function: <bound method Global.setupNetworks of <vdsm.API.Global
object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603
2022-09-04 18:01:11,262-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks
finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com'
from data-center 'Default'
2022-09-04 18:01:11,262-04 ERROR
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command
'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS,
error = Internal JSON-RPC error: {'reason': "Attempt to call function:
<bound method Global.setupNetworks of <vdsm.API.Global object at
0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask':
'255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true',
'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP':
'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {},
{'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck':
'true'}) error: Must be number, not str"}, code = -32603 (Failed with error
unexpected and code 16)
2022-09-04 18:01:11,263-04 INFO
[org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object
'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]',
sharedLocks=''}'
2022-09-04 18:01:11,269-04 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID:
SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to
configure management network on host vmh1.my.domain.com due to setup
networks failure.
1 year, 7 months
VM has been paused due to no Storage space error on ovirt 4.5
by Jirka Simon
Hi there,
last days we are facing issues with paused VMs (in past it was for few
second to resize lv device), but now it doesn't resume. we migrated to
4.5.2 cluster, this never happened before with the same storage.
there is almost notning in engine log
2022-09-06 09:47:11,160+02 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-9) [51eb7178] VM
'cfff0648-6502-4977-95a8-c6f95c723f6d'(cm1.util.prod.hq.slde
v.cz) moved from 'Up' --> 'Paused'
2022-09-06 09:47:11,264+02 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED(1,025), VM
cm1.util.prod.hq.
sldev.cz has been paused.
2022-09-06 09:47:11,271+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED_ENOSPC(138), VM
cm1.util.pro
d.hq.sldev.cz has been paused due to no Storage space error.
but there are erros with LVM in vdsmlog. (attached)
ovirt 4.5.2
OS - ovirt-node-ng.
Thank you for any hint.
Jirka
1 year, 7 months
storage domain
by izidor.stokin＠gmail.com
Hello,
One of our admins accidentally added one of available LUN disks to storage domain. Now we are not able to remove it as additional storage from it.
There was no data written on it.
on host it was removed from vg group (vgreduce) , but on site it is still in use (greyed out) (vgreduce had no issues removing pvdisk from it)
multipath -ll shows it as unused,
dmsetup status shows some values related to this lun:
OS: rhel 7.9
package ovirt 4.3
cluster: yes
Any idea how to remove it without corrupting datadomain ?
is it safe to remove it with dmsetum remove?
BRI
1 year, 7 months
Some problems with ovirt
by אריה קלטר
Hi,
I have two ovirt clusters.
1. 4.3 cluster with single node (was 3 nodes in the past)
2. 4.5.2 cluster with 3 nodes. the problem was also on oVirt-4.5 node with a matching engine version but the cluster configured on version 4.6 because I did have a problem upgrading one of the nodes, I also lost the engine because of gluster volume sync issue between the hosts, so I formatted the problematic node using the 4.5.2 node iso and created a new engine on it, and attached 2 of the existing nodes and did upgrade on the nodes using the new engine.
About the new cluster:
1. migration between hosts - almost all the time looks like it succeeds, but the VM is not responding after the successful migration.
2. run reboot inside of the VM - in a high percent of the time it just hung instead of reboot. With 100% CPU. In status "Reboot in Progress", but after some time the status changed to "Up" incorrectly. I tried on the clean images of ubuntu and centos from ovirt-image-repository. I imported a server from the 4.3 cluster to the new cluster, on the old cluster it reboots correctly but in the new cluster it doesn't. I tried it both with gluster and with nfs.
Any idea how to solve both problems?
Regards, Arye
1 year, 7 months
Re: How many oVirt cluster, hosts and VMs do you have running?
by Jonas Lindholm
I was looking at those studies but not sure how old they are.
What I can see is that our oVirt deployment seems bigger than any of those
studies but I can be wrong.
------ Original Message ------
Received: 01:08 AM MDT, 09/02/2022
From: Sandro Bonazzola <sbonazzo(a)redhat.com>
To: jlhm(a)usa.net Cc: oVirt Users <users(a)ovirt.org>
Subject:
Re: [ovirt-users] How many oVirt cluster, hosts and VMs do you have running?
> Would something like an oVirt counter help?
> https://github.com/oVirt/ovirt-site/pull/3056
>
>
> Il giorno ven 2 set 2022 alle ore 08:26 ha scritto:
>
> > Hi, just trying to understand how our oVirt deployment compare to others.
> > - 75 cluster (and 75 Data centers as we map them 1 to 1 due to security
> > requirements) spanning +3 data centers
> > - 328 oVirt servers
> > - +1900 VMs running
> >
> > Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs
> > RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or
> > Rocky 8)/oVirt 4.4.
> > When that is done we will start to upgrade to latest oVirt version but it
> > takes time due to the size of our environment and we move slowly to
ensure
> > stability.
> > _______________________________________________
> > Users mailing list -- users(a)ovirt.org
> > To unsubscribe send an email to users-leave(a)ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
https://lists.ovirt.org/archives/list/users@ovirt.org/message/LM3GRRJDXVM...
> >
>
>
> --
>
> Sandro Bonazzola
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D PERFORMANCE & SCALE
>
> Red Hat EMEA
>
> sbonazzo(a)redhat.com
>
>
> *Red Hat respects your work life balance. Therefore there is no need to
> answer this email out of your office hours.*
>
1 year, 7 months
How many oVirt cluster, hosts and VMs do you have running?
by jlhm＠usa.net
Hi, just trying to understand how our oVirt deployment compare to others.
- 75 cluster (and 75 Data centers as we map them 1 to 1 due to security requirements) spanning +3 data centers
- 328 oVirt servers
- +1900 VMs running
Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or Rocky 8)/oVirt 4.4.
When that is done we will start to upgrade to latest oVirt version but it takes time due to the size of our environment and we move slowly to ensure stability.
1 year, 7 months
Ubuntu NFS
by thilburn＠generalpacific.com
Hello,
I was having trouble with getting an Ubuntu 22.04 NFS share working and after searching for hours I was able to figure out what was needed. Below is what I found if anyone else runs into this.
My error was
engine.log
"...Unexpected return value: Status [code=701, message=Could not initialize cluster lock: ()]"
Host
supervdsm.log
-open error -13 EACCES: no permission to open /ThePath/ids
-check that daemon user sanlock *** group sanlock *** has access to disk or file.
The fix was
changing /etc/nfs.conf manage-gids=y ( Which is the default ) to # manage-gids=y ( Commenting this sets the default which is no )
It looks like in the past the fix was to change /etc/default/nfs-kernel-server Line RPCMOUNTDOPTS="--manage-gids" which I didn't need to change.
1 year, 7 months
how kill backup operation
by Diego Ercolani
Hello I saw there are other thread asking how to delete disk snapshots from backup operation.
We definitively need a tool to kill pending backup operations and locked snapshots.
I Think this is very frustrating ovirt is a good piece of software but it's very immature in a dirty asyncronous world.
We need a unified toolbox to clean manually and do database housekeeping.
1 year, 7 months
Interested in contributing with Spanish translation.
by Luis Pereida
Hello,
My name is Luis Pereida, I am Mexican, from Guadalajara and I am currently
an application security specialist.
Some time ago I met the O-Virt project and it helped me a lot to solve many
situations where virtualization was the perfect option.
Since some time ago I have been thinking about how to contribute to the
project, and talking with some friends, they would like to have
documentation in Spanish. Although English is basic for us, many times the
context or expressions are hard to understand.
I would like to help with that. How can I do it? I see that it is necessary
to use a Zanata account. How can I get an account?
Regards and thanks for being so supportive of the community.
1 year, 7 months
