September 2022 - Users - oVirt List Archives

oVirt 4.5.2 - ovirt-hosted-engine-setup fails with "error: Must be number, not str"}]" when creating ovirtmgmt network
by Thomas Simmons 06 Sep '22

06 Sep '22

Hello, I am trying to deploy the latest oVirt (4.5.2), on a fully patched Rocky 8.6 system and am having and issue where "ovirt-hosted-engine-setup" is failing when it tries to create the ovirtmgmt network with the error "error: Must be number, not str"}]". When this happens, the engine setup pauses and if I can login to the bootstrap engine UI and when I attempt to manually assign the ovirtmgmt network to the correct nic on the host, I get the same error message. This server has (2) active network interfaces - a gigabit NIC that will be a VM network for all networks except gluster and migration and a 40Gbps Infiniband adapter in connected mode (IPoIB) for gluster and migration (I previously had these servers in the same hardware configuration running oVirt 4.3 on CentOS 7 and would like to have the same setup again - just with latest versions of EL and oVirt). I don't believe it's related, however for transparency I should note that the server is running kernel-lt from elrepo (5.4.212-1.el8.elrepo.x86_64) because both native EL and elrepo support for my Infiniband HBA was dropped in the standard EL8 kernel due to known bugs with that version of the kernel. Thanks in advance for any assistance. Here is the specific error from engine.log on the bootstrap engine. I see similar messages in vdsm.log on the host. 2022-09-04 18:01:10,725-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] START, HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'}), log id: 6bc2c376 2022-09-04 18:01:10,726-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] FINISH, HostSetupNetworksVDSCommand, return: , log id: 6bc2c376 2022-09-04 18:01:11,251-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,252-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Failed in 'HostSetupNetworksVDS' method 2022-09-04 18:01:11,252-04 WARN [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Unexpected return value: Status [code=-32603, message=Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}] 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM vmh1.my.domain.com command HostSetupNetworksVDS failed: Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"} 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Error: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,261-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HostSetupNetworksVDSCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'HostSetupNetworksVDSCommand(HostName = vmh1.my.domain.com, HostSetupNetworksVdsCommandParameters:{hostId='1def9b77-b268-4a64-bac0-3e51c1d16b10', vds='Host[vmh1.my.domain.com,1def9b77-b268-4a64-bac0-3e51c1d16b10]', rollbackOnFailure='true', commitOnSuccess='true', connectivityTimeout='120', networks='[HostNetwork:{defaultRoute='true', bonding='false', networkName='ovirtmgmt', vdsmName='ovirtmgmt', nicName='enp3s0', vlan='null', vmNetwork='true', stp='false', properties='null', ipv4BootProtocol='STATIC_IP', ipv4Address='10.10.65.101', ipv4Netmask='255.255.255.0', ipv4Gateway='10.10.65.1', ipv6BootProtocol='NONE', ipv6Address='null', ipv6Prefix='null', ipv6Gateway='null', nameServers='null'}]', removedNetworks='[]', bonds='[]', removedBonds='[]', clusterSwitchType='LEGACY', managementNetworkChanged='true'})' execution failed: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 2022-09-04 18:01:11,262-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Host setup networks finished. Lock released. Monitoring can run now for host 'vmh1.my.domain.com' from data-center 'Default' 2022-09-04 18:01:11,262-04 ERROR [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Command 'org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to HostSetupNetworksVDS, error = Internal JSON-RPC error: {'reason': "Attempt to call function: <bound method Global.setupNetworks of <vdsm.API.Global object at 0x7fb24046f240>> with arguments: ({'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp3s0', 'bridged': 'true', 'ipaddr': '10.10.65.101', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '10.10.65.1', 'mtu': 1500, 'switch': 'legacy'}}, {}, {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) error: Must be number, not str"}, code = -32603 (Failed with error unexpected and code 16) 2022-09-04 18:01:11,263-04 INFO [org.ovirt.engine.core.bll.network.host.HostSetupNetworksCommand] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] Lock freed to object 'EngineLock:{exclusiveLocks='[HOST_NETWORK1def9b77-b268-4a64-bac0-3e51c1d16b10=HOST_NETWORK]', sharedLocks=''}' 2022-09-04 18:01:11,269-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [2a6921b2] EVENT_ID: SETUP_NETWORK_FAILED_FOR_MANAGEMENT_NETWORK_CONFIGURATION(1,120), Failed to configure management network on host vmh1.my.domain.com due to setup networks failure.

1 1

VM has been paused due to no Storage space error on ovirt 4.5
by Jirka Simon 06 Sep '22

06 Sep '22

Hi there, last days we are facing issues with paused VMs (in past it was for few second to resize lv device), but now it doesn't resume. we migrated to 4.5.2 cluster, this never happened before with the same storage. there is almost notning in engine log 2022-09-06 09:47:11,160+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [51eb7178] VM 'cfff0648-6502-4977-95a8-c6f95c723f6d'(cm1.util.prod.hq.slde v.cz) moved from 'Up' --> 'Paused' 2022-09-06 09:47:11,264+02 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED(1,025), VM cm1.util.prod.hq. sldev.cz has been paused. 2022-09-06 09:47:11,271+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-9) [51eb7178] EVENT_ID: VM_PAUSED_ENOSPC(138), VM cm1.util.pro d.hq.sldev.cz has been paused due to no Storage space error. but there are erros with LVM in vdsmlog. (attached) ovirt 4.5.2 OS - ovirt-node-ng. Thank you for any hint. Jirka

1 0

storage domain
by izidor.stokin＠gmail.com 05 Sep '22

05 Sep '22

Hello, One of our admins accidentally added one of available LUN disks to storage domain. Now we are not able to remove it as additional storage from it. There was no data written on it. on host it was removed from vg group (vgreduce) , but on site it is still in use (greyed out) (vgreduce had no issues removing pvdisk from it) multipath -ll shows it as unused, dmsetup status shows some values related to this lun: OS: rhel 7.9 package ovirt 4.3 cluster: yes Any idea how to remove it without corrupting datadomain ? is it safe to remove it with dmsetum remove? BRI

1 0

Some problems with ovirt
by אריה קלטר 03 Sep '22

03 Sep '22

Hi, I have two ovirt clusters. 1. 4.3 cluster with single node (was 3 nodes in the past) 2. 4.5.2 cluster with 3 nodes. the problem was also on oVirt-4.5 node with a matching engine version but the cluster configured on version 4.6 because I did have a problem upgrading one of the nodes, I also lost the engine because of gluster volume sync issue between the hosts, so I formatted the problematic node using the 4.5.2 node iso and created a new engine on it, and attached 2 of the existing nodes and did upgrade on the nodes using the new engine. About the new cluster: 1. migration between hosts - almost all the time looks like it succeeds, but the VM is not responding after the successful migration. 2. run reboot inside of the VM - in a high percent of the time it just hung instead of reboot. With 100% CPU. In status "Reboot in Progress", but after some time the status changed to "Up" incorrectly. I tried on the clean images of ubuntu and centos from ovirt-image-repository. I imported a server from the 4.3 cluster to the new cluster, on the old cluster it reboots correctly but in the new cluster it doesn't. I tried it both with gluster and with nfs. Any idea how to solve both problems? Regards, Arye

2 1

Re: How many oVirt cluster, hosts and VMs do you have running?
by Jonas Lindholm 02 Sep '22

02 Sep '22

I was looking at those studies but not sure how old they are. What I can see is that our oVirt deployment seems bigger than any of those studies but I can be wrong. ------ Original Message ------ Received: 01:08 AM MDT, 09/02/2022 From: Sandro Bonazzola <sbonazzo(a)redhat.com> To: jlhm(a)usa.net Cc: oVirt Users <users(a)ovirt.org> Subject: Re: [ovirt-users] How many oVirt cluster, hosts and VMs do you have running? > Would something like an oVirt counter help? > https://github.com/oVirt/ovirt-site/pull/3056 > > > Il giorno ven 2 set 2022 alle ore 08:26 ha scritto: > > > Hi, just trying to understand how our oVirt deployment compare to others. > > - 75 cluster (and 75 Data centers as we map them 1 to 1 due to security > > requirements) spanning +3 data centers > > - 328 oVirt servers > > - +1900 VMs running > > > > Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs > > RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or > > Rocky 8)/oVirt 4.4. > > When that is done we will start to upgrade to latest oVirt version but it > > takes time due to the size of our environment and we move slowly to ensure > > stability. > > _______________________________________________ > > Users mailing list -- users(a)ovirt.org > > To unsubscribe send an email to users-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/privacy-policy.html > > oVirt Code of Conduct: > > https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: > > https://lists.ovirt.org/archives/list/users@ovirt.org/message/LM3GRRJDXVMJI… > > > > > -- > > Sandro Bonazzola > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D PERFORMANCE & SCALE > > Red Hat EMEA > > sbonazzo(a)redhat.com > > > *Red Hat respects your work life balance. Therefore there is no need to > answer this email out of your office hours.* >

1 0

How many oVirt cluster, hosts and VMs do you have running?
by jlhm＠usa.net 02 Sep '22

02 Sep '22

Hi, just trying to understand how our oVirt deployment compare to others. - 75 cluster (and 75 Data centers as we map them 1 to 1 due to security requirements) spanning +3 data centers - 328 oVirt servers - +1900 VMs running Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or Rocky 8)/oVirt 4.4. When that is done we will start to upgrade to latest oVirt version but it takes time due to the size of our environment and we move slowly to ensure stability.

2 1

Async release for ovirt-engine (4.5.2.5) is now available
by Lev Veyde 02 Sep '22

02 Sep '22

The oVirt Team has just released a new version of ovirt-engine package (4.5.2.5) that fixes an important bug: - Bug 2122174 <https://bugzilla.redhat.com/2122174> - engine-setup on separate DWH machine fails: Failed to execute stage 'Closing up': 'NoneType' object has no attribute 'open_sftp_client' The update is already available on resources.ovirt.org and should land on oVirt mirrors within 24 hours. Thanks in advance, -- Lev Veyde Senior Software Engineer, RHCE | RHCVA | MCITP Red Hat Israel <https://www.redhat.com> lev(a)redhat.com | lveyde(a)redhat.com <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

1 0

Ubuntu NFS
by thilburn＠generalpacific.com 01 Sep '22

01 Sep '22

Hello, I was having trouble with getting an Ubuntu 22.04 NFS share working and after searching for hours I was able to figure out what was needed. Below is what I found if anyone else runs into this. My error was engine.log "...Unexpected return value: Status [code=701, message=Could not initialize cluster lock: ()]" Host supervdsm.log -open error -13 EACCES: no permission to open /ThePath/ids -check that daemon user sanlock *** group sanlock *** has access to disk or file. The fix was changing /etc/nfs.conf manage-gids=y ( Which is the default ) to # manage-gids=y ( Commenting this sets the default which is no ) It looks like in the past the fix was to change /etc/default/nfs-kernel-server Line RPCMOUNTDOPTS="--manage-gids" which I didn't need to change.

2 5

how kill backup operation
by Diego Ercolani 01 Sep '22

01 Sep '22

Hello I saw there are other thread asking how to delete disk snapshots from backup operation. We definitively need a tool to kill pending backup operations and locked snapshots. I Think this is very frustrating ovirt is a good piece of software but it's very immature in a dirty asyncronous world. We need a unified toolbox to clean manually and do database housekeeping.

3 12

Interested in contributing with Spanish translation.
by Luis Pereida 01 Sep '22

01 Sep '22

Hello, My name is Luis Pereida, I am Mexican, from Guadalajara and I am currently an application security specialist. Some time ago I met the O-Virt project and it helped me a lot to solve many situations where virtualization was the perfect option. Since some time ago I have been thinking about how to contribute to the project, and talking with some friends, they would like to have documentation in Spanish. Although English is basic for us, many times the context or expressions are hard to understand. I would like to help with that. How can I do it? I see that it is necessary to use a Zanata account. How can I get an account? Regards and thanks for being so supportive of the community.

3 2