July 2024 - Users - oVirt List Archives

rocky 9.4 hypervisor - failed to create ovirmgmt iface
by marek 12 Jul '24

12 Jul '24

hi, today i tried to add new hypervisor - rocky 9.4 ovirt engine - 4.5 ovirtmgmt iface is not functional when i try add host to cluster it exist for while (seconds) but then i see message "ovirtmgmt: port 1(eno3) entered disabled state" and ovirtmgmt iface is gone and network is unreacheable i have working hypervisors based on rocky 9.3 i tried downgrade nmstate to version from 9.3 dnf downgrade https://dl.rockylinux.org/vault/rocky/9.3/AppStream/x86_64/os/Packages/p/py… https://dl.rockylinux.org/vault/rocky/9.3/AppStream/x86_64/os/Packages/n/nm… https://dl.rockylinux.org/vault/rocky/9.3/AppStream/x86_64/os/Packages/n/nm… any ideas/tips howto debug ovirmgmt iface creation process? vdsm.log 2024-07-11 17:42:07,305+0200 INFO (jsonrpc/6) [api.network] START setupNetworks(networks={'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'eno3', 'bridged': 'true', 'ipaddr': '10.10.10.237', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, bondings={}, options={'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) from=::ffff:10.10.12.12,33842 (api:31) 2024-07-11 17:43:07,312+0200 WARN (vdsm.Scheduler) [Executor] Worker blocked: <Worker name=jsonrpc/6 running <Task <JsonRpcTask {'jsonrpc': '2.0', 'method': 'Host.setupNetworks', 'params': {'networks': {'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'eno3', 'bridged': 'true', 'ipaddr': '10.10.10.237', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'mtu': 1500, 'switch': 'legacy'}}, 'bondings': {}, 'options': {'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}}, 'id': '65b89a79-0e7a-4a05-b310-f9903bdec2ce'} at 0x7f5e8c245b20> timeout=60, duration=60.01 at 0x7f5e8c245df0> task#=0 at 0x7f5e8c27eaf0>, traceback: File: "/usr/lib64/python3.9/threading.py", line 937, in _bootstrap self._bootstrap_inner() File: "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner self.run() File: "/usr/lib64/python3.9/threading.py", line 917, in run self._target(*self._args, **self._kwargs) File: "/usr/lib/python3.9/site-packages/vdsm/common/concurrent.py", line 243, in run ret = func(*args, **kwargs) File: "/usr/lib/python3.9/site-packages/vdsm/executor.py", line 284, in _run self._execute_task() File: "/usr/lib/python3.9/site-packages/vdsm/executor.py", line 298, in _execute_task task() File: "/usr/lib/python3.9/site-packages/vdsm/executor.py", line 374, in __call__ self._callable() File: "/usr/lib/python3.9/site-packages/yajsonrpc/__init__.py", line 253, in __call__ self._handler(self._ctx, self._req) File: "/usr/lib/python3.9/site-packages/yajsonrpc/__init__.py", line 296, in _serveRequest response = self._handle_request(req, ctx) File: "/usr/lib/python3.9/site-packages/yajsonrpc/__init__.py", line 338, in _handle_request res = method(**params) File: "/usr/lib/python3.9/site-packages/vdsm/rpc/Bridge.py", line 186, in _dynamicMethod result = fn(*methodArgs) File: "<decorator-gen-508>", line 2, in setupNetworks File: "/usr/lib/python3.9/site-packages/vdsm/common/api.py", line 33, in method ret = func(*args, **kwargs) File: "/usr/lib/python3.9/site-packages/vdsm/API.py", line 1576, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File: "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 38, in __call__ return callMethod() File: "/usr/lib/python3.9/site-packages/vdsm/common/supervdsm.py", line 35, in <lambda> getattr(self._supervdsmProxy._svdsm, self._funcName)(*args, File: "<string>", line 2, in setupNetworks File: "/usr/lib64/python3.9/multiprocessing/managers.py", line 810, in _callmethod kind, result = conn.recv() File: "/usr/lib64/python3.9/multiprocessing/connection.py", line 254, in recv buf = self._recv_bytes() File: "/usr/lib64/python3.9/multiprocessing/connection.py", line 418, in _recv_bytes buf = self._recv(4) File: "/usr/lib64/python3.9/multiprocessing/connection.py", line 383, in _recv chunk = read(handle, remaining) (executor:345) installed ovirt packages on host centos-release-ovirt45-9.2-1.el9s.noarch ovirt-vmconsole-1.0.9-3.el9.noarch ovirt-imageio-common-2.5.0-1.el9.x86_64 python3-ovirt-engine-sdk4-4.6.2-1.el9.x86_64 ovirt-openvswitch-ovn-2.17-1.el9.noarch ovirt-openvswitch-ovn-common-2.17-1.el9.noarch ovirt-imageio-client-2.5.0-1.el9.x86_64 ovirt-imageio-daemon-2.5.0-1.el9.x86_64 ovirt-openvswitch-ovn-host-2.17-1.el9.noarch ovirt-vmconsole-host-1.0.9-3.el9.noarch ovirt-python-openvswitch-2.17-1.el9.noarch ovirt-openvswitch-2.17-1.el9.noarch python3.11-ovirt-imageio-common-2.5.0-1.el9.x86_64 python3.11-ovirt-engine-sdk4-4.6.2-1.el9.x86_64 python3.11-ovirt-imageio-client-2.5.0-1.el9.x86_64 ovirt-openvswitch-ipsec-2.17-1.el9.noarch python3-ovirt-setup-lib-1.3.3-1.el9.noarch ovirt-ansible-collection-3.2.0-1.el9.noarch ovirt-hosted-engine-ha-2.5.1-1.el9.noarch ovirt-provider-ovn-driver-1.2.36-1.el9.noarch ovirt-host-dependencies-4.5.0-3.el9.x86_64 ovirt-hosted-engine-setup-2.7.1-1.el9.noarch ovirt-host-4.5.0-3.el9.x86_64

1 1

Failed to configure management network on the host.
by d_hashima＠sagaidc.co.jp 12 Jul '24

12 Jul '24

RHEL9 (KVM server) cannot be added from oVirt Engine built on centos9 stream. The oVirt management NW and the RHEL9 (KVM server) management NW are in the same segment and are in an environment where they can resolve each other's names. I'm new to oVirt, so I'd appreciate it if you could explain it in an easy-to-understand manner.

2 1

Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
by Brent Saner 11 Jul '24

11 Jul '24

Whoops, sent before I finished replying- I'll look into first trying the Nightly, to see if that has the same behavior, and then trying the _setupAuth patching et. al. to see if I can narrow down what might be happening here. Brent Saner SENIOR SYSTEMS ENGINEER Follow us on LinkedIn! brent.saner(a)netfire.com 855-696-3834 Ext. 110 www.netfire.com ________________________________ From: Brent Saner <brent.saner(a)netfire.com> Sent: Thursday, July 11, 2024 5:38 PM To: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute" Looking into this, I can confirm that *after* the packages are installed and *before* engine-setup is run, /etc/ovirt-engine/aaa/ does not even exist: # ls /etc/ovirt-engine/aaa ls: cannot access '/etc/ovirt-engine/aaa': No such file or directory Which may be expected, I'm unclear on that. Further details inline below: ________________________________ From: Yedidyah Bar David <didi(a)redhat.com> Sent: Thursday, July 4, 2024 2:13 AM To: Brent Saner <brent.saner(a)netfire.com> Cc: users(a)ovirt.org <users(a)ovirt.org> Subject: Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute" You don't often get email from didi(a)redhat.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Caution: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Thu, Jul 4, 2024 at 9:11 AM Yedidyah Bar David <didi(a)redhat.com<mailto:didi@redhat.com>> wrote: On Wed, Jun 19, 2024 at 10:38 PM Brent S. <brent.saner(a)netfire.com<mailto:brent.saner@netfire.com>> wrote: As a quick update to this: # ovirt-aaa-jdbc-tool Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Jun 19, 2024 7:28:14 PM org.ovirt.engine.extension.aaa.jdbc.binding.cli.Cli main SEVERE: Unexpected Exception invoking Cli: Could not read properties from: /etc/ovirt-engine/aaa/internal.properties Which is, of course, the same message in the log. This is probably expected, since *engine-setup never actually created the file*: Are you sure about this? Presumably it was removed, but I do know the directory is not present before engine-setup is run and the file is not present after. # ls -la /etc/ovirt-engine/aaa total 4 drwxr-xr-x. 2 root root 6 Jun 19 19:27 . drwxr-xr-x. 18 root root 4096 Jun 19 19:27 .. # I guess you checked the above only after engine-setup failed/finished, right? At the time, yes; see above inlines; the directory itself does not even exist before engine-setup. And: 2024-06-19 19:27:10,917+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:923 execute-output: ['/usr/share/ovirt-engine-extension-aaa-jdbc/dbscripts/schema.sh', '-s', '[REDACTED_REMOTE_DB_HOST]', '-p', '5432', '-u', '[REDACTED_REMOTE_DB_USER]', '-d', '[REDACTED_REMOTE_DB_NAME]', '-e', 'aaa_jdbc', '-l', '/root/ovirt-engine-setup.log', '-c', 'apply'] stderr: 2024-06-19 19:27:10,917+0000 DEBUG otopi.transaction transaction._prepare:61 preparing 'File transaction for '/etc/ovirt-engine/aaa/internal.properties'' 2024-06-19 19:27:10,917+0000 DEBUG otopi.filetransaction filetransaction.prepare:184 file '/etc/ovirt-engine/aaa/internal.properties' missing Indeed (SNIP) If you grep ovirt-engine sources, you'll find internal.properties in: packaging/setup/ovirt_engine_setup/engine/constants.py: AAA_JDBC_CONFIG_DB = os.path.join( OVIRT_ENGINE_SYSCONFDIR, 'aaa', 'internal.properties' ) If you then grep for AAA_JDBC_CONFIG_DB, you see it in: packaging/setup/plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py: def _setupAuth(self): self.environment[otopicons.CoreEnv.MAIN_TRANSACTION].append( filetransaction.FileTransaction( name=oenginecons.FileLocations.AAA_JDBC_CONFIG_DB, ... visibleButUnsafe=True, Forgot to mention: You can check otopi sources:src/otopi/filetransaction.py to see what this means. ... def _setupAdminUser(self): toolArgs = ( oenginecons.FileLocations.AAA_JDBC_TOOL, '--db-config=%s' % oenginecons.FileLocations.AAA_JDBC_CONFIG_DB, ) ... @plugin.event( stage=plugin.Stages.STAGE_MISC, name=AAA_JDBC_SETUP_ADMIN_USER, after=( oengcommcons.Stages.DB_SCHEMA, oengcommcons.Stages.DB_CONNECTION_AVAILABLE, oenginecons.Stages.CONFIG_EXTENSIONS_UPGRADE, ), before=( oenginecons.Stages.CONFIG_AAA_ADMIN_USER_SETUP, ), condition=lambda self: self.environment[ oenginecons.ConfigEnv.ADMIN_USER_AUTHZ_TYPE ] == self.AAA_JDBC_AUTHZ_TYPE, ) def _misc(self): # TODO: if we knew that aaa-jdbc package was upgraded by engine-setup # TODO: we could display summary note that custom profiles have to be # TODO: upgraded manually self._setupSchema() self._setupAuth() self._setupAdminUser() ... This means that: At STAGE_MISC, _misc calls _setupAuth, which creates this file, and then it calls _setupAdminUser which tries to use it. Latter fails, and engine-setup rolls back the MAIN_TRANSACTION, including removing the file. I'd start debugging this issue by: 1. Patching _setupAuth to wait (e.g. using dialog.queryBoolean, search the source for examples) after it creates the file, so that I can investigate it 2. Patching _setupAdminUser to wait after it runs the tool, so that I can try to investigate the failure - e.g. run it myself under strace, if the existing logging is not enough. You can try using the otopi plugin wait_on_error for this, instead of patching. Good luck and best regards, -- Didi -- Didi

1 0

Recall: Re: Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
by Brent Saner 11 Jul '24

11 Jul '24

brent.saner(a)netfire.com would like to recall the message, "Re: [ovirt-users] Re: engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"". Brent Saner SENIOR SYSTEMS ENGINEER Follow us on LinkedIn! brent.saner(a)netfire.com 855-696-3834 Ext. 110 www.netfire.com

1 0

engine-setup fails: "Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute"
by brent.saner＠netfire.com 11 Jul '24

11 Jul '24

I can find no reason why this would be happening. AlmaLinux 9.4, oVirt 4.5.6, brand new/fresh cluster/install. From the log, these seem to be the only relevant lines: 2024-06-18 22:02:52,720+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:918 execute-output: ('/usr/bin/ovirt-aaa-jdbc-tool', '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'query', '--what= user', '--pattern=name=admin') stdout: 2024-06-18 22:02:52,720+0000 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.config.aaajdbc plugin.execute:923 execute-output: ('/usr/bin/ovirt-aaa-jdbc-tool', '--db-config=/etc/ovirt-engine/aaa/internal.properties', 'query', '--what= user', '--pattern=name=admin') stderr: Picked up JAVA_TOOL_OPTIONS: -Dcom.redhat.fips=false Jun 18, 2024 10:02:52 PM org.ovirt.engine.extension.aaa.jdbc.binding.cli.Cli main SEVERE: Unexpected Exception invoking Cli: com/ongres/scram/common/stringprep/StringPreparation 2024-06-18 22:02:52,720+0000 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/otopi/context.py", line 132, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py", line 414, in _misc self._setupAdminUser() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py", line 298, in _setupAdminUser if not self._userExists( File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/config/aaajdbc.py", line 49, in _userExists rc, stdout, stderr = self.execute( File "/usr/lib/python3.9/site-packages/otopi/plugin.py", line 929, in execute raise RuntimeError( RuntimeError: Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute 2024-06-18 22:02:52,721+0000 ERROR otopi.context context._executeMethod:151 Failed to execute stage 'Misc configuration': Command '/usr/bin/ovirt-aaa-jdbc-tool' failed to execute 2024-06-18 22:02:52,721+0000 DEBUG otopi.transaction transaction.abort:124 aborting 'DNF Transaction' Any ideas?

5 7

Get Host Capabilities failed: Connection timeout for host
by Frederico Madeira 11 Jul '24

11 Jul '24

Hi guys I'm getting these messages from engine-gui interfaces.: Get Host Capabilities failed: Connection timeout for host My hosts are in irresponsible state My VMs are in unknown state My Storage are in inactive state From the engine, I did successfully : ping ip and name and ssh connect to host my hosts, so I didn´t have network connectivity problems. error in engine.log 2024-07-10 03:47:30,088-03 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM servernode10 command Get Host Capabilities failed: Connection timeout for host '10.180.18.116', last response arrived 6609 ms ago. 2024-07-10 03:47:30,089-03 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedThreadFactory-engineScheduled-Thread-13) [] Unable to RefreshCapabilities: VDSNetworkException: VDSGenericException: VDSNetworkException: Connection timeout for host '10.180.18.116', last response arrived 6609 ms ago. 2024-07-10 03:49:45,027-03 INFO Any hint how to troubleshoot this issue? Regards,*o Madeira * fred(a)madeira.eng.br www.madeira.eng.br Cisco CCNA, LPIC-1, LPIC-2 Registered GNU/Linux nº 206120 GPG-Key-ID: 1024D/0F0A721D Key fingerprint = C424 D86B 57D5 BE55 767A 6ED1 53F8 254E 0F0A 721D MSN: fttmadeira(a)hotmail.com GTalk:fmadeira@gmail.com SKYPE: fred_madeira

1 0

VM live migration fails after update to qemu-kvm-9.0.0-4.el9.x86_64
by g.vasilopoulos＠uoc.gr 11 Jul '24

11 Jul '24

Hello After updating a hypervisor with qemu-kvm-9.0.0-4.el9.x86_64 live migration stopped working with errors related to the cpu topology. When I downgraded to qemu-kvm-9.0.0-3.el9.x86_64 migration worked again. Is this a known bug? Has anyone else encountered this?

3 3

vm stays locked after failed export
by h aanst 10 Jul '24

10 Jul '24

Hi, Running ovirt for many years, but this baffles me. An export was taking too long, seems to hang. Due to the heat a switch failedt, so export storage is umounted, at this time. Now my VM has a little lock, and can't be restarted. (shutdown worked) Checked all DB entries and vdsm. update vm_static set template_status = 0 where template_status = 1; select * from vm_static where template_status = 1; 0 update vm_dynamic set status = 0 where status = 15; select * from vm_dynamic where status = 15; 0 update images set imagestatus = 1 where imagestatus = 2; select * from images where imagestatus = 2; 0 update snapshots set status = 'OK' where status like 'LOCKED'; select * from snapshots where status like 'LOCKED'; 0 UPDATE images SET imagestatus = 1 WHERE imagestatus = 4; select * from images where imagestatus = 4; 0 restarted the hosted-engine I dont know what to check next. Any hints? thanks

1 0

CentOS9 Stream -- Only way forward?
by eshwayri＠gmail.com 10 Jul '24

10 Jul '24

I've been running 4.5.4-1 on RHEL8 for a while now; been running fine for my needs. I was getting around the unholy dependency issue caused by the centos-release package conflicting with the redhat-release package by upgrading everything else -- not it. It's been necessary to do this since RHEL versions were higher than any CentOS 8 Stream versions. This time round since CentOS has been archived, I made sure to install anything I hadn't and then manually purged centos-release from the rpm database. This allowed RHEL to update to 8.10, including the release package this time. Not expecting an issue since there won't be any future patches coming from that repo. If I decide to stick with oVirt and install daily snapshots, I assume what I will have to do is setup a new CentOS Steam 9 server, create a backup of the old server, and then restore on the new server. Like what I did when I was forced to migrate from CentOS 8 to RHEL8 (with CentOS 8 Stream packages). Is this right? I haven't decided if I want to do this. CentOS Stream 9 is supported until mid 2027; RHEL 8.10 is supported until mid 2029. Barring any new features or major incompatibility, I really don't see a pressing reason to do this. In fact, with no one stepping in to take over the project, there is a better chance of being supported by sticking with RHEL 8.10 (2+ years).

5 5

VM Migration Failed
by KSNull Zero 10 Jul '24

10 Jul '24

Running oVirt 4.4.5 VM cannot migrate between hosts. vdsm.log contains the following error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ovhost01.local/system: authentication failed: Failed to verify peer's certificate Certificates on hosts was renewed some time ago. How this issue can be fixed ? Thank you.

4 6