August 2022 - Users - oVirt List Archives

Hosted Engine Deployment timeout waiting for VM
by Jonas Liechti 09 Feb '26

09 Feb '26

Hello users, I am currently trying to deploy the self-hosted engine via the web interface but it seems stuck at the task "Wait for the local VM" (https://github.com/oVirt/ovirt-ansible-collection/blob/master/roles/hosted_…) I am unsure what to look at for getting more info as I haven't worked a lot with Ansible before. Do you have any idea how to debug? The temporary IP is added to /etc/hosts and I can also login to the VM via SSH: [root@server-005 ~]# cat /etc/hosts 192.168.1.97 ovirt-engine-test.admin.int.rabe.ch # temporary entry added by hosted-engine-setup for the bootstrap VM 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.128.16.5 server-005.admin.int.rabe.ch 10.128.16.6 server-006.admin.int.rabe.ch 10.128.16.7 server-007.admin.int.rabe.ch #10.128.32.2 ovirt-engine-test.admin.int.rabe.ch 10.132.16.5 server-005.storage.int.rabe.ch 10.132.16.6 server-006.storage.int.rabe.ch 10.132.16.7 server-007.storage.int.rabe.ch [root@server-005 ~]# ssh ovirt-engine-test.admin.int.rabe.ch root(a)ovirt-engine-test.admin.int.rabe.ch's password: Web console: https://ovirt-engine-test.admin.int.rabe.ch:9090/ or https://192.168.1.97:9090/ Last login: Mon Apr 18 11:33:53 2022 from 192.168.1.1 [root@ovirt-engine-test ~]# ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 00:16:3e:58:7a:a3 brd ff:ff:ff:ff:ff:ff inet 192.168.1.97/24 brd 192.168.1.255 scope global dynamic noprefixroute eth0 valid_lft 2313sec preferred_lft 2313sec inet6 fe80::216:3eff:fe58:7aa3/64 scope link valid_lft forever preferred_lft forever Thank you for any tips for debugging. Jonas

5 7

Unable to install oVirt on RHEL7.5
by SS00514758＠techmahindra.com 15 Jan '26

15 Jan '26

Hi All, I am unable to install oVirt on RHEL7.5, to install it I am taking reference of below link, https://www.ovirt.org/documentation/install-guide/chap-Installing_oVirt.html But though it is not working for me, couple of dependencies is not getting installed, and because of this I am not able to run the ovirt-engine, below are the depencies packages that unable to install, Error: Package: collectd-write_http-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) Requires: collectd(x86-64) = 5.8.0-6.1.el7 Removing: collectd-5.8.0-6.1.el7.x86_64 (@ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-6.1.el7 Updated By: collectd-5.8.1-1.el7.x86_64 (epel) collectd(x86-64) = 5.8.1-1.el7 Available: collectd-5.7.2-1.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-1.el7 Available: collectd-5.7.2-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.7.2-3.el7 Available: collectd-5.8.0-2.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-2.el7 Available: collectd-5.8.0-3.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-3.el7 Available: collectd-5.8.0-5.el7.x86_64 (ovirt-4.2-centos-opstools) collectd(x86-64) = 5.8.0-5.el7 Help me to install this. Looking forward to resolve this issue. Regards Sumit Sahay

8 10

Installing oVirt on RHEL vs CentOS Stream, Alma or Rocky Linux
by Richard W.M. Jones 06 Oct '25

06 Oct '25

I'm still struggling to get oVirt installed. It absolutely will not install on RHEL 8, the documentation here is flat out wrong: https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_manage… javapackages-tools is not a module that exists in RHEL 8. Apparently it's a CentOS PowerTools module. Since original CentOS no longer really exists either, does anyone have any opinions or experience with installing oVirt on any of: - CentOS Stream 8 - Alma Linux 8.5 - Rocky Linux Alma in particular looks like a straight replacement for old CentOS. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top

8 10

How to list all snapshots?
by jorgevisentini＠gmail.com 14 Feb '25

14 Feb '25

Hello everyone! First, I would like to thank everyone involved in this wonderful project. I leave here my sincere thanks! Does anyone know if it is possible to list all snapshots automatically? It can be by ansible, python, shell... any way that helps to list them all without having to enter Domain by Domain. Thank you all!

4 3

snapshot solution: Existing snapshots that were taken after this one will be erased.
by dhanaraj.ramesh＠yahoo.com 29 Jan '25

29 Jan '25

Hi Team, when I want to commit the older snapshots I'm getting warning stating " Existing snapshots that were taken after this one will be erased.". is there any way we can retain the latest snapshots as is in the chain? I knew cloning and template export options are there to secure that latest snapshot data but these are will consume additional space in storage and take time.

4 3

Restart oVirt-Engine
by Jeremey Wise 09 Jan '25

09 Jan '25

How ,without reboot of hosting system, do I restart the oVirt engine? # I tried below but do not seem to effect the virtual machine [root@thor iso]# systemctl restart ov ovirt-ha-agent.service ovirt-imageio.service ovn-controller.service ovs-delete-transient-ports.service ovirt-ha-broker.service ovirt-vmconsole-host-sshd.service ovsdb-server.service ovs-vswitchd.service [root@thor iso]# # You cannot restart the VM " HostedEngine " as it responses: Error while executing action: HostedEngine: - Cannot restart VM. This VM is not managed by the engine. Reason is I had to do some work on a node. Reboot it.. it is back up.. network is all fine.. Cockpit working fine... and gluster fine.. But oVirt-Engine refuses to accept the node is up. -- p <jeremey.wise(a)gmail.com>enguinpages

4 4

Unable to access ovirt Admin Screen from ovirt Host
by louisb＠ameritech.net 07 Dec '24

07 Dec '24

I've reinstalled ovirt 4.4 on my server remotely via cockpit terminal. I'm able to access the ovirt admin screen remotely from the laptop that I used for the install. However, using the same URL I'm unable to gain access to the admin screen. Following the instruction in the documentation I've modified the file: /etc/ovirt-engine/engine.conf.d/99-custom-sso-setup.conf, to reflect the DNS name and I enter in the IP address. But I'm still unable to access the screen from the server console. What else needs to change in order to gain access from the server console? Thanks

5 8

SPM and Task error ...
by Enrico 22 Nov '24

22 Nov '24

Hi all, my ovirt cluster has got 3 Hypervisors runnig Centos 7.5.1804 vdsm is 4.20.39.1-1.el7, ovirt engine is 4.2.4.5-1.el7, the storage systems are HP MSA P2000 and 2050 (fibre channel). I need to stop one of the hypervisors for maintenance but this system is the storage pool manager. For this reason I decided to manually activate SPM in one of the other nodes but this operation is not successful. In the ovirt engine (engine.log) the error is this: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. while in the hypervisor (SPM) vdsm.log: 2019-07-25 12:39:16,744+02 INFO [org.ovirt.engine.core.bll.storage.pool.ForceSelectSPMCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] Running command: ForceSelectSPMCommand internal: false. Entities affected : ID: 81c9bd3c-ae0a-467f-bf7f-63ab30cd8d9e Type: VDSAction group MANIPULATE_HOST with role type ADMIN 2019-07-25 12:39:16,745+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopOnIrsVDSCommand( SpmStopOnIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false'}), log id: 37bf4639 2019-07-25 12:39:16,747+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, ResetIrsVDSCommand( ResetIrsVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', vdsId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', ignoreStopFailed='false'}), log id: 2522686f 2019-07-25 12:39:16,749+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] START, SpmStopVDSCommand(HostName = infn-vm05.management, SpmStopVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7'}), log id: 1810fd8b 2019-07-25 12:39:16,758+02 *ERROR* [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] SpmStopVDSCommand::Not stopping SPM on vds 'infn-vm05.management', pool id '18d57688-6ed4-43b8-bd7c-0665b55950b7' as there are uncleared tasks 'Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e', status 'running'' 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopVDSCommand, log id: 1810fd8b 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, ResetIrsVDSCommand, log id: 2522686f 2019-07-25 12:39:16,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] FINISH, SpmStopOnIrsVDSCommand, log id: 37bf4639 2019-07-25 12:39:16,760+02 *ERROR* [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-30) [7c374384-f884-4dc9-87d0-7af27dce706b] EVENT_ID: USER_FORCE_SELECTED_SPM_STOP_FAILED(4,096), Failed to force select infn-vm07.management as the SPM due to a failure to stop the current SPM. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,660+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,750+02 *ERROR* [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::logEndTaskFailure: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended with failure: 2019-07-25 12:39:18,750+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,751+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 34ae2b2f 2019-07-25 12:39:18,752+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: d3a78ad 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 34ae2b2f 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] BaseAsyncTask::onTaskEndSuccess: Task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' (Parent Command 'Unknown', Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended successfully. 2019-07-25 12:39:18,757+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,758+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 42de0c2b 2019-07-25 12:39:18,759+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 4895c79c 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: 42de0c2b 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Task id 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' has passed pre-polling period time and should be polled. Pre-polling period is 60000 millis. 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.AsyncTaskManager] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] Cleaning zombie tasks: Clearing async task 'Unknown' that started at 'Fri May 03 14:48:50 CEST 2019' 2019-07-25 12:39:18,764+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: Attempting to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' 2019-07-25 12:39:18,765+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, SPMClearTaskVDSCommand( SPMTaskGuidBaseVDSCommandParameters:{storagePoolId='18d57688-6ed4-43b8-bd7c-0665b55950b7', ignoreFailoverLimit='false', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: da77af2 2019-07-25 12:39:18,766+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] START, HSMClearTaskVDSCommand(HostName = infn-vm05.management, HSMTaskGuidBaseVDSCommandParameters:{hostId='751f3e99-b95e-4c31-bc38-77f5661a0bdc', taskId='fdcf4d1b-82fe-49a6-b233-323ebe568f8e'}), log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, HSMClearTaskVDSCommand, log id: 530694fb 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] FINISH, SPMClearTaskVDSCommand, log id: da77af2 2019-07-25 12:39:18,771+02 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask] (EE-ManagedThreadFactory-engineScheduled-Thread-67) [] SPMAsyncTask::ClearAsyncTask: At time of attempt to clear task 'fdcf4d1b-82fe-49a6-b233-323ebe568f8e' the response code was 'TaskStateError' and message was 'Operation is not allowed in this task state: ("can't clean in state running",)'. Task will not be cleaned there is some relation between this error and a task that has remained hanging, from SPM server: # vdsm-client Task getInfo taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "verb": "prepareMerge", "id": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e" } # vdsm-client Task getStatus taskID=fdcf4d1b-82fe-49a6-b233-323ebe568f8e { "message": "running job 1 of 1", "code": 0, "taskID": "fdcf4d1b-82fe-49a6-b233-323ebe568f8e", "taskResult": "", "taskState": "running" } How can I solve this problem ? Thanks a lot for your help !! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it _______________________________________________________________________

5 15

oVirt networks
by Enrico Becchetti 28 Aug '24

28 Aug '24

Dear all, Ineed your help to understand how to configure the network of a new oVirt cluster. Mynew system will have a 4.3 engine thatruns in a virtual machine, andsome Dell R7525 AMD EPYC hypervisors, eachholding two 4-port PCI network cards. These servers will have node-ovirt image again in version 4.3. As for the network, there are two HPE Aruba 2540G, non-stackable, with 24 1Gbs ports and 2 10Gbs uplinks to the star center. This is a simplified scheme: My goal is to make the most of the server's 8 ethernet interfaces to have both reliability and maximum possible throughput. This cluster will have two virtual networks, one forovirt management and one for the traffic of individual virtual machines. With that said here's what my idea is. I would like to have two links aggregated by 4Gbs, one for ovrtmgt and the other for vmnet. With the ovirt web interface I can createan active-passive "Mode 1" bond, but this won'tallow me to go beyond 1Gbs. Alternatively I could create a "Mode 4" bond 802.3ad but unfortunately the switches are not stacked and therefore not even this solution applies. This is an example with active passive configuration: Can you tell me if ovirt can generate//nested bonds? Or do you have other solutions ? Thanks a lot ! Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Skype:enrico_becchetti Mail: Enrico.Becchetti<at>pg.infn.it Pagina web personale: https://www.pg.infn.it/home/enrico-becchetti/ ______________________________________________________________________

5 6

boot from cdrom & error code 0005
by edp＠maddalena.it 21 Aug '24

21 Aug '24

Hi. I have created a new storage domain (data domain, storage type nfs) to use it to upload iso images. I have so uploaded a new iso and then attach the iso to a new vm. But when I try to boot the vm I obtain this error: booting from dvd/cd... boot failed: could not read from cdrom (code 0005) no bootable device The iso file has been uploaded with success in the data storage domain and so the vm lets my attach the iso to the vm in the boot settings. Can you help me? Thank you

4 6

VM Migration Failed
by KSNull Zero 10 Jul '24

10 Jul '24

Running oVirt 4.4.5 VM cannot migrate between hosts. vdsm.log contains the following error: libvirt.libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tls://ovhost01.local/system: authentication failed: Failed to verify peer's certificate Certificates on hosts was renewed some time ago. How this issue can be fixed ? Thank you.

4 6

How to re-enroll (or renew) host certificates for a single-host hosted-engine deployment?
by Derek Atkins 28 Jun '24

28 Jun '24

Hi, I've got a single-host hosted-engine deployment that I originally installed with 4.0 and have upgraded over the years to 4.3.10. I and some of my users have upgraded remote-viewer and now I get an error when I try to view the console of my VMs: (remote-viewer:8252): Spice-WARNING **: 11:30:41.806: ../subprojects/spice-common/common/ssl_verify.c:477:openssl_verify: Error in server certificate verification: CA signature digest algorithm too weak (num=68:depth0:/O=<My Org Name>/CN=<Host's Name>) I am 99.99% sure this is because the old certs use SHA1. I reran engine-setup on the engine and it asked me if I wanted to renew the PKI, and I answered yes. This replaced many[1] of the certificates in /etc/pki/ovirt-engine/certs on the engine, but it did not update the Host's certificate. All the documentation I've seen says that to refresh this certificate I need to put the host into maintenance mode and then re-enroll.. However I cannot do that, because this is a single-host system so I cannot put the host in local mode -- there is no place to migrate the VMs (let alone the Engine VM). So.... Is there a command-line way to re-enroll manually and update the host certs? Or some other way to get all the leftover certs renewed? Thanks, -derek [1] Not only did it not update the Host's cert, it did not update any of the vmconsole-proxy certs, nor the certs in /etc/pki/ovirt-vmconsole/, and obviously nothing in /etc/pki/ on the host itself. -- Derek Atkins 617-623-3745 derek(a)ihtfp.com www.ihtfp.com Computer and Internet Security Consultant

8 16

Changing disk QoS causes segfault with IO-Threads enabled (oVirt 4.3.0.4-1.el7)
by jloh＠squiz.net 23 Apr '24

23 Apr '24

We recently upgraded to 4.3.0 and have found that when changing disk QoS settings on VMs whilst IO-Threads is enabled causes them to segfault and the VM to reboot. We've been able to replicate this across several VMs. VMs with IO-Threads disabled/turned off do not segfault when changing the QoS. Mar 1 11:49:06 srvXX kernel: IO iothread1[30468]: segfault at fffffffffffffff8 ip 0000557649f2bd24 sp 00007f80de832f60 error 5 in qemu-kvm[5576498dd000+a03000] Mar 1 11:49:06 srvXX abrt-hook-ccpp: invalid number 'iothread1' Mar 1 11:49:11 srvXX libvirtd: 2019-03-01 00:49:11.116+0000: 13365: error : qemuMonitorIORead:609 : Unable to read from monitor: Connection reset by peer Happy to supply some more logs to someone if they'll help but just wondering whether anyone else has experienced this or knows of a current fix other than turning io-threads off. Cheers.

4 3

Deploy oVirt Engine fail behind proxy
by Matteo Bonardi 21 Mar '24

21 Mar '24

Hi, I am trying to deploy the ovirt engine following self-hosted engine installation procedure on documentation. Deployment servers are behind a proxy and I have set it in environment and in yum.conf before run deploy. Deploy fails because ovirt engine vm cannot resolve AppStream repository url: [ INFO ] TASK [ovirt.engine-setup : Install oVirt Engine package] [ ERROR ] fatal: [localhost -> ovirt-manager.mydomain]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'AppStream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=st… [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ INFO ] TASK [ovirt.hosted_engine_setup : Execute just a specific set of steps] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Force facts gathering] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Fetch logs from the engine VM] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set destination directory path] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Create destination directory] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Find the local appliance image] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Set local_vm_disk_path] [ INFO ] skipping: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Give the vm time to flush dirty buffers] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Copy engine logs] [ INFO ] TASK [ovirt.hosted_engine_setup : include_tasks] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove local vm dir] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Remove temporary entry in /etc/hosts for the local VM] [ INFO ] changed: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Clean local storage pools] [ INFO ] ok: [localhost] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ he_local_vm_dir | basename }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Destroy local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] TASK [ovirt.hosted_engine_setup : Undefine local storage-pool {{ local_vm_disk_path.split('/')[5] }}] [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20201109165237.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed: please check the logs for the issue, fix accordingly or re-deploy from scratch. Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20201109164244-b3e8sd.log How I can set proxy for the engine vm? Ovirt version: [root@myhost ~]# rpm -qa | grep ovirt-engine-appliance ovirt-engine-appliance-4.4-20200916125954.1.el8.x86_64 [root@myhost ~]# rpm -qa | grep ovirt-hosted-engine-setup ovirt-hosted-engine-setup-2.4.6-1.el8.noarch OS version: [root@myhost ~]# cat /etc/centos-release CentOS Linux release 8.2.2004 (Core) [root@myhost ~]# uname -a Linux myhost.mydomain 4.18.0-193.28.1.el8_2.x86_64 #1 SMP Thu Oct 22 00:20:22 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux Thanks for the help. Regards, Matteo

7 20

Cannot restart ovirt after massive failure.
by Gilboa Davara 18 Jan '24

18 Jan '24

Hello all, During the night, one of my (smaller) setups, a single node self hosted engine (localhost NFS) crashed due to what-looks-like a massive disk failure (Software RAID6, with 10 drives + spare). After a reboot, I let the RAID resync with a fresh drive) and went on to start oVirt. However, no such luck. Two issues: 1. ovirt-ha-broker fails due to broken hosted engine state (log attached). 2. ovirt-ha-agent fails due to network test (tcp) even though both remote-host and DNS servers are active. (log attached). Two questions: 1. Can I somehow force the agent to disable the network liveliness test? 2. Can I somehow force the broker to rebuild / fix the hosted engine state? - Gilboa

4 17

how to renew expired ovirt node vdsm cert manually ?
by dhanaraj.ramesh＠yahoo.com 11 Jan '24

11 Jan '24

below are the steps to renew the expired vdsm cert of ovirt node # To check CERT expired # openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates 1. Backup vdsm folder # cd /etc/pki # mv vdsm vdsm.orig # mkdir vdsm ; chown vdsm:kvm vdsm # cd vdsm # mkdir libvirt-vnc certs keys libvirt-spice libvirt-migrate # chown vdsm:kvm libvirt-vnc certs keys libvirt-spice libvirt-migrate 2. Regenerate cert & keys # vdsm-tool configure --module certificates 3. Copy the cert to destination location chmod 440 /etc/pki/vdsm/keys/vdsmkey.pem chown root /etc/pki/vdsmcerts/*pem chmod 644 /etc/pki/vdsmcerts/*pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-spice/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-spice/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem cp /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-vnc/ca-cert.pem cp /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-vnc/server-key.pem cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/vdsm/libvirt-migrate/ca-cert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem chown root:qemu /etc/pki/vdsm/libvirt-migrate/server-key.pem cp -p /etc/pki/vdsm.orig/keys/libvirt_password /etc/pki/vdsm/keys/ mv /etc/pki/libvirt/clientcert.pem /etc/pki/libvirt/clientcert.pem.orig mv /etc/pki/libvirt/private/clientkey.pem /etc/pki/libvirt/private/clientkey.pem.orig mv /etc/pki/CA/cacert.pem /etc/pki/CA/cacert.pem.orig cp -p /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem cp -p /etc/pki/vdsm/keys/vdsmkey.pem /etc/pki/libvirt/private/clientkey.pem cp -p /etc/pki/vdsm/certs/cacert.pem /etc/pki/CA/cacert.pem 3. cross check the backup folder /etc/pki/vdsm.orig vs /etc/pki/vdsm # refer to /etc/pki/vdsm.orig/*/ and set the correct owner & group permission in /etc/pki/vdsm/*/ 4. restart services # Make sure both services are up systemctl restart vdsmd libvirtd

5 4

Grafana - Origin Not Allowed
by Maton, Brett 14 Dec '23

14 Dec '23

oVirt 4.5.0.8-1.el8 I tried to connect to grafana via the monitoring portal link from the dash and all panels are failing to display any data with varying error messages, but all include 'Origin Not Allowed' I navigated to Data Sources and ran a test on the PostgreSQL connection (localhost) which threw the same Origin Not Allowed error message. Any suggestions?

6 8

Re: Failed to synchronize networks of Provider ovirt-provider-ovn
by Mail SET Inc. Group 27 Oct '23

27 Oct '23

Yes, i use same manual to change WebUI SSL. ovirt-ca-file= is a same SSL file which use WebUI. Yes, i restart ovirt-provider-ovn, i restart engine, i restart all what i can restart. Nothing... > 12 сент. 2018 г., в 16:11, Dominik Holler <dholler(a)redhat.com> написал(а): > > On Wed, 12 Sep 2018 14:23:54 +0300 > "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: > >> Ok! > > Not exactly, please use users(a)ovirt.org for such questions. > Other should benefit from this questions, too. > Please write the next mail to users(a)ovirt.org and keep me in CC. > >> What i did: >> >> 1) install oVirt «from box» (4.2.5.2-1.el7); >> 2) generate own ssl for my engine using my FreeIPA CA, Install it and > > What means "Install it"? You can use the doc from the following link > https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/ht… > > Ensure that ovirt-ca-file= in > /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf > points to the correct file and ovirt-provider-ovn is restarted. > >> get tis issue; >> >> >> [root@engine ~]# tail -n 50 /var/log/ovirt-provider-ovn.log >> 2018-09-12 14:10:23,828 root [SSL: CERTIFICATE_VERIFY_FAILED] >> certificate verify failed (_ssl.c:579) Traceback (most recent call >> last): File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", >> line 133, in _handle_request method, path_parts, content >> File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", >> line 175, in handle_request return >> self.call_response_handler(handler, content, parameters) File >> "/usr/share/ovirt-provider-ovn/handlers/keystone.py", line 33, in >> call_response_handler return response_handler(content, parameters) >> File "/usr/share/ovirt-provider-ovn/handlers/keystone_responses.py", >> line 62, in post_tokens user_password=user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 26, in >> create_token return auth.core.plugin.create_token(user_at_domain, >> user_password) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/plugin.py", line >> 48, in create_token timeout=self._timeout()) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 75, >> in create_token username, password, engine_url, ca_file, timeout) >> File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line >> 91, in _get_sso_token timeout=timeout File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 54, >> in wrapper response = func(*args, **kwargs) File >> "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 47, >> in wrapper raise BadGateway(e) BadGateway: [SSL: >> CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579) >> >> >> [root@engine ~]# tail -n 20 /var/log/ovirt-engine/engine.log >> 2018-09-12 14:10:23,773+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:10:23,778+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:10:23,836+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:10:23,837+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-47) [316db685] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:14:12,477+03 INFO >> [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default >> task-6) [] User admin@internal successfully logged in with scopes: >> ovirt-app-admin ovirt-app-api ovirt-app-portal >> ovirt-ext=auth:sequence-priority=~ ovirt-ext=revoke:revoke-all >> ovirt-ext=token-info:authz-search >> ovirt-ext=token-info:public-authz-search >> ovirt-ext=token-info:validate ovirt-ext=token:password-access >> 2018-09-12 14:14:12,587+03 INFO >> [org.ovirt.engine.core.bll.aaa.CreateUserSessionCommand] (default >> task-6) [1bf1b763] Running command: CreateUserSessionCommand >> internal: false. 2018-09-12 14:14:12,628+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [1bf1b763] EVENT_ID: USER_VDC_LOGIN(30), User >> admin@internal-authz connecting from '10.0.3.61' using session >> 's8jAm7BUJGlicthm6yZBA3CUM8QpRdtwFaK3M/IppfhB3fHFB9gmNf0cAlbl1xIhcJ2WX+ww7e71Ri+MxJSsIg==' >> logged in. 2018-09-12 14:14:30,972+03 INFO >> [org.ovirt.engine.core.bll.provider.ImportProviderCertificateCommand] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] Running >> command: ImportProviderCertificateCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:30,982+03 INFO >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-6) [ee3cc8a7-4485-4fdf-a0c2-e9d67b5cfcd3] EVENT_ID: >> PROVIDER_CERTIFICATE_IMPORTED(213), Certificate for provider >> ovirt-provider-ovn was imported. (User: admin@internal-authz) >> 2018-09-12 14:14:31,006+03 INFO >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Running >> command: TestProviderConnectivityCommand internal: false. Entities >> affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: >> SystemAction group CREATE_STORAGE_POOL with role type ADMIN >> 2018-09-12 14:14:31,058+03 ERROR >> [org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand] >> (default task-6) [a48d94ab-b0b2-42a2-a667-0525b4c652ea] Command >> 'org.ovirt.engine.core.bll.provider.TestProviderConnectivityCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'default' is using 0 threads out of 1, 5 threads waiting for >> tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engine' is using 0 threads out of 500, 16 threads waiting for >> tasks and 0 tasks in queue. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineScheduled' is using 0 threads out of 100, 100 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'engineThreadMonitoring' is using 1 threads out of 1, 0 threads >> waiting for tasks. 2018-09-12 14:15:10,954+03 INFO >> [org.ovirt.engine.core.bll.utils.ThreadPoolMonitoringService] >> (EE-ManagedThreadFactory-engineThreadMonitoring-Thread-1) [] Thread >> pool 'hostUpdatesChecker' is using 0 threads out of 5, 2 threads >> waiting for tasks. 2018-09-12 14:15:23,843+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> Acquired to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' 2018-09-12 14:15:23,849+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Running command: SyncNetworkProviderCommand internal: true. >> 2018-09-12 14:15:23,900+03 ERROR >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] >> Command >> 'org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand' >> failed: EngineException: (Failed with error Bad Gateway and code >> 5050) 2018-09-12 14:15:23,901+03 INFO >> [org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-61) [2455041f] Lock >> freed to object >> 'EngineLock:{exclusiveLocks='[14e4fb72-9764-4757-b37d-4d487995571a=PROVIDER]', >> sharedLocks=''}' >> >> >> [root@engine ~]# >> cat /etc/ovirt-provider-ovn/conf.d/10-setup-ovirt-provider-ovn.conf # >> This file is automatically generated by engine-setup. Please do not >> edit manually [OVN REMOTE] ovn-remote=ssl:127.0.0.1:6641 >> [SSL] >> https-enabled=true >> ssl-cacert-file=/etc/pki/ovirt-engine/ca.pem >> ssl-cert-file=/etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer >> ssl-key-file=/etc/pki/ovirt-engine/keys/ovirt-provider-ovn.key.nopass >> [OVIRT] >> ovirt-sso-client-secret=Ms7Gw9qNT6IkXu7oA54tDmxaZDIukABV >> ovirt-host=https://engine.set.local:443 >> ovirt-sso-client-id=ovirt-provider-ovn >> ovirt-ca-file=/etc/pki/ovirt-engine/apache-ca.pem >> [PROVIDER] >> provider-host=engine.set.local >> >> >>> 12 сент. 2018 г., в 13:59, Dominik Holler <dholler(a)redhat.com> >>> написал(а): >>> >>> On Wed, 12 Sep 2018 13:04:53 +0300 >>> "Mail SET Inc. Group" <mail(a)set-pro.net> wrote: >>> >>>> Hello Dominik! >>>> I have a same issue with OVN provider and SSL >>>> https://www.mail-archive.com/users@ovirt.org/msg47020.html >>>> <https://www.mail-archive.com/users@ovirt.org/msg47020.html> But >>>> certificate changes not helps to resolve it. Maybe you can help me >>>> with this? >>> >>> Sure. Can you please share the relevant lines of >>> ovirt-provider-ovn.log and engine.log, and the information if you >>> are using the certificates generated by engine-setup with >>> users(a)ovirt.org ? Thanks, >>> Dominik >>> >> > >

5 13

engine-setup failing on 4.3.2 -> 4.3.3 fails during Engine schema refresh fail
by Edward Berger 18 Jul '23

18 Jul '23

I was trying to upgrade a hyperconverged oVirt hosted engine and failed in the engine-setup command with these error and warnings. ... [ INFO ] Creating/refreshing Engine database schema [ ERROR ] schema.sh: FATAL: Cannot execute sql command: --file=/usr/share/ovirt-engine/dbscripts/upgrade/04_03_0830_add_foreign_key_to_image_transfers.sql [ ERROR ] Failed to execute stage 'Misc configuration': Engine schema refresh failed ... [ INFO ] Yum Verify: 16/16: ovirt-engine-tools.noarch 0:4.3.3.5-1.el7 - e [WARNING] Rollback of DWH database postponed to Stage "Clean up" [ INFO ] Rolling back database schema ... [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed Attaching engine-setup logfile.

4 6

4.4.9 -> 4.4.10 Cannot start or migrate any VM (hotpluggable cpus requested exceeds the maximum cpus supported by KVM)
by Jillian Morgan 10 Apr '23

10 Apr '23

After upgrading the engine from 4.4.9 to 4.4.10, and then upgrading one host, any attempt to migrate a VM to that host or start a VM on that host results in the following error: Number of hotpluggable cpus requested (16) exceeds the maximum cpus supported by KVM (8) While the version of qemu is the same across hosts, ( qemu-kvm-6.0.0-33.el8s.x86_64), I traced the difference to the upgraded kernel on the new host. I have always run elrepo's kernel-ml on these hosts to support bcache which RHEL's kernel doesn't support. The working hosts still run kernel-ml-5.15.12. The upgraded host ran kernel-ml-5.17.0. In case anyone else runs kernel-ml, have you run into this issue? Does anyone know why KVM's KVM_CAP_MAX_VCPUS value is lowered on the new kernel? Does anyone know how to query the KVM capabilities from userspace without writing a program leveraging kvm_ioctl()'s? Related to this, it seems that ovirt and/or libvirtd always runs qmu-kvm with an -smp argument of "maxcpus=16". This causes qemu's built-in check to fail on the new kernel which is supporting max_vpus of 8. Why does ovirt always request maxcpus=16? And yes, before you say it, I know you're going to say that running kernel-ml isn't supported. -- Jillian Morgan (she/her) 🏳️‍⚧️ Systems & Networking Specialist Primordial Software Group & I.T. Consultancy https://www.primordial.ca

2 1

oVirt host deploy: disable multipath
by femi adegoke 27 Mar '23

27 Mar '23

https://bugzilla.redhat.com/show_bug.cgi?id=1472356 There are 2 drives in my install that oVirt is claiming as multipath. I have tried blacklisting at /etc/multipath/ but that hasn't worked. Any better solutions? I need these 2 drives to be non-multipathed.

4 19

Out-of-sync networks can only be detached
by Sakhi Hadebe 27 Feb '23

27 Feb '23

Hi, I have a 3-node oVirt cluster. I have configured 2 logical networks: ovirtmgmt and public. Public logical network is attached in only 2 nodes and failing to attach on the 3rd node with the below error Invalid operation, out-of-sync network 'public' can only be detached. Please have been stuck on this for almost the whole day now. How do I fix this error? -- Regards, Sakhi Hadebe

3 5

Re: Self-hosted engine 4.5.0 deployment fails
by John 01 Feb '23

01 Feb '23

I would run the deploy again, wait until the engine is up and then from the server you are deploying the engine on # virsh list obtain the virtual machine number from the above command, in the example below we assume the number is 1 # virsh console 1 login as root wait for all packages to finish updating: tail -f /var/log/dnf.rpm.log then: # dnf downgrade postgresql-jdbc once that's done the deployment of the engine should get past the [ INFO ] TASK [ovirt.ovirt.engine_setup : Check if Engine health page is up] step. On 03/05/2022 17:03, Mohamed Roushdy wrote: > > Hello, > > I’m deploying on Ovirt nodes v4.5.0.1, but it fails to deploy the > self-hosted engine with the following errors in the web installer: > > [ INFO ] skipping: [localhost] > [ INFO ] TASK [ovirt.ovirt.engine_setup : Check if Engine health page > is up] > [ ERROR ] fatal: [localhost -> 192.168.222.197]: FAILED! => > {"attempts": 30, "changed": false, "connection": "close", > "content_encoding": "identity", "content_length": "86", > "content_type": "text/html; charset=UTF-8", "date": "Tue, 03 May 2022 > 15:57:20 GMT", "elapsed": 0, "msg": "Status code was 500 and not > [200]: HTTP Error 500: Internal Server Error", "redirected": false, > "server": "Apache/2.4.37 (centos) OpenSSL/1.1.1k > mod_auth_gssapi/1.6.1", "status": 500, "url": > http://localhost/ovirt-engine/services/health} > [ INFO ] TASK [ovirt.ovirt.engine_setup : Clean temporary files] > [ INFO ] changed: [localhost -> 192.168.222.197] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine] > [ INFO ] changed: [localhost -> 192.168.222.197] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination > directory path] > [ INFO ] ok: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Create destination > directory] > [ INFO ] changed: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : include_tasks] > [ INFO ] ok: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Find the local > appliance image] > [ INFO ] ok: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set local_vm_disk_path] > [ INFO ] ok: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Give the vm time to > flush dirty buffers] > [ INFO ] ok: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Copy engine logs] > [ INFO ] changed: [localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Change ownership of > copied engine logs] > [ INFO ] changed: [localhost -> localhost] > [ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about > a failure] > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > "There was a failure deploying the engine on the local engine VM. The > system may not be provisioned according to the playbook results: > please check the logs for the issue, fix accordingly or re-deploy from > scratch.\n"} > > Any idea if this is a bug or something? I tried to deploy for eight > times with different configuration > > Thank y9ou, > > > _______________________________________________ > Users mailing list --users(a)ovirt.org > To unsubscribe send an email tousers-leave(a)ovirt.org > Privacy Statement:https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct:https://www.ovirt.org/community/about/community-guidelines/ > List Archives:https://lists.ovirt.org/archives/list/users@ovirt.org/message/XL3Q…

2 1

oVirt + TrueNAS: Unable to create iSCSI domain - I am missing something obvious
by David Johnson 20 Jan '23

20 Jan '23

Good morning folks, and thank you in advance. I am working on migrating my oVirt backing store from NFS to iSCSI. *oVirt Environment:* oVirt Open Virtualization Manager Software Version:4.4.4.7-1.el8 *TrueNAS environment:* FreeBSD truenas.local 12.2-RELEASE-p11 75566f060d4(HEAD) TRUENAS amd64 The iSCSI share is on a TrueNAS server, exposed to user VDSM and group 36. oVirt sees the targeted share, but is unable to make use of it. The latest issue is "Error while executing action New SAN Storage Domain: Volume Group block size error, please check your Volume Group configuration, Supported block size is 512 bytes." As near as I can tell, oVirt does not support any block size other than 512 bytes, while TrueNAS's smallest OOB block size is 4k. I know that oVirt on TrueNAS is a common configuration, so I expect I am missing something really obvious here, probably a TrueNAS configuration needed to make TrueNAS work with 512 byte blocks. Any advice would be helpful. *David Johnson*

5 9

Network Address Change
by Paul.LKW 28 Dec '22

28 Dec '22

Hi All: I just has a case, I need to change the oVirt host and engine IP address due to data center decommission I checked in the hosted-engine host there are some files I could change ; in ovirt-hosted-engine/hosted-engine.conf ca_subject="O=simple.com, CN=1.2.3.4" gateway=1.2.3.254 and of course I need to change the ovirtmgmt interface IP too, I think just change the above line could do the tick, but where could I change the other host IP in the cluster ? I think I have to be lost all the host as once changed the hosted-engine host IP as it is in diff. sub net. Does there any command line tools could do that or someone has such experience could share? Best Regards, Paul.LKW

4 6

Re: Migrated a VM from VMware to Ovirt using import ova option. Task stuck and running for day 2 now.
by Saif Abu Saleh 14 Dec '22

14 Dec '22

+ @users <users(a)ovirt.org> So it will be more visible to others Thanks, Saif On Wed, Dec 8, 2021 at 3:53 PM Sudhakar S <sudhakar.sankaralingam(a)gmail.com> wrote: > Hi Team, > > It automatically failed at last. > Thanks for the help guys. > > [image: image.png] > > On Wed, Dec 8, 2021 at 6:34 PM Tomáš Golembiovský <tgolembi(a)redhat.com> > wrote: > >> This is another instance of bug >> https://bugzilla.redhat.com/show_bug.cgi?id=2026809 >> >> To get rid of the import task you have to: >> >> 1) move the host to maintenance >> 2) restart VDSM with: systemctl restart vdsmd >> 3) move the host back UP >> 4) restart engine >> >> >> By the way, the VMware imports finished fine but the OVA is broken and >> will fail the import. >> >> Tomas >> >> On Wed, Dec 08, 2021 at 03:27:32PM +0530, Sudhakar S wrote: >> > Hi Team, >> > >> > not intentionally, did mistakenly. >> > >> > Please find attached /var/log/vdsm/import FYR. >> > >> > On Wed, Dec 8, 2021 at 3:12 PM Arik Hadas <ahadas(a)redhat.com> wrote: >> > >> > > Hi, >> > > Did you intentionally send it directly to me and not to the list? >> > > Adding Saif and Tomas >> > > But I think they'll need the import logs (AFAIR in >> /var/log/vdsm/import) >> > > to investigate it further >> > > >> > > On Wed, Dec 8, 2021 at 11:34 AM Sudhakar S < >> > > sudhakar.sankaralingam(a)gmail.com> wrote: >> > > >> > >> HI Team, >> > >> >> > >> Please find the logs FYR. >> > >> >> > >> I just want to kill this task alone since it is creating 7000 events >> per >> > >> day. Please give me commands to stop this task. >> > >> >> > >> [image: image.png] >> > >> >> > >> Regards, >> > >> Sudhakar >> > >> >> > >> On Wed, Dec 8, 2021 at 2:07 PM Arik Hadas <ahadas(a)redhat.com> wrote: >> > >> >> > >>> >> > >>> >> > >>> On Wed, Dec 8, 2021 at 8:44 AM <sudhakar.sankaralingam(a)gmail.com> >> wrote: >> > >>> >> > >>>> 1. First node that was upgraded now continuously logs this error: >> > >>>> VDSM <node name> command Get Host Statistics failed: Internal >> JSON-RPC >> > >>>> error: >> > >>>> {'reason': "'str' object has no attribute 'decode'"} >> > >>>> 2. During import of two virtual machines from VMware (has done >> multible >> > >>>> before upgrade) >> > >>>> the import seems to never finish - currently running on day 2-3... >> any >> > >>>> clues how to fix >> > >>>> this? >> > >>>> >> > >>>> Events VDSM <node name> command Get Host Statistics failed: >> Internal >> > >>>> JSON-RPC error: >> > >>>> {'reason': "'str' object has no attribute 'decode'"} running for 2 >> days >> > >>>> every 20 seconds. Please suggest >> > >>>> >> > >>> >> > >>> Looks like [1], however in that report the import process succeeded >> > >>> despite those errors. >> > >>> Can you please share the import logs? >> > >>> >> > >>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=2026809 >> > >>> >> > >>> >> > >>>> _______________________________________________ >> > >>>> Users mailing list -- users(a)ovirt.org >> > >>>> To unsubscribe send an email to users-leave(a)ovirt.org >> > >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> > >>>> oVirt Code of Conduct: >> > >>>> https://www.ovirt.org/community/about/community-guidelines/ >> > >>>> List Archives: >> > >>>> >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/37YK2JLWZQX66… >> > >>>> >> > >>> >> >> >> >> >> >> >> >> >> -- >> Tomáš Golembiovský <tgolembi(a)redhat.com> >> >>

2 1

Can't upgrade Cluster Compatibility Version
by Gillingham, Eric J (US 393D) 13 Dec '22

13 Dec '22

When trying to edit my default clusters compat version from 4.5 to anything newer it fails with " Error while executing action Edit Cluster properties: Internal Engine Error" engine.log contains the very long stack traces: ###################################################################################################################################################### 2022-08-31 23:43:08,647Z ERROR [org.ovirt.engine.core.bll.UpdateVmCommand] (default task-26) [590a6527] Error during ValidateFailure.: java.lang.NullPointerException at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateVmCommand.validate(UpdateVmCommand.java:1015) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.internalValidateInTransaction(CommandBase.java:824) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:157) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:803) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:417) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:638) at jdk.internal.reflect.GeneratedMethodAccessor380.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view2.runInternalAction(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor379.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2388) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVms(UpdateClusterCommand.java:502) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVmsAndTemplates(UpdateClusterCommand.java:444) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.executeCommand(UpdateClusterCommand.java:373) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) at jdk.internal.reflect.GeneratedMethodAccessor827.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:176) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:523) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at deployment.engine.ear.webadmin.war//org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) at io.undertow.core@2.2.8.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) at io.undertow.core@2.2.8.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841) at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280) at java.base/java.lang.Thread.run(Thread.java:829) 2022-08-31 23:43:08,651Z ERROR [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-26) [590a6527] Command 'org.ovirt.engine.core.bll.UpdateClusterCommand' failed: java.lang.NullPointerException 2022-08-31 23:43:08,651Z ERROR [org.ovirt.engine.core.bll.UpdateClusterCommand] (default task-26) [590a6527] Exception: javax.ejb.EJBException: java.lang.NullPointerException at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:239) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view2.runInternalAction(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor379.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2388) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVms(UpdateClusterCommand.java:502) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVmsAndTemplates(UpdateClusterCommand.java:444) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.executeCommand(UpdateClusterCommand.java:373) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) at jdk.internal.reflect.GeneratedMethodAccessor827.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:176) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:523) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at deployment.engine.ear.webadmin.war//org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) at io.undertow.core@2.2.8.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) at io.undertow.core@2.2.8.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841) at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateVmCommand.getCallback(UpdateVmCommand.java:1780) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.shouldPersistCommand(CommandBase.java:2266) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.updateCommandIfNeeded(CommandBase.java:2276) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:429) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:638) at jdk.internal.reflect.GeneratedMethodAccessor380.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) ... 202 more ###################################################################################################################################################### server.log contains: ###################################################################################################################################################### 2022-08-31 23:43:08,650Z ERROR [org.jboss.as.ejb3.invocation] (default task-26) WFLYEJB0034: Jakarta Enterprise Beans Invocation failed on component Backend for method public abstract org.ovirt.engine.core.common.action.ActionReturnValue org.ovirt.engine.core.bll.interfaces.BackendInternal.runInternalAction(org.ovirt.engine.core.common.action.ActionType,org.ovirt.engine.core.common.action.ActionParametersBase,org.ovirt.engine.core.bll.context.CommandContext): javax.ejb.EJBException: java.lang.NullPointerException at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:239) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view2.runInternalAction(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor379.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:410) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:134) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.InjectionPointPropagatingEnterpriseTargetBeanInstance.invoke(InjectionPointPropagatingEnterpriseTargetBeanInstance.java:68) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interfaces.BackendCommandObjectsHandler$BackendInternal$BackendLocal$2049259618$Proxy$_$$_Weld$EnterpriseProxy$.runInternalAction(Unknown Source) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInternalAction(CommandBase.java:2388) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVms(UpdateClusterCommand.java:502) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.updateVmsAndTemplates(UpdateClusterCommand.java:444) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateClusterCommand.executeCommand(UpdateClusterCommand.java:373) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1174) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1332) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2010) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:140) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:79) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1392) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:424) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:387) at jdk.internal.reflect.GeneratedMethodAccessor827.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.interceptorChainCompleted(WeldInvocationContextImpl.java:107) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:126) at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) at jdk.internal.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.invokeNext(WeldInvocationContextImpl.java:92) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.WeldInvocationContextImpl.proceed(WeldInvocationContextImpl.java:124) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:105) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) at org.jboss.as.weld.ejb@24.0.1.Final//org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:77) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:446) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:164) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:81) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438) at org.wildfly.security.elytron-base@1.16.1.Final//org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:633) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:191) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runAction(Unknown Source) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runAction(GenericApiGWTServiceImpl.java:176) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373) at deployment.engine.ear.webadmin.war//com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:523) at deployment.engine.ear.webadmin.war//org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78) at javax.servlet.api@2.0.0.Final//javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.CachingFilter.doFilter(CachingFilter.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at deployment.engine.ear.webadmin.war//org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils//org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:65) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.2.8.Final//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) at io.undertow.core@2.2.8.Final//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at org.wildfly.extension.undertow@24.0.1.Final//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1535) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78) at io.undertow.servlet@2.2.8.Final//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99) at io.undertow.core@2.2.8.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387) at io.undertow.core@2.2.8.Final//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:841) at org.jboss.threads@2.4.0.Final//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.4.0.Final//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at org.jboss.xnio@3.8.4.Final//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.UpdateVmCommand.getCallback(UpdateVmCommand.java:1780) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.shouldPersistCommand(CommandBase.java:2266) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.updateCommandIfNeeded(CommandBase.java:2276) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:429) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runAction(Backend.java:450) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:432) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:638) at jdk.internal.reflect.GeneratedMethodAccessor380.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:79) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:89) at org.jboss.as.weld.common@24.0.1.Final//org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:102) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53) at org.jboss.as.ee@24.0.1.Final//org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) at org.jboss.invocation@1.6.0.Final//org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422) at org.jboss.as.ejb3@24.0.1.Final//org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:232) ... 202 more ###################################################################################################################################################### Versions of installed packages: centos-release-ceph-pacific.noarch 1.0-2.el8 @extras centos-release-gluster10.noarch 1.0-1.el8s @extras-common centos-release-nfv-common.noarch 1-3.el8 @extras centos-release-nfv-openvswitch.noarch 1-3.el8 @extras centos-release-opstools.noarch 1-12.el8 @extras centos-release-ovirt45.noarch 8.7-2.el8s @extras-common centos-release-storage-common.noarch 2-2.el8 @extras centos-release-virt-common.noarch 1-2.el8 @extras centos-stream-release.noarch 8.6-1.el8 @baseos centos-stream-repos.noarch 8-6.el8 @baseos glusterfs.x86_64 10.2-1.el8s @centos-gluster10 vdsm.x86_64 4.50.1.4-1.el8 @centos-ovirt45 Any ideas what's going wrong?

3 16

what happens to vms when a host shutdowns?
by Nathanaël Blanchet 15 Nov '22

15 Nov '22

Hi, We are installing UPS powerchute client on hypervisors. What is the default vms behaviour of running vms when an hypervisor is ordered to shutdown: do the vms live migrate or do they shutdown properly (even the restart on an other host because of HA) ? -- Nathanaël Blanchet Supervision réseau SIRE 227 avenue Professeur-Jean-Louis-Viala 34193 MONTPELLIER CEDEX 5 Tél. 33 (0)4 67 54 84 55 Fax 33 (0)4 67 54 84 14 blanchet(a)abes.fr

7 11

[4.4.7] The latest guest agent needs to be installed and running on the guest
by Christoph Timm 28 Oct '22

28 Oct '22

Hi list, I have upgraded my oVirt from 4.4.4 to 4.4.7 and notice that some VMs have issues to report the performance information via the guest agent. I see the following message on the VMs: The latest guest agent needs to be installed and running on the guest The qemu-guest agent is installed on VMs (the VM OS is CentOS 8). Any advice how to troubleshoot this? I can see the stats will be presented in the GUI, if I migrate the VM to a different host. Best regards Christoph

4 8

Failed to delete snapshot
by Jirka Simon 24 Oct '22

24 Oct '22

Hello there. we have troubles with one VM ad it's snapshot here is short log from engine.log 2022-08-23 09:02:38,735+02 INFO [org.ovirt.engine.core.bll.tasks.CommandAsyncTask] (EE-ManagedThreadFactory-engine-Thread-1265784) [4d79272b-ffac-4b2b-a987-8c19577cd8c4] CommandAsyncTask:: HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity '5842cafc-60d4-4131-8da1-8a909123f08c' 2022-08-23 09:02:41,435+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Command id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15 failed child command status for step 'REDUCE_IMAGE' 2022-08-23 09:02:41,435+02 INFO [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommandCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-68) [ 4d79272b-ffac-4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshotSingleDiskLive' id: 'ead3fbba-7439-4337-8aaf-1d6cf65bbb15' child commands '[5b33d970-6ea0-4b8b-bc31-607afd8af1ca, 185d3fec-8c76- 44d3-a0a1-fe8fff331354, 308baf9d-1fc5-43a1-a624-b5f08f7bec1b, f11f3550-4356-417e-af10-a3afb2051dec, 5842cafc-60d4-4131-8da1-8a909123f08c]' executions were completed, status 'FAILED' 2022-08-23 09:02:42,445+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Merging of snapshot '6634db4e-23d8-4849-a008-864f15d28c3d' images 'd24a9199-741b-49cb-96d2-0d76fcd21f48'..'48af3301-0cbb-4a6e-97d1-7299e7de883f' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2022-08-23 09:02:42,451+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b -ffac-4b2b-a987-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand' with failure. 2022-08-23 09:02:42,457+02 INFO [org.ovirt.engine.core.bll.ConcurrentChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-79) [4d79272b-ffac -4b2b-a987-8c19577cd8c4] Command 'RemoveSnapshot' id: '73340ac8-07d7-4de2-bfeb-8062fa1e8cfd' child commands '[ead3fbba-7439-4337-8aaf-1d6cf65bbb15]' executions were completed, status 'FAILE D' 2022-08-23 09:02:43,470+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ffac-4b2b-a98 7-8c19577cd8c4] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2022-08-23 09:02:43,490+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [4d79272b-ff ac-4b2b-a987-8c19577cd8c4] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'vProtect 2022-08-05 22:33:44.088143' for VM 'web1.wiki.prod.hq.sldev.cz-2'. Snapshot can't be deleted, and VM is not possible to clone. thank you for any help. Jirka

7 12

VM Disk extend not reflected in VM oS
by simon＠justconnect.ie 12 Oct '22

12 Oct '22

Hi All, I extended a VM disk via oVirt Manager (4.4.8) without error, but when I checked the VM (fdisk & lsblk) the size has not changed. Any help as always would be appreciated. Kind Regards Simon...

5 11

Re: oVirt 4.5 self-hosted network not working
by Paul-Erik Törrönen 05 Oct '22

05 Oct '22

On 2022-08-25 22:56, Facundo Badaracco wrote: > Have u checked the IP of the engine VM? Yes. It is what I configured during the setup, and which I use when I ssh from the host to the engine VM. Sorry, I should have mentioned this in my previous message. Poltsi

2 4

oVirt 4.5.2 /var growing rapidly due to ovirt_engine_history db
by sohail_akhter3＠hotmail.com 05 Oct '22

05 Oct '22

Hi, We have recently upgrade our oVirt environment to 4.5.2 version. Environment is based on hosted-engine. Since we have upgrade we noticed rapid incrase in /var partition in engine VM. It is increasing very rapidly. If we vacuum ovirt_engine_history db, /var size reduces but next day size will increase again upto 5-10%. We did db vacuuming couple of time but not sure we it is increasing so rapidly. Here is the partial output of vacuuming that was done on 26-08-22. Table "host_interface_hourly_history" had more entries to be removed. Rest of the table had not much entries. Previously table "host_interface_samples_history" had entries to be removed. Any Idea what can be the reason for that? # dwh-vacuum -f -v SELECT pg_catalog.set_config('search_path', '', false); vacuumdb: vacuuming database "ovirt_engine_history" RESET search_path; SELECT c.relname, ns.nspname FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace ns ON c.relnamespace OPERATOR(pg_catalog.=) ns.oid LEFT JOIN pg_catalog.pg_class t ON c.reltoastrelid OPERATOR(pg_catalog.=) t.oid WHERE c.relkind OPERATOR(pg_catalog.=) ANY (array['r', 'm']) ORDER BY c.relpages DESC; SELECT pg_catalog.set_config('search_path', '', false); VACUUM (FULL, VERBOSE) public.host_interface_samples_history; INFO: vacuuming "public.host_interface_samples_history" INFO: "host_interface_samples_history": found 3135 removable, 84609901 nonremovable row versions in 1564960 pages DETAIL: 0 dead row versions cannot be removed yet. CPU: user: 41.88 s, system: 14.93 s, elapsed: 422.83 s. VACUUM (FULL, VERBOSE) public.host_interface_hourly_history; INFO: vacuuming "public.host_interface_hourly_history" INFO: "host_interface_hourly_history": found 252422 removable, 39904650 nonremovable row versions in 473269 pages Please let me know if any further information is required. Regards Sohail

2 4

Notified that Engine's certification is about to expire but no documentation to renew it
by Guillaume Pavese 04 Oct '22

04 Oct '22

Hello We are receiving the following notifications from our ovirt manager : Message:Engine's certification is about to expire at 2022-05-03. Please renew the engine's certification. Severity:WARNING Effectively : # openssl x509 -in /etc/pki/ovirt-engine/certs/engine.cer -startdate -enddate -noout notBefore=Mar 30 04:48:15 2021 GMT notAfter=May 3 04:48:15 2022 GMT However I can not find any documentation on how to renew this certificate. The following doc only convers changing apache-ca.pem & apache.cer, and not engine.cer Doc oVirt : https://ovirt.org/documentation/administration_guide/index.html#Replacing_t… Doc RHV : https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/ht… Any help ? Guillaume Pavese Ingénieur Système et Réseau Interactiv-Group -- Ce message et toutes les pièces jointes (ci-après le “message”) sont établis à l’intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur, merci de le détruire et d’en avertir immédiatement l’expéditeur. Toute utilisation de ce message non conforme a sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse. L’internet ne permettant pas d’assurer l’intégrité de ce message . Interactiv-group (et ses filiales) décline(nt) toute responsabilité au titre de ce message, dans l’hypothèse ou il aurait été modifié. IT, ES, UK. <https://interactiv-group.com/disclaimer.html>

7 7

Certificate expiration
by Joseph Gelinas 23 Sep '22

23 Sep '22

Hi, The certificates on our oVirt stack recently expired, while all the VMs are still up, I can't put the cluster into global maintenance via ovirt-engine, or do anything via ovirt-engine for that matter. Just get event logs about cert validity. VDSM ovirt-1.xxxxx.com command Get Host Capabilities failed: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed VDSM ovirt-2.xxxxx.com command Get Host Capabilities failed: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed VDSM ovirt-3.xxxxx.com command Get Host Capabilities failed: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed Under Compute -> Hosts, all are status Unassigned. Default data center is status Non Responsive. I have tried a couple of solutions to regenerate the certificates without much luck and have copied the originals back in place. https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/ht… https://access.redhat.com/solutions/2409751 I have seen things saying running engine-setup will generate new certs, however engine doesn't think the cluster is in global maintenance so won't run that, I believe I can get around the check with `engine-setup --otopi-environment=OVESETUP_CONFIG/continueSetupOnHEVM=bool:True` but is that the right thing to do? Will it deploy the certs on to the hosts as well so things communicate properly? Looks like one is supposed to put a node into maintenance and reenroll it after doing the engine-setup, but will it even be able to put the nodes into maintenance given I can't do anything with them now? Appreciate any ideas.

3 18

Re: Should I migrate existing oVirt Engine, or deploy new?
by David White 19 Sep '22

19 Sep '22

Hi Paul, Thanks for the response. I think you're suggesting that I take a hybrid approach, and do a restore of the current Engine onto the new VM. I hadn't thought about this option. Essentially what I was considering was either: - Export to OVA or something OR - Build a completely new oVirt engine with a completely new domain, etc... and try to live migrate the VMs from the old engine to the new engine. Do I understand you correctly that you're suggesting I install the OS onto a new VM, and try to do a restore of the oVirt settings onto the new VM (after I put the cluster into Global maintenance mode and shutdown the old oVirt)? Sent with Proton Mail secure email. ------- Original Message ------- On Friday, August 19th, 2022 at 10:46 AM, Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk> wrote: > Hello David, > I don't think there's a documentated method to go from a Hosted Engine to standalone just the other way standalone to HE. > > I would suggest doing a full backup of the engine prepare the new VM and restore to that rather than trying to export it. > This way you can shut down the original engine and run the new engine VM to test it works as you will be able to restart the original engine if it doesn't work. > > Regards, > Paul S. > > > > > > From: David White via Users <users(a)ovirt.org> > Sent: 19 August 2022 15:27 > To: David White <dmwhite823(a)protonmail.com> > Cc: oVirt Users <users(a)ovirt.org> > Subject: [ovirt-users] Re: Should I migrate existing oVirt Engine, or deploy new? > > Caution External Mail: Do not click any links or open any attachments unless you trust the sender and know that the content is safe. > > In other words, I want to migrate the Engine from a hyperconverged environment into a stand-alone setup. > > > Sent with Proton Mail secure email. > > ------- Original Message ------- > On Friday, August 19th, 2022 at 10:17 AM, David White via Users <users(a)ovirt.org> wrote: > > > > Hello, > > I have just purchased a Synology SA3400 which I plan to use for my oVirt storage domain(s) going forward. I'm currently using Gluster storage in a hyperconverged environment. > > > > My goal now is to: > > > > - Use the Synology Virtual Machine manager to host the oVirt Engine on the Synology > > - Setup NFS storage on the Synology as the storage domain for all VMs in our environment > > - Migrate all VM storage onto the new NFS domain > > - Get rid of Gluster > > > > > > My first step is to migrate the oVirt Engine off of Gluster storage / off the Hyperconverged hosts into the Synology Virtual Machine manager. > > > > Is it possible to migrate the existing oVirt Engine (put the cluster into Global Maintenance Mode, shutdown oVirt, export to VDI or something, and then import into Synology's virtualization)? Or would it be better for me to install a completely new Engine, and then somehow migrate all of the VMs from the old engine into the new engine? > > > > Thanks, > > David > > > > > > Sent with Proton Mail secure email. > > To view the terms under which this email is distributed, please go to:- > https://leedsbeckett.ac.uk/disclaimer/email

2 2

Certificate doesn't contain valid subject alternative name
by Ayansh Rocks 15 Sep '22

15 Sep '22

Hi All, I am getting below alert on my single node ovirt. Can anyone tell me more about it and how I can fix it.I am using ovirt 4.3.8.2. Certificate of host host-name.example.com is invalid. The certificate doesn't contain valid subject alternative name, please enroll new certificate for the host. Thank you

3 5

hosted-engine -vm-status show a ghost node that is not anymore in the cluster: how to remove?
by Diego Ercolani 08 Sep '22

08 Sep '22

engine 4.5.2.4 The issue is that in my cluster when I use the: [root@ovirt-node3 ~]# hosted-engine --vm-status --== Host ovirt-node3.ovirt (id: 1) status ==-- Host ID : 1 Host timestamp : 1633143 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : ovirt-node3.ovirt Local maintenance : False stopped : False crc32 : 1cbfcd19 conf_on_shared_storage : True local_conf_timestamp : 1633143 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1633143 (Wed Aug 31 14:37:53 2022) host-id=1 score=3400 vm_conf_refresh_time=1633143 (Wed Aug 31 14:37:53 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host ovirt-node1.ovirt (id: 2) status ==-- Host ID : 2 Host timestamp : 373629 Score : 0 Engine status : unknown stale-data Hostname : ovirt-node1.ovirt Local maintenance : True stopped : False crc32 : 12a6eb81 conf_on_shared_storage : True local_conf_timestamp : 373630 Status up-to-date : False Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=373629 (Tue Jun 14 16:48:50 2022) host-id=2 score=0 vm_conf_refresh_time=373630 (Tue Jun 14 16:48:50 2022) conf_on_shared_storage=True maintenance=True state=LocalMaintenance stopped=False --== Host ovirt-node2.ovirt (id: 3) status ==-- Host ID : 3 Host timestamp : 434247 Score : 3400 Engine status : {"vm": "down", "health": "bad", "detail": "unknown", "reason": "vm not running on this host"} Hostname : ovirt-node2.ovirt Local maintenance : False stopped : False crc32 : badb3751 conf_on_shared_storage : True local_conf_timestamp : 434247 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=434247 (Wed Aug 31 14:37:45 2022) host-id=3 score=3400 vm_conf_refresh_time=434247 (Wed Aug 31 14:37:45 2022) conf_on_shared_storage=True maintenance=False state=EngineDown stopped=False --== Host ovirt-node4.ovirt (id: 4) status ==-- Host ID : 4 Host timestamp : 1646655 Score : 3400 Engine status : {"vm": "up", "health": "good", "detail": "Up"} Hostname : ovirt-node4.ovirt Local maintenance : False stopped : False crc32 : 1a16027e conf_on_shared_storage : True local_conf_timestamp : 1646655 Status up-to-date : True Extra metadata (valid at timestamp): metadata_parse_version=1 metadata_feature_version=1 timestamp=1646655 (Wed Aug 31 14:37:43 2022) host-id=4 score=3400 vm_conf_refresh_time=1646655 (Wed Aug 31 14:37:43 2022) conf_on_shared_storage=True maintenance=False state=EngineUp stopped=False The problem is that ovirt-node1.ovirt is not anymore in ther cluster, in the host list presented by the ui there is correctly no ovirt-node1, the ovirt-node1 appears only in the commandline. I did a full text search in the engine DB, but node1 doesn't appear anywhere, even in the filesystem, a grep doesn't find anything

2 5

How many oVirt cluster, hosts and VMs do you have running?
by jlhm＠usa.net 02 Sep '22

02 Sep '22

Hi, just trying to understand how our oVirt deployment compare to others. - 75 cluster (and 75 Data centers as we map them 1 to 1 due to security requirements) spanning +3 data centers - 328 oVirt servers - +1900 VMs running Majority are still on 4.3 (CentOS 7) but our engines (4 of them) runs RedHat 8/oVirt 4.4. Working to upgrade all hypervisors to RedHat 8 (or Rocky 8)/oVirt 4.4. When that is done we will start to upgrade to latest oVirt version but it takes time due to the size of our environment and we move slowly to ensure stability.

2 1

Ubuntu NFS
by thilburn＠generalpacific.com 01 Sep '22

01 Sep '22

Hello, I was having trouble with getting an Ubuntu 22.04 NFS share working and after searching for hours I was able to figure out what was needed. Below is what I found if anyone else runs into this. My error was engine.log "...Unexpected return value: Status [code=701, message=Could not initialize cluster lock: ()]" Host supervdsm.log -open error -13 EACCES: no permission to open /ThePath/ids -check that daemon user sanlock *** group sanlock *** has access to disk or file. The fix was changing /etc/nfs.conf manage-gids=y ( Which is the default ) to # manage-gids=y ( Commenting this sets the default which is no ) It looks like in the past the fix was to change /etc/default/nfs-kernel-server Line RPCMOUNTDOPTS="--manage-gids" which I didn't need to change.

2 5

how kill backup operation
by Diego Ercolani 01 Sep '22

01 Sep '22

Hello I saw there are other thread asking how to delete disk snapshots from backup operation. We definitively need a tool to kill pending backup operations and locked snapshots. I Think this is very frustrating ovirt is a good piece of software but it's very immature in a dirty asyncronous world. We need a unified toolbox to clean manually and do database housekeeping.

3 12

Interested in contributing with Spanish translation.
by Luis Pereida 01 Sep '22

01 Sep '22

Hello, My name is Luis Pereida, I am Mexican, from Guadalajara and I am currently an application security specialist. Some time ago I met the O-Virt project and it helped me a lot to solve many situations where virtualization was the perfect option. Since some time ago I have been thinking about how to contribute to the project, and talking with some friends, they would like to have documentation in Spanish. Although English is basic for us, many times the context or expressions are hard to understand. I would like to help with that. How can I do it? I see that it is necessary to use a Zanata account. How can I get an account? Regards and thanks for being so supportive of the community.

3 2

Q: Engione 4.4.10 and New Host on CentOS Stream 9
by Andrei Verovski 31 Aug '22

31 Aug '22

Hi, I have engine version 4.4.10.7-1.el8, is it possible to set up new node host on CentOS Stream 9, or I need to upgrade engine to version 4.5 first (which is not right now possible because of some quite old nodes) ? Thanks Andrei

4 11

Moving SelfHosted Engine
by murat.celebi＠dbpro.com.tr 31 Aug '22

31 Aug '22

Hello We have got an Production ovirt installation with 1 datacenter and 2 clusters. Default Cluster has got 2 hosts in it. kvm01 and kvm02. PrimarySite cluster has got 2 hosts in it. kvm03 and kvm04 Our Self hosted engine is running on kvm01. We need to migrate the Selfhosted-engine to kvm03 or kvm04 since the hosts kvm01 and kvm02 hosts are going to be retired. Anyone has any idea how this can be accomplished?

3 2

Failed to delete snapshot (again)
by Giulio Casella 30 Aug '22

30 Aug '22

Hi folks, since some month I have an issue with snapshot removals (I have storware vprotect backup system, heavily using snapshots). After some time spent on a bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=1948599) we discovered that my issue is not depending on that bug :-( So they pointed me here again. Briefly: sometime snapshot removal fails, leaving snapshot in illegal state. Trying to remove again (via ovirt UI) keeps failing and doesn't solve. The only way to rebuild a consistent situation is live migrating affected disk to another storage domain; after moving the disk, snapshot is no more marked illegal and then I can remove it. You can imagine this is a bit tricky, specially for large disks. In my logs I can find: 2022-08-29 09:17:11,890+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-1) [0eced56f-689d-422b-b15c-20b824377b08] Failed to live merge. Top volume f8f84b1c-53ab-4c99-a01d-743ed3d7859b is still in qemu chain [0ea89fbc-d39a-48ff-aa2b-0381d79d7714, 55bb387f-01a6-41b6-b585-4bcaf2ea5e32, f8f84b1c-53ab-4c99-a01d-743ed3d7859b] My setup is ovirt-engine-4.5.2.4-1.el8.noarch, with hypervisors based on oVirt Node 4.5.2 (vdsm-4.50.2.2-1.el8). Thank you in advance. Regards, gc

3 4

Ovirt 4.4.7, can't renew certificate of ovirt engine (certificates expired)
by vk＠itiviti.com 30 Aug '22

30 Aug '22

Hi Team, I'm looking for your help since I didn't find any clear documentation. Is there somewhere in ovirt website a clear documentation about how to renew the engine certificates located in /etc/pki/ovirt-engine/certs/ We have an engine GUI not working, showing error message "PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed". After checking, all the cert in /etc/pki/ovirt-engine/certs/ are expired. I didn't find a clear documentation on ovirt website, or even on redhat website (it was always about host but not the engine) Anyway I've read that the renew process can be done via "engine-setup --offline", but when I try it, it generates this error: --== PKI CONFIGURATION ==-- [ ERROR ] Failed to execute stage 'Environment customization': Unable to load certificate. See https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for more details. and in log file: File "/usr/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1371, in load_pem_x509_certificate "Unable to load certificate. See https://cryptography.io/en/la" ValueError: Unable to load certificate. See https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for more details. 2022-08-29 19:16:29,502+0200 ERROR otopi.context context._executeMethod:154 Failed to execute stage 'Environment customization': Unable to load certificate. See https://cryptography.io/en/latest/faq/#why-can-t-i-import-my-pem-file for more details. I've also tried the manual procedure (using /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh) mentioned in https://users.ovirt.narkive.com/4ugjgicE/ovirt-regenerating-new-ssl-certifi… (message from Alon Bar-Lev), but the 4th command always says I enter a wrogn apssword, but it's not. we are blocked here and we can't use our ovirt cluster, so it's pretty blocking. Thx a lot in advance

2 3

unable to bring up gluster bricks after 4.5 upgrade
by Jayme 29 Aug '22

29 Aug '22

Hello All, I've been struggling with a few issues upgrading my 3-node HCI custer from 4.4 to 4.5. At present the self hosted engine VM is properly running oVirt 4.5 on CentOS 8x stream. First host node, I set in maintenance and installed new node-ng image. I ran into issue with rescue mode on boot which appears to have been related to LVM devices bug. I was able to work past that and get the node to boot. The node running 4.5.2 image is booting properly and gluster/lvm mounts etc all look good. I am able to activate the host and run VMs on it etc. however, oVirt cli is showing that all bricks on host are DOWN. I was unable to get the bricks back up even after doing a force start of the volumes. Here is the glusterd log from the host in question when I try force start on the engine volume (other volumes are similar: ==> glusterd.log <== The message "I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: bitd service is stopped" repeated 2 times between [2022-08-29 18:09:56.027147 +0000] and [2022-08-29 18:10:34.694144 +0000] [2022-08-29 18:10:34.695348 +0000] I [MSGID: 106618] [glusterd-svc-helper.c:909:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=engine) to existing process with pid 2473 [2022-08-29 18:10:34.695669 +0000] I [MSGID: 106131] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped [2022-08-29 18:10:34.695691 +0000] I [MSGID: 106568] [glusterd-svc-mgmt.c:266:glusterd_svc_stop] 0-management: scrub service is stopped [2022-08-29 18:10:34.695832 +0000] I [MSGID: 106617] [glusterd-svc-helper.c:698:glusterd_svc_attach_cbk] 0-management: svc glustershd of volume engine attached successfully to pid 2473 [2022-08-29 18:10:34.703718 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster2.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703774 +0000] E [MSGID: 106115] [glusterd-mgmt.c:119:gd_mgmt_v3_collate_errors] 0-management: Post commit failed on gluster1.xxxxx. Please check log file for details. [2022-08-29 18:10:34.703797 +0000] E [MSGID: 106664] [glusterd-mgmt.c:1969:glusterd_mgmt_v3_post_commit] 0-management: Post commit failed on peers [2022-08-29 18:10:34.703800 +0000] E [MSGID: 106664] [glusterd-mgmt.c:2664:glusterd_mgmt_v3_initiate_all_phases] 0-management: Post commit Op Failed If I run start command manually on host cli: gluster volume start engine force volume start: engine: failed: Post commit failed on gluster1.xxxx. Please check log file for details. Post commit failed on gluster2.xxxx. Please check log file for details. I feel like this may be some issue with the difference in major versions of GlusterFS on the nodes but I am unsure. The other nodes are running ovirt-node-ng-4.4.6.3 At this point I am afraid to bring down any other node to attempt upgrading it without the bricks in UP status on the first host. I do not want to lose quorum and potentially disrupt running VMs. Any idea why I can't seem to start the volumes on the upgraded host? Thanks!

1 2

(no subject)
by parallax 29 Aug '22

29 Aug '22

oVirt version:4.4.4.7-1.el8 I have several servers in cluster and I got this error: Data Center is being initialized, please wait for initialization to complete. VDSM command GetStoragePoolInfoVDS failed: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed the SPM role is constantly being transferred to the servers and I can't do anything in StorageDomains storages are in inactive status but virtual machines are runnig how ti fix it?

3 6

Commissioning hew host on centos9
by David Johnson 29 Aug '22

29 Aug '22

Good evening all, I am trying to commission an ovirt host on new hardware using Centos 9. I followed the pre-install instructions at https://www.ovirt.org/download/install_on_rhel.html, and eventually figured out that this line was missing from the preparation scripts: dnf install rdo-openvswitch-2.15 With that, I was able to continue installation. After completing the instructions, DNF update reports this: [root@localhost administrator]# dnf update --nobest Last metadata expiration check: 0:01:06 ago on Fri 26 Aug 2022 08:20:27 PM CDT. Dependencies resolved. Problem 1: package ovirt-openvswitch-2.15-4.el9.noarch requires openvswitch2.15, but none of the providers can be installed - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-99.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-51.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-56.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-81.el9s.x86_64 - cannot install the best update candidate for package ovirt-openvswitch-2.15-4.el9.noarch - cannot install the best update candidate for package openvswitch2.15-2.15.0-99.el9s.x86_64 Problem 2: problem with installed package ovirt-openvswitch-2.15-4.el9.noarch - package ovirt-openvswitch-2.15-4.el9.noarch requires openvswitch2.15, but none of the providers can be installed - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-99.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-51.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-56.el9s.x86_64 - package rdo-openvswitch-2:2.17-2.el9s.noarch obsoletes openvswitch2.15 < 2.17 provided by openvswitch2.15-2.15.0-81.el9s.x86_64 - cannot install the best update candidate for package rdo-openvswitch-1:2.15-2.el9s.noarch DNF info lists these [root@localhost administrator]# dnf list installed |grep openvswitch centos-release-nfv-openvswitch.noarch 1-4.el9s @c9s-extras-common network-scripts-openvswitch2.15.x86_64 2.15.0-99.el9s @centos-nfv-openvswitch openvswitch-selinux-extra-policy.noarch 1.0-31.el9s @centos-nfv-openvswitch *openvswitch2.15.x86_64 2.15.0-99.el9s @centos-nfv-openvswitch*ovirt-openvswitch.noarch 2.15-4.el9 @centos-ovirt45 python3-openvswitch2.15.x86_64 2.15.0-99.el9s @centos-nfv-openvswitch rdo-openvswitch.noarch 1:2.15-2.el9s @centos-openstack-yoga *Questions:* 1. Is this a problem? 2. If this is a problem, is there an FAQ sheet or known steps to work around/fix this? 3. Does there need to be an update to the pre-installation instructions? *David Johnson* *Director of Development, Maxis Technology* 844.696.2947 ext 702 (o) | 479.531.3590 (c) <https://www.linkedin.com/in/pojoguy/> <https://maxistechnology.com/wp-content/uploads/vcards/vcard-David_Johnson.v…> <https://maxistechnology.com/> *Follow us:* <https://www.linkedin.com/company/maxis-tech-inc/>

2 1

oVirt 4.5 self-hosted network not working
by Paul-Erik Törrönen 25 Aug '22

25 Aug '22

Specifically, the installation (ovirt-hosted-engine-setup) completes successfully, and the engine VM is up and running, but it can not be accessed from any other machine in the same subnet but the host machine. I can ssh (from the host) to the engine VM, and confirm that initiating the connection from the engine VM to any other machine, except the host, fails. The routing looks correct and stopping the firewalld on the engine VM does not make any difference. Nor does stopping the firewalld on the host either make any difference. This seems like a failure in correctly setting up the host routing for the engine VM. How do I fix this? Poltsi

1 0