- Users - oVirt List Archives

oVirt 4.4 - Integrate event notifications with System Center Operations Manager
by angus＠ajct.uk 02 Jun '23

02 Jun '23

Hello I am looking to integrate oVirt 4.4 event notifications with an incident management tool - M$ System Center Operations Manager. I need to give examples to SCOM of what an unhealthy email looks like and what a recover/healthy email looks like for each event I want it to manage. Is there a way of triggering sample notifications (something failed/something recovered) for events or perhaps I can see the email templates used somewhere? (I did try a grep -Hr on the engine VM for "alertMessage" and "resolveMessage" for a VM migration notification but only found an entry in the in the notifier log. Thanks Angus

1 0

Regarding oVirt IP Address Change
by Pro Cloudguru 02 Jun '23

02 Jun '23

Hello oVIrt Community, Presently, I've a healthy & running oVirt Hosting that includes single DC, Cluster, SPM & Storage. The LAN IP Address range is 10.151.65.0/24. As we're in the process of migrating office premises, I'm worried about the existing oVirt IP Address Schema as the new range is 10.101.160/0/20. In this scenario, I'd like to know how the migration happens successfully without disturbing the oVirt Hosting & all hosted VMs. Please Guide, *Nishith N. Vyas*

1 0

novnc console error: promise.js missing?
by karl.morgan＠gmail.com 02 Jun '23

02 Jun '23

Seeing the following on the engine when attempting to start a novnc console 2023-03-31 11:57:34,988-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-29) [307c22a9] START, SetVmTicketVDSCommand(HostName = ov2node02-mn, SetVmTicketVDSCommandParameters:{hostId='19a92de3-c6e4-4f4e-be31-1d5533a2b6b6', vmId='4481b0e9-a96c-4ee7-8bd2-572558eb9fda', protocol='VNC', ticket='LMl903dd', validTime='120', userName='admin@ovirt', userId='9e88a363-34cb-466a-8ba9-25e46819423b', disconnectAction='LOCK_SCREEN', consoleDisconnectActionDelay='0'}), log id: 1d6520d5 2023-03-31 11:57:35,031-07 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (default task-29) [307c22a9] FINISH, SetVmTicketVDSCommand, return: , log id: 1d6520d5 2023-03-31 11:57:35,044-07 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-29) [307c22a9] EVENT_ID: VM_SET_TICKET(164), User admin@ovirt@internalkeycloak-authz initiated console session for VM first 2023-03-31 11:57:35,158-07 INFO [org.ovirt.engine.core.utils.servlet.ServletUtils] (default task-29) [] Can't read file '/usr/share/ovirt-engine/files/novnc/vendor/promise.js' for request '/ovirt-engine/services/files/novnc/vendor/promise.js' -- 404 Curious if this is obvious to anyone? How to resolve? ovirt-engine-webadmin-portal-4.5.4-1.el9.noarch ovirt-engine-4.5.4-1.el9.noarch

2 1

troubleshooting SHE - Run engine-setup with answerfile - ?
by lejeczek 01 Jun '23

01 Jun '23

Hi guys. I'm revisiting my failed attempt to make friends with oVirt - novice so go easy on me. Deployment fails sooner that what can troubleshoot - at least I manged to find in docs so far - and I wanted to ask hot to troubleshot such failure as this: ... INFO ] TASK [ovirt.ovirt.engine_setup : Run engine-setup with answerfile] [ ERROR ] fatal: [localhost -> 192.168.1.209]: FAILED! => {"changed": true, "cmd": ["engine-setup", "--accept-defaults", "--config-append=/root/ovirt-engine-answers"], "delta": "0:08:05.997322", "end": "2023-06-01 13:07:15.932517", "msg": "non-zero return code", "rc": 1, "start": "2023-06-01 12:59:09.935195", "stderr": "", "stderr_lines": [], "stdout": "[ INFO ] Stage: Initializing\n[ INFO ] Stage: Environment setup\n Configuration files:\n ... Start with setting up Keycloak for Ovirt Engine", "[ ERROR ] Failed to execute stage 'Closing up': Command '/usr/share/ovirt-engine-keycloak/bin/kk_cli.sh' failed to execute", "[ INFO ] Stage: Clean up", " Log file is located at", " /var/log/ovirt-engine/setup/ovirt-engine-setup-20230601125911-448xvj.log", ... [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"} [ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook [ INFO ] Stage: Clean up ... Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230601123237-rhwx8e.log Logfile to which 'stdout' points, shows nothing or I fail to find anything in it and/or understand it: ... 2023-06-01 13:07:17,716+0100 ERROR otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:113 fatal: [localhost -> 192.168.1.209]: FAILED! => {"changed": true, "cmd": ["engine-setup", "--accept-defaults", "--config-append=/root/ovirt-engine-answers"], "delta": "0:08:05.997322", "end": "2023-06-01 13:07:15.932517", "msg": "non-zero return code", "rc": 1, "start": "2023-06-01 12:59:09.935195", "stderr": "", "stderr_lines": [], "stdout": ... or Ovirt Engine\n[ ERROR ] Failed to execute stage 'Closing up': Command '/usr/share/ovirt-engine-keycloak/bin/kk_cli.sh' failed to execute\n[ INFO ] Stage: Clean up\n Log file is located at\n /var/log/ovirt-engine/setup/ovirt-engine-setup-20230601125911-448xvj.log\n[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20230601130714-setup.conf'\n[ INFO ] Stage: Pre-termination\n[ INFO ] Stage: Termination\n[ ERROR ] Execution of setup failed", "stdout_lines": ["[ INFO ] Stage: Initializing", "[ INFO ] Stage: Environment setup", " Configuration files:", " /etc/ovirt-engine-setup.conf.d/10-packaging-jboss.conf,", " /etc/ovirt-engine-setup.conf.d/10-packaging.conf," ... ... 2023-06-01 13:08:52,946+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'msg': "The tas k includes an option with an undefined variable. The error was: 'local_vm_ip' is undefined. 'local_vm_ip' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_engine_setup/tasks/sync_on_engine_ma chine.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appear s to be:\n\n---\n- name: Set the name for add_host\n ^ here\n", '_ansible_no_log': False} 2023-06-01 13:08:53,047+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localh ost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'local_vm_ip' is undefined. 'local _vm_ip' is undefined\n\nThe error appears to be in '/usr/share/ansible/collections/ansible_collections/ovirt/ovirt/roles/hosted_eng ine_setup/tasks/sync_on_engine_machine.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax prob lem.\n\nThe offending line appears to be:\n\n---\n- name: Set the name for add_host\n ^ here\n"} ... 2023-06-01 13:09:16,818+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 {'changed': True, 'stdout': '', 'stderr': "error: Failed to destroy pool 9ab14cca-78e0-4a8e-8a1b-00e98c300208\nerror: Requested operation is not valid: storage pool '9ab14cca-78e0-4a8e-8a1b-00e98c300208' is not active", 'rc': 1, 'cmd': ['virsh', '-c', 'qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf', 'pool-destroy', '9ab14cca-78e0-4a8e-8a1b-00e98c300208'], 'start': '2023-06-01 13:09:16.614978', 'end': '2023-06-01 13:09:16.672260', 'delta': '0:00:00.057282', 'msg': 'non-zero return code', 'invocation': {'module_args': {'_raw_params': 'virsh -c qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf pool-destroy 9ab14cca-78e0-4a8e-8a1b-00e98c300208', '_uses_shell': False, 'stdin_add_newline': True, 'strip_empty_ends': True, 'argv': None, 'chdir': None, 'executable': None, 'creates': None, 'removes': None, 'stdin': None}}, 'stdout_lines': [], 'stderr_lines': ['error: Failed to destroy pool 9ab14cca-78e0-4a8e-8a1b-00e98c300208', "error: Requested operation is not valid: storage pool '9ab14cca-78e0-4a8e-8a1b-00e98c300208' is not active"], '_ansible_no_log': None} 2023-06-01 13:09:16,918+0100 DEBUG otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:109 ignored: [localhost]: FAILED! => {"changed": true, "cmd": ["virsh", "-c", "qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf", "pool-destroy", "9ab14cca-78e0-4a8e-8a1b-00e98c300208"], "delta": "0:00:00.057282", "end": "2023-06-01 13:09:16.672260", "msg": "non-zero return code", "rc": 1, "start": "2023-06-01 13:09:16.614978", "stderr": "error: Failed to destroy pool 9ab14cca-78e0-4a8e-8a1b-00e98c300208\nerror: Requested operation is not valid: storage pool '9ab14cca-78e0-4a8e-8a1b-00e98c300208' is not active", "stderr_lines": ["error: Failed to destroy pool 9ab14cca-78e0-4a8e-8a1b-00e98c300208", "error: Requested operation is not valid: storage pool '9ab14cca-78e0-4a8e-8a1b-00e98c300208' is not active"], "stdout": "", "stdout_lines": []} 2023-06-01 13:09:17,023+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 TASK [ovirt.ovirt.hosted_engine_setup : Undefine local storage-pool 9ab14cca-78e0-4a8e-8a1b-00e98c300208] 2023-06-01 13:09:17,224+0100 INFO otopi.ovirt_hosted_engine_setup.ansible_utils ansible_utils._process_output:115 changed: [localhost] .. I think that last two snippets-blocks, might be most useful - but what does it says? When 'installer' is finished, there is no aforementioned pool however there is: -> $ virsh pool-list | grep '9ab14cca-78e0-4a8e-8a1b-00e98c300208' 9ab14cca-78e0-4a8e-8a1b-00e98c300208-1 active yes Engine VM remains is up & running, How far from successful deployment that is & what would be advised to do next? many thanks, L.

1 0

OKD Installer Provisioned Infrastructure in oVirt > 4.3
by Fredrik Arneving 30 May '23

30 May '23

Hi, I've successfully installed OKD <= 4.8 in my ovirt 4.3 self-hosted engine setup. However, I have now found out that the imageio-proxy and/or imageio-deamon has changed it's API calls and ports to make the OKD install fail on self-hosted (all-in-one) setups on ovirt 4.4 and 4.5. Since I've upgraded to version 4.5 I can no longer install OKD with Installer Provisioned Infrastructure. I recently read throough a long thread in bugzilla (https://bugzilla.redhat.com/show_bug.cgi?id=1871348) on how to get image uploads working from the GUI but haven't found any solutions for making the OKD install work. The suggested solution, engine-config -s "EnableImageioProxy=false", didn't solve my problem. Would it be possible to tweak the setup so that the OKD installer can upload it's image through imageio? I don't use the imageio service for anything else since I (almost) only kickstart my new vm's from my foreman servers so if other use cases than the OKD install would fail I can live with that. I take it the problem only appears for a self-hosted setup. If it was possible to migrate the engine to a standalone host (not a powerful compute node but somewhere with 4vCPU and 16 GB RAM) would that make the solution easier? I would lose a lot of my current infrastructure if I had to reinstall but if I can't find a better solution I will have to. Any help would be appreciated.

1 0

VM host deploy failed
by Latchezar Filtchev 30 May '23

30 May '23

Hi there, Centos Stream 8 fresh standalone install from nightly build of ovirt-engine. VM host Centos Stream 8 node nightly build repo enabled. Cannot add host to engine. The error message: Host n1 installation failed. Task Install ovirt-hosted-engine-setup package failed to execute. Please check logs for more details: /var/log/ovirt-engine/host-deploy/ovirt-host-deploy-ansible-20230530094214-n1............. Any hint will be appreciated. Thank you! Best, Latcho 2023-05-30 09:42:26 EEST - { "uuid" : "e905de36-7b04-4b52-96b8-71855d561446", "counter" : 99, "stdout" : "fatal: [n1.aubgin.local]: FAILED! => {\"changed\": false, \"failures\": [], \"msg\": \"Depsolve Error occurred: \\n Problem: conflicting requests\\n - nothing provides python3-importlib-resources needed by ovirt-hosted-engine-setup-2.7.1-0.0.master.20230414113600.git340e19b.el8.noarch\", \"rc\": 1, \"results\": []}", "start_line" : 87, "end_line" : 88, "runner_ident" : "c7d833e9-8de3-4d53-87a3-72c3d80177ce", "event" : "runner_on_failed", "pid" : 41043, "created" : "2023-05-30T06:42:25.111204", "parent_uuid" : "ac162d01-a065-3587-db9e-000000000228", "event_data" : { "playbook" : "ovirt-host-deploy.yml", "playbook_uuid" : "de0dd632-7d53-4e08-a06c-610334200c49", "play" : "all", "play_uuid" : "ac162d01-a065-3587-db9e-000000000006", "play_pattern" : "all", "task" : "Install ovirt-hosted-engine-setup package", "task_uuid" : "ac162d01-a065-3587-db9e-000000000228", "task_action" : "ansible.builtin.yum", "task_args" : "", "task_path" : "/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-vdsm/tasks/packages.yml:6", "role" : "ovirt-host-deploy-vdsm", "host" : "n1.aubgin.local", "remote_addr" : "n1.aubgin.local", "res" : { "failures" : [ ], "results" : [ ], "rc" : 1, "msg" : "Depsolve Error occurred: \n Problem: conflicting requests\n - nothing provides python3-importlib-resources needed by ovirt-hosted-engine-setup-2.7.1-0.0.master.20230414113600.git340e19b.el8.noarch", "invocation" : { "module_args" : { "name" : [ "ovirt-hosted-engine-setup" ], "state" : "present", "allow_downgrade" : false, "autoremove" : false, "bugfix" : false, "cacheonly" : false, "disable_gpg_check" : false, "disable_plugin" : [ ], "disablerepo" : [ ], "download_only" : false, "enable_plugin" : [ ], "enablerepo" : [ ], "exclude" : [ ], "installroot" : "/", "install_repoquery" : true, "install_weak_deps" : true, "security" : false, "skip_broken" : false, "update_cache" : false, "update_only" : false, "validate_certs" : true, "sslverify" : true, "lock_timeout" : 30, "allowerasing" : false, "nobest" : false, "conf_file" : null, "disable_excludes" : null, "download_dir" : null, "list" : null, "releasever" : null } }, "_ansible_no_log" : null, "changed" : false }, "start" : "2023-05-30T06:42:23.217686", "end" : "2023-05-30T06:42:25.111007", "duration" : 1.893321, "ignore_errors" : null, "event_loop" : null, "uuid" : "e905de36-7b04-4b52-96b8-71855d561446" } } [root@e host-deploy]#

1 0

NIC Ordering
by Alan G 25 May '23

25 May '23

Is there any way to enforce NIC ordering so the vNICs match the ordering in the Engine UI? I found this but not clear if it was ever implemented? https://www.ovirt.org/develop/release-management/features/network/predictab…

3 6

ERROR HSMGetAllTasksStatusesVDS Cannot deactivate Logical Volume:
by parallax 25 May '23

25 May '23

Hello environment: oVirt 4.5.4-1 messages like this appear in the event log what do these errors mean? the volume 114f08c4-f864-44e1-a937-19405307e643 has 128m size ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-18) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM kvm3 command HSMGetAllTasksStatusesVDS failed: value=Cannot deactivate Logical Volume: 'error=General Storage Exception: ("5 [] [\' Logical volume 3526ebaa-31de-4c9f-b724-f0fbe5448a78/114f08c4-f864-44e1-a937-19405307e643 in use.\']\\n3526ebaa-31de-4c9f-b724-f0fbe5448a78/[\'114f08c4-f864-44e1-a937-19405307e643\']",), users={\'/dev/3526ebaa-31de-4c9f-b724-f0fbe5448a78/114f08c4-f864-44e1-a937-19405307e643\': [{\'pid\': 2227780, \'command\': \'systemd-udevd\', \'user\': \'root\', \'fd\': 7}]}' abortedcode=552

1 0

Extended ovirt image disk and now virtual size is < 1 GiB and actual size i +1TB
by arc＠b4restore.com 24 May '23

24 May '23

Hi, i extended 16 virtual image disks today by 1000GiB through the GUI and now we have a problem with some of them... 9 of them are shown if "edited" that they have a size of 0GiB and not 16360GiB as expected? 5 of them are not able to be extended either within the VM. we tried to power it off and then on again (not reboot, but actual power off / power on) any suggestions to get out of this weird mode? disks seems as if they have the "old" size within the VM but are shown in oVirt as if they are 0GiB.? They are relying on a posix compliant fs if that makes any difference, and they are thin-provisioned and discard-enabled. Thanks in advance.

2 3

VM causes CPU blocks and forces reboot of host
by dominik.drazyk＠blackrack.pl 23 May '23

23 May '23

Hello, I have recently migrated our customer's cluster to newer hardware (CentOS 8 Stream, 4 hypervisor nodes, 3 hosts with GlusterFS 5x 6TB SSD as JBOD with replica 3). After 1 month from the switch we encounter frequent vm locks that need host reboot in order to unlock the VM. Affected vms cannot be powered down from ovirt UI. Even if ovirt is successful in powering down affected vms, they cannot be booted again with information that OS disk is used. Once I reboot the host, vms can be turned on and everything works fine. In vdsm logs I can note the following error: 2023-05-11 19:33:12,339+0200 ERROR (qgapoller/1) [virt.periodic.Operation] <bound method QemuGuestAgentPoller._poller of <vdsm.virt.qemuguestagent.QemuGuestAgentPoller object at 0x7f553aa3e470>> operation failed (periodic:187) Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/virt/periodic.py", line 185, in __call__ self._func() File "/usr/lib/python3.6/site-packages/vdsm/virt/qemuguestagent.py", line 476, in _poller vm_id, self._qga_call_get_vcpus(vm_obj)) File "/usr/lib/python3.6/site-packages/vdsm/virt/qemuguestagent.py", line 797, in _qga_call_get_vcpus if 'online' in vcpus: TypeError: argument of type 'NoneType' is not iterable /var/log/messages reports: May 11 19:35:15 kernel: task:CPU 7/KVM state:D stack: 0 pid: 7065 ppid: 1 flags: 0x80000182 May 11 19:35:15 kernel: Call Trace: May 11 19:35:15 kernel: __schedule+0x2d1/0x870 May 11 19:35:15 kernel: schedule+0x55/0xf0 May 11 19:35:15 kernel: schedule_preempt_disabled+0xa/0x10 May 11 19:35:15 kernel: rwsem_down_read_slowpath+0x26e/0x3f0 May 11 19:35:15 kernel: down_read+0x95/0xa0 May 11 19:35:15 kernel: get_user_pages_unlocked+0x66/0x2a0 May 11 19:35:15 kernel: hva_to_pfn+0xf5/0x430 [kvm] May 11 19:35:15 kernel: kvm_faultin_pfn+0x95/0x2e0 [kvm] May 11 19:35:15 kernel: ? select_task_rq_fair+0x355/0x990 May 11 19:35:15 kernel: ? sched_clock+0x5/0x10 May 11 19:35:15 kernel: ? sched_clock_cpu+0xc/0xb0 May 11 19:35:15 kernel: direct_page_fault+0x3b4/0x860 [kvm] May 11 19:35:15 kernel: kvm_mmu_page_fault+0x114/0x680 [kvm] May 11 19:35:15 kernel: ? vmx_vmexit+0x9f/0x70d [kvm_intel] May 11 19:35:15 kernel: ? vmx_vmexit+0xae/0x70d [kvm_intel] May 11 19:35:15 kernel: ? gfn_to_pfn_cache_invalidate_start+0x190/0x190 [kvm] May 11 19:35:15 kernel: vmx_handle_exit+0x177/0x770 [kvm_intel] May 11 19:35:15 kernel: ? gfn_to_pfn_cache_invalidate_start+0x190/0x190 [kvm] May 11 19:35:15 kernel: vcpu_enter_guest+0xafd/0x18e0 [kvm] May 11 19:35:15 kernel: ? hrtimer_try_to_cancel+0x7b/0x100 May 11 19:35:15 kernel: kvm_arch_vcpu_ioctl_run+0x112/0x600 [kvm] May 11 19:35:15 kernel: kvm_vcpu_ioctl+0x2c9/0x640 [kvm] May 11 19:35:15 kernel: ? pollwake+0x74/0xa0 May 11 19:35:15 kernel: ? wake_up_q+0x70/0x70 May 11 19:35:15 kernel: ? __wake_up_common+0x7a/0x190 May 11 19:35:15 kernel: do_vfs_ioctl+0xa4/0x690 May 11 19:35:15 kernel: ksys_ioctl+0x64/0xa0 May 11 19:35:15 kernel: __x64_sys_ioctl+0x16/0x20 May 11 19:35:15 kernel: do_syscall_64+0x5b/0x1b0 May 11 19:35:15 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6 May 11 19:35:15 kernel: RIP: 0033:0x7faf1a1387cb May 11 19:35:15 kernel: Code: Unable to access opcode bytes at RIP 0x7faf1a1387a1. May 11 19:35:15 kernel: RSP: 002b:00007fa6f5ffa6e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 May 11 19:35:15 kernel: RAX: ffffffffffffffda RBX: 000055be52e7bcf0 RCX: 00007faf1a1387cb May 11 19:35:15 kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000027 May 11 19:35:15 kernel: RBP: 0000000000000000 R08: 000055be5158c6a8 R09: 00000007d9e95a00 May 11 19:35:15 kernel: R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000000 May 11 19:35:15 kernel: R13: 000055be515bcfc0 R14: 00007fffec958800 R15: 00007faf1d6c6000 May 11 19:35:15 kernel: INFO: task worker:714626 blocked for more than 120 seconds. May 11 19:35:15 kernel: Not tainted 4.18.0-489.el8.x86_64 #1 May 11 19:35:15 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 11 19:35:15 kernel: task:worker state:D stack: 0 pid:714626 ppid: 1 flags:0x00000180 May 11 19:35:15 kernel: Call Trace: May 11 19:35:15 kernel: __schedule+0x2d1/0x870 May 11 19:35:15 kernel: schedule+0x55/0xf0 May 11 19:35:15 kernel: schedule_preempt_disabled+0xa/0x10 May 11 19:35:15 kernel: rwsem_down_read_slowpath+0x26e/0x3f0 May 11 19:35:15 kernel: down_read+0x95/0xa0 May 11 19:35:15 kernel: do_madvise.part.30+0x2c3/0xa40 May 11 19:35:15 kernel: ? syscall_trace_enter+0x1ff/0x2d0 May 11 19:35:15 kernel: ? __x64_sys_madvise+0x26/0x30 May 11 19:35:15 kernel: __x64_sys_madvise+0x26/0x30 May 11 19:35:15 kernel: do_syscall_64+0x5b/0x1b0 May 11 19:35:15 kernel: entry_SYSCALL_64_after_hwframe+0x61/0xc6 May 11 19:35:15 kernel: RIP: 0033:0x7faf1a138a4b May 11 19:35:15 kernel: Code: Unable to access opcode bytes at RIP 0x7faf1a138a21. May 11 19:35:15 kernel: RSP: 002b:00007faf151ea7f8 EFLAGS: 00000206 ORIG_RAX: 000000000000001c May 11 19:35:15 kernel: RAX: ffffffffffffffda RBX: 00007faf149eb000 RCX: 00007faf1a138a4b May 11 19:35:15 kernel: RDX: 0000000000000004 RSI: 00000000007fb000 RDI: 00007faf149eb000 May 11 19:35:15 kernel: RBP: 0000000000000000 R08: 00000007faf080ba R09: 00000000ffffffff May 11 19:35:15 kernel: R10: 00007faf151ea760 R11: 0000000000000206 R12: 00007faf15aec48e May 11 19:35:15 kernel: R13: 00007faf15aec48f R14: 00007faf151eb700 R15: 00007faf151ea8c0 May 11 19:35:15 kernel: INFO: task worker:714628 blocked for more than 120 seconds. May 11 19:35:15 kernel: Not tainted 4.18.0-489.el8.x86_64 #1 May 11 19:35:15 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Installed VDSM packages: vdsm-api-4.50.3.4-1.el8.noarch vdsm-network-4.50.3.4-1.el8.x86_64 vdsm-yajsonrpc-4.50.3.4-1.el8.noarch vdsm-http-4.50.3.4-1.el8.noarch vdsm-client-4.50.3.4-1.el8.noarch vdsm-4.50.3.4-1.el8.x86_64 vdsm-gluster-4.50.3.4-1.el8.x86_64 vdsm-python-4.50.3.4-1.el8.noarch vdsm-jsonrpc-4.50.3.4-1.el8.noarch vdsm-common-4.50.3.4-1.el8.noarch Libvirt: libvirt-client-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-nodedev-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-logical-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-network-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-qemu-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-scsi-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-core-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-config-network-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-iscsi-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-rbd-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-libs-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-config-nwfilter-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-secret-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-disk-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-mpath-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-gluster-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 python3-libvirt-8.0.0-2.module_el8.7.0+1218+f626c2ff.x86_64 libvirt-daemon-driver-nwfilter-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-lock-sanlock-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-interface-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-driver-storage-iscsi-direct-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 libvirt-daemon-kvm-8.0.0-14.module_el8.8.0+1257+0c3374ae.x86_64 During the issue with locked vms, they do not respond from network, I cannot use VNC console (or any other) to check what is happening from VM perspective. The host cannot list running processes. There are plenty of resources left and each host runs about 30-35 vms. First I thought that it might be related to glusterfs (I use gluster on other clusters and it usually works fine), so we migrated all vms back to old storage (NFS). The problem came back today on two hosts. I do not have such issues on other cluster which runs on Rocky 8.6 with hyperconverged glusterfs. Hence as a last resort I'll be migrating to Rocky 8 from CentOS 8-stream. Has anyone observed such issues with ovirt hosts on CentOS 8-stream? Any form of help is welcome, as I'm running out of ideas.

4 4