- Users - oVirt List Archives

oVirt thrashes Docker network during installation
by thomas＠hoberg.net 30 Jul '20

30 Jul '20

I want to run containers and VMs side by side and not necessarily nested. The main reason for that is GPUs, Voltas mostly, used for CUDA machine learning not for VDI, which is what most of the VM orchestrators like oVirt or vSphere seem to focus on. And CUDA drivers are notorious for refusing to work under KVM unless you pay $esla. oVirt is more of a side show in my environment, used to run some smaller functional VMs alongside bigger containers, but also in order to consolidate and re-distribute the local compute node storage as a Gluster storage pool: Kibbutz storage and compute, if you want, very much how I understand the HCI philosophy behind oVirt. The full integration of containers and VMs is still very much on the roadmap I believe, but I was surprised to see that even co-existence seems to be a problem currently. So I set-up a 3-node HCI on CentOS7 (GPU-less and older) hosts and then added additional (beefier GPGPU) CentOS7 hosts, that have been running CUDA workloads on the latest Docker-CE v19 something. The installation works fine, I can migrate VMs to these extra hosts etc., but to my dismay Docker containers on these hosts lose access to the local network, that is the entire subnet the host is in. For some strange reason I can still ping Internet hosts, perhaps even everything behind the host's gateway, but local connections are blocked. It would seem that the ovritmgmt network that the oVirt installation puts in breaks the docker0 bridge that Docker put there first. I'd consider that a bug, but I'd like to gather some feedback first, if anyone else has run into this problem. I've repeated this several times in completely distinct environments with the same results: Simply add a host with a working Docker-CE as an oVirt host to an existing DC/cluster and then try if you can still ping anyone on that net, including the Docker host from a busybox container afterwards (should try that ping just before you actually add it). No, I didn't try this with podman yet, because that's separate challenge with CUDA: Would love to know if that is part of QA for oVirt already.

5 9

Hosted Engine Deployement
by Vijay Sachdeva 30 Jul '20

30 Jul '20

Hi All, It tried deploying hosted engine using cli and got this error every time: Can anyone please let me know what here is missing..!! As it would be a great help.patri Thanks Vijay Sachdeva

3 3

Non storage nodes erronously included in quota calculations for HCI?
by thomas＠hoberg.net 30 Jul '20

30 Jul '20

For my home-lab I operate a 3 node HCI cluster on 100% passive Atoms, mostly to run light infrastructure services such as LDAP and NextCloud. I then add workstations or even laptops as pure compute hosts to the cluster for bigger but temporary things, that might actually run a different OS most of the time or just be shut off. From oVirt's point of view, these are just first put into maintenance and then shut down until needed again. No fencing or power management, all manual. All nodes, even the HCI ones, run CentOS7 with more of a workstation configuration, so updates pile up pretty quickly. After I recently upgraded one of these extra compute nodes, I found my three node HCI cluster not just faltering, but indeed very hard to reactivate at all. The faltering is a distinct issue: I have the impression that reboots of oVirt nodes cause broadcast storms on my rather simplistic 10Gibt L2 switch, which a normal CentOS instance (or any other OS) doesn't, but that's for another post. No what struck me, was that the gluster daemons on the three HCI nodes kept complaining about a lack of quorum long after the network was all back to normal, even if all three of them were there, saw each other perfectly on "gluster show status all", ready and without any healing issues pending at all. Glusterd would complain on all three nodes that there was no quota for the bricks and stop them. That went away as soon as I started one additional compute node, a node that was a gluster peer (because an oVirt host added to a HCI cluster always gets put into the Gluster, even if it's not contributing storage) but had no bricks. Immediately the gluster daemon on the three nodes with contributing bricks would report back good quota and launch the volumes (and thus all the rest of oVirt), even if in terms of *storage bricks* nothing had changed. I am afraid that downing the extra compute-only oVirtNode will bring down the HCI: Clearly not the type of redundancy it's designed to deliver. Evidently such compute-only hosts (and gluster members) get included into some quorum deliberations even if they hold not a single brick, neither storage nor arbitration. To me that seems like a bug, if that is indeed what happens: There I need your advice and suggestions. AFAIK HCI is a late addition to oVirt/RHEV as storage and compute were orginally designed to be completely distinct. In fact there are still remnants of documentation which seem to prohibit using a node for both compute and storage... what HCI is all about. And I have seen compute nodes with "matching" storage (parts of a distinct HCI setup, that was taken down but still had all the storage and Gluster elements operable), being happliy absorbed into a HCI cluster with all Gluster storage appearing in the GUI etc., without any manual creation or inclusion of bricks: Fully automatic (and undocumented)! In that case it makes sense to widen the scope of quota calculations when additional nodes are hyperconverged elements with contributing bricks. It also seems the only way to turn a 3 node HCI into 6 or 9 node one. But if you really just want to add compute nodes without bricks, those can't get "quota votes" without storage to play a role in the redundancy. I can easily imagine the missing "if then else" in the code here, but I was actually very surprised to see those failure and success messages coming from glusterd itself, which to my understanding is pretty unrelated to oVirt on top. Not from the management engine (wasn't running anyway), not from VDSM. Re-creating the scenario is very scary even if I have gone through this three times already, trying to just bring my HCI back up. And then there is so verbose logs all over the place that I'd like some advice which ones I should post. But simply speaking: Gluster peers should get no quota voting rights on volumes unless they contribute bricks. That rule seems broken. Those in the know, please let me know if am on a goose chase or if there is a real issue here that deserves a bug report.

2 5

OVA import does not upload disks
by Paolo Airaldi 29 Jul '20

29 Jul '20

Hello, I’m running 4.3.5.4-1.el7. I’ve exported a VM as OVA and now I’m trying to import it in another Data Center (it’s like a DR Data Center) Import runs successfully both from GUI and from Ansible, but imported VM disks are empty and the VM fails to boot. How can I import disks content too? Thanks in advance for any help. Paolo

2 2

Problem with paused VMs in ovirt 4.3.10.
by Damien 29 Jul '20

29 Jul '20

Hi! We have a problem with paused VMs in ovirt cluster. Please, help to solve this. In ovirt manager massege "VM rtb-stagedsw02-ovh has been paused." Resume fails with error "Failed to resume VM rtb-stagedsw02-ovh (Host: ovirt-node09-ovh.local, User: admin@internal-authz)." In oVirt Cluster 38 VM, paused VM only ubuntu 20.04 focal with docker swarm. Archived logs in attach. Packeges on ovirt nodes: python2-ovirt-setup-lib-1.2.0-1.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-provider-ovn-driver-1.2.29-1.el7.noarch ovirt-vmconsole-host-1.0.7-2.el7.noarch python2-ovirt-host-deploy-1.8.5-1.el7.noarch ovirt-imageio-common-1.5.3-0.el7.x86_64 cockpit-machines-ovirt-195.6-1.el7.centos.noarch ovirt-ansible-engine-setup-1.1.9-1.el7.noarch ovirt-host-dependencies-4.3.5-1.el7.x86_64 ovirt-host-4.3.5-1.el7.x86_64 python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64 ovirt-host-deploy-common-1.8.5-1.el7.noarch ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch ovirt-hosted-engine-setup-2.3.13-1.el7.noarch ovirt-ansible-repositories-1.1.5-1.el7.noarch ovirt-imageio-daemon-1.5.3-0.el7.noarch cockpit-ovirt-dashboard-0.13.10-1.el7.noarch ovirt-release43-4.3.10-1.el7.noarch ovirt-hosted-engine-ha-2.3.6-1.el7.noarch Packeges on HostedEngine: ovirt-ansible-infra-1.1.13-1.el7.noarch ovirt-vmconsole-1.0.7-2.el7.noarch ovirt-engine-setup-plugin-websocket-proxy-4.3.10.4-1.el7.noarch ovirt-engine-websocket-proxy-4.3.10.4-1.el7.noarch ovirt-engine-restapi-4.3.10.4-1.el7.noarch ovirt-ansible-engine-setup-1.1.9-1.el7.noarch ovirt-ansible-shutdown-env-1.0.3-1.el7.noarch ovirt-iso-uploader-4.3.2-1.el7.noarch ovirt-provider-ovn-1.2.29-1.el7.noarch ovirt-imageio-proxy-setup-1.5.3-0.el7.noarch ovirt-engine-extension-aaa-ldap-setup-1.3.10-1.el7.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64 python2-ovirt-host-deploy-1.8.5-1.el7.noarch ovirt-ansible-vm-infra-1.1.22-1.el7.noarch ovirt-engine-metrics-1.3.7-1.el7.noarch ovirt-ansible-disaster-recovery-1.2.0-1.el7.noarch ovirt-engine-wildfly-overlay-17.0.1-1.el7.noarch ovirt-ansible-roles-1.1.7-1.el7.noarch ovirt-engine-dwh-setup-4.3.8-1.el7.noarch python2-ovirt-engine-lib-4.3.10.4-1.el7.noarch ovirt-engine-extension-aaa-ldap-1.3.10-1.el7.noarch ovirt-engine-setup-plugin-ovirt-engine-4.3.10.4-1.el7.noarch ovirt-engine-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch ovirt-engine-tools-backup-4.3.10.4-1.el7.noarch ovirt-engine-webadmin-portal-4.3.10.4-1.el7.noarch ovirt-host-deploy-common-1.8.5-1.el7.noarch ovirt-ansible-image-template-1.1.12-1.el7.noarch ovirt-ansible-manageiq-1.1.14-1.el7.noarch ovirt-engine-wildfly-17.0.1-1.el7.x86_64 ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch ovirt-imageio-common-1.5.3-0.el7.x86_64 ovirt-imageio-proxy-1.5.3-0.el7.noarch python2-ovirt-setup-lib-1.2.0-1.el7.noarch ovirt-vmconsole-proxy-1.0.7-2.el7.noarch ovirt-engine-setup-base-4.3.10.4-1.el7.noarch ovirt-engine-setup-plugin-cinderlib-4.3.10.4-1.el7.noarch ovirt-engine-extensions-api-impl-4.3.10.4-1.el7.noarch ovirt-release43-4.3.10-1.el7.noarch ovirt-engine-backend-4.3.10.4-1.el7.noarch ovirt-engine-tools-4.3.10.4-1.el7.noarch ovirt-web-ui-1.6.0-1.el7.noarch ovirt-ansible-cluster-upgrade-1.1.14-1.el7.noarch ovirt-cockpit-sso-0.1.1-1.el7.noarch ovirt-engine-ui-extensions-1.0.10-1.el7.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.3.10.4-1.el7.noarch ovirt-engine-4.3.10.4-1.el7.noarch ovirt-ansible-repositories-1.1.5-1.el7.noarch ovirt-engine-extension-aaa-jdbc-1.1.10-1.el7.noarch ovirt-host-deploy-java-1.8.5-1.el7.noarch ovirt-engine-dwh-4.3.8-1.el7.noarch ovirt-engine-api-explorer-0.0.5-1.el7.noarch ovirt-guest-agent-common-1.0.16-1.el7.noarch ovirt-engine-setup-4.3.10.4-1.el7.noarch ovirt-engine-dbscripts-4.3.10.4-1.el7.noarch In /var/log/ovirt-engine/engine.log: 2020-07-24 09:38:44,472+03 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] VM '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99'(rtb-stagedsw02-ovh) move d from 'Up' --> 'Paused' 2020-07-24 09:38:44,493+03 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID: VM_PAUSED(1,025), VM rtb-stagedsw02-ovh has been paused. In /var/log/vdsm/vdsm.log 2020-07-24 09:38:42,771+0300 INFO (libvirt/events) [virt.vm] (vmId='18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99') CPU stopped: onSuspend (vm:6100) 2020-07-24 09:38:44,328+0300 INFO (jsonrpc/1) [api.host] FINISH getAllVmIoTunePolicies return={'status': {'message': 'Done', 'code': 0}, 'io_tune_policies_dict': {'4d9519f6-1ab9-4032-8fdf-4c6118531544': {'poli cy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/glust erSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/e1cfb9ec-39d8-416d-9f5f-0b54765301d4/8f95d60d-931b-4764-993c-ba9373efe361', 'name': 'sda'}]}, 'b031a269-6bcd-40b7-9737-e47112a54b3a': {'po licy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/glu sterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/9c3cc7a0-254e-4756-91b6-fb54e21abf38/71dd8024-8aec-46da-a80f-34260655e929', 'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L, 'total_io ps_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/glusterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/ 3e3a5064-5fe1-40c0-81f5-44f1a3a4d503/13549972-82de-4746-aeea-3e1531f9c180', 'name': 'sdb'}]}, 'b5fad17c-fa9d-4a80-99e7-6f86e6e19c9b': {'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_ iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': '/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/image s/15ce6cb0-6f06-4a31-92d8-b6e1bcabf3bc/613de344-d1ad-49aa-a2d0-d60ca9eb7cd3', 'name': 'sda'}]}, '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99': {'policy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'tota l_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path': u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/im ages/7978e2db-c560-4315-a775-223f1b13ae31/d927eea8-e588-449e-b07b-c845d15b082e', 'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_s ec': 0L, 'total_bytes_sec': 0L}, 'path': u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/b925dc2e-17ba-470d-a9be-cb96d4ef1f0d/951d9712-7160-4f88-a838-970aec8 2b3ea', 'name': 'sdb'}]}}} from=::1,34598 (api:54) 2020-07-24 09:38:49,747+0300 WARN (qgapoller/1) [virt.periodic.VmDispatcher] could not run <function <lambda> at 0x7fe5c84de6e0> on ['18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99'] (periodic:289) In /var/log/libvirt/qemu/rtb-stagedsw03-ovh.log KVM: entry failed, hardware error 0x80000021 If you're running a guest on an Intel machine without unrestricted mode support, the failure can be most likely due to the guest entering an invalid state for Intel VT. For example, the guest maybe running in big real mode which is not supported on less recent Intel processors. EAX=00001000 EBX=43117da8 ECX=0000000c EDX=00000121 ESI=00000003 EDI=17921000 EBP=43117cb0 ESP=43117c98 EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0 ES =0000 00000000 ffffffff 00809300 CS =9b00 7ff9b000 ffffffff 00809300 SS =0000 00000000 ffffffff 00809300 DS =0000 00000000 ffffffff 00809300 FS =0000 00000000 ffffffff 00809300 GS =0000 00000000 ffffffff 00809300 LDT=0000 00000000 000fffff 00000000 TR =0040 001ce000 0000206f 00008b00 GDT= 001cc000 0000007f IDT= 00000000 00000000 CR0=00050032 CR2=17921000 CR3=2b92a003 CR4=00000000 DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 DR6=00000000fffe0ff0 DR7=0000000000000400 EFER=0000000000000000 Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

2 1

OVA exported from oVirt cannot be imported in oVirt
by thomas＠hoberg.net 29 Jul '20

29 Jul '20

I have successfully imported OVA files which came from VirtualBox before, something I'd consider the slightly greater hurdle. Exporting a VM to an OVA file and then importing it on another platform can obviously create compatibility issues, but oVirt <-> oVirt exports and imports with the same (or similar) release should obviously just work, right? Sadly, that doesn't seem to be the case. I've just exported a Linux VM, a Univention domain controller using Debian 9 to an OVA and tried importing it on another farm. The process finished without an error, but the machine wouldn't boot. Closer inspection reveals that the exported OVA file is about 100GB in size, which corresponds to the size of the thinly allocated primary disk, but actually contains only 28kb of data (the XML header), while the disk is all zeros ('strings <machine-name>.ova') Another VM I had exported a week ago contains about 2.3GB of data, but that machine also doesn't boot. When I attach its disk to another VM as a secondary, the file system seems to contain bogus data, most directories are unreadable and an fsck goes on for minutes. Export domains are deprecated but when I export the original and runnable VM there, I get 23GB, which corresponds to what the VM is actually using. Infortunately that doesn't give me the mobility for the VM that I desire, especially since I cannot have a shared export/import domain between farms. And then I really might want to use a different hypervisor for a VM that was developed on oVirt, e.g. to put on a laptop. I've been trying to find clues as to what's going on, but generally the OVA exports don't seem to be logged in any obvious place and even the imports on seem to get logged, when the OVAs need to be converted from a foreign format such as VirtualBox where the entire, seemingly rather complex process, is logged in /var/log/vdsm/import/<something> Am I the only one trying to use OVA export/import or is that part of standard Q&A testing?

1 0

Update OVF disks fails on each gluster 7.6 volumes using ovirt 4.4.1.1
by shadow emy 29 Jul '20

29 Jul '20

Failed to update OVF disks 1798e945-5be9-466e-b52d-f7f0a3bb2043, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain hosted_storage). I did not see anything with error in SPM host3 in /var/log/vdsm/vdsm.log In /var/log/vdsm/supervdsm.log i see MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:05,845::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call webhookAdd with ('http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents', None) {} MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:05,845::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-39 /sbin/gluster-eventsapi webhook-add http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents (cwd None) MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:06,599::commands::98::common.commands::(run) FAILED: <err> = b'Webhook already exists\n'; <rc> = 5 MainProcess|jsonrpc/6::ERROR::2020-07-29 19:05:06,600::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in webhookAdd Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/gluster/events.py", line 42, in webhookAdd commands.run(command) File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 101, in run raise cmdutils.Error(args, p.returncode, out, err) vdsm.common.cmdutils.Error: Command ['/sbin/gluster-eventsapi', 'webhook-add', 'http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents'] failed with rc=5 out=b'' err=b'Webhook already exists\n' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper res = func(*args, **kwargs) File "/usr/lib/python3.6/site-packages/vdsm/gluster/events.py", line 44, in webhookAdd raise ge.GlusterWebhookAddException(rc=e.rc, err=e.err) vdsm.gluster.exception.GlusterWebhookAddException: Failed to add webhook: rc=5 out=() err=b'Webhook already exists\n' MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:10,818::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call tasksList with ([],) {} MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:10,819::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-39 /usr/sbin/gluster --mode=script volume status all tasks --xml (cwd None) SPM message logs : Jul 29 18:58:22 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf Jul 29 18:58:31 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf Jul 29 18:58:42 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf Jul 29 18:58:52 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf Jul 29 18:59:03 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf Hosted-Engine /var/log/ovirt-engine/engine.log 2020-07-29 18:52:40,901+03 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] The message key 'UpdateOvfStoreForStorageDomain' is missing from 'bundles/ExecutionMessages' 2020-07-29 18:52:40,933+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[affd38b2-457d-4c9f-9802-1f5fadd7cd34=STORAGE]', sharedLocks=''}' 2020-07-29 18:52:40,989+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] Running command: UpdateOvfStoreForStorageDomainCommand internal: false. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2020-07-29 18:52:41,002+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}' 2020-07-29 18:52:41,003+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}' 2020-07-29 18:52:41,023+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 15ea58fc-9435-11e9-b093-00163e11a571 Type: StoragePool 2020-07-29 18:52:41,039+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to update VM OVFs in Data Center 'Default' 2020-07-29 18:52:41,060+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully updated VM OVFs in Data Center 'Default' 2020-07-29 18:52:41,060+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to update template OVFs in Data Center 'Default' 2020-07-29 18:52:41,062+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully updated templates OVFs in Data Center 'Default' 2020-07-29 18:52:41,062+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to remove unneeded template/vm OVFs in Data Center 'Default' 2020-07-29 18:52:41,073+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully removed unneeded template/vm OVFs in Data Center 'Default' 2020-07-29 18:52:41,091+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Lock freed to object 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}' 2020-07-29 18:52:41,418+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]'}' 2020-07-29 18:52:41,440+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2020-07-29 18:52:41,465+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='15ea58fc-9435-11e9-b093-00163e11a571', ignoreFailoverLimit='false', storageDomainId='affd38b2-457d-4c9f-9802-1f5fadd7cd34', imageGroupId='1798e945-5be9-466e-b52d-f7f0a3bb2043', imageId='8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb'}), log id: 1f1cc366 2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters: 2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ spUUID=15ea58fc-9435-11e9-b093-00163e11a571 2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ sdUUID=affd38b2-457d-4c9f-9802-1f5fadd7cd34 2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ imageGroupGUID=1798e945-5be9-466e-b52d-f7f0a3bb2043 2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ volUUID=8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb 2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ description={"Updated":false,"Last Updated":null,"Storage Domains":[{"uuid":"affd38b2-457d-4c9f-9802-1f5fadd7cd34"}],"Disk Description":"OVF_STORE"} 2020-07-29 18:52:41,545+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 1f1cc366 2020-07-29 18:52:41,570+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[33cde062-b8be-46af-b528-5963eb5d8e0d=VDS_EXECUTION]'}' 2020-07-29 18:52:41,615+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Running command: UploadStreamCommand internal: true. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: Storage 2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] START, UploadStreamVDSCommand(HostName = host3.domain.local, UploadStreamVDSCommandParameters:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d'}), log id: 10e8c127 2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] -- executeVdsBrokerCommand, parameters: 2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ spUUID=15ea58fc-9435-11e9-b093-00163e11a571 2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ sdUUID=affd38b2-457d-4c9f-9802-1f5fadd7cd34 2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ imageGUID=1798e945-5be9-466e-b52d-f7f0a3bb2043 2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ volUUID=8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb 2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ size=26112 2020-07-29 18:52:41,664+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UploadStreamVDSCommand(HostName = host3.domain.local, UploadStreamVDSCommandParameters:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d'})' execution failed: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <host3.domain.local> doesn't match any of the subject alternative names: [host3.domain.local] 2020-07-29 18:52:41,664+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] FINISH, UploadStreamVDSCommand, return: , log id: 10e8c127 2020-07-29 18:52:41,664+03 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] Clearing domains data for host host3.domain.local 2020-07-29 18:52:41,665+03 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] Host 'host3.domain.local' is not responding. 2020-07-29 18:52:41,664+03 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <host3.domain.local> doesn't match any of the subject alternative names: [host3.domain.local] (Failed with error VDS_NETWORK_ERROR and code 5022) 2020-07-29 18:52:41,686+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] EVENT_ID: VDS_HOST_NOT_RESPONDING(9,027), Host host3.domain.local is not responding. Host cannot be fenced automatically because power management for the host is disabled. 2020-07-29 18:52:41,711+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[33cde062-b8be-46af-b528-5963eb5d8e0d=VDS_EXECUTION]'}' 2020-07-29 18:52:41,740+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EVENT_ID: UPDATE_FOR_OVF_STORES_FAILED(1,016), Failed to update OVF disks 1798e945-5be9-466e-b52d-f7f0a3bb2043, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain hosted_storage). 2020-07-29 18:52:41,777+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EVENT_ID: UPDATE_OVF_FOR_STORAGE_DOMAIN_FAILED(190), Failed to update VMs/Templates OVF data for Storage Domain hosted_storage in Data Center Default. 2020-07-29 18:52:41,788+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]'}' 2020-07-29 18:52:41,797+03 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UpdateOvfStoreForStorageDomain' id: '9578b72d-bb33-4de6-b177-b851f7c367c8' with children [] failed when attempting to perform the next operation, marking as 'FAILED' 2020-07-29 18:52:41,797+03 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EngineException: ENGINE (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.EngineException: EngineException: ENGINE (Failed with error ENGINE and code 5001) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand.executeNextOperation(UpdateOvfStoreForStorageDomainCommand.java:124) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand.performNextOperation(UpdateOvfStoreForStorageDomainCommand.java:112) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175) at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) 2020-07-29 18:52:41,798+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UpdateOvfStoreForStorageDomain' id: '9578b72d-bb33-4de6-b177-b851f7c367c8' child commands '[]' executions were completed, status 'FAILED' 2020-07-29 18:52:42,725+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] START, GlusterServersListVDSCommand(HostName = host1.domain.local, VdsIdVDSCommandParametersBase:{hostId='157d2f37-6387-47a1-8bf9-bc20231726e7'}), log id: 42f3f507 2020-07-29 18:52:42,822+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [b4f2874] Command 'ProcessOvfUpdateForStorageDomain' id: 'c83199b3-626d-494d-ab93-6ffadcc4e47a' child commands '[55b09b51-04d7-4257-ba05-8fbdd2f236fb]' executions were completed, status 'FAILED' 2020-07-29 18:52:42,851+03 ERROR [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [1ec2d268-8d00-42ad-9660-0a05362a878b] Ending command 'org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand' with failure. 2020-07-29 18:52:42,860+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [1ec2d268-8d00-42ad-9660-0a05362a878b] Lock freed to object 'EngineLock:{exclusiveLocks='[affd38b2-457d-4c9f-9802-1f5fadd7cd34=STORAGE]', sharedLocks=''}' 2020-07-29 18:52:43,254+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] FINISH, GlusterServersListVDSCommand, return: [192.168.10.20/24:CONNECTED, gluster3.domain.local:CONNECTED, gluster2.domain.local:CONNECTED], log id: 42f3f507 2020-07-29 18:52:43,266+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] START, GlusterVolumesListVDSCommand(HostName = host1.domain.local, GlusterVolumesListVDSParameters:{hostId='157d2f37-6387-47a1-8bf9-bc20231726e7'}), log id: 69d508fc 2020-07-29 18:52:43,605+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] FINISH, GlusterVolumesListVDSCommand, return: {46d80a34-6871-43a4-8b40-e5efe75bf578=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4ff50ace, 4ef84624-8219-4850-96d5-ceeb7ed073ae=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@193d2158, 2169ec3c-b8fb-482e-9cd4-fdb751a037c5=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e0fb06d6}, log id: 69d508fc 2020-07-29 18:52:43,898+03 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [b4f2874] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' with failure. 2020-07-29 18:52:47,327+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] START, GetHardwareInfoAsyncVDSCommand(HostName = host3.domain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d', vds='Host[host3.domain.local,33cde062-b8be-46af-b528-5963eb5d8e0d]'}), log id: 9a0e1c5 2020-07-29 18:52:47,328+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 9a0e1c5

1 0

ovirt-engine DB datastore full
by Askhat ECC 29 Jul '20

29 Jul '20

Hello! I have ovirt cluster Version 4.2.4.5-1.el7 and hosted-engine with local database. HE-virtio-disk is full directory /var and can not be moved another domain storage. Can you say how migrate or another things to solve this problem? Thanks. Regards [image: Screenshot_1.jpg] [image: Screenshot_13.jpg] [image: image_2020_07_13T06_51_27_733Z.png]

2 1

VM locked and script unlock_entity.sh don't show any problem
by Thomas BURGUIERE 29 Jul '20

29 Jul '20

The WEB interface show that the VM is locked, but when we run unlock_entity.sh there is no problem reported. How we think the VM were locked : There were filesystem issues (shortage disk space on the ovirt engine HOST) during a backup sequence, those consequences were discovered a few days later. The VMs are backed-up with this way: https://github.com/wefixit-AT/oVirtBackup The ovirt-engine logs tell us that Postgres DB has encountered problem like this one during the backup process: Message: PL/pgSQL function updatevdsdynamic(integer,integer,character varying,numeric,character varying,boolean,integer,integer,integer,uuid,integer,integer,integer,integer,integer,integer,integer,integer,character varying,character varying,character varying,character varying,integer,character varying,integer,integer,integer,boolean,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,integer,text,integer,character varying,character varying,character varying,character varying,character varying,character varying,text,text,smallint,integer,smallint,boolean,character varying,text,text,boolean,boolean,text,character varying,boolean,uuid,boolean,jsonb) line 4 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: could not extend file "base/16385/190997": No space left on device Later, we saw that those VMs they got a lock icon in the status column since this episode. This did not appear immediatly, but maybe a few hours after. It happens that those locked VMs are sort-of limited in the way that there are numerous things we cannot do with them, such as: - edit parameters (i.e. memory). - start them again after they get halted by unix command (halt -p) - copy the related disk. The version of ovirt used 4.2.1 Is there an other way to unlock the VMs ? Or to get unlock_entity.sh to work ?

1 1

New ovirt 4.4.0.3-1.el8 leaves disks in illegal state on all snapshot actions
by Henri Aanstoot 28 Jul '20

28 Jul '20

Hi all, I've got 2 two node setup, image based installs. When doing ova exports or generic snapshots, things seem in order. Removing snapshots shows warning 'disk in illegal state' Mouse hover shows .. please do not shutdown before succesfully remove snapshot ovirt-engine log 2020-07-22 16:40:37,549+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedExecutorService-commandCoordinator-Thread-2) [264b0047-5aa6-4380-9d32-eb328fd6bed0] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM node2.lab command MergeVDS failed: Merge failed 2020-07-22 16:40:37,549+02 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-2) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Command 'MergeVDSCommand(HostName = node2.lab, MergeVDSCommandParameters:{hostId='02df5213-1243-4671-a1c6-6489d7146319', vmId='64c25543-bef7-4fdd-8204-6507046f5a34', storagePoolId='5a4ea80c-b3b2-11ea-a890-00163e3cb866', storageDomainId='9a12f1b2-5378-46cc-964d-3575695e823f', imageGroupId='3f7ac8d8-f1ab-4c7a-91cc-f34d0b8a1cb8', imageId='c757e740-9013-4ae0-901d-316932f4af0e', baseImageId='ebe50730-dec3-4f29-8a38-9ae7c59f2aef', topImageId='c757e740-9013-4ae0-901d-316932f4af0e', bandwidth='0'})' execution failed: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 2020-07-22 16:40:37,549+02 ERROR [org.ovirt.engine.core.bll.MergeCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-2) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Engine exception thrown while sending merge command: org.ovirt.engine.core.common.errors.EngineException: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 (Failed with error mergeErr and code 52) Caused by: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge failed, code = 52 <driver name='qemu' error_policy='report'/> <driver name='qemu' type='qcow2' cache='none' error_policy='stop' io='threads'/> 2020-07-22 16:40:39,659+02 ERROR [org.ovirt.engine.core.bll.MergeStatusCommand] (EE-ManagedExecutorService-commandCoordinator-Thread-3) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Failed to live merge. Top volume c757e740-9013-4ae0-901d-316932f4af0e is still in qemu chain [ebe50730-dec3-4f29-8a38-9ae7c59f2aef, c757e740-9013-4ae0-901d-316932f4af0e] 2020-07-22 16:40:41,524+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Command id: 'e0b2bce7-afe0-4955-ae46-38bcb8719852 failed child command status for step 'MERGE_STATUS' 2020-07-22 16:40:42,597+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-53) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Merging of snapshot 'ef8f7e06-e48c-4a8c-983c-64e3d4ebfcf9' images 'ebe50730-dec3-4f29-8a38-9ae7c59f2aef'..'c757e740-9013-4ae0-901d-316932f4af0e' failed. Images have been marked illegal and can no longer be previewed or reverted to. Please retry Live Merge on the snapshot to complete the operation. 2020-07-22 16:40:42,603+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-53) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand' with failure. 2020-07-22 16:40:43,679+02 ERROR [org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-15) [264b0047-5aa6-4380-9d32-eb328fd6bed0] Ending command 'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure. 2020-07-22 16:40:43,774+02 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-15) [264b0047-5aa6-4380-9d32-eb328fd6bed0] EVENT_ID: USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot 'Auto-generated for Export To OVA' for VM 'Adhoc'. VDSM on hypervisor 2020-07-22 14:14:30,220+0200 ERROR (jsonrpc/5) [virt.vm] (vmId='14283e6d-c3f0-4011-b90f-a1272f0fbc10') Live merge failed (job: e59c54d9-b8d3-44d0-9147-9dd40dff57b9) (vm:5381) if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self) libvirt.libvirtError: internal error: qemu block name 'json:{"backing": {"driver": "qcow2", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/10.12.0.9:_exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/bb3aed4b-fc41-456a-9c18-1409a9aa6d14"}}, "driver": "qcow2", "file": {"driver": "file", "filename": "/rhev/data-center/mnt/10.12.0.9:_exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/3995b256-2afb-4853-9360-33d0c12e5fd1"}}' doesn't match expected '/rhev/data-center/mnt/10.12.0.9: _exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/3995b256-2afb-4853-9360-33d0c12e5fd1' 2020-07-22 14:14:30,234+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC call VM.merge failed (error 52) in 0.17 seconds (__init__:312) 2020-07-22 14:17:28,798+0200 INFO (jsonrpc/2) [api] FINISH getStats error=Virtual machine does not exist: {'vmId': '698d486c-edbf-4e28-a199-31a2e27bd808'} (api:129) 2020-07-22 14:17:28,798+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC call VM.getStats failed (error 1) in 0.00 seconds (__init__:312) Also in log, INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1, 'message': "Virtual machine does not exist: But is is there and accessible Any advice here? Henri ovirt 4.4.0.3-1.el8 OS Version: RHEL - 8 - 1.1911.0.9.el8 OS Description: oVirt Node 4.4.0 Kernel Version: 4.18.0 - 147.8.1.el8_1.x86_64 KVM Version: 4.1.0 - 23.el8.1 LIBVIRT Version: libvirt-5.6.0-10.el8 VDSM Version: vdsm-4.40.16-1.el8 SPICE Version: 0.14.2 - 1.el8 GlusterFS Version: glusterfs-7.5-1.el8 CEPH Version: librbd1-12.2.7-9.el8 Open vSwitch Version: openvswitch-2.11.1-5.el8 Nmstate Version: nmstate-0.2.10-1.el8

3 4