Non storage nodes erronously included in quota calculations for HCI?
by thomas＠hoberg.net
For my home-lab I operate a 3 node HCI cluster on 100% passive Atoms, mostly to run light infrastructure services such as LDAP and NextCloud.
I then add workstations or even laptops as pure compute hosts to the cluster for bigger but temporary things, that might actually run a different OS most of the time or just be shut off. From oVirt's point of view, these are just first put into maintenance and then shut down until needed again. No fencing or power management, all manual.
All nodes, even the HCI ones, run CentOS7 with more of a workstation configuration, so updates pile up pretty quickly.
After I recently upgraded one of these extra compute nodes, I found my three node HCI cluster not just faltering, but indeed very hard to reactivate at all.
The faltering is a distinct issue: I have the impression that reboots of oVirt nodes cause broadcast storms on my rather simplistic 10Gibt L2 switch, which a normal CentOS instance (or any other OS) doesn't, but that's for another post.
No what struck me, was that the gluster daemons on the three HCI nodes kept complaining about a lack of quorum long after the network was all back to normal, even if all three of them were there, saw each other perfectly on "gluster show status all", ready and without any healing issues pending at all.
Glusterd would complain on all three nodes that there was no quota for the bricks and stop them.
That went away as soon as I started one additional compute node, a node that was a gluster peer (because an oVirt host added to a HCI cluster always gets put into the Gluster, even if it's not contributing storage) but had no bricks. Immediately the gluster daemon on the three nodes with contributing bricks would report back good quota and launch the volumes (and thus all the rest of oVirt), even if in terms of *storage bricks* nothing had changed.
I am afraid that downing the extra compute-only oVirtNode will bring down the HCI: Clearly not the type of redundancy it's designed to deliver.
Evidently such compute-only hosts (and gluster members) get included into some quorum deliberations even if they hold not a single brick, neither storage nor arbitration.
To me that seems like a bug, if that is indeed what happens: There I need your advice and suggestions.
AFAIK HCI is a late addition to oVirt/RHEV as storage and compute were orginally designed to be completely distinct. In fact there are still remnants of documentation which seem to prohibit using a node for both compute and storage... what HCI is all about.
And I have seen compute nodes with "matching" storage (parts of a distinct HCI setup, that was taken down but still had all the storage and Gluster elements operable), being happliy absorbed into a HCI cluster with all Gluster storage appearing in the GUI etc., without any manual creation or inclusion of bricks: Fully automatic (and undocumented)!
In that case it makes sense to widen the scope of quota calculations when additional nodes are hyperconverged elements with contributing bricks. It also seems the only way to turn a 3 node HCI into 6 or 9 node one.
But if you really just want to add compute nodes without bricks, those can't get "quota votes" without storage to play a role in the redundancy.
I can easily imagine the missing "if then else" in the code here, but I was actually very surprised to see those failure and success messages coming from glusterd itself, which to my understanding is pretty unrelated to oVirt on top. Not from the management engine (wasn't running anyway), not from VDSM.
Re-creating the scenario is very scary even if I have gone through this three times already, trying to just bring my HCI back up. And then there is so verbose logs all over the place that I'd like some advice which ones I should post.
But simply speaking: Gluster peers should get no quota voting rights on volumes unless they contribute bricks. That rule seems broken.
Those in the know, please let me know if am on a goose chase or if there is a real issue here that deserves a bug report.
4 years, 4 months
OVA import does not upload disks
by Paolo Airaldi
Hello,
I’m running 4.3.5.4-1.el7. I’ve exported a VM as OVA and now
I’m trying to import it in another Data Center (it’s like a DR Data Center)
Import runs successfully both from GUI and from
Ansible, but imported VM disks are empty and the VM fails to boot.
How can I import disks content too?
Thanks in advance for any help.
Paolo
4 years, 4 months
Problem with paused VMs in ovirt 4.3.10.
by Damien
Hi!
We have a problem with paused VMs in ovirt cluster. Please, help to
solve this.
In ovirt manager massege "VM rtb-stagedsw02-ovh has been paused."
Resume fails with error "Failed to resume VM rtb-stagedsw02-ovh (Host:
ovirt-node09-ovh.local, User: admin@internal-authz)."
In oVirt Cluster 38 VM, paused VM only ubuntu 20.04 focal with docker swarm.
Archived logs in attach.
Packeges on ovirt nodes:
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-provider-ovn-driver-1.2.29-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
cockpit-machines-ovirt-195.6-1.el7.centos.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-host-dependencies-4.3.5-1.el7.x86_64
ovirt-host-4.3.5-1.el7.x86_64
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-hosted-engine-setup-2.3.13-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-imageio-daemon-1.5.3-0.el7.noarch
cockpit-ovirt-dashboard-0.13.10-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-hosted-engine-ha-2.3.6-1.el7.noarch
Packeges on HostedEngine:
ovirt-ansible-infra-1.1.13-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-restapi-4.3.10.4-1.el7.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-ansible-shutdown-env-1.0.3-1.el7.noarch
ovirt-iso-uploader-4.3.2-1.el7.noarch
ovirt-provider-ovn-1.2.29-1.el7.noarch
ovirt-imageio-proxy-setup-1.5.3-0.el7.noarch
ovirt-engine-extension-aaa-ldap-setup-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-ansible-vm-infra-1.1.22-1.el7.noarch
ovirt-engine-metrics-1.3.7-1.el7.noarch
ovirt-ansible-disaster-recovery-1.2.0-1.el7.noarch
ovirt-engine-wildfly-overlay-17.0.1-1.el7.noarch
ovirt-ansible-roles-1.1.7-1.el7.noarch
ovirt-engine-dwh-setup-4.3.8-1.el7.noarch
python2-ovirt-engine-lib-4.3.10.4-1.el7.noarch
ovirt-engine-extension-aaa-ldap-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-engine-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
ovirt-engine-tools-backup-4.3.10.4-1.el7.noarch
ovirt-engine-webadmin-portal-4.3.10.4-1.el7.noarch
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-image-template-1.1.12-1.el7.noarch
ovirt-ansible-manageiq-1.1.14-1.el7.noarch
ovirt-engine-wildfly-17.0.1-1.el7.x86_64
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
ovirt-imageio-proxy-1.5.3-0.el7.noarch
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-proxy-1.0.7-2.el7.noarch
ovirt-engine-setup-base-4.3.10.4-1.el7.noarch
ovirt-engine-setup-plugin-cinderlib-4.3.10.4-1.el7.noarch
ovirt-engine-extensions-api-impl-4.3.10.4-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-engine-backend-4.3.10.4-1.el7.noarch
ovirt-engine-tools-4.3.10.4-1.el7.noarch
ovirt-web-ui-1.6.0-1.el7.noarch
ovirt-ansible-cluster-upgrade-1.1.14-1.el7.noarch
ovirt-cockpit-sso-0.1.1-1.el7.noarch
ovirt-engine-ui-extensions-1.0.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.3.10.4-1.el7.noarch
ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-engine-extension-aaa-jdbc-1.1.10-1.el7.noarch
ovirt-host-deploy-java-1.8.5-1.el7.noarch
ovirt-engine-dwh-4.3.8-1.el7.noarch
ovirt-engine-api-explorer-0.0.5-1.el7.noarch
ovirt-guest-agent-common-1.0.16-1.el7.noarch
ovirt-engine-setup-4.3.10.4-1.el7.noarch
ovirt-engine-dbscripts-4.3.10.4-1.el7.noarch
In /var/log/ovirt-engine/engine.log:
2020-07-24 09:38:44,472+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [] VM
'18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99'(rtb-stagedsw02-ovh) move
d from 'Up' --> 'Paused'
2020-07-24 09:38:44,493+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID:
VM_PAUSED(1,025), VM rtb-stagedsw02-ovh has been paused.
In /var/log/vdsm/vdsm.log
2020-07-24 09:38:42,771+0300 INFO (libvirt/events) [virt.vm]
(vmId='18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99') CPU stopped: onSuspend
(vm:6100)
2020-07-24 09:38:44,328+0300 INFO (jsonrpc/1) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'message': 'Done', 'code':
0}, 'io_tune_policies_dict':
{'4d9519f6-1ab9-4032-8fdf-4c6118531544': {'poli
cy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glust
erSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/e1cfb9ec-39d8-416d-9f5f-0b54765301d4/8f95d60d-931b-4764-993c-ba9373efe361',
'name': 'sda'}]}, 'b031a269-6bcd-40b7-9737-e47112a54b3a': {'po
licy': [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glu
sterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/9c3cc7a0-254e-4756-91b6-fb54e21abf38/71dd8024-8aec-46da-a80f-34260655e929',
'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L, 'total_io
ps_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glusterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/
3e3a5064-5fe1-40c0-81f5-44f1a3a4d503/13549972-82de-4746-aeea-3e1531f9c180',
'name': 'sdb'}]}, 'b5fad17c-fa9d-4a80-99e7-6f86e6e19c9b': {'policy':
[], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'total_
iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/image
s/15ce6cb0-6f06-4a31-92d8-b6e1bcabf3bc/613de344-d1ad-49aa-a2d0-d60ca9eb7cd3',
'name': 'sda'}]}, '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99': {'policy':
[], 'current_values': [{'ioTune': {'write_bytes_sec': 0L, 'tota
l_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/im
ages/7978e2db-c560-4315-a775-223f1b13ae31/d927eea8-e588-449e-b07b-c845d15b082e',
'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L, 'total_iops_sec':
0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_s
ec': 0L, 'total_bytes_sec': 0L}, 'path':
u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/b925dc2e-17ba-470d-a9be-cb96d4ef1f0d/951d9712-7160-4f88-a838-970aec8
2b3ea', 'name': 'sdb'}]}}} from=::1,34598 (api:54)
2020-07-24 09:38:49,747+0300 WARN (qgapoller/1)
[virt.periodic.VmDispatcher] could not run <function <lambda> at
0x7fe5c84de6e0> on ['18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99']
(periodic:289)
In /var/log/libvirt/qemu/rtb-stagedsw03-ovh.log
KVM: entry failed, hardware error 0x80000021
If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an
invalid
state for Intel VT. For example, the guest maybe running in big real
mode
which is not supported on less recent Intel processors.
EAX=00001000 EBX=43117da8 ECX=0000000c EDX=00000121
ESI=00000003 EDI=17921000 EBP=43117cb0 ESP=43117c98
EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =9b00 7ff9b000 ffffffff 00809300
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 000fffff 00000000
TR =0040 001ce000 0000206f 00008b00
GDT= 001cc000 0000007f
IDT= 00000000 00000000
CR0=00050032 CR2=17921000 CR3=2b92a003 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000fffe0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
<ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff
4 years, 4 months
OVA exported from oVirt cannot be imported in oVirt
by thomas＠hoberg.net
I have successfully imported OVA files which came from VirtualBox before, something I'd consider the slightly greater hurdle.
Exporting a VM to an OVA file and then importing it on another platform can obviously create compatibility issues, but oVirt <-> oVirt exports and imports with the same (or similar) release should obviously just work, right?
Sadly, that doesn't seem to be the case.
I've just exported a Linux VM, a Univention domain controller using Debian 9 to an OVA and tried importing it on another farm.
The process finished without an error, but the machine wouldn't boot.
Closer inspection reveals that the exported OVA file is about 100GB in size, which corresponds to the size of the thinly allocated primary disk, but actually contains only 28kb of data (the XML header), while the disk is all zeros ('strings <machine-name>.ova')
Another VM I had exported a week ago contains about 2.3GB of data, but that machine also doesn't boot. When I attach its disk to another VM as a secondary, the file system seems to contain bogus data, most directories are unreadable and an fsck goes on for minutes.
Export domains are deprecated but when I export the original and runnable VM there, I get 23GB, which corresponds to what the VM is actually using. Infortunately that doesn't give me the mobility for the VM that I desire, especially since I cannot have a shared export/import domain between farms. And then I really might want to use a different hypervisor for a VM that was developed on oVirt, e.g. to put on a laptop.
I've been trying to find clues as to what's going on, but generally the OVA exports don't seem to be logged in any obvious place and even the imports on seem to get logged, when the OVAs need to be converted from a foreign format such as VirtualBox where the entire, seemingly rather complex process, is logged in /var/log/vdsm/import/<something>
Am I the only one trying to use OVA export/import or is that part of standard Q&A testing?
4 years, 4 months
Update OVF disks fails on each gluster 7.6 volumes using ovirt 4.4.1.1
by shadow emy
Failed to update OVF disks 1798e945-5be9-466e-b52d-f7f0a3bb2043, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain hosted_storage).
I did not see anything with error in SPM host3 in /var/log/vdsm/vdsm.log
In /var/log/vdsm/supervdsm.log i see
MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:05,845::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call webhookAdd with ('http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents', None) {}
MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:05,845::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-39 /sbin/gluster-eventsapi webhook-add http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents (cwd None)
MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:06,599::commands::98::common.commands::(run) FAILED: <err> = b'Webhook already exists\n'; <rc> = 5
MainProcess|jsonrpc/6::ERROR::2020-07-29 19:05:06,600::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in webhookAdd
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/gluster/events.py", line 42, in webhookAdd
commands.run(command)
File "/usr/lib/python3.6/site-packages/vdsm/common/commands.py", line 101, in run
raise cmdutils.Error(args, p.returncode, out, err)
vdsm.common.cmdutils.Error: Command ['/sbin/gluster-eventsapi', 'webhook-add', 'http://ovirt-engine.domain.local:80/ovirt-engine/services/glusterevents'] failed with rc=5 out=b'' err=b'Webhook already exists\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper
res = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdsm/gluster/events.py", line 44, in webhookAdd
raise ge.GlusterWebhookAddException(rc=e.rc, err=e.err)
vdsm.gluster.exception.GlusterWebhookAddException: Failed to add webhook: rc=5 out=() err=b'Webhook already exists\n'
MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:10,818::supervdsm_server::93::SuperVdsm.ServerCallback::(wrapper) call tasksList with ([],) {}
MainProcess|jsonrpc/6::DEBUG::2020-07-29 19:05:10,819::commands::153::common.commands::(start) /usr/bin/taskset --cpu-list 0-39 /usr/sbin/gluster --mode=script volume status all tasks --xml (cwd None)
SPM message logs :
Jul 29 18:58:22 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf
Jul 29 18:58:31 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf
Jul 29 18:58:42 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf
Jul 29 18:58:52 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf
Jul 29 18:59:03 host3 journal[460535]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed extracting VM OVF from the OVF_STORE volume, falling back to initial vm.conf
Hosted-Engine /var/log/ovirt-engine/engine.log
2020-07-29 18:52:40,901+03 WARN [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] The message key 'UpdateOvfStoreForStorageDomain' is missing from 'bundles/ExecutionMessages'
2020-07-29 18:52:40,933+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] Lock Acquired to object 'EngineLock:{exclusiveLocks='[affd38b2-457d-4c9f-9802-1f5fadd7cd34=STORAGE]', sharedLocks=''}'
2020-07-29 18:52:40,989+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (default task-153) [1ec2d268-8d00-42ad-9660-0a05362a878b] Running command: UpdateOvfStoreForStorageDomainCommand internal: false. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2020-07-29 18:52:41,002+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Before acquiring and wait lock 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}'
2020-07-29 18:52:41,003+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Lock-wait acquired to object 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}'
2020-07-29 18:52:41,023+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID: 15ea58fc-9435-11e9-b093-00163e11a571 Type: StoragePool
2020-07-29 18:52:41,039+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to update VM OVFs in Data Center 'Default'
2020-07-29 18:52:41,060+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully updated VM OVFs in Data Center 'Default'
2020-07-29 18:52:41,060+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to update template OVFs in Data Center 'Default'
2020-07-29 18:52:41,062+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully updated templates OVFs in Data Center 'Default'
2020-07-29 18:52:41,062+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Attempting to remove unneeded template/vm OVFs in Data Center 'Default'
2020-07-29 18:52:41,073+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Successfully removed unneeded template/vm OVFs in Data Center 'Default'
2020-07-29 18:52:41,091+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStoragePoolCommand] (default task-153) [71d7de10] Lock freed to object 'EngineLock:{exclusiveLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]', sharedLocks=''}'
2020-07-29 18:52:41,418+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]'}'
2020-07-29 18:52:41,440+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Running command: ProcessOvfUpdateForStorageDomainCommand internal: true. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN
2020-07-29 18:52:41,465+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] START, SetVolumeDescriptionVDSCommand( SetVolumeDescriptionVDSCommandParameters:{storagePoolId='15ea58fc-9435-11e9-b093-00163e11a571', ignoreFailoverLimit='false', storageDomainId='affd38b2-457d-4c9f-9802-1f5fadd7cd34', imageGroupId='1798e945-5be9-466e-b52d-f7f0a3bb2043', imageId='8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb'}), log id: 1f1cc366
2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] -- executeIrsBrokerCommand: calling 'setVolumeDescription', parameters:
2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ spUUID=15ea58fc-9435-11e9-b093-00163e11a571
2020-07-29 18:52:41,470+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ sdUUID=affd38b2-457d-4c9f-9802-1f5fadd7cd34
2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ imageGroupGUID=1798e945-5be9-466e-b52d-f7f0a3bb2043
2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ volUUID=8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb
2020-07-29 18:52:41,471+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ description={"Updated":false,"Last Updated":null,"Storage Domains":[{"uuid":"affd38b2-457d-4c9f-9802-1f5fadd7cd34"}],"Disk Description":"OVF_STORE"}
2020-07-29 18:52:41,545+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SetVolumeDescriptionVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] FINISH, SetVolumeDescriptionVDSCommand, return: , log id: 1f1cc366
2020-07-29 18:52:41,570+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock Acquired to object 'EngineLock:{exclusiveLocks='', sharedLocks='[33cde062-b8be-46af-b528-5963eb5d8e0d=VDS_EXECUTION]'}'
2020-07-29 18:52:41,615+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Running command: UploadStreamCommand internal: true. Entities affected : ID: affd38b2-457d-4c9f-9802-1f5fadd7cd34 Type: Storage
2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] START, UploadStreamVDSCommand(HostName = host3.domain.local, UploadStreamVDSCommandParameters:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d'}), log id: 10e8c127
2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] -- executeVdsBrokerCommand, parameters:
2020-07-29 18:52:41,618+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ spUUID=15ea58fc-9435-11e9-b093-00163e11a571
2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ sdUUID=affd38b2-457d-4c9f-9802-1f5fadd7cd34
2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ imageGUID=1798e945-5be9-466e-b52d-f7f0a3bb2043
2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ volUUID=8c0afaac-6c1a-40e5-ad21-54ce53ee6fdb
2020-07-29 18:52:41,619+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] ++ size=26112
2020-07-29 18:52:41,664+03 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UploadStreamVDSCommand(HostName = host3.domain.local, UploadStreamVDSCommandParameters:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d'})' execution failed: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <host3.domain.local> doesn't match any of the subject alternative names: [host3.domain.local]
2020-07-29 18:52:41,664+03 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UploadStreamVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] FINISH, UploadStreamVDSCommand, return: , log id: 10e8c127
2020-07-29 18:52:41,664+03 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] Clearing domains data for host host3.domain.local
2020-07-29 18:52:41,665+03 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] Host 'host3.domain.local' is not responding.
2020-07-29 18:52:41,664+03 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: javax.net.ssl.SSLPeerUnverifiedException: Certificate for <host3.domain.local> doesn't match any of the subject alternative names: [host3.domain.local] (Failed with error VDS_NETWORK_ERROR and code 5022)
2020-07-29 18:52:41,686+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-29741) [b4f2874] EVENT_ID: VDS_HOST_NOT_RESPONDING(9,027), Host host3.domain.local is not responding. Host cannot be fenced automatically because power management for the host is disabled.
2020-07-29 18:52:41,711+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.UploadStreamCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock freed to object 'EngineLock:{exclusiveLocks='', sharedLocks='[33cde062-b8be-46af-b528-5963eb5d8e0d=VDS_EXECUTION]'}'
2020-07-29 18:52:41,740+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EVENT_ID: UPDATE_FOR_OVF_STORES_FAILED(1,016), Failed to update OVF disks 1798e945-5be9-466e-b52d-f7f0a3bb2043, OVF data isn't updated on those OVF stores (Data Center Default, Storage Domain hosted_storage).
2020-07-29 18:52:41,777+03 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EVENT_ID: UPDATE_OVF_FOR_STORAGE_DOMAIN_FAILED(190), Failed to update VMs/Templates OVF data for Storage Domain hosted_storage in Data Center Default.
2020-07-29 18:52:41,788+03 INFO [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Lock freed to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='[15ea58fc-9435-11e9-b093-00163e11a571=OVF_UPDATE]'}'
2020-07-29 18:52:41,797+03 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UpdateOvfStoreForStorageDomain' id: '9578b72d-bb33-4de6-b177-b851f7c367c8' with children [] failed when attempting to perform the next operation, marking as 'FAILED'
2020-07-29 18:52:41,797+03 ERROR [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] EngineException: ENGINE (Failed with error ENGINE and code 5001): org.ovirt.engine.core.common.errors.EngineException: EngineException: ENGINE (Failed with error ENGINE and code 5001)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand.executeNextOperation(UpdateOvfStoreForStorageDomainCommand.java:124)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand.performNextOperation(UpdateOvfStoreForStorageDomainCommand.java:112)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback.childCommandsExecutionEnded(SerialChildCommandsExecutionCallback.java:32)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.ChildCommandsCallbackBase.doPolling(ChildCommandsCallbackBase.java:80)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethodsImpl(CommandCallbacksPoller.java:175)
at deployment.engine.ear.bll.jar//org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller.invokeCallbackMethods(CommandCallbacksPoller.java:109)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360)
at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227)
2020-07-29 18:52:41,798+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-90) [b4f2874] Command 'UpdateOvfStoreForStorageDomain' id: '9578b72d-bb33-4de6-b177-b851f7c367c8' child commands '[]' executions were completed, status 'FAILED'
2020-07-29 18:52:42,725+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] START, GlusterServersListVDSCommand(HostName = host1.domain.local, VdsIdVDSCommandParametersBase:{hostId='157d2f37-6387-47a1-8bf9-bc20231726e7'}), log id: 42f3f507
2020-07-29 18:52:42,822+03 INFO [org.ovirt.engine.core.bll.SerialChildCommandsExecutionCallback] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [b4f2874] Command 'ProcessOvfUpdateForStorageDomain' id: 'c83199b3-626d-494d-ab93-6ffadcc4e47a' child commands '[55b09b51-04d7-4257-ba05-8fbdd2f236fb]' executions were completed, status 'FAILED'
2020-07-29 18:52:42,851+03 ERROR [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [1ec2d268-8d00-42ad-9660-0a05362a878b] Ending command 'org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand' with failure.
2020-07-29 18:52:42,860+03 INFO [org.ovirt.engine.core.bll.storage.domain.UpdateOvfStoreForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-60) [1ec2d268-8d00-42ad-9660-0a05362a878b] Lock freed to object 'EngineLock:{exclusiveLocks='[affd38b2-457d-4c9f-9802-1f5fadd7cd34=STORAGE]', sharedLocks=''}'
2020-07-29 18:52:43,254+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] FINISH, GlusterServersListVDSCommand, return: [192.168.10.20/24:CONNECTED, gluster3.domain.local:CONNECTED, gluster2.domain.local:CONNECTED], log id: 42f3f507
2020-07-29 18:52:43,266+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] START, GlusterVolumesListVDSCommand(HostName = host1.domain.local, GlusterVolumesListVDSParameters:{hostId='157d2f37-6387-47a1-8bf9-bc20231726e7'}), log id: 69d508fc
2020-07-29 18:52:43,605+03 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-17) [] FINISH, GlusterVolumesListVDSCommand, return: {46d80a34-6871-43a4-8b40-e5efe75bf578=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@4ff50ace, 4ef84624-8219-4850-96d5-ceeb7ed073ae=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@193d2158, 2169ec3c-b8fb-482e-9cd4-fdb751a037c5=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@e0fb06d6}, log id: 69d508fc
2020-07-29 18:52:43,898+03 ERROR [org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-22) [b4f2874] Ending command 'org.ovirt.engine.core.bll.storage.ovfstore.ProcessOvfUpdateForStorageDomainCommand' with failure.
2020-07-29 18:52:47,327+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] START, GetHardwareInfoAsyncVDSCommand(HostName = host3.domain.local, VdsIdAndVdsVDSCommandParametersBase:{hostId='33cde062-b8be-46af-b528-5963eb5d8e0d', vds='Host[host3.domain.local,33cde062-b8be-46af-b528-5963eb5d8e0d]'}), log id: 9a0e1c5
2020-07-29 18:52:47,328+03 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoAsyncVDSCommand] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-37) [] FINISH, GetHardwareInfoAsyncVDSCommand, return: , log id: 9a0e1c5
4 years, 4 months
ovirt-engine DB datastore full
by Askhat ECC
Hello!
I have ovirt cluster Version 4.2.4.5-1.el7 and hosted-engine with local
database.
HE-virtio-disk is full directory /var and can not be moved another domain
storage.
Can you say how migrate or another things to solve this problem? Thanks.
Regards
[image: Screenshot_1.jpg]
[image: Screenshot_13.jpg]
[image: image_2020_07_13T06_51_27_733Z.png]
4 years, 4 months
VM locked and script unlock_entity.sh don't show any problem
by Thomas BURGUIERE
The WEB interface show that the VM is locked, but when we run unlock_entity.sh there is no problem reported.
How we think the VM were locked :
There were filesystem issues (shortage disk space on the ovirt engine HOST) during a backup sequence, those consequences were discovered a few days later.
The VMs are backed-up with this way:
https://github.com/wefixit-AT/oVirtBackup
The ovirt-engine logs tell us that Postgres DB has encountered problem like this one during the backup process:
Message: PL/pgSQL function updatevdsdynamic(integer,integer,character varying,numeric,character varying,boolean,integer,integer,integer,uuid,integer,integer,integer,integer,integer,integer,integer,integer,character varying,character varying,character varying,character varying,integer,character varying,integer,integer,integer,boolean,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,character varying,integer,text,integer,character varying,character varying,character varying,character varying,character varying,character varying,text,text,smallint,integer,smallint,boolean,character varying,text,text,boolean,boolean,text,character varying,boolean,uuid,boolean,jsonb) line 4 at SQL statement; nested exception is org.postgresql.util.PSQLException: ERROR: could not extend file "base/16385/190997": No space left on device
Later, we saw that those VMs they got a lock icon in the status column since this episode. This did not appear immediatly, but maybe a few hours after.
It happens that those locked VMs are sort-of limited in the way that there are numerous things we cannot do with them, such as:
- edit parameters (i.e. memory).
- start them again after they get halted by unix command (halt -p)
- copy the related disk.
The version of ovirt used 4.2.1
Is there an other way to unlock the VMs ?
Or to get unlock_entity.sh to work ?
4 years, 4 months
New ovirt 4.4.0.3-1.el8 leaves disks in illegal state on all snapshot actions
by Henri Aanstoot
Hi all,
I've got 2 two node setup, image based installs.
When doing ova exports or generic snapshots, things seem in order.
Removing snapshots shows warning 'disk in illegal state'
Mouse hover shows .. please do not shutdown before succesfully remove
snapshot
ovirt-engine log
2020-07-22 16:40:37,549+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedExecutorService-commandCoordinator-Thread-2)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] EVENT_ID:
VDS_BROKER_COMMAND_FAILURE(10,802), VDSM node2.lab command MergeVDS failed:
Merge failed
2020-07-22 16:40:37,549+02 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.MergeVDSCommand]
(EE-ManagedExecutorService-commandCoordinator-Thread-2)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Command 'MergeVDSCommand(HostName =
node2.lab,
MergeVDSCommandParameters:{hostId='02df5213-1243-4671-a1c6-6489d7146319',
vmId='64c25543-bef7-4fdd-8204-6507046f5a34',
storagePoolId='5a4ea80c-b3b2-11ea-a890-00163e3cb866',
storageDomainId='9a12f1b2-5378-46cc-964d-3575695e823f',
imageGroupId='3f7ac8d8-f1ab-4c7a-91cc-f34d0b8a1cb8',
imageId='c757e740-9013-4ae0-901d-316932f4af0e',
baseImageId='ebe50730-dec3-4f29-8a38-9ae7c59f2aef',
topImageId='c757e740-9013-4ae0-901d-316932f4af0e', bandwidth='0'})'
execution failed: VDSGenericException: VDSErrorException: Failed to
MergeVDS, error = Merge failed, code = 52
2020-07-22 16:40:37,549+02 ERROR [org.ovirt.engine.core.bll.MergeCommand]
(EE-ManagedExecutorService-commandCoordinator-Thread-2)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Engine exception thrown while
sending merge command: org.ovirt.engine.core.common.errors.EngineException:
EngineException:
org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge
failed, code = 52 (Failed with error mergeErr and code 52)
Caused by: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException:
VDSGenericException: VDSErrorException: Failed to MergeVDS, error = Merge
failed, code = 52
<driver name='qemu' error_policy='report'/>
<driver name='qemu' type='qcow2' cache='none' error_policy='stop'
io='threads'/>
2020-07-22 16:40:39,659+02 ERROR
[org.ovirt.engine.core.bll.MergeStatusCommand]
(EE-ManagedExecutorService-commandCoordinator-Thread-3)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Failed to live merge. Top volume
c757e740-9013-4ae0-901d-316932f4af0e is still in qemu chain
[ebe50730-dec3-4f29-8a38-9ae7c59f2aef, c757e740-9013-4ae0-901d-316932f4af0e]
2020-07-22 16:40:41,524+02 ERROR
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-58)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Command id:
'e0b2bce7-afe0-4955-ae46-38bcb8719852 failed child command status for step
'MERGE_STATUS'
2020-07-22 16:40:42,597+02 ERROR
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-53)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Merging of snapshot
'ef8f7e06-e48c-4a8c-983c-64e3d4ebfcf9' images
'ebe50730-dec3-4f29-8a38-9ae7c59f2aef'..'c757e740-9013-4ae0-901d-316932f4af0e'
failed. Images have been marked illegal and can no longer be previewed or
reverted to. Please retry Live Merge on the snapshot to complete the
operation.
2020-07-22 16:40:42,603+02 ERROR
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-53)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Ending command
'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotSingleDiskLiveCommand'
with failure.
2020-07-22 16:40:43,679+02 ERROR
[org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-15)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] Ending command
'org.ovirt.engine.core.bll.snapshots.RemoveSnapshotCommand' with failure.
2020-07-22 16:40:43,774+02 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-15)
[264b0047-5aa6-4380-9d32-eb328fd6bed0] EVENT_ID:
USER_REMOVE_SNAPSHOT_FINISHED_FAILURE(357), Failed to delete snapshot
'Auto-generated for Export To OVA' for VM 'Adhoc'.
VDSM on hypervisor
2020-07-22 14:14:30,220+0200 ERROR (jsonrpc/5) [virt.vm]
(vmId='14283e6d-c3f0-4011-b90f-a1272f0fbc10') Live merge failed (job:
e59c54d9-b8d3-44d0-9147-9dd40dff57b9) (vm:5381)
if ret == -1: raise libvirtError ('virDomainBlockCommit() failed',
dom=self)
libvirt.libvirtError: internal error: qemu block name 'json:{"backing":
{"driver": "qcow2", "file": {"driver": "file", "filename":
"/rhev/data-center/mnt/10.12.0.9:_exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/bb3aed4b-fc41-456a-9c18-1409a9aa6d14"}},
"driver": "qcow2", "file": {"driver": "file", "filename":
"/rhev/data-center/mnt/10.12.0.9:_exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/3995b256-2afb-4853-9360-33d0c12e5fd1"}}'
doesn't match expected '/rhev/data-center/mnt/10.12.0.9:
_exports_data/9a12f1b2-5378-46cc-964d-3575695e823f/images/3206de41-ccdc-4f2d-a968-5e4da6c2ca3e/3995b256-2afb-4853-9360-33d0c12e5fd1'
2020-07-22 14:14:30,234+0200 INFO (jsonrpc/5) [jsonrpc.JsonRpcServer] RPC
call VM.merge failed (error 52) in 0.17 seconds (__init__:312)
2020-07-22 14:17:28,798+0200 INFO (jsonrpc/2) [api] FINISH getStats
error=Virtual machine does not exist: {'vmId':
'698d486c-edbf-4e28-a199-31a2e27bd808'} (api:129)
2020-07-22 14:17:28,798+0200 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC
call VM.getStats failed (error 1) in 0.00 seconds (__init__:312)
Also in log,
INFO (jsonrpc/1) [api.virt] FINISH getStats return={'status': {'code': 1,
'message': "Virtual machine does not exist:
But is is there and accessible
Any advice here?
Henri
ovirt 4.4.0.3-1.el8
OS Version:
RHEL - 8 - 1.1911.0.9.el8
OS Description:
oVirt Node 4.4.0
Kernel Version:
4.18.0 - 147.8.1.el8_1.x86_64
KVM Version:
4.1.0 - 23.el8.1
LIBVIRT Version:
libvirt-5.6.0-10.el8
VDSM Version:
vdsm-4.40.16-1.el8
SPICE Version:
0.14.2 - 1.el8
GlusterFS Version:
glusterfs-7.5-1.el8
CEPH Version:
librbd1-12.2.7-9.el8
Open vSwitch Version:
openvswitch-2.11.1-5.el8
Nmstate Version:
nmstate-0.2.10-1.el8
4 years, 4 months
Re: Issue with ovirt 4.4 after doing some incremental backups.
by Eyal Shenitzky
Hi Łukasz,
Can you please provide vdsm.log and libvirt.log?
On Tue, 28 Jul 2020 at 16:05, Łukasz Kołaciński <l.kolacinski(a)storware.eu>
wrote:
> Hello,
>
> After doing a few vm backups, something breaks and I am unable to perform
> any operations. I cannot do incremental backups and even full backups
> doesn't work. I have this issue third time. I don't know how to fix this so
> I am currently making new vms for testing purposes
>
> VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup
> Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup':
> <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error
> starting backup: internal error: unable to execute QEMU command
> 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not
> found"}
>
>
> Best Regards,
>
> Łukasz Kołaciński
>
> Junior Java Developer
>
> e-mail: l.kolacinski(a)storware.eu
> <m.helbert(a)storware.eu>
>
>
>
>
> *[image: STORWARE]* <http://www.storware.eu/>
>
>
>
> *ul. Leszno 8/44 01-192 Warszawa www.storware.eu
> <https://www.storware.eu/>*
>
> *[image: facebook]* <https://www.facebook.com/storware>
>
> *[image: twitter]* <https://twitter.com/storware>
>
> *[image: linkedin]* <https://www.linkedin.com/company/storware>
>
> *[image: Storware_Stopka_09]*
> <https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw>
>
>
>
> *Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa
> 000510131* *, NIP 5213672602.** Wiadomość ta jest przeznaczona jedynie
> dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne
> i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie,
> przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub
> podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub
> podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez
> pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie
> tej wiadomości z wszelkich komputerów. **This message is intended only
> for the person or entity to which it is addressed and may contain
> confidential and/or privileged material. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon,
> this information by persons or entities other than the intended recipient
> is prohibited. If you have received this message in error, please contact
> the sender and remove the material from all of your computer systems.*
>
>
--
Regards,
Eyal Shenitzky
4 years, 4 months
Issue with ovirt 4.4 after doing some incremental backups.
by Łukasz Kołaciński
Hello,
After doing a few vm backups, something breaks and I am unable to perform any operations. I cannot do incremental backups and even full backups doesn't work. I have this issue third time. I don't know how to fix this so I am currently making new vms for testing purposes
VDSM ovirt44-h2.storware.local command StartVmBackupVDS failed: Backup Error: {'vm_id': '116aa6eb-31a1-43db-9b1e-ad6e32fb9260', 'backup': <vdsm.virt.backup.BackupConfig object at 0x7f42602bba20>, 'reason': "Error starting backup: internal error: unable to execute QEMU command 'transaction': Dirty bitmap 'ef0dfe55-c08c-4d9e-ad32-d6b6d5cbdac6' not found"}
Best Regards,
Łukasz Kołaciński
Junior Java Developer
e-mail: l.kolacinski(a)storware.eu<mailto:l.kolacinski@storware.eu>
<mailto:m.helbert@storware.eu>
[STORWARE]<http://www.storware.eu/>
ul. Leszno 8/44
01-192 Warszawa
www.storware.eu <https://www.storware.eu/>
[facebook]<https://www.facebook.com/storware>
[twitter]<https://twitter.com/storware>
[linkedin]<https://www.linkedin.com/company/storware>
[Storware_Stopka_09]<https://www.youtube.com/channel/UCKvLitYPyAplBctXibFWrkw>
Storware Spółka z o.o. nr wpisu do ewidencji KRS dla M.St. Warszawa 000510131 , NIP 5213672602. Wiadomość ta jest przeznaczona jedynie dla osoby lub podmiotu, który jest jej adresatem i może zawierać poufne i/lub uprzywilejowane informacje. Zakazane jest jakiekolwiek przeglądanie, przesyłanie, rozpowszechnianie lub inne wykorzystanie tych informacji lub podjęcie jakichkolwiek działań odnośnie tych informacji przez osoby lub podmioty inne niż zamierzony adresat. Jeżeli Państwo otrzymali przez pomyłkę tę informację prosimy o poinformowanie o tym nadawcy i usunięcie tej wiadomości z wszelkich komputerów. This message is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you have received this message in error, please contact the sender and remove the material from all of your computer systems.
4 years, 4 months
