6:44 a.m.
Hi!
We have a problem with paused VMs in ovirt cluster. Please, help to
solve this.
In ovirt manager massege "VM rtb-stagedsw02-ovh has been paused."
Resume fails with error "Failed to resume VM rtb-stagedsw02-ovh (Host:
ovirt-node09-ovh.local, User: admin(a)internal-authz)."
In oVirt Cluster 38 VM, paused VM only ubuntu 20.04 focal with docker swarm.
Archived logs in attach.
Packeges on ovirt nodes:
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-provider-ovn-driver-1.2.29-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
cockpit-machines-ovirt-195.6-1.el7.centos.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-host-dependencies-4.3.5-1.el7.x86_64
ovirt-host-4.3.5-1.el7.x86_64
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-hosted-engine-setup-2.3.13-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-imageio-daemon-1.5.3-0.el7.noarch
cockpit-ovirt-dashboard-0.13.10-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-hosted-engine-ha-2.3.6-1.el7.noarch
Packeges on HostedEngine:
ovirt-ansible-infra-1.1.13-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-restapi-4.3.10.4-1.el7.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-ansible-shutdown-env-1.0.3-1.el7.noarch
ovirt-iso-uploader-4.3.2-1.el7.noarch
ovirt-provider-ovn-1.2.29-1.el7.noarch
ovirt-imageio-proxy-setup-1.5.3-0.el7.noarch
ovirt-engine-extension-aaa-ldap-setup-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-ansible-vm-infra-1.1.22-1.el7.noarch
ovirt-engine-metrics-1.3.7-1.el7.noarch
ovirt-ansible-disaster-recovery-1.2.0-1.el7.noarch
ovirt-engine-wildfly-overlay-17.0.1-1.el7.noarch
ovirt-ansible-roles-1.1.7-1.el7.noarch
ovirt-engine-dwh-setup-4.3.8-1.el7.noarch
python2-ovirt-engine-lib-4.3.10.4-1.el7.noarch
ovirt-engine-extension-aaa-ldap-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-engine-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
ovirt-engine-tools-backup-4.3.10.4-1.el7.noarch
ovirt-engine-webadmin-portal-4.3.10.4-1.el7.noarch
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-image-template-1.1.12-1.el7.noarch
ovirt-ansible-manageiq-1.1.14-1.el7.noarch
ovirt-engine-wildfly-17.0.1-1.el7.x86_64
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
ovirt-imageio-proxy-1.5.3-0.el7.noarch
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-proxy-1.0.7-2.el7.noarch
ovirt-engine-setup-base-4.3.10.4-1.el7.noarch
ovirt-engine-setup-plugin-cinderlib-4.3.10.4-1.el7.noarch
ovirt-engine-extensions-api-impl-4.3.10.4-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-engine-backend-4.3.10.4-1.el7.noarch
ovirt-engine-tools-4.3.10.4-1.el7.noarch
ovirt-web-ui-1.6.0-1.el7.noarch
ovirt-ansible-cluster-upgrade-1.1.14-1.el7.noarch
ovirt-cockpit-sso-0.1.1-1.el7.noarch
ovirt-engine-ui-extensions-1.0.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.3.10.4-1.el7.noarch
ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-engine-extension-aaa-jdbc-1.1.10-1.el7.noarch
ovirt-host-deploy-java-1.8.5-1.el7.noarch
ovirt-engine-dwh-4.3.8-1.el7.noarch
ovirt-engine-api-explorer-0.0.5-1.el7.noarch
ovirt-guest-agent-common-1.0.16-1.el7.noarch
ovirt-engine-setup-4.3.10.4-1.el7.noarch
ovirt-engine-dbscripts-4.3.10.4-1.el7.noarch
In /var/log/ovirt-engine/engine.log:
2020-07-24 09:38:44,472+03 INFO
[org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [] VM
'18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99'(rtb-stagedsw02-ovh) move
d from 'Up' --> 'Paused'
2020-07-24 09:38:44,493+03 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID:
VM_PAUSED(1,025), VM rtb-stagedsw02-ovh has been paused.
In /var/log/vdsm/vdsm.log
2020-07-24 09:38:42,771+0300 INFO (libvirt/events) [virt.vm]
(vmId='18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99') CPU stopped: onSuspend
(vm:6100)
2020-07-24 09:38:44,328+0300 INFO (jsonrpc/1) [api.host] FINISH
getAllVmIoTunePolicies return={'status': {'message': 'Done',
'code':
0}, 'io_tune_policies_dict':
{'4d9519f6-1ab9-4032-8fdf-4c6118531544': {'poli
cy': [], 'current_values': [{'ioTune': {'write_bytes_sec':
0L,
'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec':
0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glust
erSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/e1cfb9ec-39d8-416d-9f5f-0b54765301d4/8f95d60d-931b-4764-993c-ba9373efe361',
'name': 'sda'}]}, 'b031a269-6bcd-40b7-9737-e47112a54b3a':
{'po
licy': [], 'current_values': [{'ioTune':
{'write_bytes_sec': 0L,
'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec':
0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glu
sterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/9c3cc7a0-254e-4756-91b6-fb54e21abf38/71dd8024-8aec-46da-a80f-34260655e929',
'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L,
'total_io
ps_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glusterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/
3e3a5064-5fe1-40c0-81f5-44f1a3a4d503/13549972-82de-4746-aeea-3e1531f9c180',
'name': 'sdb'}]}, 'b5fad17c-fa9d-4a80-99e7-6f86e6e19c9b':
{'policy':
[], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'total_
iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/image
s/15ce6cb0-6f06-4a31-92d8-b6e1bcabf3bc/613de344-d1ad-49aa-a2d0-d60ca9eb7cd3',
'name': 'sda'}]}, '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99':
{'policy':
[], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'tota
l_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/im
ages/7978e2db-c560-4315-a775-223f1b13ae31/d927eea8-e588-449e-b07b-c845d15b082e',
'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L,
'total_iops_sec':
0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_s
ec': 0L, 'total_bytes_sec': 0L}, 'path':
u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/b925dc2e-17ba-470d-a9be-cb96d4ef1f0d/951d9712-7160-4f88-a838-970aec8
2b3ea', 'name': 'sdb'}]}}} from=::1,34598 (api:54)
2020-07-24 09:38:49,747+0300 WARN (qgapoller/1)
[virt.periodic.VmDispatcher] could not run <function <lambda> at
0x7fe5c84de6e0> on ['18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99']
(periodic:289)
In /var/log/libvirt/qemu/rtb-stagedsw03-ovh.log
KVM: entry failed, hardware error 0x80000021
If you're running a guest on an Intel machine without unrestricted mode
support, the failure can be most likely due to the guest entering an
invalid
state for Intel VT. For example, the guest maybe running in big real
mode
which is not supported on less recent Intel processors.
EAX=00001000 EBX=43117da8 ECX=0000000c EDX=00000121
ESI=00000003 EDI=17921000 EBP=43117cb0 ESP=43117c98
EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
ES =0000 00000000 ffffffff 00809300
CS =9b00 7ff9b000 ffffffff 00809300
SS =0000 00000000 ffffffff 00809300
DS =0000 00000000 ffffffff 00809300
FS =0000 00000000 ffffffff 00809300
GS =0000 00000000 ffffffff 00809300
LDT=0000 00000000 000fffff 00000000
TR =0040 001ce000 0000206f 00008b00
GDT= 001cc000 0000007f
IDT= 00000000 00000000
CR0=00050032 CR2=17921000 CR3=2b92a003 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
DR3=0000000000000000
DR6=00000000fffe0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
<ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
ff ff ff ff ff ff ff ff