Problem with paused VMs in ovirt 4.3.10.

Friday, 24 July 2020

Hi!
We have a problem with paused VMs in ovirt cluster. Please, help to 
solve this.
In ovirt manager massege "VM rtb-stagedsw02-ovh has been paused."
Resume fails with error "Failed to resume VM rtb-stagedsw02-ovh (Host: 
ovirt-node09-ovh.local, User: admin(a)internal-authz).&quot;
In oVirt Cluster 38 VM, paused VM only ubuntu 20.04 focal with docker swarm.
Archived logs in attach.

Packeges on ovirt nodes:
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-provider-ovn-driver-1.2.29-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
cockpit-machines-ovirt-195.6-1.el7.centos.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-host-dependencies-4.3.5-1.el7.x86_64
ovirt-host-4.3.5-1.el7.x86_64
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-hosted-engine-setup-2.3.13-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-imageio-daemon-1.5.3-0.el7.noarch
cockpit-ovirt-dashboard-0.13.10-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-hosted-engine-ha-2.3.6-1.el7.noarch

Packeges on HostedEngine:

ovirt-ansible-infra-1.1.13-1.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-websocket-proxy-4.3.10.4-1.el7.noarch
ovirt-engine-restapi-4.3.10.4-1.el7.noarch
ovirt-ansible-engine-setup-1.1.9-1.el7.noarch
ovirt-ansible-shutdown-env-1.0.3-1.el7.noarch
ovirt-iso-uploader-4.3.2-1.el7.noarch
ovirt-provider-ovn-1.2.29-1.el7.noarch
ovirt-imageio-proxy-setup-1.5.3-0.el7.noarch
ovirt-engine-extension-aaa-ldap-setup-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
python-ovirt-engine-sdk4-4.3.4-2.el7.x86_64
python2-ovirt-host-deploy-1.8.5-1.el7.noarch
ovirt-ansible-vm-infra-1.1.22-1.el7.noarch
ovirt-engine-metrics-1.3.7-1.el7.noarch
ovirt-ansible-disaster-recovery-1.2.0-1.el7.noarch
ovirt-engine-wildfly-overlay-17.0.1-1.el7.noarch
ovirt-ansible-roles-1.1.7-1.el7.noarch
ovirt-engine-dwh-setup-4.3.8-1.el7.noarch
python2-ovirt-engine-lib-4.3.10.4-1.el7.noarch
ovirt-engine-extension-aaa-ldap-1.3.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-engine-vmconsole-proxy-helper-4.3.10.4-1.el7.noarch
ovirt-engine-tools-backup-4.3.10.4-1.el7.noarch
ovirt-engine-webadmin-portal-4.3.10.4-1.el7.noarch
ovirt-host-deploy-common-1.8.5-1.el7.noarch
ovirt-ansible-image-template-1.1.12-1.el7.noarch
ovirt-ansible-manageiq-1.1.14-1.el7.noarch
ovirt-engine-wildfly-17.0.1-1.el7.x86_64
ovirt-ansible-hosted-engine-setup-1.0.32-1.el7.noarch
ovirt-imageio-common-1.5.3-0.el7.x86_64
ovirt-imageio-proxy-1.5.3-0.el7.noarch
python2-ovirt-setup-lib-1.2.0-1.el7.noarch
ovirt-vmconsole-proxy-1.0.7-2.el7.noarch
ovirt-engine-setup-base-4.3.10.4-1.el7.noarch
ovirt-engine-setup-plugin-cinderlib-4.3.10.4-1.el7.noarch
ovirt-engine-extensions-api-impl-4.3.10.4-1.el7.noarch
ovirt-release43-4.3.10-1.el7.noarch
ovirt-engine-backend-4.3.10.4-1.el7.noarch
ovirt-engine-tools-4.3.10.4-1.el7.noarch
ovirt-web-ui-1.6.0-1.el7.noarch
ovirt-ansible-cluster-upgrade-1.1.14-1.el7.noarch
ovirt-cockpit-sso-0.1.1-1.el7.noarch
ovirt-engine-ui-extensions-1.0.10-1.el7.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.3.10.4-1.el7.noarch
ovirt-engine-4.3.10.4-1.el7.noarch
ovirt-ansible-repositories-1.1.5-1.el7.noarch
ovirt-engine-extension-aaa-jdbc-1.1.10-1.el7.noarch
ovirt-host-deploy-java-1.8.5-1.el7.noarch
ovirt-engine-dwh-4.3.8-1.el7.noarch
ovirt-engine-api-explorer-0.0.5-1.el7.noarch
ovirt-guest-agent-common-1.0.16-1.el7.noarch
ovirt-engine-setup-4.3.10.4-1.el7.noarch
ovirt-engine-dbscripts-4.3.10.4-1.el7.noarch

In /var/log/ovirt-engine/engine.log:

    2020-07-24 09:38:44,472+03 INFO
    [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
    (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] VM
    '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99'(rtb-stagedsw02-ovh) move
    d from 'Up' --> 'Paused'
    2020-07-24 09:38:44,493+03 INFO
    [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
    (EE-ManagedThreadFactory-engineScheduled-Thread-91) [] EVENT_ID:
    VM_PAUSED(1,025), VM rtb-stagedsw02-ovh has been paused.

In /var/log/vdsm/vdsm.log

    2020-07-24 09:38:42,771+0300 INFO  (libvirt/events) [virt.vm]
    (vmId='18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99') CPU stopped: onSuspend
    (vm:6100)
    2020-07-24 09:38:44,328+0300 INFO  (jsonrpc/1) [api.host] FINISH
    getAllVmIoTunePolicies return={'status': {'message': 'Done',
'code':
    0}, 'io_tune_policies_dict':
    {'4d9519f6-1ab9-4032-8fdf-4c6118531544': {'poli
    cy': [], 'current_values': [{'ioTune': {'write_bytes_sec':
0L,
    'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec':
0L,
    'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
    '/rhev/data-center/mnt/glust

erSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/e1cfb9ec-39d8-416d-9f5f-0b54765301d4/8f95d60d-931b-4764-993c-ba9373efe361',
    'name': 'sda'}]}, 'b031a269-6bcd-40b7-9737-e47112a54b3a':
{'po
    licy': [], 'current_values': [{'ioTune':
{'write_bytes_sec': 0L,
    'total_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec':
0L,
    'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':
    '/rhev/data-center/mnt/glu

sterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/9c3cc7a0-254e-4756-91b6-fb54e21abf38/71dd8024-8aec-46da-a80f-34260655e929',
    'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L,
'total_io
    ps_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
    'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':

'/rhev/data-center/mnt/glusterSD/10.0.11.101:_vmstore01/5e05fed3-448b-4f86-b5ba-004982194c90/images/
    3e3a5064-5fe1-40c0-81f5-44f1a3a4d503/13549972-82de-4746-aeea-3e1531f9c180',
    'name': 'sdb'}]}, 'b5fad17c-fa9d-4a80-99e7-6f86e6e19c9b':
{'policy':
    [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'total_
    iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
    'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':

'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/image
    s/15ce6cb0-6f06-4a31-92d8-b6e1bcabf3bc/613de344-d1ad-49aa-a2d0-d60ca9eb7cd3',
    'name': 'sda'}]}, '18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99':
{'policy':
    [], 'current_values': [{'ioTune': {'write_bytes_sec': 0L,
'tota
    l_iops_sec': 0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L,
    'write_iops_sec': 0L, 'total_bytes_sec': 0L}, 'path':

u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/im
    ages/7978e2db-c560-4315-a775-223f1b13ae31/d927eea8-e588-449e-b07b-c845d15b082e',
    'name': 'sda'}, {'ioTune': {'write_bytes_sec': 0L,
'total_iops_sec':
    0L, 'read_iops_sec': 0L, 'read_bytes_sec': 0L, 'write_iops_s
    ec': 0L, 'total_bytes_sec': 0L}, 'path':

u'/rhev/data-center/mnt/glusterSD/10.0.11.107:_vmstore02/16c5070c-cc5f-4595-965f-66838c7c17a5/images/b925dc2e-17ba-470d-a9be-cb96d4ef1f0d/951d9712-7160-4f88-a838-970aec8
    2b3ea', 'name': 'sdb'}]}}} from=::1,34598 (api:54)
    2020-07-24 09:38:49,747+0300 WARN  (qgapoller/1)
    [virt.periodic.VmDispatcher] could not run <function <lambda> at
    0x7fe5c84de6e0> on ['18f6bb79-ba9b-4a0e-bcb2-b4ef4904ef99']
    (periodic:289)

In /var/log/libvirt/qemu/rtb-stagedsw03-ovh.log

    KVM: entry failed, hardware error 0x80000021

    If you're running a guest on an Intel machine without unrestricted mode
    support, the failure can be most likely due to the guest entering an
    invalid
    state for Intel VT. For example, the guest maybe running in big real
    mode
    which is not supported on less recent Intel processors.

    EAX=00001000 EBX=43117da8 ECX=0000000c EDX=00000121
    ESI=00000003 EDI=17921000 EBP=43117cb0 ESP=43117c98
    EIP=00008000 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=1 HLT=0
    ES =0000 00000000 ffffffff 00809300
    CS =9b00 7ff9b000 ffffffff 00809300
    SS =0000 00000000 ffffffff 00809300
    DS =0000 00000000 ffffffff 00809300
    FS =0000 00000000 ffffffff 00809300
    GS =0000 00000000 ffffffff 00809300
    LDT=0000 00000000 000fffff 00000000
    TR =0040 001ce000 0000206f 00008b00
    GDT=     001cc000 0000007f
    IDT=     00000000 00000000
    CR0=00050032 CR2=17921000 CR3=2b92a003 CR4=00000000
    DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000
    DR3=0000000000000000
    DR6=00000000fffe0ff0 DR7=0000000000000400
    EFER=0000000000000000
    Code=ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    <ff> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
    ff ff ff ff ff ff ff ff

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011