Hi !

When I try to migrate a powered VM (regardless of OS) from one hypervisor to another, the VM is immediately shut down with this error message:

Migration failed: Admin shut down from the engine (VM: VM, Source: HP11).

The oVirt engine has been upgraded from version 4.3 to version 4.4.
Some nodes are in version 4.3 and others in version 4.4.

Here are the oVirt versions for selected hypervisors:

HP11 : 4.4
HP5 : 4.4
HP6 : 4.3

Here are the migration attempts I tried with a powered VM :

From HP > to HP
HP6 > HP5 : OK
HP6 > HP11 : OK
HP5 > HP11 : OK
HP5 > HP6 : OK
HP11 > HP5 : NOK
HP11 > HP6 : OK

As mentioned above the migration of a VM between two versions of ovirt is not a problem.
The migration of the VM between two HPs with the same 4.4 version works only in one direction (HP5 to HP11) and doesn't work in the other way.

I already tried to reinstall both HPs in version 4.4 but without success.

Here are the logs on the HP5 concerning the VM:

/var/log/vdsm/vdsm.log:2023-05-11 14:32:56,303+0200 INFO (jsonrpc/3) [api.virt] START destroy(gracefulAttempts=1) from=::ffff:172.20.3.250,37534, flow_id=43364065, vmId=d14f75cd-1cb1-440b-9780-6b6ee78149ac (api:48)
/var/log/vdsm/vdsm.log:2023-05-11 14:32:56,303+0200 INFO (jsonrpc/3) [api] FINISH destroy error=Virtual machine does not exist: {'vmId': 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'} (api:129)
/var/log/vdsm/vdsm.log:2023-05-11 14:32:56,303+0200 INFO (jsonrpc/3) [api.virt] FINISH destroy return={'status': {'code': 1, 'message': "Virtual machine does not exist: {'vmId': 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'}"}} from=::ffff:172.20.3.250,37534, flow_id=43364065, vmId=d14f75cd-1cb1-440b-9780-6b6ee78149ac (api:54)

/var/log/libvirt/qemu/VM.log:2023-03-24 14:56:51.474+0000: initiating migration
/var/log/libvirt/qemu/VM.log:2023-03-24 14:56:54.342+0000: shutting down, reason=migrated
/var/log/libvirt/qemu/VM.log:2023-03-24T14:56:54.870528Z qemu-kvm: terminating on signal 15 from pid 4379 (<unknown process>)

Here are the log on the engine concerning the VM:

/var/log/ovirt-engine/engine.log:2023-05-11 14:32:53,333+02 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (default task-18197) [3f672d7f-f617-47a2-b0e9-c521656e8c01] START, MigrateVDSCommand( MigrateVDSCommandParameters:{hostId='6817e182-f163-4a44-9ad6-53156b8bb5a0', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', srcHost='HP11', dstVdsId='d2481de5-5ad2-4d06-9545-d5628cb87bcb', dstHost='HP5:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='null', consoleAddress='null', maxBandwidth='256', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='192.168.1.1'}), log id: 6a3507d0
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:53,334+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-18197) [3f672d7f-f617-47a2-b0e9-c521656e8c01] START, MigrateBrokerVDSCommand(HostName = HP11, MigrateVDSCommandParameters:{hostId='6817e182-f163-4a44-9ad6-53156b8bb5a0', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', srcHost='HP11', dstVdsId='d2481de5-5ad2-4d06-9545-d5628cb87bcb', dstHost='HP5:54321', migrationMethod='ONLINE', tunnelMigration='false', migrationDowntime='0', autoConverge='true', migrateCompressed='false', migrateEncrypted='null', consoleAddress='null', maxBandwidth='256', enableGuestEvents='true', maxIncomingMigrations='2', maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime, params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2, action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime, params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6, action={name=setDowntime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]', dstQemu='192.168.1.1'}), log id: f254f72
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,246+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [3f0e966d] VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac' was reported as Down on VDS '6817e182-f163-4a44-9ad6-53156b8bb5a0'(HP11)
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,296+02 INFO [org.ovirt.engine.core.bll.SaveVmExternalDataCommand] (ForkJoinPool-1-worker-9) [43364065] Running command: SaveVmExternalDataCommand internal: true. Entities affected : ID: d14f75cd-1cb1-440b-9780-6b6ee78149ac Type: VM
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,299+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] START, DestroyVDSCommand(HostName = HP11, DestroyVmVDSCommandParameters:{hostId='6817e182-f163-4a44-9ad6-53156b8bb5a0', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 20bf3f27
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,303+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] Failed to destroy VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac' because VM does not exist, ignoring
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,303+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [43364065] VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'(VM) moved from 'MigratingFrom' --> 'Down'
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,303+02 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (ForkJoinPool-1-worker-9) [43364065] START, DestroyVmVDSCommand( DestroyVmVDSCommandParameters:{hostId='d2481de5-5ad2-4d06-9545-d5628cb87bcb', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 1734109b
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,303+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] START, DestroyVDSCommand(HostName = HP5, DestroyVmVDSCommandParameters:{hostId='d2481de5-5ad2-4d06-9545-d5628cb87bcb', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 2679b538
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,307+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] Failed to destroy VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac' because VM does not exist, ignoring
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,310+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [43364065] Stopped migrating VM: 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'(VM) on VDS: 'd2481de5-5ad2-4d06-9545-d5628cb87bcb'
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,329+02 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand] (ForkJoinPool-1-worker-9) [43364065] Lock freed to object 'EngineLock:{exclusiveLocks='[d14f75cd-1cb1-440b-9780-6b6ee78149ac=VM]', sharedLocks=''}'
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,333+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [43364065] VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac' was reported as Down on VDS '6817e182-f163-4a44-9ad6-53156b8bb5a0'(HP11)
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,333+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [43364065] VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'(VM) was unexpectedly detected as 'Down' on VDS '6817e182-f163-4a44-9ad6-53156b8bb5a0'(HP11) (expected on 'null')
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,333+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] START, DestroyVDSCommand(HostName = HP11, DestroyVmVDSCommandParameters:{hostId='6817e182-f163-4a44-9ad6-53156b8bb5a0', vmId='d14f75cd-1cb1-440b-9780-6b6ee78149ac', secondsToWait='0', gracefully='false', reason='', ignoreNoVm='true'}), log id: 6a04ab1
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,358+02 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-9) [43364065] Failed to destroy VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac' because VM does not exist, ignoring
/var/log/ovirt-engine/engine.log:2023-05-11 14:32:56,358+02 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (ForkJoinPool-1-worker-9) [43364065] VM 'd14f75cd-1cb1-440b-9780-6b6ee78149ac'(VM) was unexpectedly detected as 'Down' on VDS '6817e182-f163-4a44-9ad6-53156b8bb5a0'(HP11) (expected on 'null')

Has anyone ever encountered this kind of problem following an oVirt cluster update?

Thanks,

Emmanuel