1:48 a.m.
Hi:
I deployed 2 ovirt hosts and an ovirt engine in a nested KVM server. I've a
windows vm setup and tried to perform live migration but failed. I checked on the hosts
and found them meeting the live migration requirements, or at least that's what I
thought. I took the requirement from the below document.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtuali...
The hosts, both source and destination are quite empty, with only the hosted engine, 1
centos vm and the windows VM in the cluster. I can do a live migration for the centos vm
successfully. But when I tried live migration on the hosted-engine vm it failed
immediately with a message "No available host to migrate VMs to". When I tried
to migrate the windows VM the message box that let me choose the destination host popped
up but failed after a while.
I'd like to ask where can I get more information with regards to live-migration
apart from /var/log/ovirt-engine/engine.log ? I also checked on the ovirt hosts'
/var/log/vdsm/vdsm.log but found nothing pointing to the reason why it failed.
Below is the extract from /var/log/ovirt-engine/engine.log when the live-migration took
place
2019-03-12 14:37:58,159+08 INFO [org.ovirt.engine.core.sso.utils.AuthenticationUtils]
(default task-131) [] User admin@internal successfully logged in with scopes:
ovirt-app-api ovirt-ext=token-info:authz-search ovirt-ext=token-info:public-authz-search
ovirt-ext=token-info:validate ovirt-ext=token:password-access
2019-03-12 14:37:58,450+08 INFO
[org.ovirt.engine.core.bll.provider.network.SyncNetworkProviderCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-59) [77888830] Lock freed to object
'EngineLock:{exclusiveLocks='[d113be83-2740-4246-a1f2-b9344889c3cf=PROVIDER]',
sharedLocks=''}'
2019-03-12 14:38:02,544+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-50) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:38:12,677+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-16) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:38:21,650+08 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer]
(EE-ManagedThreadFactory-engineScheduled-Thread-51) [] Not removing session
'xDiHqqa6l+g8cngM26TTCfW7NeLN3WgWChsx28wUM391vAngSxwtyCkLbQxZR1AbJ5I+2bkPZNQijMUk0jLZcA==',
session has running commands for user 'admin@internal-authz'.
2019-03-12 14:38:22,782+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-49) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:38:33,018+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-74) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:38:43,261+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-59) [77888830]
BaseAsyncTask::onTaskEndSuccess: Task '67631cf6-4c75-4681-88ef-fd4af56c0363'
(Parent Command 'RemoveDisk', Parameters Type
'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters') ended
successfully.
2019-03-12 14:38:53,528+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-13) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:39:03,759+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-43) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:39:14,011+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-60) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:39:21,660+08 INFO [org.ovirt.engine.core.bll.aaa.SessionDataContainer]
(EE-ManagedThreadFactory-engineScheduled-Thread-79) [] Not removing session
'xDiHqqa6l+g8cngM26TTCfW7NeLN3WgWChsx28wUM391vAngSxwtyCkLbQxZR1AbJ5I+2bkPZNQijMUk0jLZcA==',
session has running commands for user 'admin@internal-authz'.
2019-03-12 14:39:24,122+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-85) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:39:29,773+08 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
(default task-131) [7f0cf113-55e8-4def-9c68-de3b91d6d641] Lock Acquired to object
'EngineLock:{exclusiveLocks='[5cad5c5f-5aab-46ec-a28e-d484abc0401d=VM]',
sharedLocks=''}'
2019-03-12 14:39:29,887+08 INFO [org.ovirt.engine.core.bll.MigrateVmToServerCommand]
(default task-131) [7f0cf113-55e8-4def-9c68-de3b91d6d641] Running command:
MigrateVmToServerCommand internal: false. Entities affected : ID:
5cad5c5f-5aab-46ec-a28e-d484abc0401d Type: VMAction group MIGRATE_VM with role type USER
2019-03-12 14:39:30,019+08 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(default task-131) [7f0cf113-55e8-4def-9c68-de3b91d6d641] START, MigrateVDSCommand(
MigrateVDSCommandParameters:{hostId='f9014bc4-485c-4eb0-a9bc-42d13ed68f41',
vmId='5cad5c5f-5aab-46ec-a28e-d484abc0401d', srcHost='host2.xxxx.com',
dstVdsId='1bc9b9e9-1e90-4570-9930-08416d1927cc',
dstHost='host3.xxxx.com:54321', migrationMethod='ONLINE',
tunnelMigration='false', migrationDowntime='0',
autoConverge='true', migrateCompressed='false',
consoleAddress='null', maxBandwidth='null',
enableGuestEvents='true', maxIncomingMigrations='2',
maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime,
params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime,
params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6,
action={name=setDowntime, params=[500]}}, {limit=-1, action={nam
e=abort, params=[]}}]]', dstQemu='192.168.138.135'}), log id: 7eeb678c
2019-03-12 14:39:30,022+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-131)
[7f0cf113-55e8-4def-9c68-de3b91d6d641] START, MigrateBrokerVDSCommand(HostName =
host2.xxxx.com,
MigrateVDSCommandParameters:{hostId='f9014bc4-485c-4eb0-a9bc-42d13ed68f41',
vmId='5cad5c5f-5aab-46ec-a28e-d484abc0401d', srcHost='host2.xxxx.com',
dstVdsId='1bc9b9e9-1e90-4570-9930-08416d1927cc',
dstHost='host3.xxxx.com:54321', migrationMethod='ONLINE',
tunnelMigration='false', migrationDowntime='0',
autoConverge='true', migrateCompressed='false',
consoleAddress='null', maxBandwidth='null',
enableGuestEvents='true', maxIncomingMigrations='2',
maxOutgoingMigrations='2', convergenceSchedule='[init=[{name=setDowntime,
params=[100]}], stalling=[{limit=1, action={name=setDowntime, params=[150]}}, {limit=2,
action={name=setDowntime, params=[200]}}, {limit=3, action={name=setDowntime,
params=[300]}}, {limit=4, action={name=setDowntime, params=[400]}}, {limit=6,
action={name=set
Downtime, params=[500]}}, {limit=-1, action={name=abort, params=[]}}]]',
dstQemu='192.168.138.135'}), log id: 5cef4981
2019-03-12 14:39:30,039+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (default task-131)
[7f0cf113-55e8-4def-9c68-de3b91d6d641] FINISH, MigrateBrokerVDSCommand, return: , log id:
5cef4981
2019-03-12 14:39:30,048+08 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
(default task-131) [7f0cf113-55e8-4def-9c68-de3b91d6d641] FINISH, MigrateVDSCommand,
return: MigratingFrom, log id: 7eeb678c
2019-03-12 14:39:30,067+08 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-131)
[7f0cf113-55e8-4def-9c68-de3b91d6d641] EVENT_ID: VM_MIGRATION_START(62), Migration started
(VM: Win_2016_1, Source: host2.xxxx.com, Destination: host3, User: admin@internal-authz).
2019-03-12 14:39:33,901+08 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-3) [] VM '5cad5c5f-5aab-46ec-a28e-d484abc0401d' was
reported as Down on VDS '1bc9b9e9-1e90-4570-9930-08416d1927cc'(host3)
2019-03-12 14:39:33,903+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-3) []
START, DestroyVDSCommand(HostName = host3,
DestroyVmVDSCommandParameters:{hostId='1bc9b9e9-1e90-4570-9930-08416d1927cc',
vmId='5cad5c5f-5aab-46ec-a28e-d484abc0401d', secondsToWait='0',
gracefully='false', reason='', ignoreNoVm='true'}), log id:
c853ba5
2019-03-12 14:39:34,211+08 INFO [org.ovirt.engine.core.bll.tasks.SPMAsyncTask]
(EE-ManagedThreadFactory-engineScheduled-Thread-73) [] BaseAsyncTask::onTaskEndSuccess:
Task '67631cf6-4c75-4681-88ef-fd4af56c0363' (Parent Command 'RemoveDisk',
Parameters Type 'org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters')
ended successfully.
2019-03-12 14:39:34,604+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-3) []
Failed to destroy VM '5cad5c5f-5aab-46ec-a28e-d484abc0401d' because VM does not
exist, ignoring
2019-03-12 14:39:34,605+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (ForkJoinPool-1-worker-3) []
FINISH, DestroyVDSCommand, return: , log id: c853ba5
2019-03-12 14:39:34,605+08 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-3) [] VM '5cad5c5f-5aab-46ec-a28e-d484abc0401d'(Win_2016_1)
was unexpectedly detected as 'Down' on VDS
'1bc9b9e9-1e90-4570-9930-08416d1927cc'(ohost3) (expected on
'f9014bc4-485c-4eb0-a9bc-42d13ed68f41')
2019-03-12 14:39:34,605+08 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-3) [] Migration of VM 'Win_2016_1' to host 'host3'
failed: VM destroyed during the startup.
2019-03-12 14:39:34,615+08 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-10) [] VM
'5cad5c5f-5aab-46ec-a28e-d484abc0401d'(Win_2016_1) moved from
'MigratingFrom' --> 'Up'
2019-03-12 14:39:34,615+08 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer]
(ForkJoinPool-1-worker-10) [] Adding VM
'5cad5c5f-5aab-46ec-a28e-d484abc0401d'(Win_2016_1) to re-run list
2019-03-12 14:39:34,621+08 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-10) []
Rerun VM '5cad5c5f-5aab-46ec-a28e-d484abc0401d'. Called from VDS
'host2.xxxx.com'
2019-03-12 14:39:34,752+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-53959) [] START, MigrateStatusVDSCommand(HostName =
host2.xxxx.com,
MigrateStatusVDSCommandParameters:{hostId='f9014bc4-485c-4eb0-a9bc-42d13ed68f41',
vmId='5cad5c5f-5aab-46ec-a28e-d484abc0401d'}), log id: 7ded4ad7
2019-03-12 14:39:34,760+08 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
(EE-ManagedThreadFactory-engine-Thread-53959) [] FINISH, MigrateStatusVDSCommand, return:
, log id: 7ded4ad7
2019-03-12 14:39:34,786+08 ERROR
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-53959) [] EVENT_ID:
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: Win_2016_1, Source:
host2.xxxx.com, Destination: host3).
Any help is greatly appreciated.
regards,
Bong SF