[ovirt-users] Migration failed, No available host found

Mon Apr 6 12:13:35 EDT 2015

Hi Artyom,

The problems were caused by an issue with MTU on the hosts.  I have 
rectified the issue and can now migrate hosts.

Jason.

On 04/06/2015 10:57 AM, Jason Keltz wrote:
> Hi Artyom,
>
> Here are the vdsm logs from  virt1, virt2 (where the node is running), 
> and virt3.
> The logs from virt2 look suspicious, but still not sure the problem.
>
> http://goo.gl/GjbWUP
>
> Jason.
>
> On 04/06/2015 09:42 AM, Artyom Lukianov wrote:
>> Engine try to migrate vm on some available host, but migration 
>> failed, so engine try another host. From some reason migration failed 
>> on all hosts:
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
>> MigrateStatusVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
>> MigrateStatusVDS, error = Fatal error during migration, code = 12
>>
>> For future investigation we need vdsm logs(/var/log/vdsm/vdsm.log) 
>> from source and also from destination hosts.
>> Thanks
>>
>>
>> ----- Original Message -----
>> From: "Jason Keltz" <jas at cse.yorku.ca>
>> To: "users" <users at ovirt.org>
>> Sent: Monday, April 6, 2015 3:47:23 PM
>> Subject: [ovirt-users] Migration failed, No available host found
>>
>> Hi.
>>
>> I have 3 nodes in one cluster and 1 VM running on node2.  I'm trying to
>> move the VM to node 1 or node 3, and it fails with the error: Migration
>> failed, No available host found
>>
>> I'm unable to decipher engine.log to determine the cause of the
>> problem.  Below is what seems to be the relevant lines from the log.
>> Any help would be appreciated.
>>
>> Thank you!
>>
>> Jason.
>>
>> ---
>>
>> 2015-04-06 08:31:56,554 INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-5)
>> [3b191496] Lock Acquired to object EngineLock [exclusiveLocks= key:
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af value: VM
>> , sharedLocks= ]
>> 2015-04-06 08:31:56,686 INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] Running command:
>> MigrateVmCommand internal: false. Entities affected :  ID:
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type: VMAction group MIGRATE_VM
>> with role type USER,  ID: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type:
>> VMAction group EDIT_VM_PROPERTIES with role type USER,  ID:
>> 8d432949-e03c-4950-a91a-160727f7bdf2 Type: VdsGroupsAction group
>> CREATE_VM with role type USER
>> 2015-04-06 08:31:56,703 INFO
>> [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit] 
>>
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] Started HA reservation
>> scoring method
>> 2015-04-06 08:31:56,727 INFO
>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] START,
>> MigrateVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
>> dstVdsId=3429b1fc-36d5-4078-831c-a5b4370a8bfc,
>> dstHost=192.168.0.36:54321, migrationMethod=ONLINE,
>> tunnelMigration=false, migrationDowntime=0), log id: 7555acbd
>> 2015-04-06 08:31:56,728 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] START,
>> MigrateBrokerVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
>> dstVdsId=3429b1fc-36d5-4078-831c-a5b4370a8bfc,
>> dstHost=192.168.0.36:54321, migrationMethod=ONLINE,
>> tunnelMigration=false, migrationDowntime=0), log id: 6d98fb94
>> 2015-04-06 08:31:56,734 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] FINISH,
>> MigrateBrokerVDSCommand, log id: 6d98fb94
>> 2015-04-06 08:31:56,769 INFO
>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] FINISH,
>> MigrateVDSCommand, return: MigratingFrom, log id: 7555acbd
>> 2015-04-06 08:31:56,778 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (org.ovirt.thread.pool-8-thread-20) [3b191496] Correlation ID: 3b191496,
>> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
>> Event ID: -1, Message: Migration started (VM: nindigo, Source: virt2,
>> Destination: virt3, User: admin at internal).
>> 2015-04-06 08:33:17,633 INFO
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-35) [71f97a52] VM nindigo
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af moved from MigratingFrom --> Up
>> 2015-04-06 08:33:17,633 INFO
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-35) [71f97a52] Adding VM
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af to re-run list
>> 2015-04-06 08:33:17,661 ERROR
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-35) [71f97a52] Rerun vm
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af. Called from vds virt2
>> 2015-04-06 08:33:17,666 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
>> MigrateStatusVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af), log id: 6c3c9923
>> 2015-04-06 08:33:17,669 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Failed in
>> MigrateStatusVDS method
>> 2015-04-06 08:33:17,670 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
>> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return
>> value
>>    StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,
>> mMessage=Fatal error during migration]]
>> 2015-04-06 08:33:17,670 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] HostName = virt2
>> 2015-04-06 08:33:17,670 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
>> MigrateStatusVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
>> MigrateStatusVDS, error = Fatal error during migration, code = 12
>> 2015-04-06 08:33:17,671 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
>> MigrateStatusVDSCommand, log id: 6c3c9923
>> 2015-04-06 08:33:17,685 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Correlation ID: 3b191496,
>> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
>> Event ID: -1, Message: Migration failed due to Error: Fatal error during
>> migration. Trying to migrate to another Host (VM: nindigo, Source:
>> virt2, Destination: virt3).
>> 2015-04-06 08:33:17,776 INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Running command:
>> MigrateVmCommand internal: false. Entities affected :  ID:
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type: VMAction group MIGRATE_VM
>> with role type USER,  ID: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type:
>> VMAction group EDIT_VM_PROPERTIES with role type USER,  ID:
>> 8d432949-e03c-4950-a91a-160727f7bdf2 Type: VdsGroupsAction group
>> CREATE_VM with role type USER
>> 2015-04-06 08:33:17,825 INFO
>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
>> MigrateVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
>> dstVdsId=e21cf0b9-a7e7-41e7-96d0-27006b42ad84,
>> dstHost=192.168.0.34:54321, migrationMethod=ONLINE,
>> tunnelMigration=false, migrationDowntime=0), log id: 5642c909
>> 2015-04-06 08:33:17,827 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
>> MigrateBrokerVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
>> dstVdsId=e21cf0b9-a7e7-41e7-96d0-27006b42ad84,
>> dstHost=192.168.0.34:54321, migrationMethod=ONLINE,
>> tunnelMigration=false, migrationDowntime=0), log id: 1d4ead35
>> 2015-04-06 08:33:17,833 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
>> MigrateBrokerVDSCommand, log id: 1d4ead35
>> 2015-04-06 08:33:17,843 INFO
>> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
>> MigrateVDSCommand, return: MigratingFrom, log id: 5642c909
>> 2015-04-06 08:33:17,851 INFO
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Correlation ID: 3b191496,
>> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
>> Event ID: -1, Message: Migration started (VM: nindigo, Source: virt2,
>> Destination: virt1, User: admin at internal).
>> 2015-04-06 08:34:42,969 INFO
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] VM nindigo
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af moved from MigratingFrom --> Up
>> 2015-04-06 08:34:42,970 INFO
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] Adding VM
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af to re-run list
>> 2015-04-06 08:34:42,983 ERROR
>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
>> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] Rerun vm
>> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af. Called from vds virt2
>> 2015-04-06 08:34:43,026 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] START,
>> MigrateStatusVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af), log id: 7b89f823
>> 2015-04-06 08:34:43,029 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Failed in
>> MigrateStatusVDS method
>> 2015-04-06 08:34:43,029 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Command
>> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return
>> value
>>    StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,
>> mMessage=Fatal error during migration]]
>> 2015-04-06 08:34:43,030 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] HostName = virt2
>> 2015-04-06 08:34:43,030 ERROR
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Command
>> MigrateStatusVDSCommand(HostName = virt2, HostId =
>> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
>> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
>> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
>> MigrateStatusVDS, error = Fatal error during migration, code = 12
>> 2015-04-06 08:34:43,030 INFO
>> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] FINISH,
>> MigrateStatusVDSCommand, log id: 7b89f823
>> 2015-04-06 08:34:43,049 ERROR
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Correlation ID: 3b191496,
>> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
>> Event ID: -1, Message: Migration failed due to Error: Fatal error during
>> migration. Trying to migrate to another Host (VM: nindigo, Source:
>> virt2, Destination: virt1).
>> 2015-04-06 08:34:43,132 WARN
>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] CanDoAction of action
>> MigrateVm failed for user admin at internal. Reasons:
>> VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_NO_HOSTS 
>>
>> 2015-04-06 08:34:43,132 INFO
>> [org.ovirt.engine.core.bll.MigrateVmCommand]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Lock freed to object
>> EngineLock [exclusiveLocks= key: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af
>> value: VM
>> , sharedLocks= ]
>> 2015-04-06 08:34:43,165 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Correlation ID: 3b191496,
>> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
>> Event ID: -1, Message: Migration failed, No available host found (VM:
>> nindigo, Source: virt2).
>> _______________________________________________
>> Users mailing list
>> Users at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users