[ovirt-users] Migration failed, No available host found

Mon Apr 6 14:57:49 UTC 2015

Hi Artyom,

Here are the vdsm logs from  virt1, virt2 (where the node is running), 
and virt3.
The logs from virt2 look suspicious, but still not sure the problem.

http://goo.gl/GjbWUP

Jason.

On 04/06/2015 09:42 AM, Artyom Lukianov wrote:
> Engine try to migrate vm on some available host, but migration failed, so engine try another host. From some reason migration failed on all hosts:
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
> MigrateStatusVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
> MigrateStatusVDS, error = Fatal error during migration, code = 12
>
> For future investigation we need vdsm logs(/var/log/vdsm/vdsm.log) from source and also from destination hosts.
> Thanks
>
>
> ----- Original Message -----
> From: "Jason Keltz" <jas at cse.yorku.ca>
> To: "users" <users at ovirt.org>
> Sent: Monday, April 6, 2015 3:47:23 PM
> Subject: [ovirt-users] Migration failed, No available host found
>
> Hi.
>
> I have 3 nodes in one cluster and 1 VM running on node2.  I'm trying to
> move the VM to node 1 or node 3, and it fails with the error: Migration
> failed, No available host found
>
> I'm unable to decipher engine.log to determine the cause of the
> problem.  Below is what seems to be the relevant lines from the log.
> Any help would be appreciated.
>
> Thank you!
>
> Jason.
>
> ---
>
> 2015-04-06 08:31:56,554 INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand] (ajp--127.0.0.1-8702-5)
> [3b191496] Lock Acquired to object EngineLock [exclusiveLocks= key:
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af value: VM
> , sharedLocks= ]
> 2015-04-06 08:31:56,686 INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] Running command:
> MigrateVmCommand internal: false. Entities affected :  ID:
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type: VMAction group MIGRATE_VM
> with role type USER,  ID: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type:
> VMAction group EDIT_VM_PROPERTIES with role type USER,  ID:
> 8d432949-e03c-4950-a91a-160727f7bdf2 Type: VdsGroupsAction group
> CREATE_VM with role type USER
> 2015-04-06 08:31:56,703 INFO
> [org.ovirt.engine.core.bll.scheduling.policyunits.HaReservationWeightPolicyUnit]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] Started HA reservation
> scoring method
> 2015-04-06 08:31:56,727 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] START,
> MigrateVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
> dstVdsId=3429b1fc-36d5-4078-831c-a5b4370a8bfc,
> dstHost=192.168.0.36:54321, migrationMethod=ONLINE,
> tunnelMigration=false, migrationDowntime=0), log id: 7555acbd
> 2015-04-06 08:31:56,728 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] START,
> MigrateBrokerVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
> dstVdsId=3429b1fc-36d5-4078-831c-a5b4370a8bfc,
> dstHost=192.168.0.36:54321, migrationMethod=ONLINE,
> tunnelMigration=false, migrationDowntime=0), log id: 6d98fb94
> 2015-04-06 08:31:56,734 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] FINISH,
> MigrateBrokerVDSCommand, log id: 6d98fb94
> 2015-04-06 08:31:56,769 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] FINISH,
> MigrateVDSCommand, return: MigratingFrom, log id: 7555acbd
> 2015-04-06 08:31:56,778 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-20) [3b191496] Correlation ID: 3b191496,
> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
> Event ID: -1, Message: Migration started (VM: nindigo, Source: virt2,
> Destination: virt3, User: admin at internal).
> 2015-04-06 08:33:17,633 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-35) [71f97a52] VM nindigo
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af moved from MigratingFrom --> Up
> 2015-04-06 08:33:17,633 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-35) [71f97a52] Adding VM
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af to re-run list
> 2015-04-06 08:33:17,661 ERROR
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-35) [71f97a52] Rerun vm
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af. Called from vds virt2
> 2015-04-06 08:33:17,666 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
> MigrateStatusVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af), log id: 6c3c9923
> 2015-04-06 08:33:17,669 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Failed in
> MigrateStatusVDS method
> 2015-04-06 08:33:17,670 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return
> value
>    StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,
> mMessage=Fatal error during migration]]
> 2015-04-06 08:33:17,670 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] HostName = virt2
> 2015-04-06 08:33:17,670 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Command
> MigrateStatusVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
> MigrateStatusVDS, error = Fatal error during migration, code = 12
> 2015-04-06 08:33:17,671 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
> MigrateStatusVDSCommand, log id: 6c3c9923
> 2015-04-06 08:33:17,685 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Correlation ID: 3b191496,
> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
> Event ID: -1, Message: Migration failed due to Error: Fatal error during
> migration. Trying to migrate to another Host (VM: nindigo, Source:
> virt2, Destination: virt3).
> 2015-04-06 08:33:17,776 INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Running command:
> MigrateVmCommand internal: false. Entities affected :  ID:
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type: VMAction group MIGRATE_VM
> with role type USER,  ID: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af Type:
> VMAction group EDIT_VM_PROPERTIES with role type USER,  ID:
> 8d432949-e03c-4950-a91a-160727f7bdf2 Type: VdsGroupsAction group
> CREATE_VM with role type USER
> 2015-04-06 08:33:17,825 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
> MigrateVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
> dstVdsId=e21cf0b9-a7e7-41e7-96d0-27006b42ad84,
> dstHost=192.168.0.34:54321, migrationMethod=ONLINE,
> tunnelMigration=false, migrationDowntime=0), log id: 5642c909
> 2015-04-06 08:33:17,827 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] START,
> MigrateBrokerVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af, srcHost=192.168.0.35,
> dstVdsId=e21cf0b9-a7e7-41e7-96d0-27006b42ad84,
> dstHost=192.168.0.34:54321, migrationMethod=ONLINE,
> tunnelMigration=false, migrationDowntime=0), log id: 1d4ead35
> 2015-04-06 08:33:17,833 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
> MigrateBrokerVDSCommand, log id: 1d4ead35
> 2015-04-06 08:33:17,843 INFO
> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] FINISH,
> MigrateVDSCommand, return: MigratingFrom, log id: 5642c909
> 2015-04-06 08:33:17,851 INFO
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-38) [71f97a52] Correlation ID: 3b191496,
> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
> Event ID: -1, Message: Migration started (VM: nindigo, Source: virt2,
> Destination: virt1, User: admin at internal).
> 2015-04-06 08:34:42,969 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] VM nindigo
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af moved from MigratingFrom --> Up
> 2015-04-06 08:34:42,970 INFO
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] Adding VM
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af to re-run list
> 2015-04-06 08:34:42,983 ERROR
> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]
> (DefaultQuartzScheduler_Worker-54) [2b07d5b3] Rerun vm
> 9de649ca-c9a9-4ba7-bb2c-61c44e2819af. Called from vds virt2
> 2015-04-06 08:34:43,026 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] START,
> MigrateStatusVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af), log id: 7b89f823
> 2015-04-06 08:34:43,029 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Failed in
> MigrateStatusVDS method
> 2015-04-06 08:34:43,029 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Command
> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return
> value
>    StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=12,
> mMessage=Fatal error during migration]]
> 2015-04-06 08:34:43,030 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] HostName = virt2
> 2015-04-06 08:34:43,030 ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Command
> MigrateStatusVDSCommand(HostName = virt2, HostId =
> 1d1d1fbb-3067-4703-8b51-e0a231d344e6,
> vmId=9de649ca-c9a9-4ba7-bb2c-61c44e2819af) execution failed. Exception:
> VDSErrorException: VDSGenericException: VDSErrorException: Failed to
> MigrateStatusVDS, error = Fatal error during migration, code = 12
> 2015-04-06 08:34:43,030 INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] FINISH,
> MigrateStatusVDSCommand, log id: 7b89f823
> 2015-04-06 08:34:43,049 ERROR
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Correlation ID: 3b191496,
> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
> Event ID: -1, Message: Migration failed due to Error: Fatal error during
> migration. Trying to migrate to another Host (VM: nindigo, Source:
> virt2, Destination: virt1).
> 2015-04-06 08:34:43,132 WARN
> [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] CanDoAction of action
> MigrateVm failed for user admin at internal. Reasons:
> VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,VAR__ACTION__MIGRATE,VAR__TYPE__VM,SCHEDULING_NO_HOSTS
> 2015-04-06 08:34:43,132 INFO
> [org.ovirt.engine.core.bll.MigrateVmCommand]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Lock freed to object
> EngineLock [exclusiveLocks= key: 9de649ca-c9a9-4ba7-bb2c-61c44e2819af
> value: VM
> , sharedLocks= ]
> 2015-04-06 08:34:43,165 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (org.ovirt.thread.pool-8-thread-14) [2b07d5b3] Correlation ID: 3b191496,
> Job ID: 0f8c2d21-201e-454f-9876-dce9a1ca56fd, Call Stack: null, Custom
> Event ID: -1, Message: Migration failed, No available host found (VM:
> nindigo, Source: virt2).
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users