[Users] Problem with live migration.
I can't live migrate the vm's in my cluster: 012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321, migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321, method=online 2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321, migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, method=online 2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.**MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.**vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-**e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-**ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-**9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-**e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-**ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.**vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.**vdsbroker.**VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.**vdsbroker.**VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.**vdsbroker.**VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-**e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-**3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.**vdsbroker.vdsbroker.**BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.**vdsbroker.vdsbroker.**BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.**vdsbroker.vdsbroker.**StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.**vdsbroker.vdsbroker.**StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.**vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.**vdsbroker.vdsbroker.**MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.**MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_**VDS_VM_CLUSTER,VAR__ACTION__** MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
______________________________**_________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/**mailman/listinfo/users<http://lists.ovirt.org/mailman/listinfo/users>
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
This is a multi-part message in MIME format. --------------000509020206090803080704 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com <mailto:iheim@redhat.com>> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too? Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. --------------000509020206090803080704 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 04/17/2012 03:45 PM, Martin Hovmöller wrote: <blockquote cite="mid:CAGKvcKXMhdcCdui97Y6MPCk8F2912ji8rfVEFhRKasrESycjAQ@mail.gmail.com" type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <span dir="ltr"><<a moz-do-not-send="true" href="mailto:iheim@redhat.com">iheim@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class="im">On 04/17/2012 12:47 PM, Martin Hovmöller wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class="im"> I can't live migrate the vm's in my cluster:<br> <br> 012-04-17 04:06:39,728 INFO<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)<br> Running command: MigrateVmCommand internal: false. Entities affected :<br> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM<br> 2012-04-17 04:06:39,743 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div class="im"><br> migrationMethod=ONLINE), log id: 7c7201be<br> 2012-04-17 04:06:39,748 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) VdsBroker::migrate::Entered<br> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, method=online <div class="im"><br> 2012-04-17 04:06:39,749 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div> <div class="h5"><br> migrationMethod=ONLINE), log id: 3c40c2de<br> 2012-04-17 04:06:39,834 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de<br> 2012-04-17 04:06:39,840 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be<br> 2012-04-17 04:06:42,166 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-18) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain><br> ignoring it in the refresh till migration is done<br> [...]<br> 2012-04-17 04:07:33,852 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-34) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds<br> rhevh1.<domain> ignoring it in the refresh till migration is done<br> 2012-04-17 04:07:37,407 ERROR<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-71) Rerun vm<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain><br> 2012-04-17 04:07:37,414 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b<br> 2012-04-17 04:07:37,496 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Failed in MigrateStatusVDS method<br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Error code createErr and error message<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Command<br> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return<br> value<br> Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br> mStatus Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br> mCode 9<br> mMessage Error creating the requested virtual machine<br> <br> <br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Vds: rhevh2.<domain><br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)<br> Command MigrateStatusVDS execution failed. Exception: VDSErrorException:<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b<br> 2012-04-17 04:07:37,528 WARN<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)<br> CanDoAction of action MigrateVm failed.<br> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br> <br> <br> There is no problem regarding storage or anything like that. If I shut<br> down the vm I can start it on the other host without any problems<br> whatsoever.<br> Do I need to do something to make live migration work? Trying to dig in<br> the logs on the hypervisors, but there's so much stuff being logged there...<br> <br> <br> </div> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br> </blockquote> <br> vdsm log from the host that failed to create the VM?<br> </blockquote> </div> <br> <div>The vdsm log is frighteningly long, so I put it in pastebin: <a moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeqFCCb</a></div> <div>I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.</div> </blockquote> Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}<br> <blockquote cite="mid:CAGKvcKXMhdcCdui97Y6MPCk8F2912ji8rfVEFhRKasrESycjAQ@mail.gmail.com" type="cite"> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </body> </html> --------------000509020206090803080704--
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed.
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a' What domain is this, is it the storage domain?
This is a multi-part message in MIME format. --------------050105090406090809010800 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com <mailto:iheim@redhat.com>> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
What domain is this, is it the storage domain?
Domain in libvirt is simply a vm.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. --------------050105090406090809010800 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 04/17/2012 05:36 PM, Martin Hovmöller wrote: <blockquote cite="mid:CAGKvcKUS=mWsNRVq=EqxXiQWXgxqSftz0Yr9f_OKuim5Yoe-UQ@mail.gmail.com" type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div class="h5"> On 04/17/2012 03:45 PM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <span dir="ltr"><<a moz-do-not-send="true" href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div>On 04/17/2012 12:47 PM, Martin Hovmöller wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> I can't live migrate the vm's in my cluster:<br> <br> 012-04-17 04:06:39,728 INFO<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)<br> Running command: MigrateVmCommand internal: false. Entities affected :<br> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM<br> 2012-04-17 04:06:39,743 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div><br> migrationMethod=ONLINE), log id: 7c7201be<br> 2012-04-17 04:06:39,748 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) VdsBroker::migrate::Entered<br> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, method=online <div><br> 2012-04-17 04:06:39,749 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div> <div><br> migrationMethod=ONLINE), log id: 3c40c2de<br> 2012-04-17 04:06:39,834 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de<br> 2012-04-17 04:06:39,840 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be<br> 2012-04-17 04:06:42,166 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-18) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain><br> ignoring it in the refresh till migration is done<br> [...]<br> 2012-04-17 04:07:33,852 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-34) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds<br> rhevh1.<domain> ignoring it in the refresh till migration is done<br> 2012-04-17 04:07:37,407 ERROR<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-71) Rerun vm<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain><br> 2012-04-17 04:07:37,414 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b<br> 2012-04-17 04:07:37,496 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Failed in MigrateStatusVDS method<br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Error code createErr and error message<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Command<br> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return<br> value<br> Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br> mStatus Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br> mCode 9<br> mMessage Error creating the requested virtual machine<br> <br> <br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Vds: rhevh2.<domain><br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)<br> Command MigrateStatusVDS execution failed. Exception: VDSErrorException:<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b<br> 2012-04-17 04:07:37,528 WARN<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)<br> CanDoAction of action MigrateVm failed.<br> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br> <br> <br> There is no problem regarding storage or anything like that. If I shut<br> down the vm I can start it on the other host without any problems<br> whatsoever.<br> Do I need to do something to make live migration work? Trying to dig in<br> the logs on the hypervisors, but there's so much stuff being logged there...<br> <br> <br> </div> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br> </blockquote> <br> vdsm log from the host that failed to create the VM?<br> </blockquote> </div> <br> <div>The vdsm log is frighteningly long, so I put it in pastebin: <a moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb" target="_blank">http://pastebin.com/BEeqFCCb</a></div> <div>I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.</div> </blockquote> </div> </div> Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}} <div class="im"><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> </div> <span class="HOEnZb"><font color="#888888"> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </font></span></div> </blockquote> </div> <div><br> </div> <div>I think this is the relevant info from the libvirt.log:</div> <div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div> </div> <div><br> </div> <div>What domain is this, is it the storage domain?</div> </blockquote> <br> Domain in libvirt is simply a vm.<br> <blockquote cite="mid:CAGKvcKUS=mWsNRVq=EqxXiQWXgxqSftz0Yr9f_OKuim5Yoe-UQ@mail.gmail.com" type="cite"> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </body> </html> --------------050105090406090809010800--
On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed.
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
What domain is this, is it the storage domain?
Domain in libvirt is simply a vm.
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
Alright, so what does this mean then? The machine I'm migrating to can't see the vm? That doesn't make sense since I can manually start it on that host..
This is a multi-part message in MIME format. --------------020800060704090308040504 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com <mailto:iheim@redhat.com>> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
What domain is this, is it the storage domain?
Domain in libvirt is simply a vm.
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
Alright, so what does this mean then? The machine I'm migrating to can't see the vm? That doesn't make sense since I can manually start it on that host..
On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination. In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration. Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',..... Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. --------------020800060704090308040504 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 04/18/2012 11:30 AM, Martin Hovmöller wrote: <blockquote cite="mid:CAGKvcKXco_=isKREAGyq+CTV_-=xVUruBHfBHDmkREJLc-W+7w@mail.gmail.com" type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div class="h5"> On 04/17/2012 05:36 PM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div> On 04/17/2012 03:45 PM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <span dir="ltr"><<a moz-do-not-send="true" href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div>On 04/17/2012 12:47 PM, Martin Hovmöller wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> I can't live migrate the vm's in my cluster:<br> <br> 012-04-17 04:06:39,728 INFO<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)<br> Running command: MigrateVmCommand internal: false. Entities affected :<br> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM<br> 2012-04-17 04:06:39,743 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div><br> migrationMethod=ONLINE), log id: 7c7201be<br> 2012-04-17 04:06:39,748 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) VdsBroker::migrate::Entered<br> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, method=online <div><br> 2012-04-17 04:06:39,749 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div> <div><br> migrationMethod=ONLINE), log id: 3c40c2de<br> 2012-04-17 04:06:39,834 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de<br> 2012-04-17 04:06:39,840 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be<br> 2012-04-17 04:06:42,166 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-18) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain><br> ignoring it in the refresh till migration is done<br> [...]<br> 2012-04-17 04:07:33,852 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-34) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds<br> rhevh1.<domain> ignoring it in the refresh till migration is done<br> 2012-04-17 04:07:37,407 ERROR<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-71) Rerun vm<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain><br> 2012-04-17 04:07:37,414 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b<br> 2012-04-17 04:07:37,496 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Failed in MigrateStatusVDS method<br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Error code createErr and error message<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Command<br> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return<br> value<br> Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br> mStatus Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br> mCode 9<br> mMessage Error creating the requested virtual machine<br> <br> <br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Vds: rhevh2.<domain><br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)<br> Command MigrateStatusVDS execution failed. Exception: VDSErrorException:<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b<br> 2012-04-17 04:07:37,528 WARN<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)<br> CanDoAction of action MigrateVm failed.<br> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br> <br> <br> There is no problem regarding storage or anything like that. If I shut<br> down the vm I can start it on the other host without any problems<br> whatsoever.<br> Do I need to do something to make live migration work? Trying to dig in<br> the logs on the hypervisors, but there's so much stuff being logged there...<br> <br> <br> </div> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br> </blockquote> <br> vdsm log from the host that failed to create the VM?<br> </blockquote> </div> <br> <div>The vdsm log is frighteningly long, so I put it in pastebin: <a moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb" target="_blank">http://pastebin.com/BEeqFCCb</a></div> <div>I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.</div> </blockquote> </div> </div> Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}} <div><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> </div> <span><font color="#888888"> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </font></span></div> </blockquote> </div> <div><br> </div> <div>I think this is the relevant info from the libvirt.log:</div> <div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div> </div> <div><br> </div> <div>What domain is this, is it the storage domain?</div> </blockquote> <br> </div> </div> Domain in libvirt is simply a vm. <div class="im"><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </div> </div> </blockquote> </div> <br> <div>Alright, so what does this mean then? The machine I'm migrating to can't see the vm?<br> That doesn't make sense since I can manually start it on that host..</div> </blockquote> <br> On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination.<br> In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration.<br> <br> <br> Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br> Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',.....<br> Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation<br> <br> 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br> <blockquote cite="mid:CAGKvcKXco_=isKREAGyq+CTV_-=xVUruBHfBHDmkREJLc-W+7w@mail.gmail.com" type="cite"> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </body> </html> --------------020800060704090308040504--
On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed.
Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
What domain is this, is it the storage domain?
Domain in libvirt is simply a vm.
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
Alright, so what does this mean then? The machine I'm migrating to can't see the vm? That doesn't make sense since I can manually start it on that host..
On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination. In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration.
Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',..... Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation
Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}
_______________________________________________ Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
The vdsm log is already included earlier in the conversation; http://pastebin.com/BEeqFCCb I don't really know where to go from here. :/
This is a multi-part message in MIME format. --------------090506080506000809040406 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit On 04/19/2012 05:17 PM, Martin Hovmöller wrote:
On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/18/2012 11:30 AM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/17/2012 05:36 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <rvaknin@redhat.com <mailto:rvaknin@redhat.com>> wrote:
On 04/17/2012 03:45 PM, Martin Hovmöller wrote:
On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <iheim@redhat.com <mailto:iheim@redhat.com>> wrote:
On 04/17/2012 12:47 PM, Martin Hovmöller wrote:
I can't live migrate the vm's in my cluster:
012-04-17 04:06:39,728 INFO [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48) Running command: MigrateVmCommand internal: false. Entities affected : ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM 2012-04-17 04:06:39,743 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 7c7201be 2012-04-17 04:06:39,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) VdsBroker::migrate::Entered (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>, method=online
2012-04-17 04:06:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130, dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0, dstHost=10.23.30.110:54321 <http://10.23.30.110:54321> <http://10.23.30.110:54321>,
migrationMethod=ONLINE), log id: 3c40c2de 2012-04-17 04:06:39,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand] (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de 2012-04-17 04:06:39,840 INFO [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48) FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be 2012-04-17 04:06:42,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-18) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain> ignoring it in the refresh till migration is done [...] 2012-04-17 04:07:33,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-34) vds::refreshVmList vm id ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds rhevh1.<domain> ignoring it in the refresh till migration is done 2012-04-17 04:07:37,407 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (QuartzScheduler_Worker-71) Rerun vm ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain> 2012-04-17 04:07:37,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008, vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b 2012-04-17 04:07:37,496 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Failed in MigrateStatusVDS method 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Error code createErr and error message VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Command org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return value Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc mStatus Class Name: org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc mCode 9 mMessage Error creating the requested virtual machine
2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase] (pool-5-thread-47) Vds: rhevh2.<domain> 2012-04-17 04:07:37,497 ERROR [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47) Command MigrateStatusVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS, error = Error creating the requested virtual machine 2012-04-17 04:07:37,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand] (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b 2012-04-17 04:07:37,528 WARN [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47) CanDoAction of action MigrateVm failed. Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM
There is no problem regarding storage or anything like that. If I shut down the vm I can start it on the other host without any problems whatsoever. Do I need to do something to make live migration work? Trying to dig in the logs on the hypervisors, but there's so much stuff being logged there...
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
vdsm log from the host that failed to create the VM?
The vdsm log is frighteningly long, so I put it in pastebin: http://pastebin.com/BEeqFCCb I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.
Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?
Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}}
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
I think this is the relevant info from the libvirt.log: 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a 2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'
What domain is this, is it the storage domain?
Domain in libvirt is simply a vm.
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
Alright, so what does this mean then? The machine I'm migrating to can't see the vm? That doesn't make sense since I can manually start it on that host..
On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination. In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration.
Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',..... Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation
Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation
14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {} Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}
_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
--
Thanks,
Rami Vaknin, QE @ Red Hat, TLV, IL.
The vdsm log is already included earlier in the conversation; http://pastebin.com/BEeqFCCb I don't really know where to go from here. :/
The exact issue that caused the prepareVolume to fail is not mentioned in the attached log. Can you please tell whether the attached vdsm log is from the migration source host or from the migration destination host, and attach as well the second missing log? The vdsm log from the host where the vm should run after the migration should contain a failed/hanged lvm command on the vm's disk.
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. --------------090506080506000809040406 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> On 04/19/2012 05:17 PM, Martin Hovmöller wrote: <blockquote cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com" type="cite"><br> <br> <div class="gmail_quote">On Wed, Apr 18, 2012 at 11:06 AM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div class="h5"> On 04/18/2012 11:30 AM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 4:38 PM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div> On 04/17/2012 05:36 PM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:55 PM, Rami Vaknin <span dir="ltr"><<a moz-do-not-send="true" href="mailto:rvaknin@redhat.com" target="_blank">rvaknin@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div bgcolor="#FFFFFF" text="#000000"> <div> <div> On 04/17/2012 03:45 PM, Martin Hovmöller wrote: <blockquote type="cite"><br> <br> <div class="gmail_quote">On Tue, Apr 17, 2012 at 2:11 PM, Itamar Heim <span dir="ltr"><<a moz-do-not-send="true" href="mailto:iheim@redhat.com" target="_blank">iheim@redhat.com</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div>On 04/17/2012 12:47 PM, Martin Hovmöller wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> I can't live migrate the vm's in my cluster:<br> <br> 012-04-17 04:06:39,728 INFO<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-48)<br> Running command: MigrateVmCommand internal: false. Entities affected :<br> ID: ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a Type: VM<br> 2012-04-17 04:06:39,743 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> START, MigrateVDSCommand(vdsId = f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div><br> migrationMethod=ONLINE), log id: 7c7201be<br> 2012-04-17 04:06:39,748 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) VdsBroker::migrate::Entered<br> (vm_guid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, method=online <div><br> 2012-04-17 04:06:39,749 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) START, MigrateBrokerVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a, srcHost=10.23.30.130,<br> dstVdsId=b4d329b4-87b6-11e1-ac0a-b70ec8cc50f0,<br> </div> dstHost=<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">10.23.30.110:54321</a> <<a moz-do-not-send="true" href="http://10.23.30.110:54321" target="_blank">http://10.23.30.110:54321</a>>;, <div> <div><br> migrationMethod=ONLINE), log id: 3c40c2de<br> 2012-04-17 04:06:39,834 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateBrokerVDSCommand]<br> (pool-5-thread-48) FINISH, MigrateBrokerVDSCommand, log id: 3c40c2de<br> 2012-04-17 04:06:39,840 INFO<br> [org.ovirt.engine.core.vdsbroker.MigrateVDSCommand] (pool-5-thread-48)<br> FINISH, MigrateVDSCommand, return: MigratingFrom, log id: 7c7201be<br> 2012-04-17 04:06:42,166 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-18) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a is migrating to vds rhevh1.<domain><br> ignoring it in the refresh till migration is done<br> [...]<br> 2012-04-17 04:07:33,852 INFO<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-34) vds::refreshVmList vm id<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a status = Paused on vds<br> rhevh1.<domain> ignoring it in the refresh till migration is done<br> 2012-04-17 04:07:37,407 ERROR<br> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo]<br> (QuartzScheduler_Worker-71) Rerun vm<br> ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a. Called from vds rhevh2.<domain><br> 2012-04-17 04:07:37,414 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) START, MigrateStatusVDSCommand(vdsId =<br> f33ffc14-87ba-11e1-b610-e3aeca1e8008,<br> vmId=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a), log id: 8dfce1b<br> 2012-04-17 04:07:37,496 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Failed in MigrateStatusVDS method<br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Error code createErr and error message<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Command<br> org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand return<br> value<br> Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusOnlyReturnForXmlRpc<br> mStatus Class Name:<br> org.ovirt.engine.core.vdsbroker.vdsbroker.StatusForXmlRpc<br> mCode 9<br> mMessage Error creating the requested virtual machine<br> <br> <br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase]<br> (pool-5-thread-47) Vds: rhevh2.<domain><br> 2012-04-17 04:07:37,497 ERROR<br> [org.ovirt.engine.core.vdsbroker.VDSCommandBase] (pool-5-thread-47)<br> Command MigrateStatusVDS execution failed. Exception: VDSErrorException:<br> VDSGenericException: VDSErrorException: Failed to MigrateStatusVDS,<br> error = Error creating the requested virtual machine<br> 2012-04-17 04:07:37,497 INFO<br> [org.ovirt.engine.core.vdsbroker.vdsbroker.MigrateStatusVDSCommand]<br> (pool-5-thread-47) FINISH, MigrateStatusVDSCommand, log id: 8dfce1b<br> 2012-04-17 04:07:37,528 WARN<br> [org.ovirt.engine.core.bll.MigrateVmCommand] (pool-5-thread-47)<br> CanDoAction of action MigrateVm failed.<br> Reasons:ACTION_TYPE_FAILED_VDS_VM_CLUSTER,VAR__ACTION__MIGRATE,VAR__TYPE__VM<br> <br> <br> There is no problem regarding storage or anything like that. If I shut<br> down the vm I can start it on the other host without any problems<br> whatsoever.<br> Do I need to do something to make live migration work? Trying to dig in<br> the logs on the hypervisors, but there's so much stuff being logged there...<br> <br> <br> </div> </div> _______________________________________________<br> Users mailing list<br> <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br> </blockquote> <br> vdsm log from the host that failed to create the VM?<br> </blockquote> </div> <br> <div>The vdsm log is frighteningly long, so I put it in pastebin: <a moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb" target="_blank">http://pastebin.com/BEeqFCCb</a></div> <div>I don't really see anything worthwhile myself, but on the other hand I'm not sure what I'm looking for, so it's mostly a massive wall of text to me, but it should be somewhere in that time frame.</div> </blockquote> </div> </div> Here is the failure, "code 9" is not too informative, can you attach the libvirtd.log file too?<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,834::clientIF::81::vds::(wrapper) return migrationCreate with {'status': {'message': 'Error creating the requested virtual machine', 'code': 9}} <div><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> </div> <span><font color="#888888"> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </font></span></div> </blockquote> </div> <div><br> </div> <div>I think this is the relevant info from the libvirt.log:</div> <div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUIDString:1984 : conn=0x7fa9a00bcde0, uuidstr=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.768+0000: 8506: debug : virDomainLookupByUUID:1941 : conn=0x7fa9a00bcde0, uuid=ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a</div> <div>2012-04-17 14:30:53.771+0000: 8506: error : virNetClientProgramDispatchError:170 : Domain not found: no domain with matching uuid 'ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a'</div> </div> <div><br> </div> <div>What domain is this, is it the storage domain?</div> </blockquote> <br> </div> </div> Domain in libvirt is simply a vm. <div><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </div> </div> </blockquote> </div> <br> <div>Alright, so what does this mean then? The machine I'm migrating to can't see the vm?<br> That doesn't make sense since I can manually start it on that host..</div> </blockquote> <br> </div> </div> On second look at the vdsm.log I see that disk preparation in the destination host failed (timed out), try to take a look at vdsm.log at the destination.<br> In addition, for some reason getVmStats returns "Virtual machine does not exist" right before the migration.<br> <br> <br> Thread-49286::DEBUG::2012-04-17 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}}<br> Thread-49287::DEBUG::2012-04-17 14:25:07,381::clientIF::76::vds::(wrapper) [10.23.30.130]::call migrationCreate with ({'bridge': 'ovirtmgmt', 'acpiEnable': 'true',.....<br> Thread-49287::DEBUG::2012-04-17 14:25:08,156::libvirtvm::1504::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::migration destination: waiting 31s for path preparation<br> <br> Thread-49287::DEBUG::2012-04-17 14:25:39,458::libvirtvm::1507::vm.Vm::(waitForMigrationDestinationPrepare) vmId=`ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a`::Timeout while waiting for path preparation<br> <br> 14:25:07,054::clientIF::76::vds::(wrapper) [10.23.30.130]::call getVmStats with ('ce9bb531-4d2a-4f5e-9935-3c8a4f5db94a',) {}<br> Thread-49286::DEBUG::2012-04-17 14:25:07,206::clientIF::81::vds::(wrapper) return getVmStats with {'status': {'message': 'Virtual machine does not exist', 'code': 1}} <div class="im"><br> <blockquote type="cite"> <br> <fieldset></fieldset> <br> <pre>_______________________________________________ Users mailing list <a moz-do-not-send="true" href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a> <a moz-do-not-send="true" href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </div> </div> </blockquote> </div> The vdsm log is already included earlier in the conversation; <a moz-do-not-send="true" href="http://pastebin.com/BEeqFCCb">http://pastebin.com/BEeqFCCb</a> <div>I don't really know where to go from here. :/</div> </blockquote> <br> The exact issue that caused the prepareVolume to fail is not mentioned in the attached log.<br> <br> Can you please tell whether the attached vdsm log is from the migration source host or from the migration destination host, and attach as well the second missing log? The vdsm log from the host where the vm should run after the migration should contain a failed/hanged lvm command on the vm's disk.<br> <br> <br> <br> <blockquote cite="mid:CAGKvcKUOXkNPKM1qGe2seVKWp2Z98JO5ehoDYs8vcgLF5p16Ww@mail.gmail.com" type="cite"> <br> <fieldset class="mimeAttachmentHeader"></fieldset> <br> <pre wrap="">_______________________________________________ Users mailing list <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> </pre> </blockquote> <br> <br> <pre class="moz-signature" cols="72">-- Thanks, Rami Vaknin, QE @ Red Hat, TLV, IL. </pre> </body> </html> --------------090506080506000809040406--
participants (3)
-
Itamar Heim -
Martin Hovmöller -
Rami Vaknin