[ovirt-users] VMs stuck in migrating state

[Milan Zamazal]

nicolas at devels.es writes:

> We're running 4.1.9 and during the weekend we had a storage issue that seemed
> to leave some hosts in an strange state. One of the hosts has almost all VMs
> migrating (although it seems to not actually being migrating them) and the
> migration state cannot be cancelled.
>
> When clicking on one of those machines and selecting 'Cancel migration', in the
> ovirt-engine log I see:
>
> 2018-02-26 08:52:07,588Z INFO
> [org.ovirt.engine.core.vdsbroker.vdsbroker.CancelMigrateVDSCommand]
> (org.ovirt.thread.pool-6-thread-36) [887dfbf9-dece-4f7b-90a8-dac02b849b7f]
> HostName = host2.domain.com
> 2018-02-26 08:52:07,588Z ERROR
> [org.ovirt.engine.core.vdsbroker.vdsbroker.CancelMigrateVDSCommand]
> (org.ovirt.thread.pool-6-thread-36) [887dfbf9-dece-4f7b-90a8-dac02b849b7f]
> Command 'CancelMigrateVDSCommand(HostName = host2.domain.com,
> CancelMigrationVDSParameters:{runAsync='true',
> hostId='e63b9146-10c4-47ad-bd6c-f053a8c5b4eb',
> vmId='26d37e43-32e2-4e55-9c1e-1438518d5021'})' execution failed:
> VDSGenericException: VDSErrorException: Failed to CancelMigrateVDS, error =
> Migration process cancelled, code = 82
>
> On the vdsm side I see:
>
> 2018-02-26 08:56:19,396+0000 INFO  (jsonrpc/0) [vdsm.api] START migrateCancel()
> from=::ffff:10.X.X.X,54654, flow_id=874d36d7-63f5-4b71-8a4d-6d9f3ec65858
> (api:46)
> 2018-02-26 08:56:19,398+0000 INFO  (jsonrpc/0) [vdsm.api] FINISH migrateCancel
> return={'status': {'message': 'Migration process cancelled', 'code': 82},
> 'progress': 0} from=::ffff:10.X.X.X,54654,
> flow_id=874d36d7-63f5-4b71-8a4d-6d9f3ec65858 (api:52)
>
> So no error on the vdsm side log.

Interesting.  The messages above indicate that the VM was attempted to
migrate, but the migration got temporarily rejected on the destination
due to the number of already running incoming migrations (the limit is 2
incoming migrations by default).  Later, Vdsm was asked to cancel the
outgoing migration and it successfully set a migration canceling flag.
However the action was reported as an error to Engine, due to hitting
the incoming migration limit on the destination.  Maybe it's a bug, I'm
not sure, resulting in minor confusion.  Normally it shouldn't matter,
the migration should be canceled shortly after anyway and Engine should
be informed about that.

However the migration apparently wasn't canceled here.  I can't say what
happened without complete Vdsm log.  One of possible reasons is that the
migration has been waiting on completion of another migration outgoing
from the source (only one outgoing migration at the time is allowed by
default).  In any case it seems the migration either wasn't actually
started at all or it just started being set up and that has never been
completely finished.

> I already tried restarting ovirt-engine but it didn't work.

Here the problem is clearly on the Vdsm side.

> Could someone shed some light on how to cancel the migration status for these
> machines? All of them seem to be running on the same host.

Did the VMs get unblocked in the meantime?  I can't know what's the
actual state of the given VMs without seeing the complete Vdsm log, so
it's difficult to give a good advice.  I think that Vdsm restart on the
given host would help BUT it's generally not a very good idea to restart
Vdsm if any real migration, outgoing or incoming, is running on the
host.  VMs that aren't actually being migrated (despite being reported
as migrating) at all should simply return to Up state after the restart,
but VMs with any real migration action pending might get return to Up
state without proper cleanup, resulting in a different kind of mess or
maybe something even worse (things should improve in oVirt 4.2, but it's
still good to avoid Vdsm restarts with migrations running).

Regards,
Milan