Re: [ovirt-users] VMs stuck in migrating state

El 2018-03-02 14:10, Milan Zamazal escribió: > nicolas(a)devels.es writes: > >> We're running 4.1.9 and during the weekend we had a storage issue that >> seemed >> to leave some hosts in an strange state. One of the hosts has almost all VMs >> migrating (although it seems to not actually being migrating them) and the >> migration state cannot be cancelled. >> >> When clicking on one of those machines and selecting 'Cancel migration', in >> the >> ovirt-engine log I see: >> >> 2018-02-26 08:52:07,588Z INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CancelMigrateVDSCommand] >> (org.ovirt.thread.pool-6-thread-36) [887dfbf9-dece-4f7b-90a8-dac02b849b7f] >> HostName = host2.domain.com >> 2018-02-26 08:52:07,588Z ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CancelMigrateVDSCommand] >> (org.ovirt.thread.pool-6-thread-36) [887dfbf9-dece-4f7b-90a8-dac02b849b7f] >> Command 'CancelMigrateVDSCommand(HostName = host2.domain.com, >> CancelMigrationVDSParameters:{runAsync='true', >> hostId='e63b9146-10c4-47ad-bd6c-f053a8c5b4eb', >> vmId='26d37e43-32e2-4e55-9c1e-1438518d5021'})' execution failed: >> VDSGenericException: VDSErrorException: Failed to CancelMigrateVDS, error = >> Migration process cancelled, code = 82 >> >> On the vdsm side I see: >> >> 2018-02-26 08:56:19,396+0000 INFO (jsonrpc/0) [vdsm.api] START >> migrateCancel() >> from=::ffff:10.X.X.X,54654, flow_id=874d36d7-63f5-4b71-8a4d-6d9f3ec65858 >> (api:46) >> 2018-02-26 08:56:19,398+0000 INFO (jsonrpc/0) [vdsm.api] FINISH >> migrateCancel >> return={'status': {'message': 'Migration process cancelled', 'code': 82}, >> 'progress': 0} from=::ffff:10.X.X.X,54654, >> flow_id=874d36d7-63f5-4b71-8a4d-6d9f3ec65858 (api:52) >> >> So no error on the vdsm side log. > > Interesting. The messages above indicate that the VM was attempted to > migrate, but the migration got temporarily rejected on the destination > due to the number of already running incoming migrations (the limit is 2 > incoming migrations by default). Later, Vdsm was asked to cancel the > outgoing migration and it successfully set a migration canceling flag. > However the action was reported as an error to Engine, due to hitting > the incoming migration limit on the destination. Maybe it's a bug, I'm > not sure, resulting in minor confusion. Normally it shouldn't matter, > the migration should be canceled shortly after anyway and Engine should > be informed about that. > > However the migration apparently wasn't canceled here. I can't say what > happened without complete Vdsm log. One of possible reasons is that the > migration has been waiting on completion of another migration outgoing > from the source (only one outgoing migration at the time is allowed by > default). In any case it seems the migration either wasn't actually > started at all or it just started being set up and that has never been > completely finished. > I'm attaching the log. Basically the storage backend was restarted by fencing and then this issue happened. This was on 26/02 at about 08:52 (log time).

>> I already tried restarting ovirt-engine but it didn't work. > > Here the problem is clearly on the Vdsm side. > >> Could someone shed some light on how to cancel the migration status for >> these >> machines? All of them seem to be running on the same host. > > Did the VMs get unblocked in the meantime? I can't know what's the No, they didn't. They're still in a "Migrating" state. > actual state of the given VMs without seeing the complete Vdsm log, so > it's difficult to give a good advice. I think that Vdsm restart on the > given host would help BUT it's generally not a very good idea to restart > Vdsm if any real migration, outgoing or incoming, is running on the > host. VMs that aren't actually being migrated (despite being reported > as migrating) at all should simply return to Up state after the restart, > but VMs with any real migration action pending might get return to Up > state without proper cleanup, resulting in a different kind of mess or > maybe something even worse (things should improve in oVirt 4.2, but it's > still good to avoid Vdsm restarts with migrations running). > I assume this is not a real migration as it has been in this state for several days. Would you advice restarting vdsm in this case then?