[ovirt-users] Cannot start VM after live storage migration - Bad volume specification

Sun Mar 18 21:15:07 UTC 2018

Hi all,

  we did a live storage migration of one of three disks of a vm that failed because the vm became not responding when deleting the auto-snapshot:

2018-03-16 15:07:32.084+01 |        0 | Snapshot 'VMNAME_Disk2 Auto-generated for Live Storage Migration' creation for VM 'VMNAME' was initiated by xxx
2018-03-16 15:07:32.097+01 |        0 | User xxx moving disk VMNAME_Disk2 to domain VMHOST_LUN_211.
2018-03-16 15:08:56.304+01 |        0 | Snapshot 'VMNAME_Disk2 Auto-generated for Live Storage Migration' creation for VM 'VMNAME' has been completed.
2018-03-16 16:40:48.89+01  |        0 | Snapshot 'VMNAME_Disk2 Auto-generated for Live Storage Migration' deletion for VM 'VMNAME' was initiated by xxx.
2018-03-16 16:44:44.813+01 |        1 | VM VMNAME is not responding.
2018-03-18 18:40:51.258+01 |        2 | Failed to delete snapshot 'VMNAME_Disk2 Auto-generated for Live Storage Migration' for VM 'VMNAME'.
2018-03-18 18:40:54.506+01 |        1 | Possible failure while deleting VMNAME_Disk2 from the source Storage Domain VMHOST_LUN_211 during the move operation. The Storage Domain may be manually cleaned-up from possible leftover
s (User:xxx).

Now we cannot start the vm anymore as long as this disk is online. Error message is "VM VMNAME is down with error. Exit message: Bad volume specification {'index': 2, 'domainID': 'ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a', 'reqsize': '0', 'name': 'vdc', 'truesize': '2147483648', 'format': 'cow', 'discard': False, 'volumeID': '4659b5e0-93c1-478d-97d0-ec1cf4052028', 'apparentsize': '2147483648', 'imageID': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'specParams': {}, 'iface': 'virtio', 'cache': 'none', 'propagateErrors': 'off', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a/images/c1a05108-90d7-421d-a9b4-d4cc65c48429/4659b5e0-93c1-478d-97d0-ec1cf4052028', 'serial': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'diskType': 'block', 'type': 'block'}."

vdsm.log:
2018-03-18 21:53:33,815+0100 ERROR (vm/7d05e511) [storage.TaskManager.Task] (Task='fc3bac16-64f3-4910-8bc4-6cfdd4d270da') Unexpected error (task:875)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
    return fn(*args, **kargs)
  File "<string>", line 2, in prepareImage
  File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 3179, in prepareImage
    raise se.prepareIllegalVolumeError(volUUID)
prepareIllegalVolumeError: Cannot prepare illegal volume: ('4c6475b1-352a-4114-b647-505cccbe6663',)
2018-03-18 21:53:33,816+0100 INFO  (vm/7d05e511) [storage.TaskManager.Task] (Task='fc3bac16-64f3-4910-8bc4-6cfdd4d270da') aborting: Task is aborted: "Cannot prepare illegal volume: ('4c6475b1-352a-4114-b647-505cccbe6663',)" - code 227 (task:1181)
2018-03-18 21:53:33,816+0100 ERROR (vm/7d05e511) [storage.Dispatcher] FINISH prepareImage error=Cannot prepare illegal volume: ('4c6475b1-352a-4114-b647-505cccbe6663',) (dispatcher:82)
2018-03-18 21:53:33,816+0100 ERROR (vm/7d05e511) [virt.vm] (vmId='7d05e511-2e97-4002-bded-285ec4e30587') The vm start process failed (vm:927)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 856, in _startUnderlyingVm
    self._run()
 File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2661, in _run
    self._devices = self._make_devices()
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2608, in _make_devices
    self._preparePathsForDrives(disk_params)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 1001, in _preparePathsForDrives
    drive['path'] = self.cif.prepareVolumePath(drive, self.id)
  File "/usr/lib/python2.7/site-packages/vdsm/clientIF.py", line 393, in prepareVolumePath
    raise vm.VolumeError(drive)
VolumeError: Bad volume specification {'index': 2, 'domainID': 'ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a', 'reqsize': '0', 'name': 'vdc', 'truesize': '2147483648', 'format': 'cow', 'discard': False, 'volumeID': '4659b5e0-93c1-478d-97d0-ec1cf4052028', 'apparentsize': '2147483648', 'imageID': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'specParams': {}, 'iface': 'virtio', 'cache': 'none', 'propagateErrors': 'off', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a/images/c1a05108-90d7-421d-a9b4-d4cc65c48429/4659b5e0-93c1-478d-97d0-ec1cf4052028', 'serial': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'diskType': 'block', 'type': 'block'}
2018-03-18 21:53:33,817+0100 INFO  (vm/7d05e511) [virt.vm] (vmId='7d05e511-2e97-4002-bded-285ec4e30587') Changed state to Down: Bad volume specification {'index': 2, 'domainID': 'ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a', 'reqsize': '0', 'name': 'vdc', 'truesize': '2147483648', 'format': 'cow', 'discard': False, 'volumeID': '4659b5e0-93c1-478d-97d0-ec1cf4052028', 'apparentsize': '2147483648', 'imageID': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'specParams': {}, 'iface': 'virtio', 'cache': 'none', 'propagateErrors': 'off', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/ecc71a64-62c1-43f4-bf1f-3bc1b22c7a8a/images/c1a05108-90d7-421d-a9b4-d4cc65c48429/4659b5e0-93c1-478d-97d0-ec1cf4052028', 'serial': 'c1a05108-90d7-421d-a9b4-d4cc65c48429', 'diskType': 'block', 'type': 'block'} (code=1) (vm:1646)

Is there a way to recover this disk?

Thank you,
Simone

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180318/a336b04b/attachment.html>