[Users] Live storage migration fails on CentOS 6.4 + ovirt3.3 cluster

Sander Grendelman sander at grendelman.com
Mon Nov 4 14:32:29 UTC 2013 

Moving the storage of a (running) VM to a different (FC) storage domain fails.

Steps to reproduce:
1) Create new VM
2) Start VM
3) Start move of the VM to a different storage domain

When I look at the logs it seems that vdsm/libvirt tries to use an
option that is unsupported by libvirt or the qemu-kvm version on
CentOS 6.4:

"libvirtError: unsupported configuration: reuse is not supported with
this QEMU binary"

Information in the "Events" section of the oVirt engine manager:

2013-Nov-04, 14:45 VM migratest powered off by grendelmans (Host: gnkvm01).
2013-Nov-04, 14:05 User grendelmans moving disk migratest_Disk1 to
domain gneva03_vmdisk02.
2013-Nov-04, 14:04 Snapshot 'Auto-generated for Live Storage
Migration' creation for VM 'migratest' has been completed.
2013-Nov-04, 14:04 Failed to create live snapshot 'Auto-generated for
Live Storage Migration' for VM 'migratest'. VM restart is recommended.
2013-Nov-04, 14:04 Snapshot 'Auto-generated for Live Storage
Migration' creation for VM 'migratest' was initiated by grendelmans.
2013-Nov-04, 14:04 VM migratest started on Host gnkvm01
2013-Nov-04, 14:03 VM migratest was started by grendelmans (Host: gnkvm01).

Information from the vdsm log:

Thread-100903::DEBUG::2013-11-04
14:04:56,548::lvm::311::Storage.Misc.excCmd::(cmd) SUCCESS: <err> =
''; <rc> = 0
Thread-100903::DEBUG::2013-11-04
14:04:56,615::lvm::448::OperationMutex::(_reloadlvs) Operation 'lvm
reload operation' released the operation mutex
Thread-100903::DEBUG::2013-11-04
14:04:56,622::blockVolume::588::Storage.Misc.excCmd::(getMetadata)
'/bin/dd iflag=direct skip=38 bs=512
if=/dev/dfbbc8dd-bfae-44e1-8876-2bb82921565a/metadata count=1' (cwd
None)
Thread-100903::DEBUG::2013-11-04
14:04:56,642::blockVolume::588::Storage.Misc.excCmd::(getMetadata)
SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B)
copied, 0.000208694 s, 2.5 MB/s\n'; <rc> = 0
Thread-100903::DEBUG::2013-11-04
14:04:56,643::misc::288::Storage.Misc::(validateDDBytes) err: ['1+0
records in', '1+0 records out', '512 bytes (512 B) copied, 0.000208694
s, 2.5 MB/s'], size: 512
Thread-100903::INFO::2013-11-04
14:04:56,644::logUtils::47::dispatcher::(wrapper) Run and protect:
prepareImage, Return response: {'info': {'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'volType': 'path'}, 'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'chain': [{'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/7af63c13-c44b-4418-a1d4-e0e092ee7f04',
'domainID': 'dfbbc8dd-bfae-44e1-8876-2bb82921565a', 'vmVolInfo':
{'path': '/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/7af63c13-c44b-4418-a1d4-e0e092ee7f04',
'volType': 'path'}, 'volumeID':
'7af63c13-c44b-4418-a1d4-e0e092ee7f04', 'imageID':
'57ff3040-0cbd-4659-bd21-f07036d84dd8'}, {'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'domainID': 'dfbbc8dd-bfae-44e1-8876-2bb82921565a', 'vmVolInfo':
{'path': '/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'volType': 'path'}, 'volumeID':
'4d05730d-433c-40d9-8600-6fb0eb5af821', 'imageID':
'57ff3040-0cbd-4659-bd21-f07036d84dd8'}]}
Thread-100903::DEBUG::2013-11-04
14:04:56,644::task::1168::TaskManager.Task::(prepare)
Task=`0f953aa3-e2b9-4008-84ad-f271136d8d23`::finished: {'info':
{'path': '/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'volType': 'path'}, 'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'chain': [{'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/7af63c13-c44b-4418-a1d4-e0e092ee7f04',
'domainID': 'dfbbc8dd-bfae-44e1-8876-2bb82921565a', 'vmVolInfo':
{'path': '/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/7af63c13-c44b-4418-a1d4-e0e092ee7f04',
'volType': 'path'}, 'volumeID':
'7af63c13-c44b-4418-a1d4-e0e092ee7f04', 'imageID':
'57ff3040-0cbd-4659-bd21-f07036d84dd8'}, {'path':
'/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'domainID': 'dfbbc8dd-bfae-44e1-8876-2bb82921565a', 'vmVolInfo':
{'path': '/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821',
'volType': 'path'}, 'volumeID':
'4d05730d-433c-40d9-8600-6fb0eb5af821', 'imageID':
'57ff3040-0cbd-4659-bd21-f07036d84dd8'}]}
Thread-100903::DEBUG::2013-11-04
14:04:56,644::task::579::TaskManager.Task::(_updateState)
Task=`0f953aa3-e2b9-4008-84ad-f271136d8d23`::moving from state
preparing -> state finished
Thread-100903::DEBUG::2013-11-04
14:04:56,645::resourceManager::939::ResourceManager.Owner::(releaseAll)
Owner.releaseAll requests {} resources
{'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a': < ResourceRef
'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a', isValid: 'True' obj:
'None'>}
Thread-100903::DEBUG::2013-11-04
14:04:56,645::resourceManager::976::ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-100903::DEBUG::2013-11-04
14:04:56,646::resourceManager::615::ResourceManager::(releaseResource)
Trying to release resource
'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a'
Thread-100903::DEBUG::2013-11-04
14:04:56,646::resourceManager::634::ResourceManager::(releaseResource)
Released resource 'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a' (0
active users)
Thread-100903::DEBUG::2013-11-04
14:04:56,647::resourceManager::640::ResourceManager::(releaseResource)
Resource 'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a' is free,
finding out if anyone is waiting for it.
Thread-100903::DEBUG::2013-11-04
14:04:56,647::resourceManager::648::ResourceManager::(releaseResource)
No one is waiting for resource
'Storage.dfbbc8dd-bfae-44e1-8876-2bb82921565a', Clearing records.
Thread-100903::DEBUG::2013-11-04
14:04:56,648::task::974::TaskManager.Task::(_decref)
Task=`0f953aa3-e2b9-4008-84ad-f271136d8d23`::ref 0 aborting False
Thread-100903::INFO::2013-11-04
14:04:56,648::clientIF::325::vds::(prepareVolumePath) prepared volume
path: /rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821
Thread-100903::DEBUG::2013-11-04
14:04:56,649::vm::3619::vm.Vm::(snapshot)
vmId=`2147dd59-6794-4be6-98b9-948636a31159`::<domainsnapshot>
        <disks>
                <disk name="vda" snapshot="external">
                        <source
file="/rhev/data-center/def9b712-876a-49a9-b4b9-df9770befac4/dfbbc8dd-bfae-44e1-8876-2bb82921565a/images/57ff3040-0cbd-4659-bd21-f07036d84dd8/4d05730d-433c-40d9-8600-6fb0eb5af821"/>
                </disk>
        </disks>
</domainsnapshot>

Thread-100903::DEBUG::2013-11-04
14:04:56,659::libvirtconnection::101::libvirtconnection::(wrapper)
Unknown libvirterror: ecode: 67 edom: 10 level: 2 message: unsupported
configuration: reuse is not supported with this QEMU binary
Thread-100903::DEBUG::2013-11-04
14:04:56,659::vm::3640::vm.Vm::(snapshot)
vmId=`2147dd59-6794-4be6-98b9-948636a31159`::Snapshot failed using the
quiesce flag, trying again without it (unsupported configuration:
reuse is not supported with this QEMU binary)
Thread-100903::DEBUG::2013-11-04
14:04:56,668::libvirtconnection::101::libvirtconnection::(wrapper)
Unknown libvirterror: ecode: 67 edom: 10 level: 2 message: unsupported
configuration: reuse is not supported with this QEMU binary
Thread-100903::ERROR::2013-11-04
14:04:56,668::vm::3644::vm.Vm::(snapshot)
vmId=`2147dd59-6794-4be6-98b9-948636a31159`::Unable to take snapshot
Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 3642, in snapshot
    self._dom.snapshotCreateXML(snapxml, snapFlags)
  File "/usr/share/vdsm/vm.py", line 826, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py",
line 76, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1623, in
snapshotCreateXML
    if ret is None:raise libvirtError('virDomainSnapshotCreateXML()
failed', dom=self)
libvirtError: unsupported configuration: reuse is not supported with
this QEMU binary
Thread-100903::DEBUG::2013-11-04
14:04:56,670::BindingXMLRPC::986::vds::(wrapper) return vmSnapshot
with {'status': {'message': 'Snapshot failed', 'code': 48}}


Version information:

oVirt Engine server (CentOS 6.4 + updates, ovirt 3.3 stable):

[root at gnovirt01 ~]# rpm -qa '*ovirt*' '*vdsm*' '*libvirt*' '*kvm*'
ovirt-host-deploy-1.1.1-1.el6.noarch
ovirt-engine-lib-3.3.0.1-1.el6.noarch
ovirt-engine-webadmin-portal-3.3.0.1-1.el6.noarch
ovirt-engine-dbscripts-3.3.0.1-1.el6.noarch
libvirt-client-0.10.2-18.el6_4.14.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64
ovirt-engine-sdk-python-3.3.0.6-1.el6.noarch
ovirt-release-el6-8-1.noarch
ovirt-host-deploy-java-1.1.1-1.el6.noarch
ovirt-engine-websocket-proxy-3.3.0.1-1.el6.noarch
ovirt-image-uploader-3.3.1-1.el6.noarch
ovirt-log-collector-3.3.1-1.el6.noarch
ovirt-engine-userportal-3.3.0.1-1.el6.noarch
ovirt-engine-restapi-3.3.0.1-1.el6.noarch
ovirt-engine-backend-3.3.0.1-1.el6.noarch
ovirt-engine-setup-3.3.0.1-1.el6.noarch
libvirt-0.10.2-18.el6_4.14.x86_64
ovirt-engine-cli-3.3.0.4-1.el6.noarch
ovirt-node-iso-3.0.1-1.0.2.vdsm.el6.noarch
ovirt-iso-uploader-3.3.1-1.el6.noarch
ovirt-engine-tools-3.3.0.1-1.el6.noarch
ovirt-engine-3.3.0.1-1.el6.noarch

oVirt hypervisor servers (CentOS 6.4 + updates, ovirt 3.3 stable):

[root at gnkvm01 vdsm]# rpm -qa '*ovirt*' '*vdsm*' '*libvirt*' '*kvm*'
libvirt-client-0.10.2-18.el6_4.14.x86_64
libvirt-lock-sanlock-0.10.2-18.el6_4.14.x86_64
qemu-kvm-tools-0.12.1.2-2.355.0.1.el6_4.9.x86_64
libvirt-python-0.10.2-18.el6_4.14.x86_64
libvirt-0.10.2-18.el6_4.14.x86_64
vdsm-python-4.12.1-4.el6.x86_64
vdsm-xmlrpc-4.12.1-4.el6.noarch
vdsm-4.12.1-4.el6.x86_64
qemu-kvm-0.12.1.2-2.355.0.1.el6_4.9.x86_64
vdsm-python-cpopen-4.12.1-4.el6.x86_64
vdsm-cli-4.12.1-4.el6.noarch
[root at gnkvm01 vdsm]#

More information about the Users mailing list