2019-04-12 10:39:25,503+0200 INFO (jsonrpc/0) [api.virt] START diskReplicateStart(srcDisk={'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}, dstDisk={'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}) from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, vmId=71f27df0-f54f-4a2e-a51c-e61aa26b370d (api:46) 2019-04-12 10:39:25,513+0200 INFO (jsonrpc/0) [vdsm.api] START prepareImage(sdUUID='244dfdfb-2662-4103-9d39-2b13153f2047', spUUID='b1a475aa-c084-46e5-b65a-bf4a47143c88', imgUUID='9a66bf0f-1333-4931-ad58-f6f1aa1143be', leafUUID='5c2738a4-4279-4cc3-a0de-6af1095f8879', allowIllegal=False) from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, task_id=78dde3c9-74fb-4588-8cfa-117f0bbe2d2d (api:46) 2019-04-12 10:39:25,630+0200 INFO (jsonrpc/0) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2 (fileSD:623) 2019-04-12 10:39:25,631+0200 INFO (jsonrpc/0) [storage.StorageDomain] Fixing permissions on /rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879 (fileSD:623) 2019-04-12 10:39:25,632+0200 INFO (jsonrpc/0) [storage.StorageDomain] Creating domain run directory u'/var/run/vdsm/storage/244dfdfb-2662-4103-9d39-2b13153f2047' (fileSD:577) 2019-04-12 10:39:25,632+0200 INFO (jsonrpc/0) [storage.fileUtils] Creating directory: /var/run/vdsm/storage/244dfdfb-2662-4103-9d39-2b13153f2047 mode: None (fileUtils:197) 2019-04-12 10:39:25,632+0200 INFO (jsonrpc/0) [storage.StorageDomain] Creating symlink from /rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be to /var/run/vdsm/storage/244dfdfb-2662-4103-9d39-2b13153f2047/9a66bf0f-1333-4931-ad58-f6f1aa1143be (fileSD:580) 2019-04-12 10:39:25,637+0200 INFO (jsonrpc/0) [vdsm.api] FINISH prepareImage return={'info': {'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'type': 'file'}, 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'imgVolumesInfo': [{'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2', 'volumeID': u'cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2', 'leasePath': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2.lease', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}, {'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'volumeID': u'5c2738a4-4279-4cc3-a0de-6af1095f8879', 'leasePath': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879.lease', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}]} from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, task_id=78dde3c9-74fb-4588-8cfa-117f0bbe2d2d (api:52) 2019-04-12 10:39:25,637+0200 INFO (jsonrpc/0) [vds] prepared volume path: /rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879 (clientIF:497) 2019-04-12 10:39:25,642+0200 INFO (jsonrpc/0) [vdsm.api] START teardownImage(sdUUID='244dfdfb-2662-4103-9d39-2b13153f2047', spUUID='b1a475aa-c084-46e5-b65a-bf4a47143c88', imgUUID='9a66bf0f-1333-4931-ad58-f6f1aa1143be', volUUID=None) from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, task_id=7975136c-d4e2-47ff-a940-d7f9c9e94b19 (api:46) 2019-04-12 10:39:25,643+0200 INFO (jsonrpc/0) [storage.StorageDomain] Removing image rundir link u'/var/run/vdsm/storage/244dfdfb-2662-4103-9d39-2b13153f2047/9a66bf0f-1333-4931-ad58-f6f1aa1143be' (fileSD:600) 2019-04-12 10:39:25,643+0200 INFO (jsonrpc/0) [vdsm.api] FINISH teardownImage return=None from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, task_id=7975136c-d4e2-47ff-a940-d7f9c9e94b19 (api:52) 2019-04-12 10:39:25,643+0200 ERROR (jsonrpc/0) [virt.vm] (vmId='71f27df0-f54f-4a2e-a51c-e61aa26b370d') Unable to start replication for vda to {'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'volumeInfo': {'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'type': 'file'}, 'diskType': 'file', 'format': 'cow', 'cache': 'none', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'device': 'disk', 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'propagateErrors': 'off', 'volumeChain': [{'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2', 'volumeID': u'cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2', 'leasePath': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/cbe93bfe-9df0-4f12-a44e-9e8fa6ec24f2.lease', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}, {'domainID': '244dfdfb-2662-4103-9d39-2b13153f2047', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879', 'volumeID': u'5c2738a4-4279-4cc3-a0de-6af1095f8879', 'leasePath': u'/rhev/data-center/mnt/removed-IP-address:_bs01aF1C1v1/244dfdfb-2662-4103-9d39-2b13153f2047/images/9a66bf0f-1333-4931-ad58-f6f1aa1143be/5c2738a4-4279-4cc3-a0de-6af1095f8879.lease', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}]} (vm:4710) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4704, in diskReplicateStart self._startDriveReplication(drive) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4843, in _startDriveReplication self._dom.blockCopy(drive.name, destxml, flags=flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 98, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 130, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 92, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 728, in blockCopy ret = libvirtmod.virDomainBlockCopy(self._o, disk, destxml, params, flags) TypeError: block params must be a dictionary 2019-04-12 10:39:25,648+0200 INFO (jsonrpc/0) [api.virt] FINISH diskReplicateStart return={'status': {'message': 'Drive replication error', 'code': 55}} from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, vmId=71f27df0-f54f-4a2e-a51c-e61aa26b370d (api:52) 2019-04-12 10:39:25,649+0200 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateStart failed (error 55) in 0.15 seconds (__init__:573) 2019-04-12 10:39:26,867+0200 INFO (jsonrpc/6) [api.virt] START diskReplicateFinish(srcDisk={'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}, dstDisk={'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}) from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, vmId=71f27df0-f54f-4a2e-a51c-e61aa26b370d (api:46) 2019-04-12 10:39:26,867+0200 ERROR (jsonrpc/6) [api] FINISH diskReplicateFinish error=Replication not in progress.: {'driveName': 'vda', 'vmId': '71f27df0-f54f-4a2e-a51c-e61aa26b370d', 'srcDisk': {'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}} (api:129) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 122, in method ret = func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 571, in diskReplicateFinish return self.vm.diskReplicateFinish(srcDisk, dstDisk) File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 4746, in diskReplicateFinish srcDisk=srcDisk) ReplicationNotInProgress: Replication not in progress.: {'driveName': 'vda', 'vmId': '71f27df0-f54f-4a2e-a51c-e61aa26b370d', 'srcDisk': {'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}} 2019-04-12 10:39:26,871+0200 INFO (jsonrpc/6) [api.virt] FINISH diskReplicateFinish return={'status': {'message': "Replication not in progress.: {'driveName': 'vda', 'vmId': '71f27df0-f54f-4a2e-a51c-e61aa26b370d', 'srcDisk': {'device': 'disk', 'poolID': 'b1a475aa-c084-46e5-b65a-bf4a47143c88', 'volumeID': '5c2738a4-4279-4cc3-a0de-6af1095f8879', 'domainID': 'e5bb3e8a-a9c6-4581-8c6a-67d4ee7609f5', 'imageID': '9a66bf0f-1333-4931-ad58-f6f1aa1143be'}}", 'code': 88}} from=::ffff:10.76.98.4,57566, flow_id=97b620d9-6e65-4573-9fdf-5b119764fbb7, vmId=71f27df0-f54f-4a2e-a51c-e61aa26b370d (api:52) 2019-04-12 10:39:26,872+0200 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer] RPC call VM.diskReplicateFinish failed (error 88) in 0.00 seconds (__init__:573)