<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Fri, Apr 7, 2017 at 2:40 AM Bill James <<a href="mailto:bill.james@j2.com">bill.james@j2.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We are trying to convert our qa environment from local nfs to gluster.<br class="gmail_msg">
When I move a disk with a VM that is running on same server as the<br class="gmail_msg">
storage it fails.<br class="gmail_msg">
When I move a disk with VM running on a different system it works.<br class="gmail_msg">
<br class="gmail_msg">
VM running on same system as disk:<br class="gmail_msg">
<br class="gmail_msg">
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]<br class="gmail_msg">
(vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start<br class="gmail_msg">
replication for vda to {u'domainID':<br class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID':<br class="gmail_msg">
u'6affd8c3-2c<br class="gmail_msg">
51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg">
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file',<br class="gmail_msg">
'format': 'cow', 'cache': 'none', u'volumeID':<br class="gmail_msg">
u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':<br class="gmail_msg">
u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':<br class="gmail_msg">
u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path':<br class="gmail_msg">
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg">
'propagateErrors': u'off', 'volumeChain': [{'domainID':<br class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',<br class="gmail_msg">
'leaseOffset': 0, 'path':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',<br class="gmail_msg">
'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',<br class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID':<br class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',<br class="gmail_msg">
'leaseOffset': 0, 'path':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg">
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)<br class="gmail_msg">
Traceback (most recent call last):<br class="gmail_msg">
File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart<br class="gmail_msg">
self._startDriveReplication(drive)<br class="gmail_msg">
File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication<br class="gmail_msg">
self._dom.blockCopy(<a href="http://drive.name" rel="noreferrer" class="gmail_msg" target="_blank">drive.name</a>, destxml, flags=flags)<br class="gmail_msg">
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line<br class="gmail_msg">
69, in f<br class="gmail_msg">
ret = attr(*args, **kwargs)<br class="gmail_msg">
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",<br class="gmail_msg">
line 123, in wrapper<br class="gmail_msg">
ret = f(*args, **kwargs)<br class="gmail_msg">
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in<br class="gmail_msg">
wrapper<br class="gmail_msg">
return func(inst, *args, **kwargs)<br class="gmail_msg">
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in<br class="gmail_msg">
blockCopy<br class="gmail_msg">
if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',<br class="gmail_msg">
dom=self)<br class="gmail_msg">
libvirtError: internal error: unable to execute QEMU command<br class="gmail_msg">
'drive-mirror': Could not open<br class="gmail_msg">
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':<br class="gmail_msg">
Permission denied<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
[root@ovirt1 test vdsm]# ls -l<br class="gmail_msg">
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br class="gmail_msg">
-rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29<br class="gmail_msg">
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Then if I try and rerun it it says, even though move failed:<br class="gmail_msg">
<br class="gmail_msg">
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:<br class="gmail_msg">
getAllTasksStatuses, Return response: {'allT<br class="gmail_msg">
asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212,<br class="gmail_msg">
'message': 'Volume already exists', 'taskState':<br class="gmail_msg">
'finished', 'taskResult': 'cleanSuccess', 'taskID':<br class="gmail_msg">
'078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
So now I have to clean up the disks that it failed to move so I can<br class="gmail_msg">
migrate the VM and then move the disk again.<br class="gmail_msg">
Or so it seems.<br class="gmail_msg">
Failed move disks do exist in new location, even though it "failed".<br class="gmail_msg">
<br class="gmail_msg">
vdsm.log attached.<br class="gmail_msg">
<br class="gmail_msg">
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch<br class="gmail_msg">
vdsm-4.19.4-1.el7.centos.x86_64<br class="gmail_msg"></blockquote><div><br></div><div>Hi Bill,</div><div><br></div><div>Does it work after setting selinux to permissive? (setenforce 0)</div><div><br></div><div>Can you share output of:</div><div><br></div><div>ps -efZ | grep vm-name<br></div><div>(filter the specific vm) <br></div><div><br></div><div>ls -lhZ /rhev/data-center/mnt</div><div><br></div><div>ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id</div><div>(assuming the volume was not deleted after the operation).</div><div><br></div><div>If the volume is not deleted after the failed move disk operation, this is likely</div><div>a bug, please file a bug for this.</div><div><br></div><div>The actual failure may be gluster configuration issue, or selinux related bug.</div><div><br></div><div>Nir</div><div> <br></div></div></div>