2:39 a.m.
We are trying to convert our qa environment from local nfs to gluster.
When I move a disk with a VM that is running on same server as the
storage it fails.
When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
(vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
replication for vda to {u'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':
u'6affd8c3-2c
51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset':
0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType':
'file',
'format': 'cow', 'cache': 'none', u'volumeID':
u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk',
'path':
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'propagateErrors': u'off', 'volumeChain': [{'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart
self._startDriveReplication(drive)
File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication
self._dom.blockCopy(drive.name, destxml, flags=flags)
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
69, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 123, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
blockCopy
if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',
dom=self)
libvirtError: internal error: unable to execute QEMU command
'drive-mirror': Could not open
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':
Permission denied
[root@ovirt1 test vdsm]# ls -l
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
-rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:
getAllTasksStatuses, Return response: {'allT
asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212,
'message': 'Volume already exists', 'taskState':
'finished', 'taskResult': 'cleanSuccess', 'taskID':
'078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can
migrate the VM and then move the disk again.
Or so it seems.
Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
vdsm-4.19.4-1.el7.centos.x86_64
10:52 a.m.
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james(a)j2.com> wrote:
We are trying to convert our qa environment from local nfs to
gluster.
When I move a disk with a VM that is running on same server as the
storage it fails.
When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
(vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
replication for vda to {u'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':
u'6affd8c3-2c
51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset':
0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType':
'file',
'format': 'cow', 'cache': 'none', u'volumeID':
u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk',
'path':
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'propagateErrors': u'off', 'volumeChain': [{'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
{'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart
self._startDriveReplication(drive)
File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication
self._dom.blockCopy(drive.name, destxml, flags=flags)
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
69, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 123, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
blockCopy
if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',
dom=self)
libvirtError: internal error: unable to execute QEMU command
'drive-mirror': Could not open
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':
Permission denied
[root@ovirt1 test vdsm]# ls -l
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
-rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:
getAllTasksStatuses, Return response: {'allT
asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212,
'message': 'Volume already exists', 'taskState':
'finished', 'taskResult': 'cleanSuccess', 'taskID':
'078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can
migrate the VM and then move the disk again.
Or so it seems.
Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
vdsm-4.19.4-1.el7.centos.x86_64
Hi Bill,
Does it work after setting selinux to permissive? (setenforce 0)
Can you share output of:
ps -efZ | grep vm-name
(filter the specific vm)
ls -lhZ /rhev/data-center/mnt
ls -lhZ
/rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id
(assuming the volume was not deleted after the operation).
If the volume is not deleted after the failed move disk operation, this is
likely
a bug, please file a bug for this.
The actual failure may be gluster configuration issue, or selinux related
bug.
Nir
5:29 p.m.
--------------705BA3C205C526559706549D
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Transfer-Encoding: 7bit
On 4/7/17 12:52 AM, Nir Soffer wrote:
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james(a)j2.com
<mailto:bill.james@j2.com>> wrote:
We are trying to convert our qa environment from local nfs to gluster.
When I move a disk with a VM that is running on same server as the
storage it fails.
When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
(vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
replication for vda to {u'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':
u'6affd8c3-2c
51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0,
'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
'diskType':
'file',
'format': 'cow', 'cache': 'none',
u'volumeID':
u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk',
'path':
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'propagateErrors': u'off', 'volumeChain':
[{'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
{'domainID':
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart
self._startDriveReplication(drive)
File "/usr/share/vdsm/virt/vm.py", line 3713, in
_startDriveReplication
self._dom.blockCopy(drive.name <http://drive.name>;, destxml,
flags=flags)
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line
69, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 123, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
blockCopy
if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',
dom=self)
libvirtError: internal error: unable to execute QEMU command
'drive-mirror': Could not open
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':
Permission denied
[root@ovirt1 test vdsm]# ls -l
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
-rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and
protect:
getAllTasksStatuses, Return response: {'allT
asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code':
212,
'message': 'Volume already exists', 'taskState':
'finished', 'taskResult': 'cleanSuccess',
'taskID':
'078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can
migrate the VM and then move the disk again.
Or so it seems.
Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
vdsm-4.19.4-1.el7.centos.x86_64
Hi Bill,
Does it work after setting selinux to permissive? (setenforce 0)
Can you share output of:
ps -efZ | grep vm-name
(filter the specific vm)
ls -lhZ /rhev/data-center/mnt
ls -lhZ
/rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id
(assuming the volume was not deleted after the operation).
If the volume is not deleted after the failed move disk operation,
this is likely
a bug, please file a bug for this.
The actual failure may be gluster configuration issue, or selinux
related bug.
Nir
SELinux status: disabled
[root@ovirt1 test images]# ps -efZ | grep darmaster
- root 5272 1 6 Mar17 ?
1-09:08:58 /usr/libexec/qemu-kvm -name
guest=darmaster1.test.j2noc.com,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/master-key.aes
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m
size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp
1,maxcpus=16,sockets=16,cores=1,threads=1 -numa
node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462
-smbios type=1,manufacturer=oVirt,product=oVirt
Node,version=7-3.1611.el7.centos,serial=30343536-3138-584D-5134-343430313833,uuid=73368460-92e1-4c9e-a162-399304f1c462
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2017-03-17T23:57:54,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
-drive if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
10.176.30.96:6,password -k en-us -device
VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg timestamp=on
[root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt
drwxr-xr-x vdsm kvm ? glusterSD
drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__images_ovirt__QA__ISOs
drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__storage1_ovirt__qa__inside
drwxr-xr-x vdsm kvm ? ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1
drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2
drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2
drwxr-xr-x vdsm kvm ? ovirt3-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? ovirt4-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? ovirt5-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? ovirt6-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? ovirt7-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_Export
drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_images
[root@ovirt1 test images]# ls -lhZa
/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793
drwxr-xr-x vdsm kvm ? .
drwxr-xr-x vdsm kvm ? ..
-rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2
-rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.lease
-rw-r--r-- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.meta
-rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6
-rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.lease
-rw-r--r-- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.meta
bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=1440198
Thank you!
--------------705BA3C205C526559706549D
Content-Type: text/html; charset="utf-8"
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 4/7/17 12:52 AM, Nir Soffer
wrote:<br>
</div>
<blockquote
cite="mid:CAMRbyyvG1X4-npxFhsJd6dgmzB6mqr+JoFN08U2aKVFsBZckgA@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">On Fri, Apr 7, 2017 at 2:40 AM Bill James
<<a
moz-do-not-send="true"
href="mailto:bill.james@j2.com">bill.james@j2.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">We are
trying to convert our qa environment from local nfs to
gluster.<br class="gmail_msg">
When I move a disk with a VM that is running on same server
as the<br class="gmail_msg">
storage it fails.<br class="gmail_msg">
When I move a disk with VM running on a different system it
works.<br class="gmail_msg">
<br class="gmail_msg">
VM running on same system as disk:<br class="gmail_msg">
<br class="gmail_msg">
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]<br
class="gmail_msg">
(vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to
start<br class="gmail_msg">
replication for vda to {u'domainID':<br
class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':<br class="gmail_msg">
u'6affd8c3-2c<br class="gmail_msg">
51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':<br
class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br
class="gmail_msg">
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br
class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
'diskType': 'file',<br class="gmail_msg">
'format': 'cow', 'cache': 'none',
u'volumeID':<br
class="gmail_msg">
u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':<br
class="gmail_msg">
u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':<br
class="gmail_msg">
u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device':
'disk',
'path':<br class="gmail_msg">
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br
class="gmail_msg">
'propagateErrors': u'off', 'volumeChain':
[{'domainID':<br
class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType':
'path',<br
class="gmail_msg">
'leaseOffset': 0, 'path':<br
class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',<br
class="gmail_msg">
'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',<br
class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
{'domainID':<br class="gmail_msg">
u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType':
'path',<br
class="gmail_msg">
'leaseOffset': 0, 'path':<br
class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br
class="gmail_msg">
'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':<br class="gmail_msg">
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br
class="gmail_msg">
'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]}
(vm:3594)<br class="gmail_msg">
Traceback (most recent call last):<br class="gmail_msg">
File "/usr/share/vdsm/virt/vm.py", line 3588, in
diskReplicateStart<br class="gmail_msg">
self._startDriveReplication(drive)<br class="gmail_msg">
File "/usr/share/vdsm/virt/vm.py", line 3713, in
_startDriveReplication<br class="gmail_msg">
self._dom.blockCopy(<a moz-do-not-send="true"
href="http://drive.name" rel="noreferrer"
class="gmail_msg"
target="_blank">drive.name</a>, destxml,
flags=flags)<br class="gmail_msg">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line<br class="gmail_msg">
69, in f<br class="gmail_msg">
ret = attr(*args, **kwargs)<br class="gmail_msg">
File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",<br
class="gmail_msg">
line 123, in wrapper<br class="gmail_msg">
ret = f(*args, **kwargs)<br class="gmail_msg">
File "/usr/lib/python2.7/site-packages/vdsm/utils.py",
line 941, in<br class="gmail_msg">
wrapper<br class="gmail_msg">
return func(inst, *args, **kwargs)<br class="gmail_msg">
File "/usr/lib64/python2.7/site-packages/libvirt.py",
line 684, in<br class="gmail_msg">
blockCopy<br class="gmail_msg">
if ret == -1: raise libvirtError ('virDomainBlockCopy()
failed',<br class="gmail_msg">
dom=self)<br class="gmail_msg">
libvirtError: internal error: unable to execute QEMU command<br
class="gmail_msg">
'drive-mirror': Could not open<br class="gmail_msg">
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':<br
class="gmail_msg">
Permission denied<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
[root@ovirt1 test vdsm]# ls -l<br class="gmail_msg">
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br
class="gmail_msg">
-rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29<br
class="gmail_msg">
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br
class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
Then if I try and rerun it it says, even though move failed:<br
class="gmail_msg">
<br class="gmail_msg">
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run
and protect:<br class="gmail_msg">
getAllTasksStatuses, Return response: {'allT<br
class="gmail_msg">
asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b':
{'code': 212,<br class="gmail_msg">
'message': 'Volume already exists',
'taskState':<br
class="gmail_msg">
'finished', 'taskResult': 'cleanSuccess',
'taskID':<br
class="gmail_msg">
'078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)<br
class="gmail_msg">
<br class="gmail_msg">
<br class="gmail_msg">
So now I have to clean up the disks that it failed to move
so I can<br class="gmail_msg">
migrate the VM and then move the disk again.<br
class="gmail_msg">
Or so it seems.<br class="gmail_msg">
Failed move disks do exist in new location, even though it
"failed".<br class="gmail_msg">
<br class="gmail_msg">
vdsm.log attached.<br class="gmail_msg">
<br class="gmail_msg">
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch<br
class="gmail_msg">
vdsm-4.19.4-1.el7.centos.x86_64<br class="gmail_msg">
</blockquote>
<div><br>
</div>
<div>Hi Bill,</div>
<div><br>
</div>
<div>Does it work after setting selinux to permissive?
(setenforce 0)</div>
<div><br>
</div>
<div>Can you share output of:</div>
<div><br>
</div>
<div>ps -efZ | grep vm-name<br>
</div>
<div>(filter the specific vm) <br>
</div>
<div><br>
</div>
<div>ls -lhZ /rhev/data-center/mnt</div>
<div><br>
</div>
<div>ls -lhZ
/rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id</div>
<div>(assuming the volume was not deleted after the
operation).</div>
<div><br>
</div>
<div>If the volume is not deleted after the failed move disk
operation, this is likely</div>
<div>a bug, please file a bug for this.</div>
<div><br>
</div>
<div>The actual failure may be gluster configuration issue, or
selinux related bug.</div>
<div><br>
</div>
<div>Nir</div>
<div> <br>
</div>
</div>
</div>
</blockquote>
<br>
SELinux status: disabled<br>
<br>
[root@ovirt1 test images]# ps -efZ | grep darmaster<br>
- root 5272 1 6 Mar17
? 1-09:08:58 /usr/libexec/qemu-kvm -name
guest=darmaster1.test.j2noc.com,debug-threads=on -S -object
secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/master-key.aes
-machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m
size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp
1,maxcpus=16,sockets=16,cores=1,threads=1 -numa
node,nodeid=0,cpus=0,mem=1024 -uuid
73368460-92e1-4c9e-a162-399304f1c462 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=7-3.1611.el7.centos,serial=30343536-3138-584D-5134-343430313833,uuid=73368460-92e1-4c9e-a162-399304f1c462
-no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/monitor.sock,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=2017-03-17T23:57:54,driftfix=slew -global
kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2
-device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
-drive if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads
-device
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-netdev tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3
-chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-device usb-tablet,id=input0,bus=usb.0,port=1 -vnc
10.176.30.96:6,password -k en-us -device
VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg timestamp=on<br>
<br>
[root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt<br>
drwxr-xr-x vdsm kvm ? glusterSD<br>
drwxr-xr-x vdsm kvm ?
netappqa3:_vol_cloud__images_ovirt__QA__ISOs<br>
drwxr-xr-x vdsm kvm ?
netappqa3:_vol_cloud__storage1_ovirt__qa__inside<br>
drwxr-xr-x vdsm kvm ?
ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1<br>
drwxr-xr-x vdsm kvm ?
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2<br>
drwxr-xr-x vdsm kvm ?
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2<br>
drwxr-xr-x vdsm kvm ?
ovirt3-ks.test.j2noc.com:_ovirt-store_nfs<br>
drwxr-xr-x vdsm kvm ?
ovirt4-ks.test.j2noc.com:_ovirt-store_nfs<br>
drwxr-xr-x vdsm kvm ?
ovirt5-ks.test.j2noc.com:_ovirt-store_nfs<br>
drwxr-xr-x vdsm kvm ?
ovirt6-ks.test.j2noc.com:_ovirt-store_nfs<br>
drwxr-xr-x vdsm kvm ?
ovirt7-ks.test.j2noc.com:_ovirt-store_nfs<br>
drwxr-xr-x vdsm kvm ?
qagenfil1-nfs1:_ovirt__inside_Export<br>
drwxr-xr-x vdsm kvm ?
qagenfil1-nfs1:_ovirt__inside_images<br>
<br>
<br>
[root@ovirt1 test images]# ls -lhZa
/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793<br>
drwxr-xr-x vdsm kvm ? .<br>
drwxr-xr-x vdsm kvm ? ..<br>
-rw-rw---- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2<br>
-rw-rw---- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2.lease<br>
-rw-r--r-- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2.meta<br>
-rw-rw---- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6<br>
-rw-rw---- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6.lease<br>
-rw-r--r-- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6.meta<br>
<br>
<br>
bug submitted: <a class="moz-txt-link-freetext"
href="https://bugzilla.redhat.com/show_bug.cgi?id=1440198">h...
<br>
<br>
Thank you!<br>
</body>
</html>
--------------705BA3C205C526559706549D--
10:16 a.m.
On Fri, Apr 7, 2017 at 5:29 PM, Bill James <bill.james(a)j2.com> wrote:
On 4/7/17 12:52 AM, Nir Soffer wrote:
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james(a)j2.com> wrote:
> We are trying to convert our qa environment from local nfs to gluster.
> When I move a disk with a VM that is running on same server as the
> storage it fails.
> When I move a disk with VM running on a different system it works.
>
> VM running on same system as disk:
>
> 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
> (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
> replication for vda to {u'domainID':
> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':
> u'6affd8c3-2c
> 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, 'path':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
'diskType': 'file',
> 'format': 'cow', 'cache': 'none',
u'volumeID':
> u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
> u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
> u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk',
'path':
> u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
> 'propagateErrors': u'off', 'volumeChain':
[{'domainID':
> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
> 'leaseOffset': 0, 'path':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',
> 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'leasePath':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
{'domainID':
> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
> 'leaseOffset': 0, 'path':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
> Traceback (most recent call last):
> File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart
> self._startDriveReplication(drive)
> File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication
> self._dom.blockCopy(drive.name, destxml, flags=flags)
> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
> 69, in f
> ret = attr(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
> line 123, in wrapper
> ret = f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in
> wrapper
> return func(inst, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
> blockCopy
> if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',
> dom=self)
> libvirtError: internal error: unable to execute QEMU command
> 'drive-mirror': Could not open
> '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':
> Permission denied
>
>
> [root@ovirt1 test vdsm]# ls -l
> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
> -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
>
>
>
> Then if I try and rerun it it says, even though move failed:
>
> 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:
> getAllTasksStatuses, Return response: {'allT
> asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code':
212,
> 'message': 'Volume already exists', 'taskState':
> 'finished', 'taskResult': 'cleanSuccess',
'taskID':
> '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
>
>
> So now I have to clean up the disks that it failed to move so I can
> migrate the VM and then move the disk again.
> Or so it seems.
> Failed move disks do exist in new location, even though it "failed".
>
> vdsm.log attached.
>
> ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
> vdsm-4.19.4-1.el7.centos.x86_64
>
Hi Bill,
Does it work after setting selinux to permissive? (setenforce 0)
Can you share output of:
ps -efZ | grep vm-name
(filter the specific vm)
ls -lhZ /rhev/data-center/mnt
ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_
id/vol_id
(assuming the volume was not deleted after the operation).
If the volume is not deleted after the failed move disk operation, this is
likely
a bug, please file a bug for this.
The actual failure may be gluster configuration issue, or selinux related
bug.
Nir
SELinux status: disabled
This is a less tested configuration. We usually run with selinux enabled.
Y.
[root@ovirt1 test images]# ps -efZ | grep darmaster
- root 5272 1 6 Mar17 ?
1-09:08:58 /usr/libexec/qemu-kvm -name guest=darmaster1.test.j2noc.com,debug-threads=on
-S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
qemu/domain-13-darmaster1.test.j2no/master-key.aes -machine
pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m
size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp
1,maxcpus=16,sockets=16,cores=1,threads=1 -numa
node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462
-smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.
centos,serial=30343536-3138-584D-5134-343430313833,uuid=
73368460-92e1-4c9e-a162-399304f1c462 -no-user-config -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
13-darmaster1.test.j2no/monitor.sock,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-17T23:57:54,driftfix=slew
-global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
-drive if=none,id=drive-ide0-1-0,readonly=on -device
ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-
47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-
a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=
raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-
40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-
virtio-disk0,id=virtio-disk0,bootindex=1 -netdev
tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=
hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device
usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.176.30.96:6,password -k
en-us -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg
timestamp=on
[root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt
drwxr-xr-x vdsm kvm ? glusterSD
drwxr-xr-x vdsm kvm ?
netappqa3:_vol_cloud__images_ovirt__QA__ISOs
drwxr-xr-x vdsm kvm ?
netappqa3:_vol_cloud__storage1_ovirt__qa__inside
drwxr-xr-x vdsm kvm ?
ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1
drwxr-xr-x vdsm kvm ?
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2
drwxr-xr-x vdsm kvm ?
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2
drwxr-xr-x vdsm kvm ?
ovirt3-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ?
ovirt4-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ?
ovirt5-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ?
ovirt6-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ?
ovirt7-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ?
qagenfil1-nfs1:_ovirt__inside_Export
drwxr-xr-x vdsm kvm ?
qagenfil1-nfs1:_ovirt__inside_images
[root@ovirt1 test images]# ls -lhZa /rhev/data-center/mnt/
glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-
4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793
drwxr-xr-x vdsm kvm ? .
drwxr-xr-x vdsm kvm ? ..
-rw-rw---- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2
-rw-rw---- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2.lease
-rw-r--r-- vdsm kvm ?
33c04305-efbe-418a-b42c-07f5f76214f2.meta
-rw-rw---- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6
-rw-rw---- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6.lease
-rw-r--r-- vdsm kvm ?
38de110d-464c-4735-97ba-3d623ee1a1b6.meta
bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=1440198
Thank you!
_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
10:29 a.m.
--Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
On 9 Apr 2017, at 09:16, Yaniv Kaul <ykaul(a)redhat.com> wrote:
=20
=20
=20
On Fri, Apr 7, 2017 at 5:29 PM, Bill James <bill.james(a)j2.com =
<mailto:bill.james@j2.com>> wrote:
=20
=20
On 4/7/17 12:52 AM, Nir Soffer wrote:
> On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james(a)j2.com =
<mailto:bill.james@j2.com>> wrote:
> We are trying to convert our qa environment from local nfs to =
gluster.
> When I move a disk with a VM that is running on same server as
the
> storage it fails.
> When I move a disk with VM running on a different system it works.
>=20
> VM running on same system as disk:
>=20
> 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
> (vmId=3D'e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
> replication for vda to {u'domainID':
> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo':
{'domainID':
> u'6affd8c3-2c
> 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
'leaseOffset': 0, =
'path':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f=
d46c9-c738-4b13-aeca-3dc9ffc677f5',
>> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f=
d46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
> 'imageID':
u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': =
'file',
>> 'format': 'cow', 'cache': 'none',
u'volumeID':
>> u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
>> u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
>> u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device':
'disk', 'path':
> =
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd=
1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c=
738-4b13-aeca-3dc9ffc677f5',
>> 'propagateErrors': u'off', 'volumeChain':
[{'domainID':
>> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType':
'path',
>> 'leaseOffset': 0, 'path':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/675=
6eb05-6803-42a7-a3a2-10233bf2ca8d',
>> 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d',
'leasePath':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/675=
6eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
>> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'},
{'domainID':
>> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType':
'path',
>> 'leaseOffset': 0, 'path':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f=
d46c9-c738-4b13-aeca-3dc9ffc677f5',
>> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
'leasePath':
> =
u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2=
c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f=
d46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
> 'imageID':
u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
> Traceback (most recent call last):
> File "/usr/share/vdsm/virt/vm.py", line 3588, in =
diskReplicateStart
> self._startDriveReplication(drive)
> File "/usr/share/vdsm/virt/vm.py", line 3713, in =
_startDriveReplication
> self._dom.blockCopy(drive.name <http://drive.name/>;,
destxml, =
flags=3Dflags)
> File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", =
line
> 69, in f
> ret =3D attr(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
> line 123, in wrapper
> ret =3D f(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, =
in
> wrapper
> return func(inst, *args, **kwargs)
> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
> blockCopy
> if ret =3D=3D -1: raise libvirtError ('virDomainBlockCopy() =
failed',
>> dom=3Dself)
>> libvirtError: internal error: unable to execute QEMU command
>> 'drive-mirror': Could not open
> =
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1=
-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c7=
38-4b13-aeca-3dc9ffc677f5':
>> Permission denied
>>=20
>>=20
>> [root@ovirt1 test vdsm]# ls -l
> =
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-=
8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c73=
8-4b13-aeca-3dc9ffc677f5
>> -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
> =
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-=
8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c73=
8-4b13-aeca-3dc9ffc677f5
>=20
>=20
>=20
> Then if I try and rerun it it says, even though move failed:
>=20
> 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and =
protect:
> getAllTasksStatuses, Return response: {'allT
> asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code':
212,
> 'message': 'Volume already exists', 'taskState':
> 'finished', 'taskResult': 'cleanSuccess',
'taskID':
> '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
>=20
>=20
> So now I have to clean up the disks that it failed to move so I can
> migrate the VM and then move the disk again.
> Or so it seems.
> Failed move disks do exist in new location, even though it "failed".
>=20
> vdsm.log attached.
>=20
> ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
> vdsm-4.19.4-1.el7.centos.x86_64
>=20
> Hi Bill,
>=20
> Does it work after setting selinux to permissive? (setenforce 0)
>=20
> Can you share output of:
>=20
> ps -efZ | grep vm-name
> (filter the specific vm)=20
>=20
> ls -lhZ /rhev/data-center/mnt
>=20
> ls -lhZ =
/rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id
> (assuming the volume was not deleted after the operation).
>=20
> If the volume is not deleted after the failed move disk operation, =
this is
likely
> a bug, please file a bug for this.
>=20
> The actual failure may be gluster configuration issue, or selinux =
related
bug.
>=20
> Nir
> =20
=20
SELinux status: disabled
=20
This is a less tested configuration. We usually run with selinux =
enabled.
Disabled specifically is untested. And doesn=E2=80=99t work.
If you for whatever reason don=E2=80=99t want it, set it to Permissive, =
but don=E2=80=99t disable it
Y.
=20
=20
[root@ovirt1 test images]# ps -efZ | grep darmaster
- root 5272 1 6 Mar17 ? =
1-09:08:58
/usr/libexec/qemu-kvm -name guest=3Ddarmaster1.test.j2noc.com =
<http://darmaster1.test.j2noc.com/>,debug-threads=3Don -S -object =
secret,id=3DmasterKey0,format=3Draw,file=3D/var/lib/libvirt/qemu/domain-13=
-darmaster1.test.j2no/master-key.aes -machine =
pc-i440fx-rhel7.3.0,accel=3Dkvm,usb=3Doff -cpu Nehalem -m =
size=3D1048576k,slots=3D16,maxmem=3D4194304k -realtime mlock=3Doff -smp =
1,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D1 -numa =
node,nodeid=3D0,cpus=3D0,mem=3D1024 -uuid =
73368460-92e1-4c9e-a162-399304f1c462 -smbios =
type=3D1,manufacturer=3DoVirt,product=3DoVirt =
Node,version=3D7-3.1611.el7.centos,serial=3D30343536-3138-584D-5134-343430=
313833,uuid=3D73368460-92e1-4c9e-a162-399304f1c462 -no-user-config =
-nodefaults -chardev =
socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/domain-13-darmaster1.=
test.j2no/monitor.sock,server,nowait -mon =
chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc =
base=3D2017-03-17T23:57:54,driftfix=3Dslew -global =
kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don=
-device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device =
virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device =
virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x=
5 -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don -device =
ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0 -drive =
file=3D/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e06=
0-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de1=
10d-464c-4735-97ba-3d623ee1a1b6,format=3Draw,if=3Dnone,id=3Ddrive-virtio-d=
isk0,serial=3D33db5688-dafe-40ab-9dd0-a826a90c3793,cache=3Dnone,werror=3Ds=
top,rerror=3Dstop,aio=3Dthreads -device =
virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk=
0,id=3Dvirtio-disk0,bootindex=3D1 -netdev =
tap,fd=3D38,id=3Dhostnet0,vhost=3Don,vhostfd=3D40 -device =
virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:16:01:69,bus=3Dp=
ci.0,addr=3D0x3 -chardev =
socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/73368460-92=
e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait -device =
virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc=
hannel0,name=3Dcom.redhat.rhevm.vdsm -chardev =
socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/73368460-92=
e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait -device =
virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc=
hannel1,name=3Dorg.qemu.guest_agent.0 -device =
usb-tablet,id=3Dinput0,bus=3Dusb.0,port=3D1 -vnc 10.176.30.96:6 =
<http://10.176.30.96:6/>,password -k en-us -device =
VGA,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpci.0,addr=3D0x2 -msg timestamp=3Don
=20
[root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt
drwxr-xr-x vdsm kvm ? glusterSD
drwxr-xr-x vdsm kvm ? =
netappqa3:_vol_cloud__images_ovirt__QA__ISOs
drwxr-xr-x vdsm kvm ? =
netappqa3:_vol_cloud__storage1_ovirt__qa__inside
drwxr-xr-x vdsm kvm ? =
ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1
drwxr-xr-x vdsm kvm ? =
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2
drwxr-xr-x vdsm kvm ? =
ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2
drwxr-xr-x vdsm kvm ? =
ovirt3-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? =
ovirt4-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? =
ovirt5-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? =
ovirt6-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? =
ovirt7-ks.test.j2noc.com:_ovirt-store_nfs
drwxr-xr-x vdsm kvm ? =
qagenfil1-nfs1:_ovirt__inside_Export
drwxr-xr-x vdsm kvm ? =
qagenfil1-nfs1:_ovirt__inside_images
=20
=20
[root@ovirt1 test images]# ls -lhZa =
/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c5=
1-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793
drwxr-xr-x vdsm kvm ? .
drwxr-xr-x vdsm kvm ? ..
-rw-rw---- vdsm kvm ? =
33c04305-efbe-418a-b42c-07f5f76214f2
-rw-rw---- vdsm kvm ? =
33c04305-efbe-418a-b42c-07f5f76214f2.lease
-rw-r--r-- vdsm kvm ? =
33c04305-efbe-418a-b42c-07f5f76214f2.meta
-rw-rw---- vdsm kvm ? =
38de110d-464c-4735-97ba-3d623ee1a1b6
-rw-rw---- vdsm kvm ? =
38de110d-464c-4735-97ba-3d623ee1a1b6.lease
-rw-r--r-- vdsm kvm ? =
38de110d-464c-4735-97ba-3d623ee1a1b6.meta
=20
=20
bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198 =
<https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198>
=20
=20
Thank you!
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
--Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><br class=3D""><div><blockquote
type=3D"cite" class=3D""><div =
class=3D"">On 9 Apr 2017, at 09:16, Yaniv Kaul <<a =
href=3D"mailto:ykaul@redhat.com"
class=3D"">ykaul(a)redhat.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D""><div =
dir=3D"ltr" style=3D"font-family: Helvetica; font-size: 12px; =
font-style: normal; font-variant-caps: normal; font-weight: normal; =
letter-spacing: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; =
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><div
=
class=3D"gmail_extra"><br
class=3D"Apple-interchange-newline"><br =
class=3D""><div class=3D"gmail_quote">On Fri, Apr 7, 2017 at
5:29 PM, =
Bill James<span
class=3D"Apple-converted-space"> </span><span =
dir=3D"ltr" class=3D""><<a
href=3D"mailto:bill.james@j2.com" =
target=3D"_blank"
class=3D"">bill.james(a)j2.com</a>&gt;</span><span =
class=3D"Apple-converted-space"> </span>wrote:<br =
class=3D""><blockquote class=3D"gmail_quote"
style=3D"margin: 0px 0px =
0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, =
204); border-left-style: solid; padding-left: 1ex;"><div =
bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""><div
class=3D""><div =
class=3D"h5"><br class=3D""><br
class=3D""><div =
class=3D"m_-7375313093972664498moz-cite-prefix">On 4/7/17 12:52 AM, Nir =
Soffer wrote:<br class=3D""></div><blockquote
type=3D"cite" =
class=3D""><div dir=3D"ltr" class=3D""><div
class=3D"gmail_quote"><div =
dir=3D"ltr" class=3D"">On Fri, Apr 7, 2017 at 2:40 AM Bill James
<<a =
href=3D"mailto:bill.james@j2.com" target=3D"_blank" =
class=3D"">bill.james(a)j2.com</a>&gt; wrote:<br =
class=3D""></div><blockquote class=3D"gmail_quote"
style=3D"margin: 0px =
0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, =
204); border-left-style: solid; padding-left: 1ex;">We are trying to =
convert our qa environment from local nfs to gluster.<br =
class=3D"m_-7375313093972664498gmail_msg">When I move a disk with a VM =
that is running on same server as the<br =
class=3D"m_-7375313093972664498gmail_msg">storage it fails.<br =
class=3D"m_-7375313093972664498gmail_msg">When I move a disk with VM =
running on a different system it works.<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">VM running on same system as =
disk:<br class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">2017-04-06 13:31:00,588 ERROR =
(jsonrpc/6) [virt.vm]<br =
class=3D"m_-7375313093972664498gmail_msg">(vmId=3D'e598485a-dc74-43f7-<wbr=
class=3D"">8447-e00ac44dae21') Unable to start<br =
class=3D"m_-7375313093972664498gmail_msg">replication for vda to =
{u'domainID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr
=
class=3D"">bfbbb14edbe9', 'volumeInfo':
{'domainID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c<br =
class=3D"m_-7375313093972664498gmail_msg">51-4cd1-8300-bfbbb14edbe9', =
'volType': 'path', 'leaseOffset': 0, 'path':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/30fd46c9-c738-<wbr =
class=3D"">4b13-aeca-3dc9ffc677f5',<br =
class=3D"m_-7375313093972664498gmail_msg">'volumeID': =
u'30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5',
'leasePath':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/30fd46c9-c738-<wbr =
class=3D"">4b13-aeca-3dc9ffc677f5.lease',<br =
class=3D"m_-7375313093972664498gmail_msg">'imageID': =
u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'},
'diskType': =
'file',<br
class=3D"m_-7375313093972664498gmail_msg">'format': 'cow', =
'cache': 'none', u'volumeID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'30fd46c9-c738-4b13-aeca-<wbr
=
class=3D"">3dc9ffc677f5', u'imageID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'7ae9b3f7-3507-4469-a080-<wbr
=
class=3D"">d0944d0ab753', u'poolID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'8b6303b3-79c6-4633-ae21-<wbr
=
class=3D"">71b15ed00675', u'device': 'disk',
'path':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/8b6303b3-<wb=
r class=3D"">79c6-4633-ae21-71b15ed00675/<wbr =
class=3D"">6affd8c3-2c51-4cd1-8300-<wbr =
class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr =
class=3D"">3507-4469-a080-d0944d0ab753/<wbr =
class=3D"">30fd46c9-c738-4b13-aeca-<wbr
class=3D"">3dc9ffc677f5',<br =
class=3D"m_-7375313093972664498gmail_msg">'propagateErrors':
u'off', =
'volumeChain': [{'domainID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr
=
class=3D"">bfbbb14edbe9', 'volType': 'path',<br =
class=3D"m_-7375313093972664498gmail_msg">'leaseOffset': 0,
'path':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/6756eb05-6803-<wbr =
class=3D"">42a7-a3a2-10233bf2ca8d',<br =
class=3D"m_-7375313093972664498gmail_msg">'volumeID': =
u'6756eb05-6803-42a7-a3a2-<wbr class=3D"">10233bf2ca8d',
'leasePath':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/6756eb05-6803-<wbr =
class=3D"">42a7-a3a2-10233bf2ca8d.lease',<br =
class=3D"m_-7375313093972664498gmail_msg">'imageID': =
u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'}, =
{'domainID':<br =
class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr
=
class=3D"">bfbbb14edbe9', 'volType': 'path',<br =
class=3D"m_-7375313093972664498gmail_msg">'leaseOffset': 0,
'path':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/30fd46c9-c738-<wbr =
class=3D"">4b13-aeca-3dc9ffc677f5',<br =
class=3D"m_-7375313093972664498gmail_msg">'volumeID': =
u'30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5',
'leasePath':<br =
class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr
=
class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a =
href=3D"http://j2noc.com"
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr=
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">7ae9b3f7-3507-4469-a080-<wbr =
class=3D"">d0944d0ab753/30fd46c9-c738-<wbr =
class=3D"">4b13-aeca-3dc9ffc677f5.lease',<br =
class=3D"m_-7375313093972664498gmail_msg">'imageID': =
u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'}]}
(vm:3594)<br =
class=3D"m_-7375313093972664498gmail_msg">Traceback (most recent call =
last):<br class=3D"m_-7375313093972664498gmail_msg">
File =
"/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart<br =
class=3D"m_-7375313093972664498gmail_msg"> =
self._startDriveReplication(<wbr class=3D"">drive)<br =
class=3D"m_-7375313093972664498gmail_msg"> File =
"/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication<br =
class=3D"m_-7375313093972664498gmail_msg"> =
self._dom.blockCopy(<a href=3D"http://drive.name/" =
rel=3D"noreferrer" class=3D"m_-7375313093972664498gmail_msg" =
target=3D"_blank">drive.<wbr class=3D"">name</a>,
destxml, =
flags=3Dflags)<br class=3D"m_-7375313093972664498gmail_msg"> =
File "/usr/lib/python2.7/site-<wbr =
class=3D"">packages/vdsm/virt/virdomain.<wbr
class=3D"">py", line<br =
class=3D"m_-7375313093972664498gmail_msg">69, in f<br =
class=3D"m_-7375313093972664498gmail_msg"> ret
=3D =
attr(*args, **kwargs)<br
class=3D"m_-7375313093972664498gmail_msg"> =
File "/usr/lib/python2.7/site-<wbr
class=3D"">packages/vdsm/<wbr =
class=3D"">libvirtconnection.py",<br =
class=3D"m_-7375313093972664498gmail_msg">line 123, in wrapper<br =
class=3D"m_-7375313093972664498gmail_msg"> ret
=3D =
f(*args, **kwargs)<br class=3D"m_-7375313093972664498gmail_msg">
=
File "/usr/lib/python2.7/site-<wbr =
class=3D"">packages/vdsm/utils.py", line 941, in<br =
class=3D"m_-7375313093972664498gmail_msg">wrapper<br =
class=3D"m_-7375313093972664498gmail_msg">
return =
func(inst, *args, **kwargs)<br =
class=3D"m_-7375313093972664498gmail_msg"> File =
"/usr/lib64/python2.7/site-<wbr class=3D"">packages/libvirt.py",
line =
684, in<br class=3D"m_-7375313093972664498gmail_msg">blockCopy<br =
class=3D"m_-7375313093972664498gmail_msg"> if
ret =3D=3D=
-1: raise libvirtError ('virDomainBlockCopy() failed',<br =
class=3D"m_-7375313093972664498gmail_msg">dom=3Dself)<br =
class=3D"m_-7375313093972664498gmail_msg">libvirtError: internal error: =
unable to execute QEMU command<br =
class=3D"m_-7375313093972664498gmail_msg">'drive-mirror': Could not
=
open<br =
class=3D"m_-7375313093972664498gmail_msg">'/rhev/data-center/8b6303b3-<wbr=
class=3D"">79c6-4633-ae21-71b15ed00675/<wbr =
class=3D"">6affd8c3-2c51-4cd1-8300-<wbr =
class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr =
class=3D"">3507-4469-a080-d0944d0ab753/<wbr =
class=3D"">30fd46c9-c738-4b13-aeca-<wbr
class=3D"">3dc9ffc677f5':<br =
class=3D"m_-7375313093972664498gmail_msg">Permission denied<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">[root@ovirt1 test vdsm]# ls =
-l<br =
class=3D"m_-7375313093972664498gmail_msg">/rhev/data-center/8b6303b3-<wbr
=
class=3D"">79c6-4633-ae21-71b15ed00675/<wbr =
class=3D"">6affd8c3-2c51-4cd1-8300-<wbr =
class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr =
class=3D"">3507-4469-a080-d0944d0ab753/<wbr =
class=3D"">30fd46c9-c738-4b13-aeca-<wbr
class=3D"">3dc9ffc677f5<br =
class=3D"m_-7375313093972664498gmail_msg">-rw-rw---- 2 vdsm kvm 197120 =
Apr 6 13:29<br =
class=3D"m_-7375313093972664498gmail_msg">/rhev/data-center/8b6303b3-<wbr
=
class=3D"">79c6-4633-ae21-71b15ed00675/<wbr =
class=3D"">6affd8c3-2c51-4cd1-8300-<wbr =
class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr =
class=3D"">3507-4469-a080-d0944d0ab753/<wbr =
class=3D"">30fd46c9-c738-4b13-aeca-<wbr
class=3D"">3dc9ffc677f5<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">Then if I try and rerun it it =
says, even though move failed:<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">2017-04-06 13:49:27,197 =
INFO (jsonrpc/1) [dispatcher] Run and protect:<br =
class=3D"m_-7375313093972664498gmail_msg">getAllTasksStatuses, Return =
response: {'allT<br
class=3D"m_-7375313093972664498gmail_msg">asksStatus':=
{'078d962c-e682-40f9-a177-<wbr class=3D"">2a8b479a7d8b':
{'code': =
212,<br class=3D"m_-7375313093972664498gmail_msg">'message':
'Volume =
already exists', 'taskState':<br =
class=3D"m_-7375313093972664498gmail_msg"> <span =
class=3D"Apple-converted-space"> </span>'finished',
'taskResult': =
'cleanSuccess', 'taskID':<br =
class=3D"m_-7375313093972664498gmail_msg">'078d962c-e682-40f9-a177-<wbr
=
class=3D"">2a8b479a7d8b'}}} (logUtils:52)<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">So now I have to clean up the =
disks that it failed to move so I can<br =
class=3D"m_-7375313093972664498gmail_msg">migrate the VM and then move =
the disk again.<br class=3D"m_-7375313093972664498gmail_msg">Or so it =
seems.<br class=3D"m_-7375313093972664498gmail_msg">Failed move disks do
=
exist in new location, even though it "failed".<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">vdsm.log attached.<br =
class=3D"m_-7375313093972664498gmail_msg"><br =
class=3D"m_-7375313093972664498gmail_msg">ovirt-engine-tools-4.1.0.4-1.<wb=
r class=3D"">el7.centos.noarch<br =
class=3D"m_-7375313093972664498gmail_msg">vdsm-4.19.4-1.el7.centos.x86_<wb=
r class=3D"">64<br =
class=3D"m_-7375313093972664498gmail_msg"></blockquote><div
class=3D""><br=
class=3D""></div><div class=3D"">Hi
Bill,</div><div class=3D""><br =
class=3D""></div><div class=3D"">Does it work after
setting selinux to =
permissive? (setenforce 0)</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Can you share output of:</div><div
class=3D""><br =
class=3D""></div><div class=3D"">ps -efZ | grep
vm-name<br =
class=3D""></div><div class=3D"">(filter the specific
vm) <br =
class=3D""></div><div class=3D""><br
class=3D""></div><div class=3D"">ls =
-lhZ /rhev/data-center/mnt</div><div class=3D""><br
class=3D""></div><div =
class=3D"">ls -lhZ /rhev/data-center/mnt/gluster-<wbr =
class=3D"">server:_path/sd_id/images/img_<wbr =
class=3D"">id/vol_id</div><div class=3D"">(assuming the
volume was not =
deleted after the operation).</div><div class=3D""><br =
class=3D""></div><div class=3D"">If the volume is not
deleted after the =
failed move disk operation, this is likely</div><div class=3D"">a
bug, =
please file a bug for this.</div><div class=3D""><br
class=3D""></div><div=
class=3D"">The actual failure may be gluster configuration issue, or =
selinux related bug.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">Nir</div><div class=3D""> <br =
class=3D""></div></div></div></blockquote><br =
class=3D""></div></div>SELinux =
status: &=
nbsp; disabled<br =
class=3D""></div></blockquote><div
class=3D""><br class=3D""></div><div =
class=3D"">This is a less tested configuration. We usually run with =
selinux
enabled.</div></div></div></div></div></blockquote><div><br
=
class=3D""></div>Disabled specifically is untested. And doesn=E2=80=99t
=
work.</div><div>If you for whatever reason don=E2=80=99t want it, set it =
to Permissive, but don=E2=80=99t disable it</div><div><br =
class=3D""></div><div><br
class=3D""><blockquote type=3D"cite" =
class=3D""><div class=3D""><div dir=3D"ltr"
style=3D"font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; orphans: auto; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: =
0px;" class=3D""><div class=3D"gmail_extra"><div =
class=3D"gmail_quote"><div class=3D"">Y.</div><div
=
class=3D""> </div><blockquote
class=3D"gmail_quote" style=3D"margin: =
0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, =
204, 204); border-left-style: solid; padding-left: 1ex;"><div =
bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""><br =
class=3D"">[root@ovirt1 test images]# ps -efZ | grep darmaster<br =
class=3D"">- &n=
bsp; &nbs=
p; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>root &=
nbsp; 5272 1 6 Mar17 =
? 1-09:08:58 =
/usr/libexec/qemu-kvm -name guest=3D<a =
href=3D"http://darmaster1.test.j2noc.com/" target=3D"_blank" =
class=3D"">darmaster1.test.j2noc.<wbr
class=3D"">com</a>,debug-threads=3Do=
n -S -object secret,id=3DmasterKey0,format=3D<wbr =
class=3D"">raw,file=3D/var/lib/libvirt/<wbr =
class=3D"">qemu/domain-13-darmaster1.<wbr =
class=3D"">test.j2no/master-key.aes -machine =
pc-i440fx-rhel7.3.0,accel=3Dkvm,<wbr class=3D"">usb=3Doff -cpu Nehalem =
-m size=3D1048576k,slots=3D16,maxmem=3D<wbr class=3D"">4194304k =
-realtime mlock=3Doff -smp 1,maxcpus=3D16,sockets=3D16,cores=3D<wbr =
class=3D"">1,threads=3D1 -numa node,nodeid=3D0,cpus=3D0,mem=3D1024 -uuid =
73368460-92e1-4c9e-a162-<wbr class=3D"">399304f1c462 -smbios =
type=3D1,manufacturer=3DoVirt,<wbr class=3D"">product=3DoVirt =
Node,version=3D7-3.1611.el7.<wbr =
class=3D"">centos,serial=3D30343536-3138-<wbr =
class=3D"">584D-5134-343430313833,uuid=3D<wbr =
class=3D"">73368460-92e1-4c9e-a162-<wbr
class=3D"">399304f1c462 =
-no-user-config -nodefaults -chardev socket,id=3Dcharmonitor,path=3D/<wbr =
class=3D"">var/lib/libvirt/qemu/domain-<wbr =
class=3D"">13-darmaster1.test.j2no/<wbr =
class=3D"">monitor.sock,server,nowait -mon
chardev=3Dcharmonitor,id=3D<wbr=
class=3D"">monitor,mode=3Dcontrol -rtc base=3D2017-03-17T23:57:54,<wbr =
class=3D"">driftfix=3Dslew -global kvm-pit.lost_tick_policy=3D<wbr =
class=3D"">discard -no-hpet -no-shutdown -boot strict=3Don -device =
piix3-usb-uhci,id=3Dusb,bus=3Dpci.<wbr class=3D"">0,addr=3D0x1.0x2 =
-device virtio-scsi-pci,id=3Dscsi0,bus=3D<wbr class=3D"">pci.0,addr=3D0x4
=
-device virtio-serial-pci,id=3Dvirtio-<wbr =
class=3D"">serial0,max_ports=3D16,bus=3Dpci.<wbr
class=3D"">0,addr=3D0x5 =
-drive if=3Dnone,id=3Ddrive-ide0-1-0,<wbr class=3D"">readonly=3Don =
-device ide-cd,bus=3Dide.1,unit=3D0,drive=3D<wbr =
class=3D"">drive-ide0-1-0,id=3Dide0-1-0 -drive =
file=3D/rhev/data-center/<wbr class=3D"">8b6303b3-79c6-4633-ae21-<wbr
=
class=3D"">71b15ed00675/7e566f55-e060-<wbr =
class=3D"">47b7-bfa4-ac3c48d70dda/images/<wbr =
class=3D"">33db5688-dafe-40ab-9dd0-<wbr =
class=3D"">a826a90c3793/38de110d-464c-<wbr =
class=3D"">4735-97ba-3d623ee1a1b6,format=3D<wbr =
class=3D"">raw,if=3Dnone,id=3Ddrive-virtio-<wbr =
class=3D"">disk0,serial=3D33db5688-dafe-<wbr =
class=3D"">40ab-9dd0-a826a90c3793,cache=3D<wbr =
class=3D"">none,werror=3Dstop,rerror=3Dstop,<wbr
class=3D"">aio=3Dthreads =
-device virtio-blk-pci,scsi=3Doff,bus=3D<wbr =
class=3D"">pci.0,addr=3D0x6,drive=3Ddrive-<wbr =
class=3D"">virtio-disk0,id=3Dvirtio-disk0,<wbr
class=3D"">bootindex=3D1 =
-netdev tap,fd=3D38,id=3Dhostnet0,vhost=3D<wbr class=3D"">on,vhostfd=3D40
=
-device virtio-net-pci,netdev=3D<wbr =
class=3D"">hostnet0,id=3Dnet0,mac=3D00:1a:4a:<wbr =
class=3D"">16:01:69,bus=3Dpci.0,addr=3D0x3 -chardev =
socket,id=3Dcharchannel0,path=3D/<wbr =
class=3D"">var/lib/libvirt/qemu/channels/<wbr =
class=3D"">73368460-92e1-4c9e-a162-<wbr =
class=3D"">399304f1c462.com.redhat.rhevm.<wbr =
class=3D"">vdsm,server,nowait -device virtserialport,bus=3Dvirtio-<wbr =
class=3D"">serial0.0,nr=3D1,chardev=3D<wbr =
class=3D"">charchannel0,id=3Dchannel0,name=3D<wbr =
class=3D"">com.redhat.rhevm.vdsm -chardev =
socket,id=3Dcharchannel1,path=3D/<wbr =
class=3D"">var/lib/libvirt/qemu/channels/<wbr =
class=3D"">73368460-92e1-4c9e-a162-<wbr =
class=3D"">399304f1c462.org.qemu.guest_<wbr =
class=3D"">agent.0,server,nowait -device virtserialport,bus=3Dvirtio-<wbr
=
class=3D"">serial0.0,nr=3D2,chardev=3D<wbr =
class=3D"">charchannel1,id=3Dchannel1,name=3D<wbr =
class=3D"">org.qemu.guest_agent.0 -device =
usb-tablet,id=3Dinput0,bus=3Dusb.<wbr class=3D"">0,port=3D1 -vnc<span
=
class=3D"Apple-converted-space"> </span><a =
href=3D"http://10.176.30.96:6/" target=3D"_blank" =
class=3D"">10.176.30.96:6</a>,password -k en-us -device =
VGA,id=3Dvideo0,vgamem_mb=3D16,<wbr class=3D"">bus=3Dpci.0,addr=3D0x2 =
-msg timestamp=3Don<br class=3D""><br
class=3D"">[root@ovirt1 test ~]# =
ls -lhZ /rhev/data-center/mnt<br class=3D"">drwxr-xr-x vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>glusterSD<br =
class=3D"">drwxr-xr-x vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>netappqa3:_vol_cloud__images_=
<wbr class=3D"">ovirt__QA__ISOs<br class=3D"">drwxr-xr-x
vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>netappqa3:_vol_cloud__<wbr
=
class=3D"">storage1_ovirt__qa__inside<br
class=3D"">drwxr-xr-x vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt1-ks.test.j2noc.com" =
class=3D"">ovirt1-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs1<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt2-ks.test.j2noc.com" =
class=3D"">ovirt2-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs2<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt2-ks.test.j2noc.com" =
class=3D"">ovirt2-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs-2<br class=3D"">drwxr-xr-x vdsm
kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt3-ks.test.j2noc.com" =
class=3D"">ovirt3-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt4-ks.test.j2noc.com" =
class=3D"">ovirt4-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt5-ks.test.j2noc.com" =
class=3D"">ovirt5-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt6-ks.test.j2noc.com" =
class=3D"">ovirt6-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span><a =
href=3D"http://ovirt7-ks.test.j2noc.com" =
class=3D"">ovirt7-ks.test.j2noc.com</a>:_<wbr =
class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>qagenfil1-nfs1:_ovirt__inside=
_<wbr class=3D"">Export<br class=3D"">drwxr-xr-x vdsm kvm
=
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>qagenfil1-nfs1:_ovirt__inside=
_<wbr class=3D"">images<br class=3D""><br
class=3D""><br =
class=3D"">[root@ovirt1 test images]# ls -lhZa =
/rhev/data-center/mnt/<wbr class=3D"">glusterSD/ovirt1-ks.test.<wbr =
class=3D""><a href=3D"http://j2noc.com" =
class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr =
class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr =
class=3D"">33db5688-dafe-40ab-9dd0-<wbr
class=3D"">a826a90c3793<br =
class=3D"">drwxr-xr-x vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>.<br
class=3D"">drwxr-xr-x =
vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>..<br
class=3D"">-rw-rw---- =
vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr
=
class=3D"">07f5f76214f2<br class=3D"">-rw-rw---- vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr
=
class=3D"">07f5f76214f2.lease<br class=3D"">-rw-r--r-- vdsm
kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr
=
class=3D"">07f5f76214f2.meta<br class=3D"">-rw-rw---- vdsm
kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr
=
class=3D"">3d623ee1a1b6<br class=3D"">-rw-rw---- vdsm kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr
=
class=3D"">3d623ee1a1b6.lease<br class=3D"">-rw-r--r-- vdsm
kvm =
? &=
nbsp; &nb=
sp; <wbr
class=3D""> <span =
class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr
=
class=3D"">3d623ee1a1b6.meta<br class=3D""><br
class=3D""><br =
class=3D"">bug submitted: <span =
class=3D"Apple-converted-space"> </span><a =
class=3D"m_-7375313093972664498moz-txt-link-freetext" =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198" =
target=3D"_blank">https://bugzilla.redhat.com/<wbr =
class=3D"">show_bug.cgi?id=3D1440198</a><br
class=3D""><br class=3D""><br =
class=3D"">Thank you!<br class=3D""></div><br =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">Users mailing
list<br =
class=3D""><a href=3D"mailto:Users@ovirt.org" =
class=3D"">Users(a)ovirt.org</a><br class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users"
rel=3D"noreferrer" =
target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/users</a><br
class=3D""><br =
class=3D""></blockquote></div><br
class=3D""></div></div><span =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; float: none; display: inline =
!important;" =
class=3D"">_______________________________________________</span><br
=
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px;" class=3D""><span
style=3D"font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant-caps: =
normal; font-weight: normal; letter-spacing: normal; orphans: auto; =
text-align: start; text-indent: 0px; text-transform: none; white-space: =
normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
float: none; display: inline !important;" class=3D"">Users mailing =
list</span><br style=3D"font-family: Helvetica; font-size: 12px; =
font-style: normal; font-variant-caps: normal; font-weight: normal; =
letter-spacing: normal; orphans: auto; text-align: start; text-indent: =
0px; text-transform: none; white-space: normal; widows: auto; =
word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><a =
href=3D"mailto:Users@ovirt.org" style=3D"font-family: Helvetica; =
font-size: 12px; font-style: normal; font-variant-caps: normal; =
font-weight: normal; letter-spacing: normal; orphans: auto; text-align: =
start; text-indent: 0px; text-transform: none; white-space: normal; =
widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; =
-webkit-text-stroke-width: 0px;"
class=3D"">Users(a)ovirt.org</a><br =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px;" class=3D""><a =
href=3D"http://lists.ovirt.org/mailman/listinfo/users" =
style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; =
font-variant-caps: normal; font-weight: normal; letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" =
class=3D"">http://lists.ovirt.org/mailman/listinfo/users<...
uote></div><br class=3D""></body></html>=
--Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794--
2784
days inactive
2787
days old
4 comments
4 participants
participants (4)
-
Bill James -
Michal Skrivanek -
Nir Soffer -
Yaniv Kaul