moving disk from one storage domain to another
We are trying to convert our qa environment from local nfs to gluster. When I move a disk with a VM that is running on same server as the storage it fails. When I move a disk with VM running on a different system it works. VM running on same system as disk: 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm] (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start replication for vda to {u'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID': u'6affd8c3-2c 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file', 'format': 'cow', 'cache': 'none', u'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID': u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path': u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'propagateErrors': u'off', 'volumeChain': [{'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart self._startDriveReplication(drive) File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication self._dom.blockCopy(drive.name, destxml, flags=flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in blockCopy if ret == -1: raise libvirtError ('virDomainBlockCopy() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5': Permission denied [root@ovirt1 test vdsm]# ls -l /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5 -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29 /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5 Then if I try and rerun it it says, even though move failed: 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect: getAllTasksStatuses, Return response: {'allT asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212, 'message': 'Volume already exists', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52) So now I have to clean up the disks that it failed to move so I can migrate the VM and then move the disk again. Or so it seems. Failed move disks do exist in new location, even though it "failed". vdsm.log attached. ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james@j2.com> wrote:
We are trying to convert our qa environment from local nfs to gluster. When I move a disk with a VM that is running on same server as the storage it fails. When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm] (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start replication for vda to {u'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID': u'6affd8c3-2c 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file', 'format': 'cow', 'cache': 'none', u'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID': u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path':
u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'propagateErrors': u'off', 'volumeChain': [{'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart self._startDriveReplication(drive) File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication self._dom.blockCopy(drive.name, destxml, flags=flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in blockCopy if ret == -1: raise libvirtError ('virDomainBlockCopy() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open
'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5': Permission denied
[root@ovirt1 test vdsm]# ls -l
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5 -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect: getAllTasksStatuses, Return response: {'allT asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212, 'message': 'Volume already exists', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can migrate the VM and then move the disk again. Or so it seems. Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64
Hi Bill, Does it work after setting selinux to permissive? (setenforce 0) Can you share output of: ps -efZ | grep vm-name (filter the specific vm) ls -lhZ /rhev/data-center/mnt ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id (assuming the volume was not deleted after the operation). If the volume is not deleted after the failed move disk operation, this is likely a bug, please file a bug for this. The actual failure may be gluster configuration issue, or selinux related bug. Nir
--------------705BA3C205C526559706549D Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit On 4/7/17 12:52 AM, Nir Soffer wrote:
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james@j2.com <mailto:bill.james@j2.com>> wrote:
We are trying to convert our qa environment from local nfs to gluster. When I move a disk with a VM that is running on same server as the storage it fails. When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm] (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start replication for vda to {u'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID': u'6affd8c3-2c 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file', 'format': 'cow', 'cache': 'none', u'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID': u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path': u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'propagateErrors': u'off', 'volumeChain': [{'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart self._startDriveReplication(drive) File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication self._dom.blockCopy(drive.name <http://drive.name>, destxml, flags=flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in blockCopy if ret == -1: raise libvirtError ('virDomainBlockCopy() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5': Permission denied
[root@ovirt1 test vdsm]# ls -l /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5 -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29 /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect: getAllTasksStatuses, Return response: {'allT asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212, 'message': 'Volume already exists', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can migrate the VM and then move the disk again. Or so it seems. Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64
Hi Bill,
Does it work after setting selinux to permissive? (setenforce 0)
Can you share output of:
ps -efZ | grep vm-name (filter the specific vm)
ls -lhZ /rhev/data-center/mnt
ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id (assuming the volume was not deleted after the operation).
If the volume is not deleted after the failed move disk operation, this is likely a bug, please file a bug for this.
The actual failure may be gluster configuration issue, or selinux related bug.
Nir
SELinux status: disabled [root@ovirt1 test images]# ps -efZ | grep darmaster - root 5272 1 6 Mar17 ? 1-09:08:58 /usr/libexec/qemu-kvm -name guest=darmaster1.test.j2noc.com,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=30343536-3138-584D-5134-343430313833,uuid=73368460-92e1-4c9e-a162-399304f1c462 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-17T23:57:54,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.176.30.96:6,password -k en-us -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg timestamp=on [root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt drwxr-xr-x vdsm kvm ? glusterSD drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__images_ovirt__QA__ISOs drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__storage1_ovirt__qa__inside drwxr-xr-x vdsm kvm ? ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1 drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2 drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2 drwxr-xr-x vdsm kvm ? ovirt3-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt4-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt5-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt6-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt7-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_Export drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_images [root@ovirt1 test images]# ls -lhZa /rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793 drwxr-xr-x vdsm kvm ? . drwxr-xr-x vdsm kvm ? .. -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2 -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.lease -rw-r--r-- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.meta -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6 -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.lease -rw-r--r-- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.meta bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=1440198 Thank you! --------------705BA3C205C526559706549D Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 8bit <html> <head> <meta content="text/html; charset=utf-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> <br> <br> <div class="moz-cite-prefix">On 4/7/17 12:52 AM, Nir Soffer wrote:<br> </div> <blockquote cite="mid:CAMRbyyvG1X4-npxFhsJd6dgmzB6mqr+JoFN08U2aKVFsBZckgA@mail.gmail.com" type="cite"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <div dir="ltr"> <div class="gmail_quote"> <div dir="ltr">On Fri, Apr 7, 2017 at 2:40 AM Bill James <<a moz-do-not-send="true" href="mailto:bill.james@j2.com">bill.james@j2.com</a>> wrote:<br> </div> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">We are trying to convert our qa environment from local nfs to gluster.<br class="gmail_msg"> When I move a disk with a VM that is running on same server as the<br class="gmail_msg"> storage it fails.<br class="gmail_msg"> When I move a disk with VM running on a different system it works.<br class="gmail_msg"> <br class="gmail_msg"> VM running on same system as disk:<br class="gmail_msg"> <br class="gmail_msg"> 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]<br class="gmail_msg"> (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start<br class="gmail_msg"> replication for vda to {u'domainID':<br class="gmail_msg"> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID':<br class="gmail_msg"> u'6affd8c3-2c<br class="gmail_msg"> 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg"> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br class="gmail_msg"> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file',<br class="gmail_msg"> 'format': 'cow', 'cache': 'none', u'volumeID':<br class="gmail_msg"> u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':<br class="gmail_msg"> u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':<br class="gmail_msg"> u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path':<br class="gmail_msg"> u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg"> 'propagateErrors': u'off', 'volumeChain': [{'domainID':<br class="gmail_msg"> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',<br class="gmail_msg"> 'leaseOffset': 0, 'path':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',<br class="gmail_msg"> 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',<br class="gmail_msg"> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID':<br class="gmail_msg"> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',<br class="gmail_msg"> 'leaseOffset': 0, 'path':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',<br class="gmail_msg"> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':<br class="gmail_msg"> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',<br class="gmail_msg"> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)<br class="gmail_msg"> Traceback (most recent call last):<br class="gmail_msg"> File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart<br class="gmail_msg"> self._startDriveReplication(drive)<br class="gmail_msg"> File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication<br class="gmail_msg"> self._dom.blockCopy(<a moz-do-not-send="true" href="http://drive.name" rel="noreferrer" class="gmail_msg" target="_blank">drive.name</a>, destxml, flags=flags)<br class="gmail_msg"> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line<br class="gmail_msg"> 69, in f<br class="gmail_msg"> ret = attr(*args, **kwargs)<br class="gmail_msg"> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",<br class="gmail_msg"> line 123, in wrapper<br class="gmail_msg"> ret = f(*args, **kwargs)<br class="gmail_msg"> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in<br class="gmail_msg"> wrapper<br class="gmail_msg"> return func(inst, *args, **kwargs)<br class="gmail_msg"> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in<br class="gmail_msg"> blockCopy<br class="gmail_msg"> if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',<br class="gmail_msg"> dom=self)<br class="gmail_msg"> libvirtError: internal error: unable to execute QEMU command<br class="gmail_msg"> 'drive-mirror': Could not open<br class="gmail_msg"> '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':<br class="gmail_msg"> Permission denied<br class="gmail_msg"> <br class="gmail_msg"> <br class="gmail_msg"> [root@ovirt1 test vdsm]# ls -l<br class="gmail_msg"> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br class="gmail_msg"> -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29<br class="gmail_msg"> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5<br class="gmail_msg"> <br class="gmail_msg"> <br class="gmail_msg"> <br class="gmail_msg"> Then if I try and rerun it it says, even though move failed:<br class="gmail_msg"> <br class="gmail_msg"> 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:<br class="gmail_msg"> getAllTasksStatuses, Return response: {'allT<br class="gmail_msg"> asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212,<br class="gmail_msg"> 'message': 'Volume already exists', 'taskState':<br class="gmail_msg"> 'finished', 'taskResult': 'cleanSuccess', 'taskID':<br class="gmail_msg"> '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)<br class="gmail_msg"> <br class="gmail_msg"> <br class="gmail_msg"> So now I have to clean up the disks that it failed to move so I can<br class="gmail_msg"> migrate the VM and then move the disk again.<br class="gmail_msg"> Or so it seems.<br class="gmail_msg"> Failed move disks do exist in new location, even though it "failed".<br class="gmail_msg"> <br class="gmail_msg"> vdsm.log attached.<br class="gmail_msg"> <br class="gmail_msg"> ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch<br class="gmail_msg"> vdsm-4.19.4-1.el7.centos.x86_64<br class="gmail_msg"> </blockquote> <div><br> </div> <div>Hi Bill,</div> <div><br> </div> <div>Does it work after setting selinux to permissive? (setenforce 0)</div> <div><br> </div> <div>Can you share output of:</div> <div><br> </div> <div>ps -efZ | grep vm-name<br> </div> <div>(filter the specific vm) <br> </div> <div><br> </div> <div>ls -lhZ /rhev/data-center/mnt</div> <div><br> </div> <div>ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id</div> <div>(assuming the volume was not deleted after the operation).</div> <div><br> </div> <div>If the volume is not deleted after the failed move disk operation, this is likely</div> <div>a bug, please file a bug for this.</div> <div><br> </div> <div>The actual failure may be gluster configuration issue, or selinux related bug.</div> <div><br> </div> <div>Nir</div> <div> <br> </div> </div> </div> </blockquote> <br> SELinux status: disabled<br> <br> [root@ovirt1 test images]# ps -efZ | grep darmaster<br> - root 5272 1 6 Mar17 ? 1-09:08:58 /usr/libexec/qemu-kvm -name guest=darmaster1.test.j2noc.com,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.centos,serial=30343536-3138-584D-5134-343430313833,uuid=73368460-92e1-4c9e-a162-399304f1c462 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-darmaster1.test.j2no/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-17T23:57:54,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.176.30.96:6,password -k en-us -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg timestamp=on<br> <br> [root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt<br> drwxr-xr-x vdsm kvm ? glusterSD<br> drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__images_ovirt__QA__ISOs<br> drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__storage1_ovirt__qa__inside<br> drwxr-xr-x vdsm kvm ? ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1<br> drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2<br> drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2<br> drwxr-xr-x vdsm kvm ? ovirt3-ks.test.j2noc.com:_ovirt-store_nfs<br> drwxr-xr-x vdsm kvm ? ovirt4-ks.test.j2noc.com:_ovirt-store_nfs<br> drwxr-xr-x vdsm kvm ? ovirt5-ks.test.j2noc.com:_ovirt-store_nfs<br> drwxr-xr-x vdsm kvm ? ovirt6-ks.test.j2noc.com:_ovirt-store_nfs<br> drwxr-xr-x vdsm kvm ? ovirt7-ks.test.j2noc.com:_ovirt-store_nfs<br> drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_Export<br> drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_images<br> <br> <br> [root@ovirt1 test images]# ls -lhZa /rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793<br> drwxr-xr-x vdsm kvm ? .<br> drwxr-xr-x vdsm kvm ? ..<br> -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2<br> -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.lease<br> -rw-r--r-- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.meta<br> -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6<br> -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.lease<br> -rw-r--r-- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.meta<br> <br> <br> bug submitted: <a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1440198">https://bugzilla.redhat.com/show_bug.cgi?id=1440198</a><br> <br> <br> Thank you!<br> </body> </html> --------------705BA3C205C526559706549D--
On Fri, Apr 7, 2017 at 5:29 PM, Bill James <bill.james@j2.com> wrote:
On 4/7/17 12:52 AM, Nir Soffer wrote:
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james@j2.com> wrote:
We are trying to convert our qa environment from local nfs to gluster. When I move a disk with a VM that is running on same server as the storage it fails. When I move a disk with VM running on a different system it works.
VM running on same system as disk:
2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm] (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start replication for vda to {u'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID': u'6affd8c3-2c 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file', 'format': 'cow', 'cache': 'none', u'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID': u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path': u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/ 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7- 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'propagateErrors': u'off', 'volumeChain': [{'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com: _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080- d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart self._startDriveReplication(drive) File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication self._dom.blockCopy(drive.name, destxml, flags=flags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 69, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in blockCopy if ret == -1: raise libvirtError ('virDomainBlockCopy() failed', dom=self) libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/ 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7- 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5': Permission denied
[root@ovirt1 test vdsm]# ls -l /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/ 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7- 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5 -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29 /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/ 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7- 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
Then if I try and rerun it it says, even though move failed:
2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect: getAllTasksStatuses, Return response: {'allT asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212, 'message': 'Volume already exists', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
So now I have to clean up the disks that it failed to move so I can migrate the VM and then move the disk again. Or so it seems. Failed move disks do exist in new location, even though it "failed".
vdsm.log attached.
ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64
Hi Bill,
Does it work after setting selinux to permissive? (setenforce 0)
Can you share output of:
ps -efZ | grep vm-name (filter the specific vm)
ls -lhZ /rhev/data-center/mnt
ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_ id/vol_id (assuming the volume was not deleted after the operation).
If the volume is not deleted after the failed move disk operation, this is likely a bug, please file a bug for this.
The actual failure may be gluster configuration issue, or selinux related bug.
Nir
SELinux status: disabled
This is a less tested configuration. We usually run with selinux enabled. Y.
[root@ovirt1 test images]# ps -efZ | grep darmaster - root 5272 1 6 Mar17 ? 1-09:08:58 /usr/libexec/qemu-kvm -name guest=darmaster1.test.j2noc.com,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/ qemu/domain-13-darmaster1.test.j2no/master-key.aes -machine pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7. centos,serial=30343536-3138-584D-5134-343430313833,uuid= 73368460-92e1-4c9e-a162-399304f1c462 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain- 13-darmaster1.test.j2no/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-17T23:57:54,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-1-0,readonly=on -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060- 47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0- a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format= raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe- 40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive- virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev= hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/ 73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev= charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/ 73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev= charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.176.30.96:6,password -k en-us -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg timestamp=on
[root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt drwxr-xr-x vdsm kvm ? glusterSD drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__images_ovirt__QA__ISOs drwxr-xr-x vdsm kvm ? netappqa3:_vol_cloud__storage1_ovirt__qa__inside drwxr-xr-x vdsm kvm ? ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1 drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2 drwxr-xr-x vdsm kvm ? ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2 drwxr-xr-x vdsm kvm ? ovirt3-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt4-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt5-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt6-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? ovirt7-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_Export drwxr-xr-x vdsm kvm ? qagenfil1-nfs1:_ovirt__inside_images
[root@ovirt1 test images]# ls -lhZa /rhev/data-center/mnt/ glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51- 4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793 drwxr-xr-x vdsm kvm ? . drwxr-xr-x vdsm kvm ? .. -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2 -rw-rw---- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.lease -rw-r--r-- vdsm kvm ? 33c04305-efbe-418a-b42c-07f5f76214f2.meta -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6 -rw-rw---- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.lease -rw-r--r-- vdsm kvm ? 38de110d-464c-4735-97ba-3d623ee1a1b6.meta
bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=1440198
Thank you!
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
On 9 Apr 2017, at 09:16, Yaniv Kaul <ykaul@redhat.com> wrote: =20 =20 =20 On Fri, Apr 7, 2017 at 5:29 PM, Bill James <bill.james@j2.com = <mailto:bill.james@j2.com>> wrote: =20 =20 On 4/7/17 12:52 AM, Nir Soffer wrote:
On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james@j2.com = <mailto:bill.james@j2.com>> wrote: We are trying to convert our qa environment from local nfs to = gluster. When I move a disk with a VM that is running on same server as the storage it fails. When I move a disk with VM running on a different system it works. =20 VM running on same system as disk: =20 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm] (vmId=3D'e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start replication for vda to {u'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID': u'6affd8c3-2c 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, = 'path': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f= d46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f= d46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': = 'file', 'format': 'cow', 'cache': 'none', u'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID': u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path': = u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd= 1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c= 738-4b13-aeca-3dc9ffc677f5', 'propagateErrors': u'off', 'volumeChain': [{'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/675= 6eb05-6803-42a7-a3a2-10233bf2ca8d', 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/675= 6eb05-6803-42a7-a3a2-10233bf2ca8d.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID': u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f= d46c9-c738-4b13-aeca-3dc9ffc677f5', 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath': = u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2= c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30f= d46c9-c738-4b13-aeca-3dc9ffc677f5.lease', 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594) Traceback (most recent call last): File "/usr/share/vdsm/virt/vm.py", line 3588, in = diskReplicateStart self._startDriveReplication(drive) File "/usr/share/vdsm/virt/vm.py", line 3713, in = _startDriveReplication self._dom.blockCopy(drive.name <http://drive.name/>, destxml, = flags=3Dflags) File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", =
69, in f ret =3D attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 123, in wrapper ret =3D f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, = in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in blockCopy if ret =3D=3D -1: raise libvirtError ('virDomainBlockCopy() = failed', dom=3Dself) libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open = '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1= -8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c7= 38-4b13-aeca-3dc9ffc677f5': Permission denied =20 =20 [root@ovirt1 test vdsm]# ls -l = /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-= 8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c73= 8-4b13-aeca-3dc9ffc677f5 -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29 = /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/6affd8c3-2c51-4cd1-= 8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-d0944d0ab753/30fd46c9-c73= 8-4b13-aeca-3dc9ffc677f5 =20 =20 =20 Then if I try and rerun it it says, even though move failed: =20 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and =
getAllTasksStatuses, Return response: {'allT asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212, 'message': 'Volume already exists', 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52) =20 =20 So now I have to clean up the disks that it failed to move so I can migrate the VM and then move the disk again. Or so it seems. Failed move disks do exist in new location, even though it "failed". =20 vdsm.log attached. =20 ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch vdsm-4.19.4-1.el7.centos.x86_64 =20 Hi Bill, =20 Does it work after setting selinux to permissive? (setenforce 0) =20 Can you share output of: =20 ps -efZ | grep vm-name (filter the specific vm)=20 =20 ls -lhZ /rhev/data-center/mnt =20 ls -lhZ = /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_id/vol_id (assuming the volume was not deleted after the operation). =20 If the volume is not deleted after the failed move disk operation, =
--Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 line protect: this is likely
a bug, please file a bug for this. =20 The actual failure may be gluster configuration issue, or selinux = related bug. =20 Nir =20 =20 SELinux status: disabled =20 This is a less tested configuration. We usually run with selinux = enabled.
Y. =20 =20 [root@ovirt1 test images]# ps -efZ | grep darmaster - root 5272 1 6 Mar17 ? = 1-09:08:58 /usr/libexec/qemu-kvm -name guest=3Ddarmaster1.test.j2noc.com = <http://darmaster1.test.j2noc.com/>,debug-threads=3Don -S -object = secret,id=3DmasterKey0,format=3Draw,file=3D/var/lib/libvirt/qemu/domain-13= -darmaster1.test.j2no/master-key.aes -machine =
Disabled specifically is untested. And doesn=E2=80=99t work. If you for whatever reason don=E2=80=99t want it, set it to Permissive, = but don=E2=80=99t disable it pc-i440fx-rhel7.3.0,accel=3Dkvm,usb=3Doff -cpu Nehalem -m = size=3D1048576k,slots=3D16,maxmem=3D4194304k -realtime mlock=3Doff -smp = 1,maxcpus=3D16,sockets=3D16,cores=3D1,threads=3D1 -numa = node,nodeid=3D0,cpus=3D0,mem=3D1024 -uuid = 73368460-92e1-4c9e-a162-399304f1c462 -smbios = type=3D1,manufacturer=3DoVirt,product=3DoVirt = Node,version=3D7-3.1611.el7.centos,serial=3D30343536-3138-584D-5134-343430= 313833,uuid=3D73368460-92e1-4c9e-a162-399304f1c462 -no-user-config = -nodefaults -chardev = socket,id=3Dcharmonitor,path=3D/var/lib/libvirt/qemu/domain-13-darmaster1.= test.j2no/monitor.sock,server,nowait -mon = chardev=3Dcharmonitor,id=3Dmonitor,mode=3Dcontrol -rtc = base=3D2017-03-17T23:57:54,driftfix=3Dslew -global = kvm-pit.lost_tick_policy=3Ddiscard -no-hpet -no-shutdown -boot strict=3Don= -device piix3-usb-uhci,id=3Dusb,bus=3Dpci.0,addr=3D0x1.0x2 -device = virtio-scsi-pci,id=3Dscsi0,bus=3Dpci.0,addr=3D0x4 -device = virtio-serial-pci,id=3Dvirtio-serial0,max_ports=3D16,bus=3Dpci.0,addr=3D0x= 5 -drive if=3Dnone,id=3Ddrive-ide0-1-0,readonly=3Don -device = ide-cd,bus=3Dide.1,unit=3D0,drive=3Ddrive-ide0-1-0,id=3Dide0-1-0 -drive = file=3D/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e06= 0-47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-a826a90c3793/38de1= 10d-464c-4735-97ba-3d623ee1a1b6,format=3Draw,if=3Dnone,id=3Ddrive-virtio-d= isk0,serial=3D33db5688-dafe-40ab-9dd0-a826a90c3793,cache=3Dnone,werror=3Ds= top,rerror=3Dstop,aio=3Dthreads -device = virtio-blk-pci,scsi=3Doff,bus=3Dpci.0,addr=3D0x6,drive=3Ddrive-virtio-disk= 0,id=3Dvirtio-disk0,bootindex=3D1 -netdev = tap,fd=3D38,id=3Dhostnet0,vhost=3Don,vhostfd=3D40 -device = virtio-net-pci,netdev=3Dhostnet0,id=3Dnet0,mac=3D00:1a:4a:16:01:69,bus=3Dp= ci.0,addr=3D0x3 -chardev = socket,id=3Dcharchannel0,path=3D/var/lib/libvirt/qemu/channels/73368460-92= e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D1,chardev=3Dcharchannel0,id=3Dc= hannel0,name=3Dcom.redhat.rhevm.vdsm -chardev = socket,id=3Dcharchannel1,path=3D/var/lib/libvirt/qemu/channels/73368460-92= e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait -device = virtserialport,bus=3Dvirtio-serial0.0,nr=3D2,chardev=3Dcharchannel1,id=3Dc= hannel1,name=3Dorg.qemu.guest_agent.0 -device = usb-tablet,id=3Dinput0,bus=3Dusb.0,port=3D1 -vnc 10.176.30.96:6 = <http://10.176.30.96:6/>,password -k en-us -device = VGA,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpci.0,addr=3D0x2 -msg timestamp=3Don
=20 [root@ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt drwxr-xr-x vdsm kvm ? glusterSD drwxr-xr-x vdsm kvm ? = netappqa3:_vol_cloud__images_ovirt__QA__ISOs drwxr-xr-x vdsm kvm ? = netappqa3:_vol_cloud__storage1_ovirt__qa__inside drwxr-xr-x vdsm kvm ? = ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1 drwxr-xr-x vdsm kvm ? = ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2 drwxr-xr-x vdsm kvm ? = ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2 drwxr-xr-x vdsm kvm ? = ovirt3-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? = ovirt4-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? = ovirt5-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? = ovirt6-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? = ovirt7-ks.test.j2noc.com:_ovirt-store_nfs drwxr-xr-x vdsm kvm ? = qagenfil1-nfs1:_ovirt__inside_Export drwxr-xr-x vdsm kvm ? = qagenfil1-nfs1:_ovirt__inside_images =20 =20 [root@ovirt1 test images]# ls -lhZa = /rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c5= 1-4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793 drwxr-xr-x vdsm kvm ? . drwxr-xr-x vdsm kvm ? .. -rw-rw---- vdsm kvm ? = 33c04305-efbe-418a-b42c-07f5f76214f2 -rw-rw---- vdsm kvm ? = 33c04305-efbe-418a-b42c-07f5f76214f2.lease -rw-r--r-- vdsm kvm ? = 33c04305-efbe-418a-b42c-07f5f76214f2.meta -rw-rw---- vdsm kvm ? = 38de110d-464c-4735-97ba-3d623ee1a1b6 -rw-rw---- vdsm kvm ? = 38de110d-464c-4735-97ba-3d623ee1a1b6.lease -rw-r--r-- vdsm kvm ? = 38de110d-464c-4735-97ba-3d623ee1a1b6.meta =20 =20 bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198 = <https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198> =20 =20 Thank you! =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users>
--Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On 9 Apr 2017, at 09:16, Yaniv Kaul <<a = href=3D"mailto:ykaul@redhat.com" class=3D"">ykaul@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"gmail_extra"><br class=3D"Apple-interchange-newline"><br = class=3D""><div class=3D"gmail_quote">On Fri, Apr 7, 2017 at 5:29 PM, = Bill James<span class=3D"Apple-converted-space"> </span><span = dir=3D"ltr" class=3D""><<a href=3D"mailto:bill.james@j2.com" = target=3D"_blank" class=3D"">bill.james@j2.com</a>></span><span = class=3D"Apple-converted-space"> </span>wrote:<br = class=3D""><blockquote class=3D"gmail_quote" style=3D"margin: 0px 0px = 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, = 204); border-left-style: solid; padding-left: 1ex;"><div = bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""><div class=3D""><div = class=3D"h5"><br class=3D""><br class=3D""><div = class=3D"m_-7375313093972664498moz-cite-prefix">On 4/7/17 12:52 AM, Nir = Soffer wrote:<br class=3D""></div><blockquote type=3D"cite" = class=3D""><div dir=3D"ltr" class=3D""><div class=3D"gmail_quote"><div = dir=3D"ltr" class=3D"">On Fri, Apr 7, 2017 at 2:40 AM Bill James <<a = href=3D"mailto:bill.james@j2.com" target=3D"_blank" = class=3D"">bill.james@j2.com</a>> wrote:<br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin: 0px = 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, = 204); border-left-style: solid; padding-left: 1ex;">We are trying to = convert our qa environment from local nfs to gluster.<br = class=3D"m_-7375313093972664498gmail_msg">When I move a disk with a VM = that is running on same server as the<br = class=3D"m_-7375313093972664498gmail_msg">storage it fails.<br = class=3D"m_-7375313093972664498gmail_msg">When I move a disk with VM = running on a different system it works.<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">VM running on same system as = disk:<br class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">2017-04-06 13:31:00,588 ERROR = (jsonrpc/6) [virt.vm]<br = class=3D"m_-7375313093972664498gmail_msg">(vmId=3D'e598485a-dc74-43f7-<wbr= class=3D"">8447-e00ac44dae21') Unable to start<br = class=3D"m_-7375313093972664498gmail_msg">replication for vda to = {u'domainID':<br = class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9', 'volumeInfo': {'domainID':<br = class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c<br = class=3D"m_-7375313093972664498gmail_msg">51-4cd1-8300-bfbbb14edbe9', = 'volType': 'path', 'leaseOffset': 0, 'path':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/30fd46c9-c738-<wbr = class=3D"">4b13-aeca-3dc9ffc677f5',<br = class=3D"m_-7375313093972664498gmail_msg">'volumeID': = u'30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5', 'leasePath':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/30fd46c9-c738-<wbr = class=3D"">4b13-aeca-3dc9ffc677f5.lease',<br = class=3D"m_-7375313093972664498gmail_msg">'imageID': = u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'}, 'diskType': = 'file',<br class=3D"m_-7375313093972664498gmail_msg">'format': 'cow', = 'cache': 'none', u'volumeID':<br = class=3D"m_-7375313093972664498gmail_msg">u'30fd46c9-c738-4b13-aeca-<wbr = class=3D"">3dc9ffc677f5', u'imageID':<br = class=3D"m_-7375313093972664498gmail_msg">u'7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753', u'poolID':<br = class=3D"m_-7375313093972664498gmail_msg">u'8b6303b3-79c6-4633-ae21-<wbr = class=3D"">71b15ed00675', u'device': 'disk', 'path':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/8b6303b3-<wb= r class=3D"">79c6-4633-ae21-71b15ed00675/<wbr = class=3D"">6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr = class=3D"">3507-4469-a080-d0944d0ab753/<wbr = class=3D"">30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5',<br = class=3D"m_-7375313093972664498gmail_msg">'propagateErrors': u'off', = 'volumeChain': [{'domainID':<br = class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9', 'volType': 'path',<br = class=3D"m_-7375313093972664498gmail_msg">'leaseOffset': 0, 'path':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/6756eb05-6803-<wbr = class=3D"">42a7-a3a2-10233bf2ca8d',<br = class=3D"m_-7375313093972664498gmail_msg">'volumeID': = u'6756eb05-6803-42a7-a3a2-<wbr class=3D"">10233bf2ca8d', 'leasePath':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/6756eb05-6803-<wbr = class=3D"">42a7-a3a2-10233bf2ca8d.lease',<br = class=3D"m_-7375313093972664498gmail_msg">'imageID': = u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'}, = {'domainID':<br = class=3D"m_-7375313093972664498gmail_msg">u'6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9', 'volType': 'path',<br = class=3D"m_-7375313093972664498gmail_msg">'leaseOffset': 0, 'path':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/30fd46c9-c738-<wbr = class=3D"">4b13-aeca-3dc9ffc677f5',<br = class=3D"m_-7375313093972664498gmail_msg">'volumeID': = u'30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5', 'leasePath':<br = class=3D"m_-7375313093972664498gmail_msg">u'/rhev/data-center/mnt/<wbr = class=3D"">glusterSD/ovirt1-ks.test.<wbr class=3D""><a = href=3D"http://j2noc.com" class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr= class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">7ae9b3f7-3507-4469-a080-<wbr = class=3D"">d0944d0ab753/30fd46c9-c738-<wbr = class=3D"">4b13-aeca-3dc9ffc677f5.lease',<br = class=3D"m_-7375313093972664498gmail_msg">'imageID': = u'7ae9b3f7-3507-4469-a080-<wbr class=3D"">d0944d0ab753'}]} (vm:3594)<br = class=3D"m_-7375313093972664498gmail_msg">Traceback (most recent call = last):<br class=3D"m_-7375313093972664498gmail_msg"> File = "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart<br = class=3D"m_-7375313093972664498gmail_msg"> = self._startDriveReplication(<wbr class=3D"">drive)<br = class=3D"m_-7375313093972664498gmail_msg"> File = "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication<br = class=3D"m_-7375313093972664498gmail_msg"> = self._dom.blockCopy(<a href=3D"http://drive.name/" = rel=3D"noreferrer" class=3D"m_-7375313093972664498gmail_msg" = target=3D"_blank">drive.<wbr class=3D"">name</a>, destxml, = flags=3Dflags)<br class=3D"m_-7375313093972664498gmail_msg"> = File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/virt/virdomain.<wbr class=3D"">py", line<br = class=3D"m_-7375313093972664498gmail_msg">69, in f<br = class=3D"m_-7375313093972664498gmail_msg"> ret =3D = attr(*args, **kwargs)<br class=3D"m_-7375313093972664498gmail_msg"> = File "/usr/lib/python2.7/site-<wbr class=3D"">packages/vdsm/<wbr = class=3D"">libvirtconnection.py",<br = class=3D"m_-7375313093972664498gmail_msg">line 123, in wrapper<br = class=3D"m_-7375313093972664498gmail_msg"> ret =3D = f(*args, **kwargs)<br class=3D"m_-7375313093972664498gmail_msg"> = File "/usr/lib/python2.7/site-<wbr = class=3D"">packages/vdsm/utils.py", line 941, in<br = class=3D"m_-7375313093972664498gmail_msg">wrapper<br = class=3D"m_-7375313093972664498gmail_msg"> return = func(inst, *args, **kwargs)<br = class=3D"m_-7375313093972664498gmail_msg"> File = "/usr/lib64/python2.7/site-<wbr class=3D"">packages/libvirt.py", line = 684, in<br class=3D"m_-7375313093972664498gmail_msg">blockCopy<br = class=3D"m_-7375313093972664498gmail_msg"> if ret =3D=3D= -1: raise libvirtError ('virDomainBlockCopy() failed',<br = class=3D"m_-7375313093972664498gmail_msg">dom=3Dself)<br = class=3D"m_-7375313093972664498gmail_msg">libvirtError: internal error: = unable to execute QEMU command<br = class=3D"m_-7375313093972664498gmail_msg">'drive-mirror': Could not = open<br = class=3D"m_-7375313093972664498gmail_msg">'/rhev/data-center/8b6303b3-<wbr= class=3D"">79c6-4633-ae21-71b15ed00675/<wbr = class=3D"">6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr = class=3D"">3507-4469-a080-d0944d0ab753/<wbr = class=3D"">30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5':<br = class=3D"m_-7375313093972664498gmail_msg">Permission denied<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">[root@ovirt1 test vdsm]# ls = -l<br = class=3D"m_-7375313093972664498gmail_msg">/rhev/data-center/8b6303b3-<wbr = class=3D"">79c6-4633-ae21-71b15ed00675/<wbr = class=3D"">6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr = class=3D"">3507-4469-a080-d0944d0ab753/<wbr = class=3D"">30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5<br = class=3D"m_-7375313093972664498gmail_msg">-rw-rw---- 2 vdsm kvm 197120 = Apr 6 13:29<br = class=3D"m_-7375313093972664498gmail_msg">/rhev/data-center/8b6303b3-<wbr = class=3D"">79c6-4633-ae21-71b15ed00675/<wbr = class=3D"">6affd8c3-2c51-4cd1-8300-<wbr = class=3D"">bfbbb14edbe9/images/7ae9b3f7-<wbr = class=3D"">3507-4469-a080-d0944d0ab753/<wbr = class=3D"">30fd46c9-c738-4b13-aeca-<wbr class=3D"">3dc9ffc677f5<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">Then if I try and rerun it it = says, even though move failed:<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">2017-04-06 13:49:27,197 = INFO (jsonrpc/1) [dispatcher] Run and protect:<br = class=3D"m_-7375313093972664498gmail_msg">getAllTasksStatuses, Return = response: {'allT<br class=3D"m_-7375313093972664498gmail_msg">asksStatus':= {'078d962c-e682-40f9-a177-<wbr class=3D"">2a8b479a7d8b': {'code': = 212,<br class=3D"m_-7375313093972664498gmail_msg">'message': 'Volume = already exists', 'taskState':<br = class=3D"m_-7375313093972664498gmail_msg"> <span = class=3D"Apple-converted-space"> </span>'finished', 'taskResult': = 'cleanSuccess', 'taskID':<br = class=3D"m_-7375313093972664498gmail_msg">'078d962c-e682-40f9-a177-<wbr = class=3D"">2a8b479a7d8b'}}} (logUtils:52)<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">So now I have to clean up the = disks that it failed to move so I can<br = class=3D"m_-7375313093972664498gmail_msg">migrate the VM and then move = the disk again.<br class=3D"m_-7375313093972664498gmail_msg">Or so it = seems.<br class=3D"m_-7375313093972664498gmail_msg">Failed move disks do = exist in new location, even though it "failed".<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">vdsm.log attached.<br = class=3D"m_-7375313093972664498gmail_msg"><br = class=3D"m_-7375313093972664498gmail_msg">ovirt-engine-tools-4.1.0.4-1.<wb= r class=3D"">el7.centos.noarch<br = class=3D"m_-7375313093972664498gmail_msg">vdsm-4.19.4-1.el7.centos.x86_<wb= r class=3D"">64<br = class=3D"m_-7375313093972664498gmail_msg"></blockquote><div class=3D""><br= class=3D""></div><div class=3D"">Hi Bill,</div><div class=3D""><br = class=3D""></div><div class=3D"">Does it work after setting selinux to = permissive? (setenforce 0)</div><div class=3D""><br class=3D""></div><div = class=3D"">Can you share output of:</div><div class=3D""><br = class=3D""></div><div class=3D"">ps -efZ | grep vm-name<br = class=3D""></div><div class=3D"">(filter the specific vm) <br = class=3D""></div><div class=3D""><br class=3D""></div><div class=3D"">ls = -lhZ /rhev/data-center/mnt</div><div class=3D""><br class=3D""></div><div = class=3D"">ls -lhZ /rhev/data-center/mnt/gluster-<wbr = class=3D"">server:_path/sd_id/images/img_<wbr = class=3D"">id/vol_id</div><div class=3D"">(assuming the volume was not = deleted after the operation).</div><div class=3D""><br = class=3D""></div><div class=3D"">If the volume is not deleted after the = failed move disk operation, this is likely</div><div class=3D"">a bug, = please file a bug for this.</div><div class=3D""><br class=3D""></div><div= class=3D"">The actual failure may be gluster configuration issue, or = selinux related bug.</div><div class=3D""><br class=3D""></div><div = class=3D"">Nir</div><div class=3D""> <br = class=3D""></div></div></div></blockquote><br = class=3D""></div></div>SELinux = status: &= nbsp; disabled<br = class=3D""></div></blockquote><div class=3D""><br class=3D""></div><div = class=3D"">This is a less tested configuration. We usually run with = selinux enabled.</div></div></div></div></div></blockquote><div><br = class=3D""></div>Disabled specifically is untested. And doesn=E2=80=99t = work.</div><div>If you for whatever reason don=E2=80=99t want it, set it = to Permissive, but don=E2=80=99t disable it</div><div><br = class=3D""></div><div><br class=3D""><blockquote type=3D"cite" = class=3D""><div class=3D""><div dir=3D"ltr" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><div class=3D"gmail_extra"><div = class=3D"gmail_quote"><div class=3D"">Y.</div><div = class=3D""> </div><blockquote class=3D"gmail_quote" style=3D"margin: = 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, = 204, 204); border-left-style: solid; padding-left: 1ex;"><div = bgcolor=3D"#FFFFFF" text=3D"#000000" class=3D""><br = class=3D"">[root@ovirt1 test images]# ps -efZ | grep darmaster<br = class=3D"">- &n= bsp; &nbs= p; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>root &= nbsp; 5272 1 6 Mar17 = ? 1-09:08:58 = /usr/libexec/qemu-kvm -name guest=3D<a = href=3D"http://darmaster1.test.j2noc.com/" target=3D"_blank" = class=3D"">darmaster1.test.j2noc.<wbr class=3D"">com</a>,debug-threads=3Do= n -S -object secret,id=3DmasterKey0,format=3D<wbr = class=3D"">raw,file=3D/var/lib/libvirt/<wbr = class=3D"">qemu/domain-13-darmaster1.<wbr = class=3D"">test.j2no/master-key.aes -machine = pc-i440fx-rhel7.3.0,accel=3Dkvm,<wbr class=3D"">usb=3Doff -cpu Nehalem = -m size=3D1048576k,slots=3D16,maxmem=3D<wbr class=3D"">4194304k = -realtime mlock=3Doff -smp 1,maxcpus=3D16,sockets=3D16,cores=3D<wbr = class=3D"">1,threads=3D1 -numa node,nodeid=3D0,cpus=3D0,mem=3D1024 -uuid = 73368460-92e1-4c9e-a162-<wbr class=3D"">399304f1c462 -smbios = type=3D1,manufacturer=3DoVirt,<wbr class=3D"">product=3DoVirt = Node,version=3D7-3.1611.el7.<wbr = class=3D"">centos,serial=3D30343536-3138-<wbr = class=3D"">584D-5134-343430313833,uuid=3D<wbr = class=3D"">73368460-92e1-4c9e-a162-<wbr class=3D"">399304f1c462 = -no-user-config -nodefaults -chardev socket,id=3Dcharmonitor,path=3D/<wbr = class=3D"">var/lib/libvirt/qemu/domain-<wbr = class=3D"">13-darmaster1.test.j2no/<wbr = class=3D"">monitor.sock,server,nowait -mon chardev=3Dcharmonitor,id=3D<wbr= class=3D"">monitor,mode=3Dcontrol -rtc base=3D2017-03-17T23:57:54,<wbr = class=3D"">driftfix=3Dslew -global kvm-pit.lost_tick_policy=3D<wbr = class=3D"">discard -no-hpet -no-shutdown -boot strict=3Don -device = piix3-usb-uhci,id=3Dusb,bus=3Dpci.<wbr class=3D"">0,addr=3D0x1.0x2 = -device virtio-scsi-pci,id=3Dscsi0,bus=3D<wbr class=3D"">pci.0,addr=3D0x4 = -device virtio-serial-pci,id=3Dvirtio-<wbr = class=3D"">serial0,max_ports=3D16,bus=3Dpci.<wbr class=3D"">0,addr=3D0x5 = -drive if=3Dnone,id=3Ddrive-ide0-1-0,<wbr class=3D"">readonly=3Don = -device ide-cd,bus=3Dide.1,unit=3D0,drive=3D<wbr = class=3D"">drive-ide0-1-0,id=3Dide0-1-0 -drive = file=3D/rhev/data-center/<wbr class=3D"">8b6303b3-79c6-4633-ae21-<wbr = class=3D"">71b15ed00675/7e566f55-e060-<wbr = class=3D"">47b7-bfa4-ac3c48d70dda/images/<wbr = class=3D"">33db5688-dafe-40ab-9dd0-<wbr = class=3D"">a826a90c3793/38de110d-464c-<wbr = class=3D"">4735-97ba-3d623ee1a1b6,format=3D<wbr = class=3D"">raw,if=3Dnone,id=3Ddrive-virtio-<wbr = class=3D"">disk0,serial=3D33db5688-dafe-<wbr = class=3D"">40ab-9dd0-a826a90c3793,cache=3D<wbr = class=3D"">none,werror=3Dstop,rerror=3Dstop,<wbr class=3D"">aio=3Dthreads = -device virtio-blk-pci,scsi=3Doff,bus=3D<wbr = class=3D"">pci.0,addr=3D0x6,drive=3Ddrive-<wbr = class=3D"">virtio-disk0,id=3Dvirtio-disk0,<wbr class=3D"">bootindex=3D1 = -netdev tap,fd=3D38,id=3Dhostnet0,vhost=3D<wbr class=3D"">on,vhostfd=3D40 = -device virtio-net-pci,netdev=3D<wbr = class=3D"">hostnet0,id=3Dnet0,mac=3D00:1a:4a:<wbr = class=3D"">16:01:69,bus=3Dpci.0,addr=3D0x3 -chardev = socket,id=3Dcharchannel0,path=3D/<wbr = class=3D"">var/lib/libvirt/qemu/channels/<wbr = class=3D"">73368460-92e1-4c9e-a162-<wbr = class=3D"">399304f1c462.com.redhat.rhevm.<wbr = class=3D"">vdsm,server,nowait -device virtserialport,bus=3Dvirtio-<wbr = class=3D"">serial0.0,nr=3D1,chardev=3D<wbr = class=3D"">charchannel0,id=3Dchannel0,name=3D<wbr = class=3D"">com.redhat.rhevm.vdsm -chardev = socket,id=3Dcharchannel1,path=3D/<wbr = class=3D"">var/lib/libvirt/qemu/channels/<wbr = class=3D"">73368460-92e1-4c9e-a162-<wbr = class=3D"">399304f1c462.org.qemu.guest_<wbr = class=3D"">agent.0,server,nowait -device virtserialport,bus=3Dvirtio-<wbr = class=3D"">serial0.0,nr=3D2,chardev=3D<wbr = class=3D"">charchannel1,id=3Dchannel1,name=3D<wbr = class=3D"">org.qemu.guest_agent.0 -device = usb-tablet,id=3Dinput0,bus=3Dusb.<wbr class=3D"">0,port=3D1 -vnc<span = class=3D"Apple-converted-space"> </span><a = href=3D"http://10.176.30.96:6/" target=3D"_blank" = class=3D"">10.176.30.96:6</a>,password -k en-us -device = VGA,id=3Dvideo0,vgamem_mb=3D16,<wbr class=3D"">bus=3Dpci.0,addr=3D0x2 = -msg timestamp=3Don<br class=3D""><br class=3D"">[root@ovirt1 test ~]# = ls -lhZ /rhev/data-center/mnt<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>glusterSD<br = class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>netappqa3:_vol_cloud__images_= <wbr class=3D"">ovirt__QA__ISOs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>netappqa3:_vol_cloud__<wbr = class=3D"">storage1_ovirt__qa__inside<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt1-ks.test.j2noc.com" = class=3D"">ovirt1-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs1<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt2-ks.test.j2noc.com" = class=3D"">ovirt2-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs2<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt2-ks.test.j2noc.com" = class=3D"">ovirt2-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs-2<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt3-ks.test.j2noc.com" = class=3D"">ovirt3-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt4-ks.test.j2noc.com" = class=3D"">ovirt4-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt5-ks.test.j2noc.com" = class=3D"">ovirt5-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt6-ks.test.j2noc.com" = class=3D"">ovirt6-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span><a = href=3D"http://ovirt7-ks.test.j2noc.com" = class=3D"">ovirt7-ks.test.j2noc.com</a>:_<wbr = class=3D"">ovirt-store_nfs<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>qagenfil1-nfs1:_ovirt__inside= _<wbr class=3D"">Export<br class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>qagenfil1-nfs1:_ovirt__inside= _<wbr class=3D"">images<br class=3D""><br class=3D""><br = class=3D"">[root@ovirt1 test images]# ls -lhZa = /rhev/data-center/mnt/<wbr class=3D"">glusterSD/ovirt1-ks.test.<wbr = class=3D""><a href=3D"http://j2noc.com" = class=3D"">j2noc.com</a>:_gv2/6affd8c3-2c51-<wbr = class=3D"">4cd1-8300-bfbbb14edbe9/images/<wbr = class=3D"">33db5688-dafe-40ab-9dd0-<wbr class=3D"">a826a90c3793<br = class=3D"">drwxr-xr-x vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>.<br class=3D"">drwxr-xr-x = vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>..<br class=3D"">-rw-rw---- = vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr = class=3D"">07f5f76214f2<br class=3D"">-rw-rw---- vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr = class=3D"">07f5f76214f2.lease<br class=3D"">-rw-r--r-- vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>33c04305-efbe-418a-b42c-<wbr = class=3D"">07f5f76214f2.meta<br class=3D"">-rw-rw---- vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr = class=3D"">3d623ee1a1b6<br class=3D"">-rw-rw---- vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr = class=3D"">3d623ee1a1b6.lease<br class=3D"">-rw-r--r-- vdsm kvm = ? &= nbsp; &nb= sp; <wbr class=3D""> <span = class=3D"Apple-converted-space"> </span>38de110d-464c-4735-97ba-<wbr = class=3D"">3d623ee1a1b6.meta<br class=3D""><br class=3D""><br = class=3D"">bug submitted: <span = class=3D"Apple-converted-space"> </span><a = class=3D"m_-7375313093972664498moz-txt-link-freetext" = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1440198" = target=3D"_blank">https://bugzilla.redhat.com/<wbr = class=3D"">show_bug.cgi?id=3D1440198</a><br class=3D""><br class=3D""><br = class=3D"">Thank you!<br class=3D""></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D"">Users mailing list<br = class=3D""><a href=3D"mailto:Users@ovirt.org" = class=3D"">Users@ovirt.org</a><br class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" = target=3D"_blank" class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/users</a><br class=3D""><br = class=3D""></blockquote></div><br class=3D""></div></div><span = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; float: none; display: inline = !important;" = class=3D"">_______________________________________________</span><br = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><span style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: normal; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = float: none; display: inline !important;" class=3D"">Users mailing = list</span><br style=3D"font-family: Helvetica; font-size: 12px; = font-style: normal; font-variant-caps: normal; font-weight: normal; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><a = href=3D"mailto:Users@ovirt.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: normal; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; = -webkit-text-stroke-width: 0px;" class=3D"">Users@ovirt.org</a><br = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><a = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = style=3D"font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: normal; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" = class=3D"">http://lists.ovirt.org/mailman/listinfo/users</a></div></blockq= uote></div><br class=3D""></body></html>= --Apple-Mail=_A958104E-D048-4E05-A186-E136F26C6794--
participants (4)
-
Bill James -
Michal Skrivanek -
Nir Soffer -
Yaniv Kaul