[ovirt-users] moving disk from one storage domain to another
Yaniv Kaul
ykaul at redhat.com
Sun Apr 9 07:16:08 UTC 2017
On Fri, Apr 7, 2017 at 5:29 PM, Bill James <bill.james at j2.com> wrote:
>
>
> On 4/7/17 12:52 AM, Nir Soffer wrote:
>
> On Fri, Apr 7, 2017 at 2:40 AM Bill James <bill.james at j2.com> wrote:
>
>> We are trying to convert our qa environment from local nfs to gluster.
>> When I move a disk with a VM that is running on same server as the
>> storage it fails.
>> When I move a disk with VM running on a different system it works.
>>
>> VM running on same system as disk:
>>
>> 2017-04-06 13:31:00,588 ERROR (jsonrpc/6) [virt.vm]
>> (vmId='e598485a-dc74-43f7-8447-e00ac44dae21') Unable to start
>> replication for vda to {u'domainID':
>> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volumeInfo': {'domainID':
>> u'6affd8c3-2c
>> 51-4cd1-8300-bfbbb14edbe9', 'volType': 'path', 'leaseOffset': 0, 'path':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
>> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
>> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, 'diskType': 'file',
>> 'format': 'cow', 'cache': 'none', u'volumeID':
>> u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', u'imageID':
>> u'7ae9b3f7-3507-4469-a080-d0944d0ab753', u'poolID':
>> u'8b6303b3-79c6-4633-ae21-71b15ed00675', u'device': 'disk', 'path':
>> u'/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
>> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
>> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
>> 'propagateErrors': u'off', 'volumeChain': [{'domainID':
>> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
>> 'leaseOffset': 0, 'path':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d',
>> 'volumeID': u'6756eb05-6803-42a7-a3a2-10233bf2ca8d', 'leasePath':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/6756eb05-6803-42a7-a3a2-10233bf2ca8d.lease',
>> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}, {'domainID':
>> u'6affd8c3-2c51-4cd1-8300-bfbbb14edbe9', 'volType': 'path',
>> 'leaseOffset': 0, 'path':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5',
>> 'volumeID': u'30fd46c9-c738-4b13-aeca-3dc9ffc677f5', 'leasePath':
>> u'/rhev/data-center/mnt/glusterSD/ovirt1-ks.test.j2noc.com:
>> _gv2/6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-3507-4469-a080-
>> d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5.lease',
>> 'imageID': u'7ae9b3f7-3507-4469-a080-d0944d0ab753'}]} (vm:3594)
>> Traceback (most recent call last):
>> File "/usr/share/vdsm/virt/vm.py", line 3588, in diskReplicateStart
>> self._startDriveReplication(drive)
>> File "/usr/share/vdsm/virt/vm.py", line 3713, in _startDriveReplication
>> self._dom.blockCopy(drive.name, destxml, flags=flags)
>> File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line
>> 69, in f
>> ret = attr(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
>> line 123, in wrapper
>> ret = f(*args, **kwargs)
>> File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 941, in
>> wrapper
>> return func(inst, *args, **kwargs)
>> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 684, in
>> blockCopy
>> if ret == -1: raise libvirtError ('virDomainBlockCopy() failed',
>> dom=self)
>> libvirtError: internal error: unable to execute QEMU command
>> 'drive-mirror': Could not open
>> '/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
>> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
>> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5':
>> Permission denied
>>
>>
>> [root at ovirt1 test vdsm]# ls -l
>> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
>> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
>> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
>> -rw-rw---- 2 vdsm kvm 197120 Apr 6 13:29
>> /rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/
>> 6affd8c3-2c51-4cd1-8300-bfbbb14edbe9/images/7ae9b3f7-
>> 3507-4469-a080-d0944d0ab753/30fd46c9-c738-4b13-aeca-3dc9ffc677f5
>>
>>
>>
>> Then if I try and rerun it it says, even though move failed:
>>
>> 2017-04-06 13:49:27,197 INFO (jsonrpc/1) [dispatcher] Run and protect:
>> getAllTasksStatuses, Return response: {'allT
>> asksStatus': {'078d962c-e682-40f9-a177-2a8b479a7d8b': {'code': 212,
>> 'message': 'Volume already exists', 'taskState':
>> 'finished', 'taskResult': 'cleanSuccess', 'taskID':
>> '078d962c-e682-40f9-a177-2a8b479a7d8b'}}} (logUtils:52)
>>
>>
>> So now I have to clean up the disks that it failed to move so I can
>> migrate the VM and then move the disk again.
>> Or so it seems.
>> Failed move disks do exist in new location, even though it "failed".
>>
>> vdsm.log attached.
>>
>> ovirt-engine-tools-4.1.0.4-1.el7.centos.noarch
>> vdsm-4.19.4-1.el7.centos.x86_64
>>
>
> Hi Bill,
>
> Does it work after setting selinux to permissive? (setenforce 0)
>
> Can you share output of:
>
> ps -efZ | grep vm-name
> (filter the specific vm)
>
> ls -lhZ /rhev/data-center/mnt
>
> ls -lhZ /rhev/data-center/mnt/gluster-server:_path/sd_id/images/img_
> id/vol_id
> (assuming the volume was not deleted after the operation).
>
> If the volume is not deleted after the failed move disk operation, this is
> likely
> a bug, please file a bug for this.
>
> The actual failure may be gluster configuration issue, or selinux related
> bug.
>
> Nir
>
>
>
> SELinux status: disabled
>
This is a less tested configuration. We usually run with selinux enabled.
Y.
>
> [root at ovirt1 test images]# ps -efZ | grep darmaster
> - root 5272 1 6 Mar17 ?
> 1-09:08:58 /usr/libexec/qemu-kvm -name guest=darmaster1.test.j2noc.com,debug-threads=on
> -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/
> qemu/domain-13-darmaster1.test.j2no/master-key.aes -machine
> pc-i440fx-rhel7.3.0,accel=kvm,usb=off -cpu Nehalem -m
> size=1048576k,slots=16,maxmem=4194304k -realtime mlock=off -smp
> 1,maxcpus=16,sockets=16,cores=1,threads=1 -numa
> node,nodeid=0,cpus=0,mem=1024 -uuid 73368460-92e1-4c9e-a162-399304f1c462
> -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-3.1611.el7.
> centos,serial=30343536-3138-584D-5134-343430313833,uuid=
> 73368460-92e1-4c9e-a162-399304f1c462 -no-user-config -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-
> 13-darmaster1.test.j2no/monitor.sock,server,nowait -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=2017-03-17T23:57:54,driftfix=slew
> -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot
> strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
> virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x4 -device
> virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x5
> -drive if=none,id=drive-ide0-1-0,readonly=on -device
> ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
> file=/rhev/data-center/8b6303b3-79c6-4633-ae21-71b15ed00675/7e566f55-e060-
> 47b7-bfa4-ac3c48d70dda/images/33db5688-dafe-40ab-9dd0-
> a826a90c3793/38de110d-464c-4735-97ba-3d623ee1a1b6,format=
> raw,if=none,id=drive-virtio-disk0,serial=33db5688-dafe-
> 40ab-9dd0-a826a90c3793,cache=none,werror=stop,rerror=stop,aio=threads
> -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-
> virtio-disk0,id=virtio-disk0,bootindex=1 -netdev
> tap,fd=38,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=
> hostnet0,id=net0,mac=00:1a:4a:16:01:69,bus=pci.0,addr=0x3 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/
> 73368460-92e1-4c9e-a162-399304f1c462.com.redhat.rhevm.vdsm,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=
> charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev
> socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/
> 73368460-92e1-4c9e-a162-399304f1c462.org.qemu.guest_agent.0,server,nowait
> -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=
> charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device
> usb-tablet,id=input0,bus=usb.0,port=1 -vnc 10.176.30.96:6,password -k
> en-us -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -msg
> timestamp=on
>
> [root at ovirt1 test ~]# ls -lhZ /rhev/data-center/mnt
> drwxr-xr-x vdsm kvm ? glusterSD
> drwxr-xr-x vdsm kvm ?
> netappqa3:_vol_cloud__images_ovirt__QA__ISOs
> drwxr-xr-x vdsm kvm ?
> netappqa3:_vol_cloud__storage1_ovirt__qa__inside
> drwxr-xr-x vdsm kvm ?
> ovirt1-ks.test.j2noc.com:_ovirt-store_nfs1
> drwxr-xr-x vdsm kvm ?
> ovirt2-ks.test.j2noc.com:_ovirt-store_nfs2
> drwxr-xr-x vdsm kvm ?
> ovirt2-ks.test.j2noc.com:_ovirt-store_nfs-2
> drwxr-xr-x vdsm kvm ?
> ovirt3-ks.test.j2noc.com:_ovirt-store_nfs
> drwxr-xr-x vdsm kvm ?
> ovirt4-ks.test.j2noc.com:_ovirt-store_nfs
> drwxr-xr-x vdsm kvm ?
> ovirt5-ks.test.j2noc.com:_ovirt-store_nfs
> drwxr-xr-x vdsm kvm ?
> ovirt6-ks.test.j2noc.com:_ovirt-store_nfs
> drwxr-xr-x vdsm kvm ?
> ovirt7-ks.test.j2noc.com:_ovirt-store_nfs
> drwxr-xr-x vdsm kvm ?
> qagenfil1-nfs1:_ovirt__inside_Export
> drwxr-xr-x vdsm kvm ?
> qagenfil1-nfs1:_ovirt__inside_images
>
>
> [root at ovirt1 test images]# ls -lhZa /rhev/data-center/mnt/
> glusterSD/ovirt1-ks.test.j2noc.com:_gv2/6affd8c3-2c51-
> 4cd1-8300-bfbbb14edbe9/images/33db5688-dafe-40ab-9dd0-a826a90c3793
> drwxr-xr-x vdsm kvm ? .
> drwxr-xr-x vdsm kvm ? ..
> -rw-rw---- vdsm kvm ?
> 33c04305-efbe-418a-b42c-07f5f76214f2
> -rw-rw---- vdsm kvm ?
> 33c04305-efbe-418a-b42c-07f5f76214f2.lease
> -rw-r--r-- vdsm kvm ?
> 33c04305-efbe-418a-b42c-07f5f76214f2.meta
> -rw-rw---- vdsm kvm ?
> 38de110d-464c-4735-97ba-3d623ee1a1b6
> -rw-rw---- vdsm kvm ?
> 38de110d-464c-4735-97ba-3d623ee1a1b6.lease
> -rw-r--r-- vdsm kvm ?
> 38de110d-464c-4735-97ba-3d623ee1a1b6.meta
>
>
> bug submitted: https://bugzilla.redhat.com/show_bug.cgi?id=1440198
>
>
> Thank you!
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170409/4a26ad79/attachment-0001.html>
More information about the Users
mailing list