[ovirt-users] Re: Unable to move or copy disks

Sunday, 29 November 2020

No errors 

# sudo -u vdsm dd
if=/rhev/data-center/mnt/glusterSD/gfs1.server.pt:_gfs1data/0e8de531-ac5e-4089-b390-cfc0adc3e79a/images/a847beca-7ed0-4ff1-8767-fc398379d85b/61d85180-65a4-452d-8773-db778f56e242
of=/dev/null bs=4M status=progress 
107336433664 bytes (107 GB) copied, 245.349334 s, 437 MB/s 
25600+0 records in 
25600+0 records out 
107374182400 bytes (107 GB) copied, 245.682 s, 437 MB/s 

After this I tried again to move the disk, and surprise, successfully 

I didn't believe it. 
Try to move another disk, the same error came back 
I did a dd to this other disk and tried again to move it, again successfully 

!!! 

De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: suporte(a)logicworks.pt 
Cc: users(a)ovirt.org 
Enviadas: Domingo, 29 De Novembro de 2020 20:22:36 
Assunto: Re: [ovirt-users] Re: Unable to move or copy disks 

Usually distributed volumes are supported on a Single-node setup, but it shouldn't be
the problem. 

As you know the affected VMs , you can easily find the disks of a VM. 

Then try to read the VM's disk: 

sudo -u vdsm dd
if=/rhev/data-center/mnt/glusterSD/gfs1.server.pt:_gfs1data/<Storage_domain_UUID>/images/<DISK-UUID>/<VM_DISK>
of=/dev/null bs=4M status=progress 

Does it give errors ? 

Best Regards, 
Strahil Nikolov 

В неделя, 29 ноември 2020 г., 20:06:42 Гринуич+2, suporte(a)logicworks.pt
<suporte(a)logicworks.pt&gt; написа: 

No heals pending 
There are some VM's I can move the disk but some others VM's I cannot move the
disk 

It's a simple gluster 
]# gluster volume info 

Volume Name: gfs1data 
Type: Distribute 
Volume ID: 7e6826b9-1220-49d4-a4bf-e7f50f38c42c 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 1 
Transport-type: tcp 
Bricks: 
Brick1: gfs1.server.pt:/home/brick1 
Options Reconfigured: 
diagnostics.brick-log-level: INFO 
performance.client-io-threads: off 
server.event-threads: 4 
client.event-threads: 4 
cluster.choose-local: yes 
user.cifs: off 
features.shard: on 
cluster.shd-wait-qlength: 10000 
cluster.locking-scheme: granular 
cluster.data-self-heal-algorithm: full 
cluster.server-quorum-type: server 
cluster.quorum-type: auto 
cluster.eager-lock: enable 
network.remote-dio: enable 
performance.low-prio-threads: 32 
performance.io-cache: off 
performance.read-ahead: off 
performance.quick-read: off 
storage.owner-gid: 36 
storage.owner-uid: 36 
transport.address-family: inet 
nfs.disable: on 

________________________________ 
De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: suporte(a)logicworks.pt 
Cc: users(a)ovirt.org 
Enviadas: Domingo, 29 De Novembro de 2020 17:27:04 
Assunto: Re: [ovirt-users] Re: Unable to move or copy disks 

Are you sure you don't have any heals pending ? 
I should admit I have never seen this type of error. 

Is it happening for all VMs or only specific ones ? 

Best Regards, 
Strahil Nikolov 

В неделя, 29 ноември 2020 г., 15:37:04 Гринуич+2, suporte(a)logicworks.pt
<suporte(a)logicworks.pt&gt; написа: 

Sorry, I found this error on gluster logs: 

[MSGID: 113040] [posix-helpers.c:1929:__posix_fd_ctx_get] 0-gfs1data-posix: Failed to get
anonymous fd for real_path:
/home/brick1/.glusterfs/bc/57/bc57653e-b08c-417b-83f3-bf234a97e30f. [No such file or
directory] 

________________________________ 
De: suporte(a)logicworks.pt 
Para: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Cc: users(a)ovirt.org 
Enviadas: Domingo, 29 De Novembro de 2020 13:13:00 
Assunto: [ovirt-users] Re: Unable to move or copy disks 

I don't find any error in the gluster logs, I just find this error in the vdsm log: 

2020-11-29 12:57:45,528+0000 INFO (tasks/1) [storage.SANLock] Successfully released
Lease(name='61d85180-65a4-452d-8773-db778f56e242',
path=u'/rhev/data-center/mnt/node2.server.pt:_home_node2data/ab4855be-0edd-4fac-b062-bded661e20a1/images/a847beca-7ed0-4ff1-8767-fc398379d85b/61d85180-65a4-452d-8773-db778f56e242.lease',
offset=0) (clusterlock:524) 
2020-11-29 12:57:45,528+0000 ERROR (tasks/1) [root] Job
u'cc8ea210-df4b-4f0b-a385-5bc3adc825f6' failed (jobs:221) 
Traceback (most recent call last): 
File "/usr/lib/python2.7/site-packages/vdsm/jobs.py", line 157, in run 
self._run() 
File "/usr/lib/python2.7/site-packages/vdsm/storage/sdm/api/copy_data.py", line
86, in _run 
self._operation.run() 
File "/usr/lib/python2.7/site-packages/vdsm/storage/qemuimg.py", line 343, in
run 
for data in self._operation.watch(): 
File "/usr/lib/python2.7/site-packages/vdsm/storage/operation.py", line 106, in
watch 
self._finalize(b"", err) 
File "/usr/lib/python2.7/site-packages/vdsm/storage/operation.py", line 179, in
_finalize 
raise cmdutils.Error(self._cmd, rc, out, err) 
Error: Command ['/usr/bin/qemu-img', 'convert', '-p',
'-t', 'none', '-T', 'none', '-f', 'raw',
u'/rhev/data-center/mnt/glusterSD/gfs1.server.pt:_gfs1data/0e8de531-ac5e-4089-b390-cfc0adc3e79a/images/a847beca-7ed0-4ff1-8767-fc398379d85b/61d85180-65a4-452d-8773-db778f56e242',
'-O', 'raw',
u'/rhev/data-center/mnt/node2.server.pt:_home_node2data/ab4855be-0edd-4fac-b062-bded661e20a1/images/a847beca-7ed0-4ff1-8767-fc398379d85b/61d85180-65a4-452d-8773-db778f56e242']
failed with rc=1 out='' err=bytearray(b'qemu-img: error while reading sector
134086625: No such file or directory\n') 
2020-11-29 12:57:45,528+0000 INFO (tasks/1) [root] Job
u'cc8ea210-df4b-4f0b-a385-5bc3adc825f6' will be deleted in 3600 seconds (jobs:249)

2020-11-29 12:57:45,529+0000 INFO (tasks/1) [storage.ThreadPool.WorkerThread] FINISH task
309c4289-fbba-489b-94c7-8aed36948c29 (threadPool:210) 

Any idea? 

Regards 
José 
________________________________ 
De: suporte(a)logicworks.pt 
Para: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Cc: users(a)ovirt.org 
Enviadas: Sábado, 28 De Novembro de 2020 18:39:47 
Assunto: [ovirt-users] Re: Unable to move or copy disks 

I really don't understand this. 
I have 2 glusters same version, 6.10 

I can move a disk from gluster2 to gluster1, but cannot move the same disk from gluster1
to gluster2 
ovirt version: 4.3.10.4-1.el7 

Regards 
José 

________________________________ 
De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: suporte(a)logicworks.pt 
Cc: users(a)ovirt.org 
Enviadas: Segunda-feira, 23 De Novembro de 2020 5:45:37 
Assunto: Re: [ovirt-users] Re: Unable to move or copy disks 

No, but keep an eye on you "/var/log" as debug is providing a lot of info. 

Usually when you got a failure to move the disk, you can disable and check the logs. 

Best Regards, 
Strahil Nikolov 

В неделя, 22 ноември 2020 г., 21:12:26 Гринуич+2, <suporte(a)logicworks.pt&gt; написа: 

Do I need to restart gluster after enable debug level? 

gluster volume set data2 diagnostics.brick-log-level DEBUG 

________________________________ 
De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: suporte(a)logicworks.pt 
Cc: users(a)ovirt.org 
Enviadas: Sábado, 21 De Novembro de 2020 19:42:44 
Assunto: Re: [ovirt-users] Re: Unable to move or copy disks 

You still haven't provided debug logs from the Gluster Bricks. 
There will be always a chance that a bug hits you ... no matter OS and tech. What matters
- is how you debug and overcome that bug. 

Check the gluster brick debug logs and you can test if the issue happens with an older
version. 

Also, consider providing oVirt version, Gluster version and some details about your setup
- otherwise helping you is almost impossible. 

Best Regards, 
Strahil Nikolov 

В събота, 21 ноември 2020 г., 18:16:13 Гринуич+2, suporte(a)logicworks.pt
<suporte(a)logicworks.pt&gt; написа: 

With older gluster verison this does not happens. 

Always get this error: VDSM NODE3 command HSMGetAllTasksStatusesVDS failed: low level
Image copy failed: () 
and in the vdsm.log: 
ERROR (tasks/7) [storage.Image] Copy image error:
image=6939cc5d-dbca-488d-ab7a-c8b8d39c3656, src
domain=70e33d55-f1be-4826-b85a-9650c76c8db8, dst
domain=0b80eac1-8bbb-4634-9098-4155602c7b38 (image:485) ERROR (tasks/7)
[storage.TaskManager.Task] (Task='fbf02f6b-c107-4c1e-a9c7-b13261bf99e0')
Unexpected error (task:875) 

For smaller installations, without the need of storage HA, maybe it's better to use
NFS? Is more stable? 

Regards 

José 

________________________________ 
De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: suporte(a)logicworks.pt 
Cc: users(a)ovirt.org 
Enviadas: Sexta-feira, 20 De Novembro de 2020 21:55:28 
Assunto: Re: [ovirt-users] Unable to move or copy disks 

I can recommend you to: 
- enable debug level of gluster's bricks 
- try to reproduce the issue 

I had similar issue with gluster v6.6 and above. 

Best Regards, 
Strahil Nikolov 

В петък, 20 ноември 2020 г., 23:35:14 Гринуич+2, <suporte(a)logicworks.pt&gt; написа: 

I tried to move the VM disk with the VM up 
I also tried to move or copy the disk with the VM down, and get the same error. 

Strange is, I have a gluster storage domain, with an older gluster version, that worked. I
was able to move the disk to this older gluster and also I was able to copy the disk from
to this older gluster storage 

What should I do with this VM? Should I shut it down and try delete the snapshot? 

Regards 
José 

________________________________ 
De: "Strahil Nikolov" <hunter86_bg(a)yahoo.com&gt; 
Para: users(a)ovirt.org, suporte(a)logicworks.pt 
Enviadas: Sexta-feira, 20 De Novembro de 2020 21:05:19 
Assunto: Re: [ovirt-users] Unable to move or copy disks 

Can you try a live migration ? 
I got a similar case and the live migration somehow triggered a fix. 

Best Regards, 
Strahil Nikolov 

В петък, 20 ноември 2020 г., 21:04:13 Гринуич+2, <suporte(a)logicworks.pt&gt; написа: 

Hi, 

I was trying to move a disk between glusters domain storage without success. 

# gluster --version glusterfs 6.10 

Now have a VM with this message: 
The VM has snapshot(s) with disk(s) in illegal status. Please don't shutdown the VM
before successfully retrying the snapshot delete 

oVirt verison is 
Version 4.3.10.4-1.el7 

I cannot delete the snapshot. What should I do? 

hanks 

-- 
________________________________ 
Jose Ferradeira 
http://www.logicworks.pt 
_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EAYBO6KEKQS...

_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/VIVNXE4C3MZ...

_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TTOPYUWNFDH...

_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3NWLJYU2RDL...

_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/KYQSHSJQ67Y...

_______________________________________________ 
Users mailing list -- users(a)ovirt.org 
To unsubscribe send an email to users-leave(a)ovirt.org 
Privacy Statement: https://www.ovirt.org/privacy-policy.html 
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ 
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/F4VXQ7QT27R...

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

[ovirt-users] Re: Unable to move or copy disks