[ovirt-users] Re: Fwd: Fwd: Issues with Gluster Domain

Strahil, I understand. Please keep me posted. Thanks For The Help ! On Sat, Jun 20, 2020 at 4:36 AM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; wrote: > Hey C Williams, > > sorry for the delay, but I couldn't get somw time to check your logs. > Will try a little bit later. > > Best Regards, > Strahil Nikolov > > На 20 юни 2020 г. 2:37:22 GMT+03:00, C Williams <cwilliams3320(a)gmail.com&gt; > написа: > >Hello, > > > >Was wanting to follow up on this issue. Users are impacted. > > > >Thank You > > > >On Fri, Jun 19, 2020 at 9:20 AM C Williams <cwilliams3320(a)gmail.com&gt; > >wrote: > > > >> Hello, > >> > >> Here are the logs (some IPs are changed ) > >> > >> ov05 is the SPM > >> > >> Thank You For Your Help ! > >> > >> On Thu, Jun 18, 2020 at 11:31 PM Strahil Nikolov > &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >> wrote: > >> > >>> Check on the hosts tab , which is your current SPM (last column in > >Admin > >>> UI). > >>> Then open the /var/log/vdsm/vdsm.log and repeat the operation. > >>> Then provide the log from that host and the engine's log (on the > >>> HostedEngine VM or on your standalone engine). > >>> > >>> Best Regards, > >>> Strahil Nikolov > >>> > >>> На 18 юни 2020 г. 23:59:36 GMT+03:00, C Williams > &gt;&lt;cwilliams3320(a)gmail.com&gt; > >>> написа: > >>> >Resending to eliminate email issues > >>> > > >>> >---------- Forwarded message --------- > >>> >From: C Williams <cwilliams3320(a)gmail.com&gt; > >>> >Date: Thu, Jun 18, 2020 at 4:01 PM > >>> >Subject: Re: [ovirt-users] Fwd: Issues with Gluster Domain > >>> >To: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > >>> > > >>> > > >>> >Here is output from mount > >>> > > >>> >192.168.24.12:/stor/import0 on > >>> >/rhev/data-center/mnt/192.168.24.12:_stor_import0 > >>> >type nfs4 > >>> > >>> > > >>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.12) > >>> >192.168.24.13:/stor/import1 on > >>> >/rhev/data-center/mnt/192.168.24.13:_stor_import1 > >>> >type nfs4 > >>> > >>> > > >>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) > >>> >192.168.24.13:/stor/iso1 on > >>> >/rhev/data-center/mnt/192.168.24.13:_stor_iso1 > >>> >type nfs4 > >>> > >>> > > >>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) > >>> >192.168.24.13:/stor/export0 on > >>> >/rhev/data-center/mnt/192.168.24.13:_stor_export0 > >>> >type nfs4 > >>> > >>> > > >>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) > >>> >192.168.24.15:/images on > >>> >/rhev/data-center/mnt/glusterSD/192.168.24.15:_images > >>> >type fuse.glusterfs > >>> > >>> > > >>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > >>> >192.168.24.18:/images3 on > >>> >/rhev/data-center/mnt/glusterSD/192.168.24.18:_images3 > >>> >type fuse.glusterfs > >>> > >>> > > >>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) > >>> >tmpfs on /run/user/0 type tmpfs > >>> >(rw,nosuid,nodev,relatime,seclabel,size=13198392k,mode=700) > >>> >[root@ov06 glusterfs]# > >>> > > >>> >Also here is a screenshot of the console > >>> > > >>> >[image: image.png] > >>> >The other domains are up > >>> > > >>> >Import0 and Import1 are NFS . GLCL0 is gluster. They all are > >running > >>> >VMs > >>> > > >>> >Thank You For Your Help ! > >>> > > >>> >On Thu, Jun 18, 2020 at 3:51 PM Strahil Nikolov > &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >>> >wrote: > >>> > > >>> >> I don't see '/rhev/data-center/mnt/192.168.24.13:_stor_import1' > >>> >mounted > >>> >> at all . > >>> >> What is the status of all storage domains ? > >>> >> > >>> >> Best Regards, > >>> >> Strahil Nikolov > >>> >> > >>> >> На 18 юни 2020 г. 21:43:44 GMT+03:00, C Williams > >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; > >>> >> написа: > >>> >> > Resending to deal with possible email issues > >>> >> > > >>> >> >---------- Forwarded message --------- > >>> >> >From: C Williams <cwilliams3320(a)gmail.com&gt; > >>> >> >Date: Thu, Jun 18, 2020 at 2:07 PM > >>> >> >Subject: Re: [ovirt-users] Issues with Gluster Domain > >>> >> >To: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; > >>> >> > > >>> >> > > >>> >> >More > >>> >> > > >>> >> >[root@ov06 ~]# for i in $(gluster volume list); do echo > >$i;echo; > >>> >> >gluster > >>> >> >volume info $i; echo;echo;gluster volume status > >>> >$i;echo;echo;echo;done > >>> >> >images3 > >>> >> > > >>> >> > > >>> >> >Volume Name: images3 > >>> >> >Type: Replicate > >>> >> >Volume ID: 0243d439-1b29-47d0-ab39-d61c2f15ae8b > >>> >> >Status: Started > >>> >> >Snapshot Count: 0 > >>> >> >Number of Bricks: 1 x 3 = 3 > >>> >> >Transport-type: tcp > >>> >> >Bricks: > >>> >> >Brick1: 192.168.24.18:/bricks/brick04/images3 > >>> >> >Brick2: 192.168.24.19:/bricks/brick05/images3 > >>> >> >Brick3: 192.168.24.20:/bricks/brick06/images3 > >>> >> >Options Reconfigured: > >>> >> >performance.client-io-threads: on > >>> >> >nfs.disable: on > >>> >> >transport.address-family: inet > >>> >> >user.cifs: off > >>> >> >auth.allow: * > >>> >> >performance.quick-read: off > >>> >> >performance.read-ahead: off > >>> >> >performance.io-cache: off > >>> >> >performance.low-prio-threads: 32 > >>> >> >network.remote-dio: off > >>> >> >cluster.eager-lock: enable > >>> >> >cluster.quorum-type: auto > >>> >> >cluster.server-quorum-type: server > >>> >> >cluster.data-self-heal-algorithm: full > >>> >> >cluster.locking-scheme: granular > >>> >> >cluster.shd-max-threads: 8 > >>> >> >cluster.shd-wait-qlength: 10000 > >>> >> >features.shard: on > >>> >> >cluster.choose-local: off > >>> >> >client.event-threads: 4 > >>> >> >server.event-threads: 4 > >>> >> >storage.owner-uid: 36 > >>> >> >storage.owner-gid: 36 > >>> >> >performance.strict-o-direct: on > >>> >> >network.ping-timeout: 30 > >>> >> >cluster.granular-entry-heal: enable > >>> >> > > >>> >> > > >>> >> >Status of volume: images3 > >>> >> >Gluster process TCP Port RDMA Port > >>> >Online > >>> >> > Pid > >>> >> > >>> >> > >>> > >>> > > >>>------------------------------------------------------------------------------ > >>> >> >Brick 192.168.24.18:/bricks/brick04/images3 49152 0 > >Y > >>> >> >6666 > >>> >> >Brick 192.168.24.19:/bricks/brick05/images3 49152 0 > >Y > >>> >> >6779 > >>> >> >Brick 192.168.24.20:/bricks/brick06/images3 49152 0 > >Y > >>> >> >7227 > >>> >> >Self-heal Daemon on localhost N/A N/A > >Y > >>> >> >6689 > >>> >> >Self-heal Daemon on ov07.ntc.srcle.com N/A N/A > >Y > >>> >> >6802 > >>> >> >Self-heal Daemon on ov08.ntc.srcle.com N/A N/A > >Y > >>> >> >7250 > >>> >> > > >>> >> >Task Status of Volume images3 > >>> >> > >>> >> > >>> > >>> > > >>>------------------------------------------------------------------------------ > >>> >> >There are no active volume tasks > >>> >> > > >>> >> > > >>> >> > > >>> >> > > >>> >> >[root@ov06 ~]# ls -l /rhev/data-center/mnt/glusterSD/ > >>> >> >total 16 > >>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 14:04 192.168.24.15:_images > >>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 14:05 192.168.24.18:_images3 > >>> >> >[root@ov06 ~]# > >>> >> > > >>> >> >On Thu, Jun 18, 2020 at 2:03 PM C Williams > &gt;&lt;cwilliams3320(a)gmail.com&gt; > >>> >> >wrote: > >>> >> > > >>> >> >> Strahil, > >>> >> >> > >>> >> >> Here you go -- Thank You For Your Help ! > >>> >> >> > >>> >> >> BTW -- I can write a test file to gluster and it replicates > >>> >properly. > >>> >> >> Thinking something about the oVirt Storage Domain ? > >>> >> >> > >>> >> >> [root@ov08 ~]# gluster pool list > >>> >> >> UUID Hostname > >>> >State > >>> >> >> 5b40c659-d9ab-43c3-9af8-18b074ea0b83 ov06 > >>> >> >Connected > >>> >> >> 36ce5a00-6f65-4926-8438-696944ebadb5 ov07.ntc.srcle.com > >>> >> >Connected > >>> >> >> c7e7abdb-a8f4-4842-924c-e227f0db1b29 localhost > >>> >> >Connected > >>> >> >> [root@ov08 ~]# gluster volume list > >>> >> >> images3 > >>> >> >> > >>> >> >> On Thu, Jun 18, 2020 at 1:13 PM Strahil Nikolov > >>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; > >>> >> >> wrote: > >>> >> >> > >>> >> >>> Log to the oVirt cluster and provide the output of: > >>> >> >>> gluster pool list > >>> >> >>> gluster volume list > >>> >> >>> for i in $(gluster volume list); do echo $i;echo; gluster > >>> >volume > >>> >> >info > >>> >> >>> $i; echo;echo;gluster volume status $i;echo;echo;echo;done > >>> >> >>> > >>> >> >>> ls -l /rhev/data-center/mnt/glusterSD/ > >>> >> >>> > >>> >> >>> Best Regards, > >>> >> >>> Strahil Nikolov > >>> >> >>> > >>> >> >>> > >>> >> >>> На 18 юни 2020 г. 19:17:46 GMT+03:00, C Williams > >>> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; > >>> >> >>> написа: > >>> >> >>> >Hello, > >>> >> >>> > > >>> >> >>> >I recently added 6 hosts to an existing oVirt > >compute/gluster > >>> >> >cluster. > >>> >> >>> > > >>> >> >>> >Prior to this attempted addition, my cluster had 3 > >Hypervisor > >>> >hosts > >>> >> >and > >>> >> >>> >3 > >>> >> >>> >gluster bricks which made up a single gluster volume > >(replica 3 > >>> >> >volume) > >>> >> >>> >. I > >>> >> >>> >added the additional hosts and made a brick on 3 of the new > >>> >hosts > >>> >> >and > >>> >> >>> >attempted to make a new replica 3 volume. I had difficulty > >>> >> >creating > >>> >> >>> >the > >>> >> >>> >new volume. So, I decided that I would make a new > >>> >compute/gluster > >>> >> >>> >cluster > >>> >> >>> >for each set of 3 new hosts. > >>> >> >>> > > >>> >> >>> >I removed the 6 new hosts from the existing oVirt > >>> >Compute/Gluster > >>> >> >>> >Cluster > >>> >> >>> >leaving the 3 original hosts in place with their bricks. At > >that > >>> >> >point > >>> >> >>> >my > >>> >> >>> >original bricks went down and came back up . The volume > >showed > >>> >> >entries > >>> >> >>> >that > >>> >> >>> >needed healing. At that point I ran gluster volume heal > >images3 > >>> >> >full, > >>> >> >>> >etc. > >>> >> >>> >The volume shows no unhealed entries. I also corrected some > >peer > >>> >> >>> >errors. > >>> >> >>> > > >>> >> >>> >However, I am unable to copy disks, move disks to another > >>> >domain, > >>> >> >>> >export > >>> >> >>> >disks, etc. It appears that the engine cannot locate disks > >>> >properly > >>> >> >and > >>> >> >>> >I > >>> >> >>> >get storage I/O errors. > >>> >> >>> > > >>> >> >>> >I have detached and removed the oVirt Storage Domain. I > >>> >reimported > >>> >> >the > >>> >> >>> >domain and imported 2 VMs, But the VM disks exhibit the same > >>> >> >behaviour > >>> >> >>> >and > >>> >> >>> >won't run from the hard disk. > >>> >> >>> > > >>> >> >>> > > >>> >> >>> >I get errors such as this > >>> >> >>> > > >>> >> >>> >VDSM ov05 command HSMGetAllTasksStatusesVDS failed: low > >level > >>> >Image > >>> >> >>> >copy > >>> >> >>> >failed: ("Command ['/usr/bin/qemu-img', 'convert', '-p', > >'-t', > >>> >> >'none', > >>> >> >>> >'-T', 'none', '-f', 'raw', > >>> >> >>> >u'/rhev/data-center/mnt/glusterSD/192.168.24.18: > >>> >> >>> > >>> >> > >>> >> > >>> > >>> > > >>>_images3/5fe3ad3f-2d21-404c-832e-4dc7318ca10d/images/3ea5afbd-0fe0-4c09-8d39-e556c66a8b3d/fe6eab63-3b22-4815-bfe6-4a0ade292510', > >>> >> >>> >'-O', 'raw', > >>> >> >>> >u'/rhev/data-center/mnt/192.168.24.13: > >>> >> >>> > >>> >> > >>> >> > >>> > >>> > > >>>_stor_import1/1ab89386-a2ba-448b-90ab-bc816f55a328/images/f707a218-9db7-4e23-8bbd-9b12972012b6/d6591ec5-3ede-443d-bd40-93119ca7c7d5'] > >>> >> >>> >failed with rc=1 out='' err=bytearray(b'qemu-img: error > >while > >>> >> >reading > >>> >> >>> >sector 135168: Transport endpoint is not > >connected\\nqemu-img: > >>> >> >error > >>> >> >>> >while > >>> >> >>> >reading sector 131072: Transport endpoint is not > >>> >> >connected\\nqemu-img: > >>> >> >>> >error while reading sector 139264: Transport endpoint is not > >>> >> >>> >connected\\nqemu-img: error while reading sector 143360: > >>> >Transport > >>> >> >>> >endpoint > >>> >> >>> >is not connected\\nqemu-img: error while reading sector > >147456: > >>> >> >>> >Transport > >>> >> >>> >endpoint is not connected\\nqemu-img: error while reading > >sector > >>> >> >>> >155648: > >>> >> >>> >Transport endpoint is not connected\\nqemu-img: error while > >>> >reading > >>> >> >>> >sector > >>> >> >>> >151552: Transport endpoint is not connected\\nqemu-img: > >error > >>> >while > >>> >> >>> >reading > >>> >> >>> >sector 159744: Transport endpoint is not connected\\n')",) > >>> >> >>> > > >>> >> >>> >oVirt version is 4.3.82-1.el7 > >>> >> >>> >OS CentOS Linux release 7.7.1908 (Core) > >>> >> >>> > > >>> >> >>> >The Gluster Cluster has been working very well until this > >>> >incident. > >>> >> >>> > > >>> >> >>> >Please help. > >>> >> >>> > > >>> >> >>> >Thank You > >>> >> >>> > > >>> >> >>> >Charles Williams > >>> >> >>> > >>> >> >> > >>> >> > >>> > >> >