[ovirt-users] Re: Fwd: Fwd: Issues with Gluster Domain

In my situation I had only the ovirt nodes. На 21 юни 2020 г. 22:43:04 GMT+03:00, C Williams <cwilliams3320(a)gmail.com&gt; написа: >Strahil, > >So should I make the target volume on 3 bricks which do not have ovirt >-- >just gluster ? In other words (3) Centos 7 hosts ? > >Thank You For Your Help ! > >On Sun, Jun 21, 2020 at 3:08 PM Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >wrote: > >> I created a fresh volume (which is not an ovirt sgorage domain), >set >> the original storage domain in maintenance and detached it. >> Then I 'cp -a ' the data from the old to the new volume. Next, I >just >> added the new storage domain (the old one was a kind of a >> 'backup') - pointing to the new volume name. >> >> If you observe issues , I would recommend you to downgrade >> gluster packages one node at a time . Then you might be able to >> restore your oVirt operations. >> >> Best Regards, >> Strahil Nikolov >> >> На 21 юни 2020 г. 18:01:31 GMT+03:00, C Williams &gt;&lt;cwilliams3320(a)gmail.com&gt; >> написа: >> >Strahil, >> > >> >Thanks for the follow up ! >> > >> >How did you copy the data to another volume ? >> > >> >I have set up another storage domain GLCLNEW1 with a new volume >imgnew1 >> >. >> >How would you copy all of the data from the problematic domain GLCL3 >> >with >> >volume images3 to GLCLNEW1 and volume imgnew1 and preserve all the >VMs, >> >VM >> >disks, settings, etc. ? >> > >> >Remember all of the regular ovirt disk copy, disk move, VM export >> >tools >> >are failing and my VMs and disks are trapped on domain GLCL3 and >volume >> >images3 right now. >> > >> >Please let me know >> > >> >Thank You For Your Help ! >> > >> > >> > >> > >> > >> >On Sun, Jun 21, 2020 at 8:27 AM Strahil Nikolov &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >wrote: >> > >> >> Sorry to hear that. >> >> I can say that for me 6.5 was working, while 6.6 didn't and I >> >upgraded >> >> to 7.0 . >> >> In the ended , I have ended with creating a new fresh volume >and >> >> physically copying the data there, then I detached the storage >> >domains and >> >> attached to the new ones (which holded the old data), but I >> >could >> >> afford the downtime. >> >> Also, I can say that v7.0 ( but not 7.1 or anything later) >also >> >> worked without the ACL issue, but it causes some trouble in >oVirt >> >- so >> >> avoid that unless you have no other options. >> >> >> >> Best Regards, >> >> Strahil Nikolov >> >> >> >> >> >> >> >> >> >> На 21 юни 2020 г. 4:39:46 GMT+03:00, C Williams >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> написа: >> >> >Hello, >> >> > >> >> >Upgrading diidn't help >> >> > >> >> >Still acl errors trying to use a Virtual Disk from a VM >> >> > >> >> >[root@ov06 bricks]# tail bricks-brick04-images3.log | grep acl >> >> >[2020-06-21 01:33:45.665888] I [MSGID: 139001] >> >> >[posix-acl.c:263:posix_acl_log_permit_denied] >> >0-images3-access-control: >> >> >client: >> >> >> >> >> >> >>>CTX_ID:3697a7f1-44fb-4258-96b0-98cb4137d195-GRAPH_ID:0-PID:6706-HOST:ov06.ntc.srcle.com-PC_NAME:images3-client-0-RECON_NO:-0, >> >> >gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >req(uid:107,gid:107,perm:1,ngrps:3), >> >> >ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >[Permission denied] >> >> >The message "I [MSGID: 139001] >> >> >[posix-acl.c:263:posix_acl_log_permit_denied] >> >0-images3-access-control: >> >> >client: >> >> >> >> >> >> >>>CTX_ID:3697a7f1-44fb-4258-96b0-98cb4137d195-GRAPH_ID:0-PID:6706-HOST:ov06.ntc.srcle.com-PC_NAME:images3-client-0-RECON_NO:-0, >> >> >gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >req(uid:107,gid:107,perm:1,ngrps:3), >> >> >ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >[Permission denied]" repeated 2 times between [2020-06-21 >> >> >01:33:45.665888] >> >> >and [2020-06-21 01:33:45.806779] >> >> > >> >> >Thank You For Your Help ! >> >> > >> >> >On Sat, Jun 20, 2020 at 8:59 PM C Williams &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >wrote: >> >> > >> >> >> Hello, >> >> >> >> >> >> Based on the situation, I am planning to upgrade the 3 affected >> >> >hosts. >> >> >> >> >> >> My reasoning is that the hosts/bricks were attached to 6.9 at >one >> >> >time. >> >> >> >> >> >> Thanks For Your Help ! >> >> >> >> >> >> On Sat, Jun 20, 2020 at 8:38 PM C Williams >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >> wrote: >> >> >> >> >> >>> Strahil, >> >> >>> >> >> >>> The gluster version on the current 3 gluster hosts is 6.7 >(last >> >> >update >> >> >>> 2/26). These 3 hosts provide 1 brick each for the replica 3 >> >volume. >> >> >>> >> >> >>> Earlier I had tried to add 6 additional hosts to the cluster. >> >Those >> >> >new >> >> >>> hosts were 6.9 gluster. >> >> >>> >> >> >>> I attempted to make a new separate volume with 3 bricks >provided >> >by >> >> >the 3 >> >> >>> new gluster 6.9 hosts. After having many errors from the >oVirt >> >> >interface, >> >> >>> I gave up and removed the 6 new hosts from the cluster. That >is >> >> >where the >> >> >>> problems started. The intent was to expand the gluster cluster >> >while >> >> >making >> >> >>> 2 new volumes for that cluster. The ovirt compute cluster >would >> >> >allow for >> >> >>> efficient VM migration between 9 hosts -- while having >separate >> >> >gluster >> >> >>> volumes for safety purposes. >> >> >>> >> >> >>> Looking at the brick logs, I see where there are acl errors >> >starting >> >> >from >> >> >>> the time of the removal of the 6 new hosts. >> >> >>> >> >> >>> Please check out the attached brick log from 6/14-18. The >events >> >> >started >> >> >>> on 6/17. >> >> >>> >> >> >>> I wish I had a downgrade path. >> >> >>> >> >> >>> Thank You For The Help !! >> >> >>> >> >> >>> On Sat, Jun 20, 2020 at 7:47 PM Strahil Nikolov >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>> wrote: >> >> >>> >> >> >>>> Hi , >> >> >>>> >> >> >>>> >> >> >>>> This one really looks like the ACL bug I was hit with when I >> >> >updated >> >> >>>> from Gluster v6.5 to 6.6 and later from 7.0 to 7.2. >> >> >>>> >> >> >>>> Did you update your setup recently ? Did you upgrade gluster >> >also ? >> >> >>>> >> >> >>>> You have to check the gluster logs in order to verify that, >so >> >you >> >> >can >> >> >>>> try: >> >> >>>> >> >> >>>> 1. Set Gluster logs to trace level (for details check: >> >> >>>> >> >> > >> >> >> > >> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/h... >> >> >>>> ) >> >> >>>> 2. Power up a VM that was already off , or retry the >procedure >> >from >> >> >the >> >> >>>> logs you sent. >> >> >>>> 3. Stop the trace level of the logs >> >> >>>> 4. Check libvirt logs on the host that was supposed to power >up >> >the >> >> >VM >> >> >>>> (in case a VM was powered on) >> >> >>>> 5. Check the gluster brick logs on all nodes for ACL errors. >> >> >>>> Here is a sample from my old logs: >> >> >>>> >> >> >>>> gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >> >> >13:19:41.489047] I >> >> >>>> [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] >> >> >>>> 0-data_fast4-access-control: client: CTX_ID:4a654305-d2e4- >> >> >>>> >> >> >> >> >> >> >>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >> >> >>>> (uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >>>> [Permission denied] >> >> >>>> gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >> >> >13:22:51.818796] I >> >> >>>> [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] >> >> >>>> 0-data_fast4-access-control: client: CTX_ID:4a654305-d2e4- >> >> >>>> >> >> >> >> >> >> >>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >> >> >>>> (uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >>>> [Permission denied] >> >> >>>> gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >> >> >13:24:43.732856] I >> >> >>>> [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] >> >> >>>> 0-data_fast4-access-control: client: CTX_ID:4a654305-d2e4- >> >> >>>> >> >> >> >> >> >> >>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >> >> >>>> (uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >>>> [Permission denied] >> >> >>>> gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >> >> >13:26:50.758178] I >> >> >>>> [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] >> >> >>>> 0-data_fast4-access-control: client: CTX_ID:4a654305-d2e4- >> >> >>>> >> >> >> >> >> >> >>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >> >> >>>> (uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) >> >> >>>> [Permission denied] >> >> >>>> >> >> >>>> >> >> >>>> In my case , the workaround was to downgrade the gluster >> >packages >> >> >on all >> >> >>>> nodes (and reboot each node 1 by 1 ) if the major version is >the >> >> >same, but >> >> >>>> if you upgraded to v7.X - then you can try the v7.0 . >> >> >>>> >> >> >>>> Best Regards, >> >> >>>> Strahil Nikolov >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> В събота, 20 юни 2020 г., 18:48:42 ч. Гринуич+3, C Williams < >> >> >>>> cwilliams3320(a)gmail.com&gt; написа: >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> Hello, >> >> >>>> >> >> >>>> Here are additional log tiles as well as a tree of the >> >problematic >> >> >>>> Gluster storage domain. During this time I attempted to copy >a >> >> >virtual disk >> >> >>>> to another domain, move a virtual disk to another domain and >run >> >a >> >> >VM where >> >> >>>> the virtual hard disk would be used. >> >> >>>> >> >> >>>> The copies/moves failed and the VM went into pause mode when >the >> >> >virtual >> >> >>>> HDD was involved. >> >> >>>> >> >> >>>> Please check these out. >> >> >>>> >> >> >>>> Thank You For Your Help ! >> >> >>>> >> >> >>>> On Sat, Jun 20, 2020 at 9:54 AM C Williams >> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> wrote: >> >> >>>> > Strahil, >> >> >>>> > >> >> >>>> > I understand. Please keep me posted. >> >> >>>> > >> >> >>>> > Thanks For The Help ! >> >> >>>> > >> >> >>>> > On Sat, Jun 20, 2020 at 4:36 AM Strahil Nikolov >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> wrote: >> >> >>>> >> Hey C Williams, >> >> >>>> >> >> >> >>>> >> sorry for the delay, but I couldn't get somw time to >check >> >your >> >> >>>> logs. Will try a little bit later. >> >> >>>> >> >> >> >>>> >> Best Regards, >> >> >>>> >> Strahil Nikolov >> >> >>>> >> >> >> >>>> >> На 20 юни 2020 г. 2:37:22 GMT+03:00, C Williams < >> >> >>>> cwilliams3320(a)gmail.com&gt; написа: >> >> >>>> >>>Hello, >> >> >>>> >>> >> >> >>>> >>>Was wanting to follow up on this issue. Users are >impacted. >> >> >>>> >>> >> >> >>>> >>>Thank You >> >> >>>> >>> >> >> >>>> >>>On Fri, Jun 19, 2020 at 9:20 AM C Williams >> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>wrote: >> >> >>>> >>> >> >> >>>> >>>> Hello, >> >> >>>> >>>> >> >> >>>> >>>> Here are the logs (some IPs are changed ) >> >> >>>> >>>> >> >> >>>> >>>> ov05 is the SPM >> >> >>>> >>>> >> >> >>>> >>>> Thank You For Your Help ! >> >> >>>> >>>> >> >> >>>> >>>> On Thu, Jun 18, 2020 at 11:31 PM Strahil Nikolov >> >> >>>> &gt;&gt;&gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >>>> wrote: >> >> >>>> >>>> >> >> >>>> >>>>> Check on the hosts tab , which is your current SPM >(last >> >> >column in >> >> >>>> >>>Admin >> >> >>>> >>>>> UI). >> >> >>>> >>>>> Then open the /var/log/vdsm/vdsm.log and repeat the >> >> >operation. >> >> >>>> >>>>> Then provide the log from that host and the engine's >log >> >(on >> >> >the >> >> >>>> >>>>> HostedEngine VM or on your standalone engine). >> >> >>>> >>>>> >> >> >>>> >>>>> Best Regards, >> >> >>>> >>>>> Strahil Nikolov >> >> >>>> >>>>> >> >> >>>> >>>>> На 18 юни 2020 г. 23:59:36 GMT+03:00, C Williams >> >> >>>> &gt;&gt;&gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> написа: >> >> >>>> >>>>> >Resending to eliminate email issues >> >> >>>> >>>>> > >> >> >>>> >>>>> >---------- Forwarded message --------- >> >> >>>> >>>>> >From: C Williams <cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> >Date: Thu, Jun 18, 2020 at 4:01 PM >> >> >>>> >>>>> >Subject: Re: [ovirt-users] Fwd: Issues with Gluster >> >Domain >> >> >>>> >>>>> >To: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >> >>>> >>>>> > >> >> >>>> >>>>> > >> >> >>>> >>>>> >Here is output from mount >> >> >>>> >>>>> > >> >> >>>> >>>>> >192.168.24.12:/stor/import0 on >> >> >>>> >>>>> >/rhev/data-center/mnt/192.168.24.12:_stor_import0 >> >> >>>> >>>>> >type nfs4 >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.12) >> >> >>>> >>>>> >192.168.24.13:/stor/import1 on >> >> >>>> >>>>> >/rhev/data-center/mnt/192.168.24.13:_stor_import1 >> >> >>>> >>>>> >type nfs4 >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >> >> >>>> >>>>> >192.168.24.13:/stor/iso1 on >> >> >>>> >>>>> >/rhev/data-center/mnt/192.168.24.13:_stor_iso1 >> >> >>>> >>>>> >type nfs4 >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >> >> >>>> >>>>> >192.168.24.13:/stor/export0 on >> >> >>>> >>>>> >/rhev/data-center/mnt/192.168.24.13:_stor_export0 >> >> >>>> >>>>> >type nfs4 >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >> >> >>>> >>>>> >192.168.24.15:/images on >> >> >>>> >>>>> >/rhev/data-center/mnt/glusterSD/192.168.24.15:_images >> >> >>>> >>>>> >type fuse.glusterfs >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >> >> >>>> >>>>> >192.168.24.18:/images3 on >> >> >>>> >>>>> >/rhev/data-center/mnt/glusterSD/192.168.24.18:_images3 >> >> >>>> >>>>> >type fuse.glusterfs >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >> >> >>>> >>>>> >tmpfs on /run/user/0 type tmpfs >> >> >>>> >>>>> >> >>(rw,nosuid,nodev,relatime,seclabel,size=13198392k,mode=700) >> >> >>>> >>>>> >[root@ov06 glusterfs]# >> >> >>>> >>>>> > >> >> >>>> >>>>> >Also here is a screenshot of the console >> >> >>>> >>>>> > >> >> >>>> >>>>> >[image: image.png] >> >> >>>> >>>>> >The other domains are up >> >> >>>> >>>>> > >> >> >>>> >>>>> >Import0 and Import1 are NFS . GLCL0 is gluster. They >all >> >are >> >> >>>> >>>running >> >> >>>> >>>>> >VMs >> >> >>>> >>>>> > >> >> >>>> >>>>> >Thank You For Your Help ! >> >> >>>> >>>>> > >> >> >>>> >>>>> >On Thu, Jun 18, 2020 at 3:51 PM Strahil Nikolov >> >> >>>> &gt;&gt;&gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >>>>> >wrote: >> >> >>>> >>>>> > >> >> >>>> >>>>> >> I don't see >> >> >'/rhev/data-center/mnt/192.168.24.13:_stor_import1' >> >> >>>> >>>>> >mounted >> >> >>>> >>>>> >> at all . >> >> >>>> >>>>> >> What is the status of all storage domains ? >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> Best Regards, >> >> >>>> >>>>> >> Strahil Nikolov >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> На 18 юни 2020 г. 21:43:44 GMT+03:00, C Williams >> >> >>>> >>>>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> >> написа: >> >> >>>> >>>>> >> > Resending to deal with possible email issues >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >---------- Forwarded message --------- >> >> >>>> >>>>> >> >From: C Williams <cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> >> >Date: Thu, Jun 18, 2020 at 2:07 PM >> >> >>>> >>>>> >> >Subject: Re: [ovirt-users] Issues with Gluster >Domain >> >> >>>> >>>>> >> >To: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >More >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >[root@ov06 ~]# for i in $(gluster volume list); do >> >echo >> >> >>>> >>>$i;echo; >> >> >>>> >>>>> >> >gluster >> >> >>>> >>>>> >> >volume info $i; echo;echo;gluster volume status >> >> >>>> >>>>> >$i;echo;echo;echo;done >> >> >>>> >>>>> >> >images3 >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >Volume Name: images3 >> >> >>>> >>>>> >> >Type: Replicate >> >> >>>> >>>>> >> >Volume ID: 0243d439-1b29-47d0-ab39-d61c2f15ae8b >> >> >>>> >>>>> >> >Status: Started >> >> >>>> >>>>> >> >Snapshot Count: 0 >> >> >>>> >>>>> >> >Number of Bricks: 1 x 3 = 3 >> >> >>>> >>>>> >> >Transport-type: tcp >> >> >>>> >>>>> >> >Bricks: >> >> >>>> >>>>> >> >Brick1: 192.168.24.18:/bricks/brick04/images3 >> >> >>>> >>>>> >> >Brick2: 192.168.24.19:/bricks/brick05/images3 >> >> >>>> >>>>> >> >Brick3: 192.168.24.20:/bricks/brick06/images3 >> >> >>>> >>>>> >> >Options Reconfigured: >> >> >>>> >>>>> >> >performance.client-io-threads: on >> >> >>>> >>>>> >> >nfs.disable: on >> >> >>>> >>>>> >> >transport.address-family: inet >> >> >>>> >>>>> >> >user.cifs: off >> >> >>>> >>>>> >> >auth.allow: * >> >> >>>> >>>>> >> >performance.quick-read: off >> >> >>>> >>>>> >> >performance.read-ahead: off >> >> >>>> >>>>> >> >performance.io-cache: off >> >> >>>> >>>>> >> >performance.low-prio-threads: 32 >> >> >>>> >>>>> >> >network.remote-dio: off >> >> >>>> >>>>> >> >cluster.eager-lock: enable >> >> >>>> >>>>> >> >cluster.quorum-type: auto >> >> >>>> >>>>> >> >cluster.server-quorum-type: server >> >> >>>> >>>>> >> >cluster.data-self-heal-algorithm: full >> >> >>>> >>>>> >> >cluster.locking-scheme: granular >> >> >>>> >>>>> >> >cluster.shd-max-threads: 8 >> >> >>>> >>>>> >> >cluster.shd-wait-qlength: 10000 >> >> >>>> >>>>> >> >features.shard: on >> >> >>>> >>>>> >> >cluster.choose-local: off >> >> >>>> >>>>> >> >client.event-threads: 4 >> >> >>>> >>>>> >> >server.event-threads: 4 >> >> >>>> >>>>> >> >storage.owner-uid: 36 >> >> >>>> >>>>> >> >storage.owner-gid: 36 >> >> >>>> >>>>> >> >performance.strict-o-direct: on >> >> >>>> >>>>> >> >network.ping-timeout: 30 >> >> >>>> >>>>> >> >cluster.granular-entry-heal: enable >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >Status of volume: images3 >> >> >>>> >>>>> >> >Gluster process TCP >Port >> >> >RDMA Port >> >> >>>> >>>>> >Online >> >> >>>> >>>>> >> > Pid >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>>------------------------------------------------------------------------------ >> >> >>>> >>>>> >> >Brick 192.168.24.18:/bricks/brick04/images3 49152 >> >0 >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >6666 >> >> >>>> >>>>> >> >Brick 192.168.24.19:/bricks/brick05/images3 49152 >> >0 >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >6779 >> >> >>>> >>>>> >> >Brick 192.168.24.20:/bricks/brick06/images3 49152 >> >0 >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >7227 >> >> >>>> >>>>> >> >Self-heal Daemon on localhost N/A >> >N/A >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >6689 >> >> >>>> >>>>> >> >Self-heal Daemon on ov07.ntc.srcle.com N/A >> >N/A >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >6802 >> >> >>>> >>>>> >> >Self-heal Daemon on ov08.ntc.srcle.com N/A >> >N/A >> >> >>>> >> >> >>>> >>>Y >> >> >>>> >>>>> >> >7250 >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >Task Status of Volume images3 >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>>------------------------------------------------------------------------------ >> >> >>>> >>>>> >> >There are no active volume tasks >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >[root@ov06 ~]# ls -l >/rhev/data-center/mnt/glusterSD/ >> >> >>>> >>>>> >> >total 16 >> >> >>>> >>>>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 14:04 >> >> >192.168.24.15:_images >> >> >>>> >>>>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 14:05 >> >192.168.24.18: >> >> >>>> _images3 >> >> >>>> >>>>> >> >[root@ov06 ~]# >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >On Thu, Jun 18, 2020 at 2:03 PM C Williams >> >> >>>> &gt;&gt;&gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> >> >wrote: >> >> >>>> >>>>> >> > >> >> >>>> >>>>> >> >> Strahil, >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >> Here you go -- Thank You For Your Help ! >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >> BTW -- I can write a test file to gluster and it >> >> >replicates >> >> >>>> >>>>> >properly. >> >> >>>> >>>>> >> >> Thinking something about the oVirt Storage Domain >? >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >> [root@ov08 ~]# gluster pool list >> >> >>>> >>>>> >> >> UUID Hostname >> >> >>>> >>>>> >State >> >> >>>> >>>>> >> >> 5b40c659-d9ab-43c3-9af8-18b074ea0b83 ov06 >> >> >>>> >>>>> >> >Connected >> >> >>>> >>>>> >> >> 36ce5a00-6f65-4926-8438-696944ebadb5 >> >> >ov07.ntc.srcle.com >> >> >>>> >>>>> >> >Connected >> >> >>>> >>>>> >> >> c7e7abdb-a8f4-4842-924c-e227f0db1b29 localhost >> >> >>>> >>>>> >> >Connected >> >> >>>> >>>>> >> >> [root@ov08 ~]# gluster volume list >> >> >>>> >>>>> >> >> images3 >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >> On Thu, Jun 18, 2020 at 1:13 PM Strahil Nikolov >> >> >>>> >>>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> >> >>>> >>>>> >> >> wrote: >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >>> Log to the oVirt cluster and provide the output >of: >> >> >>>> >>>>> >> >>> gluster pool list >> >> >>>> >>>>> >> >>> gluster volume list >> >> >>>> >>>>> >> >>> for i in $(gluster volume list); do echo >$i;echo; >> >> >gluster >> >> >>>> >>>>> >volume >> >> >>>> >>>>> >> >info >> >> >>>> >>>>> >> >>> $i; echo;echo;gluster volume status >> >> >$i;echo;echo;echo;done >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >>> ls -l /rhev/data-center/mnt/glusterSD/ >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >>> Best Regards, >> >> >>>> >>>>> >> >>> Strahil Nikolov >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >>> На 18 юни 2020 г. 19:17:46 GMT+03:00, C Williams >> >> >>>> >>>>> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >> >> >>>> >>>>> >> >>> написа: >> >> >>>> >>>>> >> >>> >Hello, >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >I recently added 6 hosts to an existing oVirt >> >> >>>> >>>compute/gluster >> >> >>>> >>>>> >> >cluster. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >Prior to this attempted addition, my cluster >had 3 >> >> >>>> >>>Hypervisor >> >> >>>> >>>>> >hosts >> >> >>>> >>>>> >> >and >> >> >>>> >>>>> >> >>> >3 >> >> >>>> >>>>> >> >>> >gluster bricks which made up a single gluster >> >volume >> >> >>>> >>>(replica 3 >> >> >>>> >>>>> >> >volume) >> >> >>>> >>>>> >> >>> >. I >> >> >>>> >>>>> >> >>> >added the additional hosts and made a brick on >3 >> >of >> >> >the new >> >> >>>> >>>>> >hosts >> >> >>>> >>>>> >> >and >> >> >>>> >>>>> >> >>> >attempted to make a new replica 3 volume. I had >> >> >difficulty >> >> >>>> >>>>> >> >creating >> >> >>>> >>>>> >> >>> >the >> >> >>>> >>>>> >> >>> >new volume. So, I decided that I would make a >new >> >> >>>> >>>>> >compute/gluster >> >> >>>> >>>>> >> >>> >cluster >> >> >>>> >>>>> >> >>> >for each set of 3 new hosts. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >I removed the 6 new hosts from the existing >oVirt >> >> >>>> >>>>> >Compute/Gluster >> >> >>>> >>>>> >> >>> >Cluster >> >> >>>> >>>>> >> >>> >leaving the 3 original hosts in place with >their >> >> >bricks. At >> >> >>>> >>>that >> >> >>>> >>>>> >> >point >> >> >>>> >>>>> >> >>> >my >> >> >>>> >>>>> >> >>> >original bricks went down and came back up . >The >> >> >volume >> >> >>>> >>>showed >> >> >>>> >>>>> >> >entries >> >> >>>> >>>>> >> >>> >that >> >> >>>> >>>>> >> >>> >needed healing. At that point I ran gluster >volume >> >> >heal >> >> >>>> >>>images3 >> >> >>>> >>>>> >> >full, >> >> >>>> >>>>> >> >>> >etc. >> >> >>>> >>>>> >> >>> >The volume shows no unhealed entries. I also >> >> >corrected some >> >> >>>> >>>peer >> >> >>>> >>>>> >> >>> >errors. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >However, I am unable to copy disks, move disks >to >> >> >another >> >> >>>> >>>>> >domain, >> >> >>>> >>>>> >> >>> >export >> >> >>>> >>>>> >> >>> >disks, etc. It appears that the engine cannot >> >locate >> >> >disks >> >> >>>> >>>>> >properly >> >> >>>> >>>>> >> >and >> >> >>>> >>>>> >> >>> >I >> >> >>>> >>>>> >> >>> >get storage I/O errors. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >I have detached and removed the oVirt Storage >> >Domain. >> >> >I >> >> >>>> >>>>> >reimported >> >> >>>> >>>>> >> >the >> >> >>>> >>>>> >> >>> >domain and imported 2 VMs, But the VM disks >> >exhibit >> >> >the >> >> >>>> same >> >> >>>> >>>>> >> >behaviour >> >> >>>> >>>>> >> >>> >and >> >> >>>> >>>>> >> >>> >won't run from the hard disk. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >I get errors such as this >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >VDSM ov05 command HSMGetAllTasksStatusesVDS >> >failed: >> >> >low >> >> >>>> >>>level >> >> >>>> >>>>> >Image >> >> >>>> >>>>> >> >>> >copy >> >> >>>> >>>>> >> >>> >failed: ("Command ['/usr/bin/qemu-img', >'convert', >> >> >'-p', >> >> >>>> >>>'-t', >> >> >>>> >>>>> >> >'none', >> >> >>>> >>>>> >> >>> >'-T', 'none', '-f', 'raw', >> >> >>>> >>>>> >> >>> >>u'/rhev/data-center/mnt/glusterSD/192.168.24.18: >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>>_images3/5fe3ad3f-2d21-404c-832e-4dc7318ca10d/images/3ea5afbd-0fe0-4c09-8d39-e556c66a8b3d/fe6eab63-3b22-4815-bfe6-4a0ade292510', >> >> >>>> >>>>> >> >>> >'-O', 'raw', >> >> >>>> >>>>> >> >>> >u'/rhev/data-center/mnt/192.168.24.13: >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >>>> >>>>> >> >> >>>> >> >> >>>> >> >> >> >> >> >> >>>>>>>>_stor_import1/1ab89386-a2ba-448b-90ab-bc816f55a328/images/f707a218-9db7-4e23-8bbd-9b12972012b6/d6591ec5-3ede-443d-bd40-93119ca7c7d5'] >> >> >>>> >>>>> >> >>> >failed with rc=1 out='' >err=bytearray(b'qemu-img: >> >> >error >> >> >>>> >>>while >> >> >>>> >>>>> >> >reading >> >> >>>> >>>>> >> >>> >sector 135168: Transport endpoint is not >> >> >>>> >>>connected\\nqemu-img: >> >> >>>> >>>>> >> >error >> >> >>>> >>>>> >> >>> >while >> >> >>>> >>>>> >> >>> >reading sector 131072: Transport endpoint is >not >> >> >>>> >>>>> >> >connected\\nqemu-img: >> >> >>>> >>>>> >> >>> >error while reading sector 139264: Transport >> >endpoint >> >> >is >> >> >>>> not >> >> >>>> >>>>> >> >>> >connected\\nqemu-img: error while reading >sector >> >> >143360: >> >> >>>> >>>>> >Transport >> >> >>>> >>>>> >> >>> >endpoint >> >> >>>> >>>>> >> >>> >is not connected\\nqemu-img: error while >reading >> >> >sector >> >> >>>> >>>147456: >> >> >>>> >>>>> >> >>> >Transport >> >> >>>> >>>>> >> >>> >endpoint is not connected\\nqemu-img: error >while >> >> >reading >> >> >>>> >>>sector >> >> >>>> >>>>> >> >>> >155648: >> >> >>>> >>>>> >> >>> >Transport endpoint is not connected\\nqemu-img: >> >error >> >> >while >> >> >>>> >>>>> >reading >> >> >>>> >>>>> >> >>> >sector >> >> >>>> >>>>> >> >>> >151552: Transport endpoint is not >> >> >connected\\nqemu-img: >> >> >>>> >>>error >> >> >>>> >>>>> >while >> >> >>>> >>>>> >> >>> >reading >> >> >>>> >>>>> >> >>> >sector 159744: Transport endpoint is not >> >> >connected\\n')",) >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >oVirt version is 4.3.82-1.el7 >> >> >>>> >>>>> >> >>> >OS CentOS Linux release 7.7.1908 (Core) >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >The Gluster Cluster has been working very well >> >until >> >> >this >> >> >>>> >>>>> >incident. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >Please help. >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >Thank You >> >> >>>> >>>>> >> >>> > >> >> >>>> >>>>> >> >>> >Charles Williams >> >> >>>> >>>>> >> >>> >> >> >>>> >>>>> >> >> >> >> >>>> >>>>> >> >> >> >>>> >>>>> >> >> >>>> >>>> >> >> >>>> >> >> >> >>>> > >> >> >>>> _______________________________________________ >> >> >>>> Users mailing list -- users(a)ovirt.org >> >> >>>> To unsubscribe send an email to users-leave(a)ovirt.org >> >> >>>> Privacy Statement: https://www.ovirt.org/privacy-policy.html >> >> >>>> oVirt Code of Conduct: >> >> >>>> https://www.ovirt.org/community/about/community-guidelines/ >> >> >>>> List Archives: >> >> >>>> >> >> > >> >> >> > >> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/YY3VUKEJLI7... >> >> >>>> >> >> >>> >> >> >>