[ovirt-users] Re: Fwd: Re: Fwd: Fwd: Issues with Gluster Domain

As I told you, you could just downgrade gluster on all nodes and later plan to live migrate the VM disks. I had to copy my data to the new volume, so I can avoid the ACL bug , when I use newer versions of gluster. Let's clarify some details: 1. Which version of oVirt and Gluster are you using ? 2. You now have your old gluster volume attached to oVirt and the new volume unused, right ? 3. Did you copy the contents of the old volume to the new one ? Best Regards, Strahil Nikolov На 23 юни 2020 г. 4:34:19 GMT+03:00, C Williams <cwilliams3320(a)gmail.com&gt; написа: >Strahil, > >Thank You For Help ! > >Downgrading Gluster to 6.5 got the original storage domain working >again ! > >After, I finished my copy of the contents of the problematic volume to >a >new volume, I did the following > >Unmounted the mount points >Stopped the original problematic Gluster volume >On each problematic peer, >I downgraded Gluster to 6.5 >(yum downgrade glusterfs-6.5-1.el7.x86_64 >vdsm-gluster-4.30.46-1.el7.x86_64 >python2-gluster-6.5-1.el7.x86_64 glusterfs-libs-6.5-1.el7.x86_64 >glusterfs-cli-6.5-1.el7.x86_64 glusterfs-fuse-6.5-1.el7.x86_64 >glusterfs-rdma-6.5-1.el7.x86_64 glusterfs-api-6.5-1.el7.x86_64 >glusterfs-server-6.5-1.el7.x86_64 glusterfs-events-6.5-1.el7.x86_64 >glusterfs-client-xlators-6.5-1.el7.x86_64 >glusterfs-geo-replication-6.5-1.el7.x86_64) >Restarted glusterd (systemctl restart glusterd) >Restarted the problematic Gluster volume >Reattached the problematic storage domain >Started the problematic storage domain > >Things work now. I can now run VMs and write data, copy virtual disks, >move >virtual disks to other storage domains, etc. > >I am very thankful that the storage domain is working again ! > >How can I safely perform upgrades on Gluster ! When will it be safe to >do >so ? > >Thank You Again For Your Help ! > >On Mon, Jun 22, 2020 at 10:58 AM C Williams <cwilliams3320(a)gmail.com&gt; >wrote: > >> Strahil, >> >> I have downgraded the target. The copy from the problematic volume to >the >> target is going on now. >> Once I have the data copied, I might downgrade the problematic >volume's >> Gluster to 6.5. >> At that point I might reattach the original ovirt domain and see if >it >> will work again. >> But the copy is going on right now. >> >> Thank You For Your Help ! >> >> On Mon, Jun 22, 2020 at 10:52 AM Strahil Nikolov &gt;&lt;hunter86_bg(a)yahoo.com&gt; >> wrote: >> >>> You should ensure that in the storage domain tab, the old storage >is >>> not visible. >>> >>> I still wander why yoiu didn't try to downgrade first. >>> >>> Best Regards, >>> Strahil Nikolov >>> >>> На 22 юни 2020 г. 13:58:33 GMT+03:00, C Williams &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> написа: >>> >Strahil, >>> > >>> >The GLCL3 storage domain was detached prior to attempting to add >the >>> >new >>> >storage domain. >>> > >>> >Should I also "Remove" it ? >>> > >>> >Thank You For Your Help ! >>> > >>> >---------- Forwarded message --------- >>> >From: Strahil Nikolov <hunter86_bg(a)yahoo.com&gt; >>> >Date: Mon, Jun 22, 2020 at 12:50 AM >>> >Subject: Re: [ovirt-users] Re: Fwd: Fwd: Issues with Gluster Domain >>> >To: C Williams <cwilliams3320(a)gmail.com&gt; >>> >Cc: users <Users(a)ovirt.org&gt; >>> > >>> > >>> >You can't add the new volume as it contains the same data (UUID) as >the >>> >old >>> >one , thus you need to detach the old one before adding the new one >- >>> >of >>> >course this means downtime for all VMs on that storage. >>> > >>> >As you see , downgrading is more simpler. For me v6.5 was working, >>> >while >>> >anything above (6.6+) was causing complete lockdown. Also v7.0 was >>> >working, but it's supported in oVirt 4.4. >>> > >>> >Best Regards, >>> >Strahil Nikolov >>> > >>> >На 22 юни 2020 г. 7:21:15 GMT+03:00, C Williams >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >написа: >>> >>Another question >>> >> >>> >>What version could I downgrade to safely ? I am at 6.9 . >>> >> >>> >>Thank You For Your Help !! >>> >> >>> >>On Sun, Jun 21, 2020 at 11:38 PM Strahil Nikolov >>> &gt;&gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>wrote: >>> >> >>> >>> You are definitely reading it wrong. >>> >>> 1. I didn't create a new storage domain ontop this new volume. >>> >>> 2. I used cli >>> >>> >>> >>> Something like this (in your case it should be 'replica 3'): >>> >>> gluster volume create newvol replica 3 arbiter 1 >>> >>ovirt1:/new/brick/path >>> >>> ovirt2:/new/brick/path ovirt3:/new/arbiter/brick/path >>> >>> gluster volume start newvol >>> >>> >>> >>> #Detach oldvol from ovirt >>> >>> >>> >>> mount -t glusterfs ovirt1:/oldvol /mnt/oldvol >>> >>> mount -t glusterfs ovirt1:/newvol /mnt/newvol >>> >>> cp -a /mnt/oldvol/* /mnt/newvol >>> >>> >>> >>> #Add only newvol as a storage domain in oVirt >>> >>> #Import VMs >>> >>> >>> >>> I still think that you should downgrade your gluster packages!!! >>> >>> >>> >>> Best Regards, >>> >>> Strahil Nikolov >>> >>> >>> >>> На 22 юни 2020 г. 0:43:46 GMT+03:00, C Williams >>> &gt;&gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> написа: >>> >>> >Strahil, >>> >>> > >>> >>> >It sounds like you used a "System Managed Volume" for the new >>> >>storage >>> >>> >domain,is that correct? >>> >>> > >>> >>> >Thank You For Your Help ! >>> >>> > >>> >>> >On Sun, Jun 21, 2020 at 5:40 PM C Williams >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >wrote: >>> >>> > >>> >>> >> Strahil, >>> >>> >> >>> >>> >> So you made another oVirt Storage Domain -- then copied the >data >>> >>with >>> >>> >cp >>> >>> >> -a from the failed volume to the new volume. >>> >>> >> >>> >>> >> At the root of the volume there will be the old domain folder >id >>> >>ex >>> >>> >> 5fe3ad3f-2d21-404c-832e-4dc7318ca10d >>> >>> >> in my case. Did that cause issues with making the new domain >>> >>since >>> >>> >it is >>> >>> >> the same folder id as the old one ? >>> >>> >> >>> >>> >> Thank You For Your Help ! >>> >>> >> >>> >>> >> On Sun, Jun 21, 2020 at 5:18 PM Strahil Nikolov >>> >>> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >> wrote: >>> >>> >> >>> >>> >>> In my situation I had only the ovirt nodes. >>> >>> >>> >>> >>> >>> На 21 юни 2020 г. 22:43:04 GMT+03:00, C Williams >>> >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> написа: >>> >>> >>> >Strahil, >>> >>> >>> > >>> >>> >>> >So should I make the target volume on 3 bricks which do not >>> >have >>> >>> >ovirt >>> >>> >>> >-- >>> >>> >>> >just gluster ? In other words (3) Centos 7 hosts ? >>> >>> >>> > >>> >>> >>> >Thank You For Your Help ! >>> >>> >>> > >>> >>> >>> >On Sun, Jun 21, 2020 at 3:08 PM Strahil Nikolov >>> >>> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >wrote: >>> >>> >>> > >>> >>> >>> >> I created a fresh volume (which is not an ovirt sgorage >>> >>> >domain), >>> >>> >>> >set >>> >>> >>> >> the original storage domain in maintenance and >detached >>> >>it. >>> >>> >>> >> Then I 'cp -a ' the data from the old to the new volume. >>> >>Next, >>> >>> >I >>> >>> >>> >just >>> >>> >>> >> added the new storage domain (the old one was a >kind >>> >>of a >>> >>> >>> >> 'backup') - pointing to the new volume name. >>> >>> >>> >> >>> >>> >>> >> If you observe issues , I would recommend you to >>> >>downgrade >>> >>> >>> >> gluster packages one node at a time . Then you might >be >>> >>able >>> >>> >to >>> >>> >>> >> restore your oVirt operations. >>> >>> >>> >> >>> >>> >>> >> Best Regards, >>> >>> >>> >> Strahil Nikolov >>> >>> >>> >> >>> >>> >>> >> На 21 юни 2020 г. 18:01:31 GMT+03:00, C Williams >>> >>> >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> написа: >>> >>> >>> >> >Strahil, >>> >>> >>> >> > >>> >>> >>> >> >Thanks for the follow up ! >>> >>> >>> >> > >>> >>> >>> >> >How did you copy the data to another volume ? >>> >>> >>> >> > >>> >>> >>> >> >I have set up another storage domain GLCLNEW1 with a new >>> >>volume >>> >>> >>> >imgnew1 >>> >>> >>> >> >. >>> >>> >>> >> >How would you copy all of the data from the problematic >>> >>domain >>> >>> >GLCL3 >>> >>> >>> >> >with >>> >>> >>> >> >volume images3 to GLCLNEW1 and volume imgnew1 and >preserve >>> >>all >>> >>> >the >>> >>> >>> >VMs, >>> >>> >>> >> >VM >>> >>> >>> >> >disks, settings, etc. ? >>> >>> >>> >> > >>> >>> >>> >> >Remember all of the regular ovirt disk copy, disk move, >VM >>> >>> >export >>> >>> >>> >> >tools >>> >>> >>> >> >are failing and my VMs and disks are trapped on domain >GLCL3 >>> >>and >>> >>> >>> >volume >>> >>> >>> >> >images3 right now. >>> >>> >>> >> > >>> >>> >>> >> >Please let me know >>> >>> >>> >> > >>> >>> >>> >> >Thank You For Your Help ! >>> >>> >>> >> > >>> >>> >>> >> > >>> >>> >>> >> > >>> >>> >>> >> > >>> >>> >>> >> > >>> >>> >>> >> >On Sun, Jun 21, 2020 at 8:27 AM Strahil Nikolov >>> >>> >>> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >wrote: >>> >>> >>> >> > >>> >>> >>> >> >> Sorry to hear that. >>> >>> >>> >> >> I can say that for me 6.5 was working, while 6.6 >>> >didn't >>> >>> >and I >>> >>> >>> >> >upgraded >>> >>> >>> >> >> to 7.0 . >>> >>> >>> >> >> In the ended , I have ended with creating a new >fresh >>> >>> >volume >>> >>> >>> >and >>> >>> >>> >> >> physically copying the data there, then I detached >the >>> >>> >storage >>> >>> >>> >> >domains and >>> >>> >>> >> >> attached to the new ones (which holded the old >data), >>> >>but >>> >>> >I >>> >>> >>> >> >could >>> >>> >>> >> >> afford the downtime. >>> >>> >>> >> >> Also, I can say that v7.0 ( but not 7.1 or >anything >>> >>later) >>> >>> >>> >also >>> >>> >>> >> >> worked without the ACL issue, but it causes some >>> >trouble >>> >>> >in >>> >>> >>> >oVirt >>> >>> >>> >> >- so >>> >>> >>> >> >> avoid that unless you have no other options. >>> >>> >>> >> >> >>> >>> >>> >> >> Best Regards, >>> >>> >>> >> >> Strahil Nikolov >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >> На 21 юни 2020 г. 4:39:46 GMT+03:00, C Williams >>> >>> >>> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> написа: >>> >>> >>> >> >> >Hello, >>> >>> >>> >> >> > >>> >>> >>> >> >> >Upgrading diidn't help >>> >>> >>> >> >> > >>> >>> >>> >> >> >Still acl errors trying to use a Virtual Disk from a >VM >>> >>> >>> >> >> > >>> >>> >>> >> >> >[root@ov06 bricks]# tail bricks-brick04-images3.log | >>> >grep >>> >>> >acl >>> >>> >>> >> >> >[2020-06-21 01:33:45.665888] I [MSGID: 139001] >>> >>> >>> >> >> >[posix-acl.c:263:posix_acl_log_permit_denied] >>> >>> >>> >> >0-images3-access-control: >>> >>> >>> >> >> >client: >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>CTX_ID:3697a7f1-44fb-4258-96b0-98cb4137d195-GRAPH_ID:0-PID:6706-HOST:ov06.ntc.srcle.com-PC_NAME:images3-client-0-RECON_NO:-0, >>> >>> >>> >> >> >gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >req(uid:107,gid:107,perm:1,ngrps:3), >>> >>> >>> >> >> >>ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >[Permission denied] >>> >>> >>> >> >> >The message "I [MSGID: 139001] >>> >>> >>> >> >> >[posix-acl.c:263:posix_acl_log_permit_denied] >>> >>> >>> >> >0-images3-access-control: >>> >>> >>> >> >> >client: >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>CTX_ID:3697a7f1-44fb-4258-96b0-98cb4137d195-GRAPH_ID:0-PID:6706-HOST:ov06.ntc.srcle.com-PC_NAME:images3-client-0-RECON_NO:-0, >>> >>> >>> >> >> >gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >req(uid:107,gid:107,perm:1,ngrps:3), >>> >>> >>> >> >> >>ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >[Permission denied]" repeated 2 times between >[2020-06-21 >>> >>> >>> >> >> >01:33:45.665888] >>> >>> >>> >> >> >and [2020-06-21 01:33:45.806779] >>> >>> >>> >> >> > >>> >>> >>> >> >> >Thank You For Your Help ! >>> >>> >>> >> >> > >>> >>> >>> >> >> >On Sat, Jun 20, 2020 at 8:59 PM C Williams >>> >>> >>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >wrote: >>> >>> >>> >> >> > >>> >>> >>> >> >> >> Hello, >>> >>> >>> >> >> >> >>> >>> >>> >> >> >> Based on the situation, I am planning to upgrade >the 3 >>> >>> >affected >>> >>> >>> >> >> >hosts. >>> >>> >>> >> >> >> >>> >>> >>> >> >> >> My reasoning is that the hosts/bricks were attached >to >>> >>6.9 >>> >>> >at >>> >>> >>> >one >>> >>> >>> >> >> >time. >>> >>> >>> >> >> >> >>> >>> >>> >> >> >> Thanks For Your Help ! >>> >>> >>> >> >> >> >>> >>> >>> >> >> >> On Sat, Jun 20, 2020 at 8:38 PM C Williams >>> >>> >>> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >> wrote: >>> >>> >>> >> >> >> >>> >>> >>> >> >> >>> Strahil, >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> The gluster version on the current 3 gluster hosts >is >>> >>6.7 >>> >>> >>> >(last >>> >>> >>> >> >> >update >>> >>> >>> >> >> >>> 2/26). These 3 hosts provide 1 brick each for the >>> >>replica >>> >>> >3 >>> >>> >>> >> >volume. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> Earlier I had tried to add 6 additional hosts to >the >>> >>> >cluster. >>> >>> >>> >> >Those >>> >>> >>> >> >> >new >>> >>> >>> >> >> >>> hosts were 6.9 gluster. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> I attempted to make a new separate volume with 3 >>> >bricks >>> >>> >>> >provided >>> >>> >>> >> >by >>> >>> >>> >> >> >the 3 >>> >>> >>> >> >> >>> new gluster 6.9 hosts. After having many errors >from >>> >>the >>> >>> >>> >oVirt >>> >>> >>> >> >> >interface, >>> >>> >>> >> >> >>> I gave up and removed the 6 new hosts from the >>> >cluster. >>> >>> >That >>> >>> >>> >is >>> >>> >>> >> >> >where the >>> >>> >>> >> >> >>> problems started. The intent was to expand the >gluster >>> >>> >cluster >>> >>> >>> >> >while >>> >>> >>> >> >> >making >>> >>> >>> >> >> >>> 2 new volumes for that cluster. The ovirt compute >>> >>cluster >>> >>> >>> >would >>> >>> >>> >> >> >allow for >>> >>> >>> >> >> >>> efficient VM migration between 9 hosts -- while >having >>> >>> >>> >separate >>> >>> >>> >> >> >gluster >>> >>> >>> >> >> >>> volumes for safety purposes. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> Looking at the brick logs, I see where there are >acl >>> >>> >errors >>> >>> >>> >> >starting >>> >>> >>> >> >> >from >>> >>> >>> >> >> >>> the time of the removal of the 6 new hosts. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> Please check out the attached brick log from >6/14-18. >>> >>The >>> >>> >>> >events >>> >>> >>> >> >> >started >>> >>> >>> >> >> >>> on 6/17. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> I wish I had a downgrade path. >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> Thank You For The Help !! >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> On Sat, Jun 20, 2020 at 7:47 PM Strahil Nikolov >>> >>> >>> >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>> wrote: >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>>> Hi , >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> This one really looks like the ACL bug I was hit >with >>> >>> >when I >>> >>> >>> >> >> >updated >>> >>> >>> >> >> >>>> from Gluster v6.5 to 6.6 and later from 7.0 to >7.2. >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Did you update your setup recently ? Did you >upgrade >>> >>> >gluster >>> >>> >>> >> >also ? >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> You have to check the gluster logs in order to >verify >>> >>> >that, >>> >>> >>> >so >>> >>> >>> >> >you >>> >>> >>> >> >> >can >>> >>> >>> >> >> >>>> try: >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> 1. Set Gluster logs to trace level (for details >>> >check: >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> > >>> >>> >>> >> >> >>> >>> >>> >> > >>> >>> >>> >> >>> >>> >>> > >>> >>> >>> >>> >>> > >>> >>> >>> >> >>> > >>> > https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/h... >>> >>> >>> >> >> >>>> ) >>> >>> >>> >> >> >>>> 2. Power up a VM that was already off , or retry >the >>> >>> >>> >procedure >>> >>> >>> >> >from >>> >>> >>> >> >> >the >>> >>> >>> >> >> >>>> logs you sent. >>> >>> >>> >> >> >>>> 3. Stop the trace level of the logs >>> >>> >>> >> >> >>>> 4. Check libvirt logs on the host that was >supposed >>> >to >>> >>> >power >>> >>> >>> >up >>> >>> >>> >> >the >>> >>> >>> >> >> >VM >>> >>> >>> >> >> >>>> (in case a VM was powered on) >>> >>> >>> >> >> >>>> 5. Check the gluster brick logs on all nodes for >ACL >>> >>> >errors. >>> >>> >>> >> >> >>>> Here is a sample from my old logs: >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >>> >>> >>> >> >> >13:19:41.489047] I >>> >>> >>> >> >> >>>> [MSGID: 139001] >>> >>> >[posix-acl.c:262:posix_acl_log_permit_denied] >>> >>> >>> >> >> >>>> 0-data_fast4-access-control: client: >>> >>> >CTX_ID:4a654305-d2e4- >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >>> >>> >>> >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >>> >>> >>> >> >> >>>> >>> >(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >>>> [Permission denied] >>> >>> >>> >> >> >>>> >gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >>> >>> >>> >> >> >13:22:51.818796] I >>> >>> >>> >> >> >>>> [MSGID: 139001] >>> >>> >[posix-acl.c:262:posix_acl_log_permit_denied] >>> >>> >>> >> >> >>>> 0-data_fast4-access-control: client: >>> >>> >CTX_ID:4a654305-d2e4- >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >>> >>> >>> >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >>> >>> >>> >> >> >>>> >>> >(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >>>> [Permission denied] >>> >>> >>> >> >> >>>> >gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >>> >>> >>> >> >> >13:24:43.732856] I >>> >>> >>> >> >> >>>> [MSGID: 139001] >>> >>> >[posix-acl.c:262:posix_acl_log_permit_denied] >>> >>> >>> >> >> >>>> 0-data_fast4-access-control: client: >>> >>> >CTX_ID:4a654305-d2e4- >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >>> >>> >>> >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >>> >>> >>> >> >> >>>> >>> >(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >>>> [Permission denied] >>> >>> >>> >> >> >>>> >gluster_bricks-data_fast4-data_fast4.log:[2020-03-18 >>> >>> >>> >> >> >13:26:50.758178] I >>> >>> >>> >> >> >>>> [MSGID: 139001] >>> >>> >[posix-acl.c:262:posix_acl_log_permit_denied] >>> >>> >>> >> >> >>>> 0-data_fast4-access-control: client: >>> >>> >CTX_ID:4a654305-d2e4- >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>4a10-bad9-58d670d99a97-GRAPH_ID:0-PID:32412-HOST:ovirt1.localdomain-PC_NAME:data_fast4-client-0-RECON_NO:-19, >>> >>> >>> >> >> >>>> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, >>> >>> >>> >> >> >>>> req(uid:36,gid:36,perm:1,ngrps:3), ctx >>> >>> >>> >> >> >>>> >>> >(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, >>> >>> >acl:-) >>> >>> >>> >> >> >>>> [Permission denied] >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> In my case , the workaround was to downgrade the >>> >>gluster >>> >>> >>> >> >packages >>> >>> >>> >> >> >on all >>> >>> >>> >> >> >>>> nodes (and reboot each node 1 by 1 ) if the major >>> >>version >>> >>> >is >>> >>> >>> >the >>> >>> >>> >> >> >same, but >>> >>> >>> >> >> >>>> if you upgraded to v7.X - then you can try the >v7.0 . >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Best Regards, >>> >>> >>> >> >> >>>> Strahil Nikolov >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> В събота, 20 юни 2020 г., 18:48:42 ч. Гринуич+3, >C >>> >>> >Williams < >>> >>> >>> >> >> >>>> cwilliams3320(a)gmail.com&gt; написа: >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Hello, >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Here are additional log tiles as well as a tree >of >>> >the >>> >>> >>> >> >problematic >>> >>> >>> >> >> >>>> Gluster storage domain. During this time I >attempted >>> >>to >>> >>> >copy >>> >>> >>> >a >>> >>> >>> >> >> >virtual disk >>> >>> >>> >> >> >>>> to another domain, move a virtual disk to another >>> >>domain >>> >>> >and >>> >>> >>> >run >>> >>> >>> >> >a >>> >>> >>> >> >> >VM where >>> >>> >>> >> >> >>>> the virtual hard disk would be used. >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> The copies/moves failed and the VM went into >pause >>> >>mode >>> >>> >when >>> >>> >>> >the >>> >>> >>> >> >> >virtual >>> >>> >>> >> >> >>>> HDD was involved. >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Please check these out. >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> Thank You For Your Help ! >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> On Sat, Jun 20, 2020 at 9:54 AM C Williams >>> >>> >>> >> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> wrote: >>> >>> >>> >> >> >>>> > Strahil, >>> >>> >>> >> >> >>>> > >>> >>> >>> >> >> >>>> > I understand. Please keep me posted. >>> >>> >>> >> >> >>>> > >>> >>> >>> >> >> >>>> > Thanks For The Help ! >>> >>> >>> >> >> >>>> > >>> >>> >>> >> >> >>>> > On Sat, Jun 20, 2020 at 4:36 AM Strahil Nikolov >>> >>> >>> >> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> wrote: >>> >>> >>> >> >> >>>> >> Hey C Williams, >>> >>> >>> >> >> >>>> >> >>> >>> >>> >> >> >>>> >> sorry for the delay, but I couldn't get somw >time >>> >>to >>> >>> >>> >check >>> >>> >>> >> >your >>> >>> >>> >> >> >>>> logs. Will try a little bit later. >>> >>> >>> >> >> >>>> >> >>> >>> >>> >> >> >>>> >> Best Regards, >>> >>> >>> >> >> >>>> >> Strahil Nikolov >>> >>> >>> >> >> >>>> >> >>> >>> >>> >> >> >>>> >> На 20 юни 2020 г. 2:37:22 GMT+03:00, C >Williams < >>> >>> >>> >> >> >>>> cwilliams3320(a)gmail.com&gt; написа: >>> >>> >>> >> >> >>>> >>>Hello, >>> >>> >>> >> >> >>>> >>> >>> >>> >>> >> >> >>>> >>>Was wanting to follow up on this issue. Users >are >>> >>> >>> >impacted. >>> >>> >>> >> >> >>>> >>> >>> >>> >>> >> >> >>>> >>>Thank You >>> >>> >>> >> >> >>>> >>> >>> >>> >>> >> >> >>>> >>>On Fri, Jun 19, 2020 at 9:20 AM C Williams >>> >>> >>> >> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>wrote: >>> >>> >>> >> >> >>>> >>> >>> >>> >>> >> >> >>>> >>>> Hello, >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >>>> Here are the logs (some IPs are changed ) >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >>>> ov05 is the SPM >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >>>> Thank You For Your Help ! >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >>>> On Thu, Jun 18, 2020 at 11:31 PM Strahil >Nikolov >>> >>> >>> >> >> >>>> &gt;&gt;&gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> >>>> wrote: >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >>>>> Check on the hosts tab , which is your >current >>> >>SPM >>> >>> >>> >(last >>> >>> >>> >> >> >column in >>> >>> >>> >> >> >>>> >>>Admin >>> >>> >>> >> >> >>>> >>>>> UI). >>> >>> >>> >> >> >>>> >>>>> Then open the /var/log/vdsm/vdsm.log and >>> >repeat >>> >>> >the >>> >>> >>> >> >> >operation. >>> >>> >>> >> >> >>>> >>>>> Then provide the log from that host and the >>> >>> >engine's >>> >>> >>> >log >>> >>> >>> >> >(on >>> >>> >>> >> >> >the >>> >>> >>> >> >> >>>> >>>>> HostedEngine VM or on your standalone >engine). >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> Best Regards, >>> >>> >>> >> >> >>>> >>>>> Strahil Nikolov >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> На 18 юни 2020 г. 23:59:36 GMT+03:00, C >>> >Williams >>> >>> >>> >> >> >>>> &gt;&gt;&gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> написа: >>> >>> >>> >> >> >>>> >>>>> >Resending to eliminate email issues >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >---------- Forwarded message --------- >>> >>> >>> >> >> >>>> >>>>> >From: C Williams <cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> >Date: Thu, Jun 18, 2020 at 4:01 PM >>> >>> >>> >> >> >>>> >>>>> >Subject: Re: [ovirt-users] Fwd: Issues >with >>> >>> >Gluster >>> >>> >>> >> >Domain >>> >>> >>> >> >> >>>> >>>>> >To: Strahil Nikolov &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >Here is output from mount >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >192.168.24.12:/stor/import0 on >>> >>> >>> >> >> >>>> >>>>> >>> >>>/rhev/data-center/mnt/192.168.24.12:_stor_import0 >>> >>> >>> >> >> >>>> >>>>> >type nfs4 >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.12) >>> >>> >>> >> >> >>>> >>>>> >192.168.24.13:/stor/import1 on >>> >>> >>> >> >> >>>> >>>>> >>> >>>/rhev/data-center/mnt/192.168.24.13:_stor_import1 >>> >>> >>> >> >> >>>> >>>>> >type nfs4 >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >>> >>> >>> >> >> >>>> >>>>> >192.168.24.13:/stor/iso1 on >>> >>> >>> >> >> >>>> >>>>> >>/rhev/data-center/mnt/192.168.24.13:_stor_iso1 >>> >>> >>> >> >> >>>> >>>>> >type nfs4 >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >>> >>> >>> >> >> >>>> >>>>> >192.168.24.13:/stor/export0 on >>> >>> >>> >> >> >>>> >>>>> >>> >>>/rhev/data-center/mnt/192.168.24.13:_stor_export0 >>> >>> >>> >> >> >>>> >>>>> >type nfs4 >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=192.168.24.18,local_lock=none,addr=192.168.24.13) >>> >>> >>> >> >> >>>> >>>>> >192.168.24.15:/images on >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>/rhev/data-center/mnt/glusterSD/192.168.24.15:_images >>> >>> >>> >> >> >>>> >>>>> >type fuse.glusterfs >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >>> >>> >>> >> >> >>>> >>>>> >192.168.24.18:/images3 on >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>/rhev/data-center/mnt/glusterSD/192.168.24.18:_images3 >>> >>> >>> >> >> >>>> >>>>> >type fuse.glusterfs >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072) >>> >>> >>> >> >> >>>> >>>>> >tmpfs on /run/user/0 type tmpfs >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >>>(rw,nosuid,nodev,relatime,seclabel,size=13198392k,mode=700) >>> >>> >>> >> >> >>>> >>>>> >[root@ov06 glusterfs]# >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >Also here is a screenshot of the console >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >[image: image.png] >>> >>> >>> >> >> >>>> >>>>> >The other domains are up >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >Import0 and Import1 are NFS . GLCL0 is >>> >gluster. >>> >>> >They >>> >>> >>> >all >>> >>> >>> >> >are >>> >>> >>> >> >> >>>> >>>running >>> >>> >>> >> >> >>>> >>>>> >VMs >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >Thank You For Your Help ! >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >On Thu, Jun 18, 2020 at 3:51 PM Strahil >>> >Nikolov >>> >>> >>> >> >> >>>> &gt;&gt;&gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> >>>>> >wrote: >>> >>> >>> >> >> >>>> >>>>> > >>> >>> >>> >> >> >>>> >>>>> >> I don't see >>> >>> >>> >> >> >'/rhev/data-center/mnt/192.168.24.13:_stor_import1' >>> >>> >>> >> >> >>>> >>>>> >mounted >>> >>> >>> >> >> >>>> >>>>> >> at all . >>> >>> >>> >> >> >>>> >>>>> >> What is the status of all storage >domains ? >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> Best Regards, >>> >>> >>> >> >> >>>> >>>>> >> Strahil Nikolov >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> На 18 юни 2020 г. 21:43:44 GMT+03:00, C >>> >>Williams >>> >>> >>> >> >> >>>> >>>>> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> написа: >>> >>> >>> >> >> >>>> >>>>> >> > Resending to deal with possible email >>> >>issues >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >---------- Forwarded message --------- >>> >>> >>> >> >> >>>> >>>>> >> >From: C Williams &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> >Date: Thu, Jun 18, 2020 at 2:07 PM >>> >>> >>> >> >> >>>> >>>>> >> >Subject: Re: [ovirt-users] Issues with >>> >>Gluster >>> >>> >>> >Domain >>> >>> >>> >> >> >>>> >>>>> >> >To: Strahil Nikolov &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >More >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >[root@ov06 ~]# for i in $(gluster >volume >>> >>list); >>> >>> > do >>> >>> >>> >> >echo >>> >>> >>> >> >> >>>> >>>$i;echo; >>> >>> >>> >> >> >>>> >>>>> >> >gluster >>> >>> >>> >> >> >>>> >>>>> >> >volume info $i; echo;echo;gluster >volume >>> >>status >>> >>> >>> >> >> >>>> >>>>> >$i;echo;echo;echo;done >>> >>> >>> >> >> >>>> >>>>> >> >images3 >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >Volume Name: images3 >>> >>> >>> >> >> >>>> >>>>> >> >Type: Replicate >>> >>> >>> >> >> >>>> >>>>> >> >Volume ID: >>> >>0243d439-1b29-47d0-ab39-d61c2f15ae8b >>> >>> >>> >> >> >>>> >>>>> >> >Status: Started >>> >>> >>> >> >> >>>> >>>>> >> >Snapshot Count: 0 >>> >>> >>> >> >> >>>> >>>>> >> >Number of Bricks: 1 x 3 = 3 >>> >>> >>> >> >> >>>> >>>>> >> >Transport-type: tcp >>> >>> >>> >> >> >>>> >>>>> >> >Bricks: >>> >>> >>> >> >> >>>> >>>>> >> >Brick1: >>> >>192.168.24.18:/bricks/brick04/images3 >>> >>> >>> >> >> >>>> >>>>> >> >Brick2: >>> >>192.168.24.19:/bricks/brick05/images3 >>> >>> >>> >> >> >>>> >>>>> >> >Brick3: >>> >>192.168.24.20:/bricks/brick06/images3 >>> >>> >>> >> >> >>>> >>>>> >> >Options Reconfigured: >>> >>> >>> >> >> >>>> >>>>> >> >performance.client-io-threads: on >>> >>> >>> >> >> >>>> >>>>> >> >nfs.disable: on >>> >>> >>> >> >> >>>> >>>>> >> >transport.address-family: inet >>> >>> >>> >> >> >>>> >>>>> >> >user.cifs: off >>> >>> >>> >> >> >>>> >>>>> >> >auth.allow: * >>> >>> >>> >> >> >>>> >>>>> >> >performance.quick-read: off >>> >>> >>> >> >> >>>> >>>>> >> >performance.read-ahead: off >>> >>> >>> >> >> >>>> >>>>> >> >performance.io-cache: off >>> >>> >>> >> >> >>>> >>>>> >> >performance.low-prio-threads: 32 >>> >>> >>> >> >> >>>> >>>>> >> >network.remote-dio: off >>> >>> >>> >> >> >>>> >>>>> >> >cluster.eager-lock: enable >>> >>> >>> >> >> >>>> >>>>> >> >cluster.quorum-type: auto >>> >>> >>> >> >> >>>> >>>>> >> >cluster.server-quorum-type: server >>> >>> >>> >> >> >>>> >>>>> >> >cluster.data-self-heal-algorithm: full >>> >>> >>> >> >> >>>> >>>>> >> >cluster.locking-scheme: granular >>> >>> >>> >> >> >>>> >>>>> >> >cluster.shd-max-threads: 8 >>> >>> >>> >> >> >>>> >>>>> >> >cluster.shd-wait-qlength: 10000 >>> >>> >>> >> >> >>>> >>>>> >> >features.shard: on >>> >>> >>> >> >> >>>> >>>>> >> >cluster.choose-local: off >>> >>> >>> >> >> >>>> >>>>> >> >client.event-threads: 4 >>> >>> >>> >> >> >>>> >>>>> >> >server.event-threads: 4 >>> >>> >>> >> >> >>>> >>>>> >> >storage.owner-uid: 36 >>> >>> >>> >> >> >>>> >>>>> >> >storage.owner-gid: 36 >>> >>> >>> >> >> >>>> >>>>> >> >performance.strict-o-direct: on >>> >>> >>> >> >> >>>> >>>>> >> >network.ping-timeout: 30 >>> >>> >>> >> >> >>>> >>>>> >> >cluster.granular-entry-heal: enable >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >Status of volume: images3 >>> >>> >>> >> >> >>>> >>>>> >> >Gluster process >>> >>TCP >>> >>> >>> >Port >>> >>> >>> >> >> >RDMA Port >>> >>> >>> >> >> >>>> >>>>> >Online >>> >>> >>> >> >> >>>> >>>>> >> > Pid >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>>------------------------------------------------------------------------------ >>> >>> >>> >> >> >>>> >>>>> >> >Brick >192.168.24.18:/bricks/brick04/images3 >>> >>> >49152 >>> >>> >>> >> >0 >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >6666 >>> >>> >>> >> >> >>>> >>>>> >> >Brick >192.168.24.19:/bricks/brick05/images3 >>> >>> >49152 >>> >>> >>> >> >0 >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >6779 >>> >>> >>> >> >> >>>> >>>>> >> >Brick >192.168.24.20:/bricks/brick06/images3 >>> >>> >49152 >>> >>> >>> >> >0 >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >7227 >>> >>> >>> >> >> >>>> >>>>> >> >Self-heal Daemon on localhost >>> >>N/A >>> >>> >>> >> >N/A >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >6689 >>> >>> >>> >> >> >>>> >>>>> >> >Self-heal Daemon on ov07.ntc.srcle.com >>> >>N/A >>> >>> >>> >> >N/A >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >6802 >>> >>> >>> >> >> >>>> >>>>> >> >Self-heal Daemon on ov08.ntc.srcle.com >>> >>N/A >>> >>> >>> >> >N/A >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>>Y >>> >>> >>> >> >> >>>> >>>>> >> >7250 >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >Task Status of Volume images3 >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>>------------------------------------------------------------------------------ >>> >>> >>> >> >> >>>> >>>>> >> >There are no active volume tasks >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >[root@ov06 ~]# ls -l >>> >>> >>> >/rhev/data-center/mnt/glusterSD/ >>> >>> >>> >> >> >>>> >>>>> >> >total 16 >>> >>> >>> >> >> >>>> >>>>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 >14:04 >>> >>> >>> >> >> >192.168.24.15:_images >>> >>> >>> >> >> >>>> >>>>> >> >drwxr-xr-x. 5 vdsm kvm 8192 Jun 18 >14:05 >>> >>> >>> >> >192.168.24.18: >>> >>> >>> >> >> >>>> _images3 >>> >>> >>> >> >> >>>> >>>>> >> >[root@ov06 ~]# >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >On Thu, Jun 18, 2020 at 2:03 PM C >Williams >>> >>> >>> >> >> >>>> &gt;&gt;&gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> >wrote: >>> >>> >>> >> >> >>>> >>>>> >> > >>> >>> >>> >> >> >>>> >>>>> >> >> Strahil, >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >> Here you go -- Thank You For Your >Help ! >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >> BTW -- I can write a test file to >gluster >>> >>and >>> >>> >it >>> >>> >>> >> >> >replicates >>> >>> >>> >> >> >>>> >>>>> >properly. >>> >>> >>> >> >> >>>> >>>>> >> >> Thinking something about the oVirt >>> >Storage >>> >>> >Domain >>> >>> >>> >? >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >> [root@ov08 ~]# gluster pool list >>> >>> >>> >> >> >>>> >>>>> >> >> UUID >>> >>> >Hostname >>> >>> >>> >> >> >>>> >>>>> >State >>> >>> >>> >> >> >>>> >>>>> >> >> 5b40c659-d9ab-43c3-9af8-18b074ea0b83 >>> >>ov06 >>> >>> >>> >> >> >>>> >>>>> >> >Connected >>> >>> >>> >> >> >>>> >>>>> >> >> 36ce5a00-6f65-4926-8438-696944ebadb5 >>> >>> >>> >> >> >ov07.ntc.srcle.com >>> >>> >>> >> >> >>>> >>>>> >> >Connected >>> >>> >>> >> >> >>>> >>>>> >> >> c7e7abdb-a8f4-4842-924c-e227f0db1b29 >>> >>> >localhost >>> >>> >>> >> >> >>>> >>>>> >> >Connected >>> >>> >>> >> >> >>>> >>>>> >> >> [root@ov08 ~]# gluster volume list >>> >>> >>> >> >> >>>> >>>>> >> >> images3 >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >> On Thu, Jun 18, 2020 at 1:13 PM >Strahil >>> >>> >Nikolov >>> >>> >>> >> >> >>>> >>>>> >> &gt;&lt;hunter86_bg(a)yahoo.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> >> wrote: >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> Log to the oVirt cluster and provide >the >>> >>> >output >>> >>> >>> >of: >>> >>> >>> >> >> >>>> >>>>> >> >>> gluster pool list >>> >>> >>> >> >> >>>> >>>>> >> >>> gluster volume list >>> >>> >>> >> >> >>>> >>>>> >> >>> for i in $(gluster volume list); do >>> >>echo >>> >>> >>> >$i;echo; >>> >>> >>> >> >> >gluster >>> >>> >>> >> >> >>>> >>>>> >volume >>> >>> >>> >> >> >>>> >>>>> >> >info >>> >>> >>> >> >> >>>> >>>>> >> >>> $i; echo;echo;gluster volume status >>> >>> >>> >> >> >$i;echo;echo;echo;done >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> ls -l >/rhev/data-center/mnt/glusterSD/ >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> Best Regards, >>> >>> >>> >> >> >>>> >>>>> >> >>> Strahil Nikolov >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> На 18 юни 2020 г. 19:17:46 >GMT+03:00, C >>> >>> >Williams >>> >>> >>> >> >> >>>> >>>>> >> &gt;&lt;cwilliams3320(a)gmail.com&gt; >>> >>> >>> >> >> >>>> >>>>> >> >>> написа: >>> >>> >>> >> >> >>>> >>>>> >> >>> >Hello, >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >I recently added 6 hosts to an >existing >>> >>> >oVirt >>> >>> >>> >> >> >>>> >>>compute/gluster >>> >>> >>> >> >> >>>> >>>>> >> >cluster. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >Prior to this attempted addition, >my >>> >>> >cluster >>> >>> >>> >had 3 >>> >>> >>> >> >> >>>> >>>Hypervisor >>> >>> >>> >> >> >>>> >>>>> >hosts >>> >>> >>> >> >> >>>> >>>>> >> >and >>> >>> >>> >> >> >>>> >>>>> >> >>> >3 >>> >>> >>> >> >> >>>> >>>>> >> >>> >gluster bricks which made up a >single >>> >>> >gluster >>> >>> >>> >> >volume >>> >>> >>> >> >> >>>> >>>(replica 3 >>> >>> >>> >> >> >>>> >>>>> >> >volume) >>> >>> >>> >> >> >>>> >>>>> >> >>> >. I >>> >>> >>> >> >> >>>> >>>>> >> >>> >added the additional hosts and made >a >>> >>brick >>> >>> >on >>> >>> >>> >3 >>> >>> >>> >> >of >>> >>> >>> >> >> >the new >>> >>> >>> >> >> >>>> >>>>> >hosts >>> >>> >>> >> >> >>>> >>>>> >> >and >>> >>> >>> >> >> >>>> >>>>> >> >>> >attempted to make a new replica 3 >>> >>volume. I >>> >>> >had >>> >>> >>> >> >> >difficulty >>> >>> >>> >> >> >>>> >>>>> >> >creating >>> >>> >>> >> >> >>>> >>>>> >> >>> >the >>> >>> >>> >> >> >>>> >>>>> >> >>> >new volume. So, I decided that I >would >>> >>make >>> >>> >a >>> >>> >>> >new >>> >>> >>> >> >> >>>> >>>>> >compute/gluster >>> >>> >>> >> >> >>>> >>>>> >> >>> >cluster >>> >>> >>> >> >> >>>> >>>>> >> >>> >for each set of 3 new hosts. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >I removed the 6 new hosts from the >>> >>existing >>> >>> >>> >oVirt >>> >>> >>> >> >> >>>> >>>>> >Compute/Gluster >>> >>> >>> >> >> >>>> >>>>> >> >>> >Cluster >>> >>> >>> >> >> >>>> >>>>> >> >>> >leaving the 3 original hosts in >place >>> >>with >>> >>> >>> >their >>> >>> >>> >> >> >bricks. At >>> >>> >>> >> >> >>>> >>>that >>> >>> >>> >> >> >>>> >>>>> >> >point >>> >>> >>> >> >> >>>> >>>>> >> >>> >my >>> >>> >>> >> >> >>>> >>>>> >> >>> >original bricks went down and came >back >>> >>up >>> >>> >. >>> >>> >>> >The >>> >>> >>> >> >> >volume >>> >>> >>> >> >> >>>> >>>showed >>> >>> >>> >> >> >>>> >>>>> >> >entries >>> >>> >>> >> >> >>>> >>>>> >> >>> >that >>> >>> >>> >> >> >>>> >>>>> >> >>> >needed healing. At that point I ran >>> >>gluster >>> >>> >>> >volume >>> >>> >>> >> >> >heal >>> >>> >>> >> >> >>>> >>>images3 >>> >>> >>> >> >> >>>> >>>>> >> >full, >>> >>> >>> >> >> >>>> >>>>> >> >>> >etc. >>> >>> >>> >> >> >>>> >>>>> >> >>> >The volume shows no unhealed >entries. I >>> >>> >also >>> >>> >>> >> >> >corrected some >>> >>> >>> >> >> >>>> >>>peer >>> >>> >>> >> >> >>>> >>>>> >> >>> >errors. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >However, I am unable to copy disks, >>> >move >>> >>> >disks >>> >>> >>> >to >>> >>> >>> >> >> >another >>> >>> >>> >> >> >>>> >>>>> >domain, >>> >>> >>> >> >> >>>> >>>>> >> >>> >export >>> >>> >>> >> >> >>>> >>>>> >> >>> >disks, etc. It appears that the >engine >>> >>> >cannot >>> >>> >>> >> >locate >>> >>> >>> >> >> >disks >>> >>> >>> >> >> >>>> >>>>> >properly >>> >>> >>> >> >> >>>> >>>>> >> >and >>> >>> >>> >> >> >>>> >>>>> >> >>> >I >>> >>> >>> >> >> >>>> >>>>> >> >>> >get storage I/O errors. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >I have detached and removed the >oVirt >>> >>> >Storage >>> >>> >>> >> >Domain. >>> >>> >>> >> >> >I >>> >>> >>> >> >> >>>> >>>>> >reimported >>> >>> >>> >> >> >>>> >>>>> >> >the >>> >>> >>> >> >> >>>> >>>>> >> >>> >domain and imported 2 VMs, But the >VM >>> >>disks >>> >>> >>> >> >exhibit >>> >>> >>> >> >> >the >>> >>> >>> >> >> >>>> same >>> >>> >>> >> >> >>>> >>>>> >> >behaviour >>> >>> >>> >> >> >>>> >>>>> >> >>> >and >>> >>> >>> >> >> >>>> >>>>> >> >>> >won't run from the hard disk. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >I get errors such as this >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >VDSM ov05 command >>> >>HSMGetAllTasksStatusesVDS >>> >>> >>> >> >failed: >>> >>> >>> >> >> >low >>> >>> >>> >> >> >>>> >>>level >>> >>> >>> >> >> >>>> >>>>> >Image >>> >>> >>> >> >> >>>> >>>>> >> >>> >copy >>> >>> >>> >> >> >>>> >>>>> >> >>> >failed: ("Command >['/usr/bin/qemu-img', >>> >>> >>> >'convert', >>> >>> >>> >> >> >'-p', >>> >>> >>> >> >> >>>> >>>'-t', >>> >>> >>> >> >> >>>> >>>>> >> >'none', >>> >>> >>> >> >> >>>> >>>>> >> >>> >'-T', 'none', '-f', 'raw', >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >>u'/rhev/data-center/mnt/glusterSD/192.168.24.18: >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>>_images3/5fe3ad3f-2d21-404c-832e-4dc7318ca10d/images/3ea5afbd-0fe0-4c09-8d39-e556c66a8b3d/fe6eab63-3b22-4815-bfe6-4a0ade292510', >>> >>> >>> >> >> >>>> >>>>> >> >>> >'-O', 'raw', >>> >>> >>> >> >> >>>> >>>>> >> >>> >>u'/rhev/data-center/mnt/192.168.24.13: >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>>>>>>>>>>_stor_import1/1ab89386-a2ba-448b-90ab-bc816f55a328/images/f707a218-9db7-4e23-8bbd-9b12972012b6/d6591ec5-3ede-443d-bd40-93119ca7c7d5'] >>> >>> >>> >> >> >>>> >>>>> >> >>> >failed with rc=1 out='' >>> >>> >>> >err=bytearray(b'qemu-img: >>> >>> >>> >> >> >error >>> >>> >>> >> >> >>>> >>>while >>> >>> >>> >> >> >>>> >>>>> >> >reading >>> >>> >>> >> >> >>>> >>>>> >> >>> >sector 135168: Transport endpoint >is >>> >not >>> >>> >>> >> >> >>>> >>>connected\\nqemu-img: >>> >>> >>> >> >> >>>> >>>>> >> >error >>> >>> >>> >> >> >>>> >>>>> >> >>> >while >>> >>> >>> >> >> >>>> >>>>> >> >>> >reading sector 131072: Transport >>> >>endpoint >>> >>> >is >>> >>> >>> >not >>> >>> >>> >> >> >>>> >>>>> >> >connected\\nqemu-img: >>> >>> >>> >> >> >>>> >>>>> >> >>> >error while reading sector 139264: >>> >>> >Transport >>> >>> >>> >> >endpoint >>> >>> >>> >> >> >is >>> >>> >>> >> >> >>>> not >>> >>> >>> >> >> >>>> >>>>> >> >>> >connected\\nqemu-img: error while >>> >>reading >>> >>> >>> >sector >>> >>> >>> >> >> >143360: >>> >>> >>> >> >> >>>> >>>>> >Transport >>> >>> >>> >> >> >>>> >>>>> >> >>> >endpoint >>> >>> >>> >> >> >>>> >>>>> >> >>> >is not connected\\nqemu-img: error >>> >while >>> >>> >>> >reading >>> >>> >>> >> >> >sector >>> >>> >>> >> >> >>>> >>>147456: >>> >>> >>> >> >> >>>> >>>>> >> >>> >Transport >>> >>> >>> >> >> >>>> >>>>> >> >>> >endpoint is not >connected\\nqemu-img: >>> >>error >>> >>> >>> >while >>> >>> >>> >> >> >reading >>> >>> >>> >> >> >>>> >>>sector >>> >>> >>> >> >> >>>> >>>>> >> >>> >155648: >>> >>> >>> >> >> >>>> >>>>> >> >>> >Transport endpoint is not >>> >>> >connected\\nqemu-img: >>> >>> >>> >> >error >>> >>> >>> >> >> >while >>> >>> >>> >> >> >>>> >>>>> >reading >>> >>> >>> >> >> >>>> >>>>> >> >>> >sector >>> >>> >>> >> >> >>>> >>>>> >> >>> >151552: Transport endpoint is not >>> >>> >>> >> >> >connected\\nqemu-img: >>> >>> >>> >> >> >>>> >>>error >>> >>> >>> >> >> >>>> >>>>> >while >>> >>> >>> >> >> >>>> >>>>> >> >>> >reading >>> >>> >>> >> >> >>>> >>>>> >> >>> >sector 159744: Transport endpoint >is >>> >not >>> >>> >>> >> >> >connected\\n')",) >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >oVirt version is 4.3.82-1.el7 >>> >>> >>> >> >> >>>> >>>>> >> >>> >OS CentOS Linux release 7.7.1908 >(Core) >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >The Gluster Cluster has been >working >>> >>very >>> >>> >well >>> >>> >>> >> >until >>> >>> >>> >> >> >this >>> >>> >>> >> >> >>>> >>>>> >incident. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >Please help. >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >Thank You >>> >>> >>> >> >> >>>> >>>>> >> >>> > >>> >>> >>> >> >> >>>> >>>>> >> >>> >Charles Williams >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >>> >> >> >>>> >>>>> >> >> >>> >>> >>> >> >> >>>> >>>>> >> >>> >>> >>> >> >> >>>> >>>>> >>> >>> >>> >> >> >>>> >>>> >>> >>> >>> >> >> >>>> >> >>> >>> >>> >> >> >>>> > >>> >>> >>> >> >> >>>> _______________________________________________ >>> >>> >>> >> >> >>>> Users mailing list -- users(a)ovirt.org >>> >>> >>> >> >> >>>> To unsubscribe send an email to &gt;users-leave(a)ovirt.org >>> >>> >>> >> >> >>>> Privacy Statement: >>> >>> >https://www.ovirt.org/privacy-policy.html >>> >>> >>> >> >> >>>> oVirt Code of Conduct: >>> >>> >>> >> >> >>>> >>> >>> >https://www.ovirt.org/community/about/community-guidelines/ >>> >>> >>> >> >> >>>> List Archives: >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> > >>> >>> >>> >> >> >>> >>> >>> >> > >>> >>> >>> >> >>> >>> >>> > >>> >>> >>> >>> >>> > >>> >>> >>> >> >>> > >>> > https://lists.ovirt.org/archives/list/users@ovirt.org/message/YY3VUKEJLI7... >>> >>> >>> >> >> >>>> >>> >>> >>> >> >> >>> >>> >>> >>> >> >> >>> >>> >>> >> >>> >>> >>> >>> >>> >> >>> >>> >>> >>