7:37 a.m.
Hi Jiffin,
Thanks for the info.
As I'm now on Gluster v7 , I hope it won't happen again.
It's nice to know it got fixed.
Best Regards,
Strahil NikolovOn Dec 2, 2019 11:32, Jiffin Thottan <jthottan(a)redhat.com> wrote:
>
> Hi Krutika,
>
> Apparently, in context acl info got corrupted see brick logs
>
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
> client:
>
CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
> req(uid:107,gid:107,perm:1,ngrps:4),
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
> [Permission denied]
>
> which resulted in the situation. There was one bug similar was reported
https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and
>
> it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/.
But here he mentioned he saw the issue when
>
> he upgraded from 6.5 to 6.6
>
> One way to workaround is to perform a dummy setfacl(preferably using root) on the
corrupted files which will forcefully fetch the acl
>
> info again from backend and update the context. Another approach to restart brick
process(kill and vol start force)
>
> Regards,
> Jiffin
>
> ----- Original Message -----
> From: "Krutika Dhananjay" <kdhananj(a)redhat.com>
> To: "Strahil Nikolov" <hunter86_bg(a)yahoo.com>, "Jiffin
Thottan" <jthottan(a)redhat.com>, "raghavendra talur"
<rtalur(a)redhat.com>
> Cc: "Nir Soffer" <nsoffer(a)redhat.com>, "Rafi Kavungal Chundattu
Parambil" <rkavunga(a)redhat.com>, "users" <users(a)ovirt.org>,
"gluster-user" <gluster-users(a)gluster.org>
> Sent: Monday, December 2, 2019 11:48:22 AM
> Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now
available for testing
>
> Sorry about the late response.
>
> I looked at the logs. These errors are originating from posix-acl
> translator -
>
>
>
> *[2019-11-17 07:55:47.090065] E [MSGID: 115050]
> [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496:
> LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6
> (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6),
> client:
>
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
> error-xlator: data_fast-access-control [Permission denied][2019-11-17
> 07:55:47.090174] I [MSGID: 139001]
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
> client:
>
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
> req(uid:36,gid:36,perm:1,ngrps:3),
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
> [Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050]
> [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497:
> LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7
> (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7),
> client:
>
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
> error-xlator: data_fast-access-control [Permission denied][2019-11-17
> 07:55:47.090299] I [MSGID: 139001]
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:
> client:
>
CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806,
> req(uid:36,gid:36,perm:1,ngrps:3),
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)
> [Permission denied]*
>
> Jiffin/Raghavendra Talur,
> Can you help?
>
> -Krutika
>
> On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg(a)yahoo.com>
> wrote:
>
> > Hi Nir,All,
> >
> > it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached
> > screenshot of oVirt running on v7 gluster).
> > It seems strange that both my serious issues with oVirt are related to
> > gluster issue (1st gluster v3 to v5 migration and now this one).
> >
> > I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all
> > nodes.
> > Now both Engine and all my VMs are back online - so if you hit issues with
> > 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before
> > deciding to wipe everything.
> >
> > @Krutika,
> >
> > I guess you will ask for the logs, so let's switch to gluster-users about
> > this one ?
> >
> > Best Regards,
> > Strahil Nikolov
> >
> > В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov <
> > hunter86_bg(a)yahoo.com> написа:
> >
> >
> > Hi Krutika,
> >
> > I have enabled TRACE log level for the volume data_fast,
> >
> > but the issue is not much clear:
> > FUSE reports:
> >
> > [2019-11-25 21:31:53.478130] I [MSGID: 133022]
> > [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of
> > gfid=6d9ed2e5-d4f2-4749-839b-2f1
> > 3a68ed472 from backend
> > [2019-11-25 21:32:43.564694] W [MSGID: 114031]
> > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:
> > remote operation failed. Path: