Hi Nir,

thanks for your feedback. My guess is that as gluster is setup to use shard - the first one (the volume file) is OK ,but then the shard is giving an error.

As gluster1=ovirt1 and gluster2=ovirt2 & ovirt3 is the arbiter , I think it's not SELINUX issue (all nodes are still in permissive - just to be sure that nothing comes up) on gluster level.

As I can't powerup any VM (in the data_fastX domains), I took more radical step and enabled Trace log on the Fuse mount and started a VM.

As per the log , I got :

[2019-11-23 17:15:16.961636] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449da
e.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-23 17:15:16.961669] D [MSGID: 0] [client-rpc-fops_v2.c:2642:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-client-2 returned -1 error: Permission denied
[Permission denied]
[2019-11-23 17:15:16.961705] D [MSGID: 0] [afr-common.c:2464:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-replicate-0 returned -1 error: Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961735] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae
[Permission denied]
[2019-11-23 17:15:16.961764] D [MSGID: 0] [shard.c:842:shard_common_failure_unwind] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-shard returned -1 error: Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961793] D [MSGID: 0] [utime-autogen-fops.c:100:gf_utime_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-utime returned -1 error: Permission denied [Perm
ission denied]
[2019-11-23 17:15:16.961821] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-write-behind returned -1 error: Permission denied [Permiss
ion denied]
[2019-11-23 17:15:16.961848] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-open-behind returned -1 error: Permission denied [Permissi
on denied]
[2019-11-23 17:15:16.961876] D [MSGID: 0] [md-cache.c:2010:mdc_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-md-cache returned -1 error: Permission denied [Permission deni
ed]
[2019-11-23 17:15:16.961902] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-io-threads returned -1 error: Permission denied [Permissio
n denied]
[2019-11-23 17:15:16.961938] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 703: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fbeb4017018 (Permission denied)

This indicates that the shard 79 for gfid b0af2b81-22cf-482e-9b2f-c431b6449dae has to be checked.

So I first check the brick logs on ovirt1 from the moment when VM was powered up:

[2019-11-23 17:15:16.962013] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842
4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-
fop:INVALID, acl:-) [Permission denied]
[2019-11-23 17:15:16.962113] E [MSGID: 115050] [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 541: LOOKUP /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (be318638-e8a0-4c6d
-977d-7a937aa84806/b0af2b81-22cf-482e-9b2f-c431b6449dae.79), client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:18424-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_N
O:-0, error-xlator: data_fast-access-control [Permission denied]
[2019-11-23 17:15:16.962096] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842
4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-
fop:INVALID, acl:-) [Permission denied]

All of the above doesn't give me any clue what is it going about.

And the actual shard:

[root@ovirt1 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79

[root@ovirt1 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1

[root@ovirt2 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79

[root@ovirt2 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1

[root@ovirt3 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79
-rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79

[root@ovirt3 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79
# file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31
trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739
trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1

Access for user qemu (uid 107/gid 107 -> gluster brick log on ovirt1) is OK:

[root@ovirt1 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,0130595 s, 5,1 GB/s

[root@ovirt2 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,0224745 s, 3,0 GB/s

[root@ovirt3 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress
0+0 прочетени блока
0+0 записани блока
изкопирани са 0 байта (0 B), 0,00103977 s, 0,0 kB/s

[root@ovirt1 .shard]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock

[root@ovirt2 ~]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock

[root@ovirt3 .shard]# grep 107 /etc/passwd /etc/group
/etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin
/etc/group:qemu:x:107:vdsm,sanlock

I've created 3 strace files (1 as root and 2 as vdsm) and attached them to this e-mail:

[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096 -o /tmp/dd-strace-vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs=8M stat
us=progress
dd: грешка при четене на „5b1d3113-5cca-4582-9029-634b16338a2f“: Отказан достъп
8+0 прочетени блока
8+0 записани блока
изкопирани са 67108864 байта (67 MB), 0,118223 s, 568 MB/s

[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# strace -fvttTyyx -s 4096 -o /tmp/dd-strace-root dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs=8M status=progress
изкопирани са 5083496448 байта (5,1 GB), 10,144199 s, 501 MB/s
640+0 прочетени блока
640+0 записани блока
изкопирани са 5368709120 байта (5,4 GB), 10,7444 s, 500 MB/s

[root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096 -o /tmp/dd-strace-vdsm-2nd-try dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs
=8M status=progress
изкопирани са 4815060992 байта (4,8 GB), 8,056124 s, 598 MB/s
640+0 прочетени блока
640+0 записани блока
изкопирани са 5368709120 байта (5,4 GB), 8,98862 s, 597 MB/s

P.S.: Sorry for the cyrrilic , it's my wife's laptop :). First case gives an error and reads 67MB, while the second one works as expected.Then vdsm can access the volume without issue (quite strange)!

В петък, 22 ноември 2019 г., 23:44:27 ч. Гринуич+2, Nir Soffer <nsoffer@redhat.com> написа:

On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:

On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose@redhat.com> wrote:

On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:

Hi All,

another clue in the logs :
[2019-11-21 00:29:50.536631] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536798] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.536959] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:29:50.537007] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission denied]
[2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission denied)
[2019-11-21 00:30:01.177665] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend
[2019-11-21 00:30:13.132756] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.132824] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133217] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:13.133238] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission denied]
[2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission denied)
[2019-11-21 00:30:38.489449] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489520] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489669] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied]
[2019-11-21 00:30:38.489717] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission denied]
[2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission denied)

Anyone got an idea why is it happening?
I checked user/group and selinux permissions - all OK

>Can you share the commands (and output) used to check this?
I first thought that the file is cached in memory and that's why vdsm user can read the file , but the following shows opposite:

[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll
total 562145
-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f
-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease
-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd
/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches

>>I would use iflag=direct instead, no need to mess with caches. Vdsm always use direct I/O.

[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied

>>You got permissions denied...

16+0 records in
16+0 records out
67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s

>>And dd continue to read data?!

>>I have never seen anything like this.

>>It will be helpful to run this with strace:

>> strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M status=progress

>>And share dd.strace.

>>Logs in /var/log/glusterfs/exportname.log will contain useful info for this test.

[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress
5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s
1280+0 records in
1280+0 records out
5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s

As you can see , once root user reads the file -> vdsm user can also do that.

>>Smells like issue on gluster side.

>I would try this on the hypervisor to check what vdsm/qemu see:

>$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path
I'm attaching the output of the find I run, but this one should be enough:
[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print

>>A full output of ls -lahRZ, showing user, group, permissions bits, and selinux label

>>of the entire tree will be more useful.

[root@ovirt2 gluster1:_data__fast]# tree
.
├── 396604d9-2a9e-49cd-9563-fdc79981f67b
│   ├── dom_md
│   │   ├── ids
│   │   ├── inbox
│   │   ├── leases
│   │   ├── metadata
│   │   ├── outbox
│   │   └── xleases
│   ├── images
│   │   ├── 225e4c45-0f70-40aa-8d51-7b705f259cd7
│   │   │   ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5
│   │   │   ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.lease
│   │   │   ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.meta
│   │   │   ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d
│   │   │   ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.lease
│   │   │   └── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.meta
│   │   ├── 25107759-f56b-4d94-872e-0f9b145886d1
│   │   │   ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2
│   │   │   ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.lease
│   │   │   ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta
│   │   │   └── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta_old
│   │   ├── 2c79e7ab-47cc-4f41-a190-40cd94053d83
│   │   │   ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d
│   │   │   ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.lease
│   │   │   ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta
│   │   │   └── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta_old
│   │   ├── 36fed07b-a08f-4fb9-a273-9dbf0ad1f27b
│   │   │   ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9
│   │   │   ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.lease
│   │   │   └── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.meta
│   │   ├── 7d11479e-1a02-4a74-a9be-14b4e56faaa1
│   │   │   ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a
│   │   │   ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.lease
│   │   │   ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.meta
│   │   │   ├── a2b94064-9e34-47c9-8eab-867673c5e48f
│   │   │   ├── a2b94064-9e34-47c9-8eab-867673c5e48f.lease
│   │   │   └── a2b94064-9e34-47c9-8eab-867673c5e48f.meta
│   │   ├── 85f18244-6d5b-4d02-ab66-140bfd2d734b
│   │   │   ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108
│   │   │   ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.lease
│   │   │   ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.meta
│   │   │   ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa
│   │   │   ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.lease
│   │   │   └── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.meta
│   │   ├── 94f763e9-fd96-4bee-a6b2-31af841a918b
│   │   │   ├── 5b1d3113-5cca-4582-9029-634b16338a2f
│   │   │   ├── 5b1d3113-5cca-4582-9029-634b16338a2f.lease
│   │   │   └── 5b1d3113-5cca-4582-9029-634b16338a2f.meta
│   │   ├── 99c3c20c-a7bb-4851-8ae9-0b39c0c86b66
│   │   │   ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661
│   │   │   ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.lease
│   │   │   ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.meta
│   │   │   ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5
│   │   │   ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.lease
│   │   │   └── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.meta
│   │   ├── e56eb458-e459-4828-a268-82a9b91c60f8
│   │   │   ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871
│   │   │   ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871.lease
│   │   │   └── 5a7ca769-4657-4f65-92bd-72fa13cc2871.meta
│   │   └── f5336a0f-de34-482e-b437-47f9d276ed85
│   │       ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4
│   │       ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.lease
│   │       └── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.meta
│   └── master
└── __DIRECT_IO_TEST__

14 directories, 51 files

Full find of "data_fastX" domains are attached.

>Also, to make sure we don't have selinux issue on the hypervisor, you can change
>selinux to permissive mode:

> setenforce 0

This is the first thing I did and the systems were still in permissive when I tried again.I'm 99.99% sure it's not selinux.

>And then try again. If this was selinux issue the permission denied issue will disappear.
>If this is the case please provide the output of:

> ausearh -m AVC -ts today

>If the issue still exists, we eliminated selinux, and you can enable it again:

> setenforce 1

[root@ovirt3 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt2 ~]# ausearch -m AVC -ts today
<no matches>
[root@ovirt1 ~]# ausearch -m AVC -ts today
<no matches>

>So this is not selinux on the hypervisor. I wonder if it can be selinux on the gluster side?

I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I several days before... So if any logs are needed (or debug enabled), just mention.

>If this is the last change, and it worked before, most likely.

Gluster was patched several days before RC3 , but all the storage-related issues lead me to that. The strage thing is that the HostedEngine is not affected, nor the "data" storage domain .

Any ideas appreciated. I would like to find out what has caused this instead of just taking the shortcut to redeploy the whole setup.

Best Regards,

Strahil Nikolov