Hi Nir, thanks for your feedback. My guess is that as gluster is setup to use shard - the first one (the volume file) is OK ,but then the shard is giving an error. As gluster1=ovirt1 and gluster2=ovirt2 & ovirt3 is the arbiter , I think it's not SELINUX issue (all nodes are still in permissive - just to be sure that nothing comes up) on gluster level. As I can't powerup any VM (in the data_fastX domains), I took more radical step and enabled Trace log on the Fuse mount and started a VM. As per the log , I got :[2019-11-23 17:15:16.961636] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449da e.79 (00000000-0000-0000-0000-000000000000) [Permission denied] [2019-11-23 17:15:16.961669] D [MSGID: 0] [client-rpc-fops_v2.c:2642:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-client-2 returned -1 error: Permission denied [Permission denied] [2019-11-23 17:15:16.961705] D [MSGID: 0] [afr-common.c:2464:afr_lookup_done] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-replicate-0 returned -1 error: Permission denied [Permissi on denied] [2019-11-23 17:15:16.961735] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission denied] [2019-11-23 17:15:16.961764] D [MSGID: 0] [shard.c:842:shard_common_failure_unwind] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-shard returned -1 error: Permission denied [Permissi on denied] [2019-11-23 17:15:16.961793] D [MSGID: 0] [utime-autogen-fops.c:100:gf_utime_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-utime returned -1 error: Permission denied [Perm ission denied] [2019-11-23 17:15:16.961821] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-write-behind returned -1 error: Permission denied [Permiss ion denied] [2019-11-23 17:15:16.961848] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-open-behind returned -1 error: Permission denied [Permissi on denied] [2019-11-23 17:15:16.961876] D [MSGID: 0] [md-cache.c:2010:mdc_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-md-cache returned -1 error: Permission denied [Permission deni ed] [2019-11-23 17:15:16.961902] D [MSGID: 0] [defaults.c:1189:default_readv_cbk] 0-stack-trace: stack-address: 0x7fbeb4005218, data_fast-io-threads returned -1 error: Permission denied [Permissio n denied] [2019-11-23 17:15:16.961938] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 703: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fbeb4017018 (Permission denied) This indicates that the shard 79 for gfid b0af2b81-22cf-482e-9b2f-c431b6449dae has to be checked. So I first check the brick logs on ovirt1 from the moment when VM was powered up:[2019-11-23 17:15:16.962013] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842 4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated- fop:INVALID, acl:-) [Permission denied] [2019-11-23 17:15:16.962113] E [MSGID: 115050] [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 541: LOOKUP /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (be318638-e8a0-4c6d -977d-7a937aa84806/b0af2b81-22cf-482e-9b2f-c431b6449dae.79), client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:18424-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_N O:-0, error-xlator: data_fast-access-control [Permission denied] [2019-11-23 17:15:16.962096] I [MSGID: 139001] [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:43c4e254-d99e-45b3-a6b2-dba28b9c2269-GRAPH_ID:0-PID:1842 4-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, req(uid:107,gid:107,perm:1,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated- fop:INVALID, acl:-) [Permission denied] All of the above doesn't give me any clue what is it going about.And the actual shard: [root@ovirt1 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79 -rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79 [root@ovirt1 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79 # file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31 trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739 trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1 [root@ovirt2 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79 -rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79 [root@ovirt2 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79 # file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31 trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739 trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1 [root@ovirt3 .shard]# ls -lahRZ b0af2b81-22cf-482e-9b2f-c431b6449dae.79 -rw-rw----. vdsm kvm system_u:object_r:glusterd_brick_t:s0 b0af2b81-22cf-482e-9b2f-c431b6449dae.79 [root@ovirt3 .shard]# getfattr -d -e hex -m. b0af2b81-22cf-482e-9b2f-c431b6449dae.79 # file: b0af2b81-22cf-482e-9b2f-c431b6449dae.79 security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000 trusted.afr.dirty=0x000000000000000000000000 trusted.gfid=0xac073db9c9934d0bb70bb50dd11bee31 trusted.gfid2path.bedc7b97ff07378d=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f62306166326238312d323263662d343832652d396232662d6334333162363434396461652e3739 trusted.glusterfs.mdata=0x010000000000000000000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1000000005dc94ff0000000001eb592b1 Access for user qemu (uid 107/gid 107 -> gluster brick log on ovirt1) is OK: [root@ovirt1 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress 8+0 прочетени блока 8+0 записани блока изкопирани са 67108864 байта (67 MB), 0,0130595 s, 5,1 GB/s [root@ovirt2 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress 8+0 прочетени блока 8+0 записани блока изкопирани са 67108864 байта (67 MB), 0,0224745 s, 3,0 GB/s [root@ovirt3 .shard]# sudo -u qemu dd if=b0af2b81-22cf-482e-9b2f-c431b6449dae.79 of=/dev/null bs=8M iflag=direct status=progress 0+0 прочетени блока 0+0 записани блока изкопирани са 0 байта (0 B), 0,00103977 s, 0,0 kB/s [root@ovirt1 .shard]# grep 107 /etc/passwd /etc/group /etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin /etc/group:qemu:x:107:vdsm,sanlock [root@ovirt2 ~]# grep 107 /etc/passwd /etc/group /etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin /etc/group:qemu:x:107:vdsm,sanlock [root@ovirt3 .shard]# grep 107 /etc/passwd /etc/group /etc/passwd:qemu:x:107:107:qemu user:/:/sbin/nologin /etc/group:qemu:x:107:vdsm,sanlock I've created 3 strace files (1 as root and 2 as vdsm) and attached them to this e-mail: [root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096 -o /tmp/dd-strace-vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs=8M stat us=progress dd: грешка при четене на „5b1d3113-5cca-4582-9029-634b16338a2f“: Отказан достъп 8+0 прочетени блока 8+0 записани блока изкопирани са 67108864 байта (67 MB), 0,118223 s, 568 MB/s [root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# strace -fvttTyyx -s 4096 -o /tmp/dd-strace-root dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs=8M status=progress изкопирани са 5083496448 байта (5,1 GB), 10,144199 s, 501 MB/s 640+0 прочетени блока 640+0 записани блока изкопирани са 5368709120 байта (5,4 GB), 10,7444 s, 500 MB/s [root@ovirt2 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm strace -fvttTyyx -s 4096 -o /tmp/dd-strace-vdsm-2nd-try dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null iflag=direct bs =8M status=progress изкопирани са 4815060992 байта (4,8 GB), 8,056124 s, 598 MB/s 640+0 прочетени блока 640+0 записани блока изкопирани са 5368709120 байта (5,4 GB), 8,98862 s, 597 MB/s P.S.: Sorry for the cyrrilic , it's my wife's laptop :). First case gives an error and reads 67MB, while the second one works as expected.Then vdsm can access the volume without issue (quite strange)! В петък, 22 ноември 2019 г., 23:44:27 ч. Гринуич+2, Nir Soffer <nsoffer@redhat.com> написа: On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote: On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <sabose@redhat.com> wrote: On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <hunter86_bg@yahoo.com> wrote: Hi All, another clue in the logs :[2019-11-21 00:29:50.536631] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536798] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.536959] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:29:50.537007] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae [Permission denied][2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12458: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission denied)[2019-11-21 00:30:01.177665] I [MSGID: 133022] [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend[2019-11-21 00:30:13.132756] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.132824] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133217] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:13.133238] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c [Permission denied][2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12660: READ => -1 gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission denied)[2019-11-21 00:30:38.489449] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489520] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489669] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2: remote operation failed. Path: /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6 (00000000-0000-0000-0000-000000000000) [Permission denied][2019-11-21 00:30:38.489717] E [MSGID: 133010] [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4 [Permission denied][2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk] 0-glusterfs-fuse: 12928: READ => -1 gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission denied) Anyone got an idea why is it happening?I checked user/group and selinux permissions - all OK
Can you share the commands (and output) used to check this?I first thought that the file is cached in memory and that's why vdsm user can read the file , but the following shows opposite: [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# lltotal 562145-rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29 5b1d3113-5cca-4582-9029-634b16338a2f-rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11 5b1d3113-5cca-4582-9029-634b16338a2f.lease-rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11 5b1d3113-5cca-4582-9029-634b16338a2f.meta[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd/rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches
I would use iflag=direct instead, no need to mess with caches. Vdsm always use direct I/O. [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progressdd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied
You got permissions denied... 16+0 records in16+0 records out67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s
And dd continue to read data?! I have never seen anything like this. It will be helpful to run this with strace: strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct bs=8M status=progress And share dd.strace. Logs in /var/log/glusterfs/exportname.log will contain useful info for this test. [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s1280+0 records in1280+0 records out5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s1280+0 records in1280+0 records out5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s[root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 > /proc/sys/vm/drop_caches [root@ovirt1 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s1280+0 records in1280+0 records out5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s As you can see , once root user reads the file -> vdsm user can also do that.
Smells like issue on gluster side. I would try this on the hypervisor to check what vdsm/qemu see:
$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_pathI'm attaching the output of the find I run, but this one should be enough:[root@ovirt1 ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/ -not -user vdsm -print
A full output of ls -lahRZ, showing user, group, permissions bits, and selinux label>>of the entire tree will be more useful. [root@ovirt2 gluster1:_data__fast]# tree . ├── 396604d9-2a9e-49cd-9563-fdc79981f67b │ ├── dom_md │ │ ├── ids │ │ ├── inbox │ │ ├── leases │ │ ├── metadata │ │ ├── outbox │ │ └── xleases │ ├── images │ │ ├── 225e4c45-0f70-40aa-8d51-7b705f259cd7 │ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5 │ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.lease │ │ │ ├── 7592ce4c-ecb9-4dbe-ba67-a343ec4cdbc5.meta │ │ │ ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d │ │ │ ├── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.lease │ │ │ └── b965dd4e-c814-4bb6-8bc5-d1bbb8dc211d.meta │ │ ├── 25107759-f56b-4d94-872e-0f9b145886d1 │ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2 │ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.lease │ │ │ ├── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta │ │ │ └── 571a5eb4-eb92-479c-8305-79eaa0673bb2.meta_old │ │ ├── 2c79e7ab-47cc-4f41-a190-40cd94053d83 │ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d │ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.lease │ │ │ ├── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta │ │ │ └── 6a6d6ff8-92b2-4a14-b54c-77239e28254d.meta_old │ │ ├── 36fed07b-a08f-4fb9-a273-9dbf0ad1f27b │ │ │ ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9 │ │ │ ├── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.lease │ │ │ └── 44ac76fe-dd0d-471d-8ffe-cb9ad28b07a9.meta │ │ ├── 7d11479e-1a02-4a74-a9be-14b4e56faaa1 │ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a │ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.lease │ │ │ ├── 3e9a41f6-652e-4439-bd24-3e7621c27f4a.meta │ │ │ ├── a2b94064-9e34-47c9-8eab-867673c5e48f │ │ │ ├── a2b94064-9e34-47c9-8eab-867673c5e48f.lease │ │ │ └── a2b94064-9e34-47c9-8eab-867673c5e48f.meta │ │ ├── 85f18244-6d5b-4d02-ab66-140bfd2d734b │ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108 │ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.lease │ │ │ ├── 0cc4fb07-8a08-4c16-bf3f-93d4462c7108.meta │ │ │ ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa │ │ │ ├── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.lease │ │ │ └── 116191b2-99a1-437b-8b7d-ab1927b5f4aa.meta │ │ ├── 94f763e9-fd96-4bee-a6b2-31af841a918b │ │ │ ├── 5b1d3113-5cca-4582-9029-634b16338a2f │ │ │ ├── 5b1d3113-5cca-4582-9029-634b16338a2f.lease │ │ │ └── 5b1d3113-5cca-4582-9029-634b16338a2f.meta │ │ ├── 99c3c20c-a7bb-4851-8ae9-0b39c0c86b66 │ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661 │ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.lease │ │ │ ├── 6ad3bd5e-6b21-4f10-83b4-07126ca14661.meta │ │ │ ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5 │ │ │ ├── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.lease │ │ │ └── 95c260da-8dfb-46be-9c4a-6a55eb3967d5.meta │ │ ├── e56eb458-e459-4828-a268-82a9b91c60f8 │ │ │ ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871 │ │ │ ├── 5a7ca769-4657-4f65-92bd-72fa13cc2871.lease │ │ │ └── 5a7ca769-4657-4f65-92bd-72fa13cc2871.meta │ │ └── f5336a0f-de34-482e-b437-47f9d276ed85 │ │ ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4 │ │ ├── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.lease │ │ └── 4f218ba9-a5a2-4f2f-ac6d-e5c9217d4ec4.meta │ └── master └── __DIRECT_IO_TEST__
14 directories, 51 files Full find of "data_fastX" domains are attached.
Also, to make sure we don't have selinux issue on the hypervisor, you can change>selinux to permissive mode: > setenforce 0 This is the first thing I did and the systems were still in permissive when I tried again.I'm 99.99% sure it's not selinux.
And then try again. If this was selinux issue the permission denied issue will disappear.>If this is the case please provide the output of: > ausearh -m AVC -ts today If the issue still exists, we eliminated selinux, and you can enable it again: > setenforce 1 [root@ovirt3 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt2 ~]# ausearch -m AVC -ts today<no matches>[root@ovirt1 ~]# ausearch -m AVC -ts today<no matches>
So this is not selinux on the hypervisor. I wonder if it can be selinux on the gluster side? I have a vague feeling that the issue is related to gluster v6.5 to 6.6 upgrade which I several days before... So if any logs are needed (or debug enabled), just mention.
If this is the last change, and it worked before, most likely. Gluster was patched several days before RC3 , but all the storage-related issues lead me to that. The strage thing is that the HostedEngine is not affected, nor the "data" storage domain . Any ideas appreciated. I would like to find out what has caused this instead of just taking the shortcut to redeploy the whole setup. Best Regards,Strahil Nikolov