Hey folks, oh Jesus. 3-Way HCI. Gluster w/o any issues: [root@node01:/var/log/glusterfs] # gluster vol info  ssd_storage Volume Name: ssd_storage Type: Replicate Volume ID: d84ec99a-5db9-49c6-aab4-c7481a1dc57b Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: node01.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick2: node02.company.com:/gluster_bricks/ssd_storage/ssd_storage Brick3: node03.company.com:/gluster_bricks/ssd_storage/ssd_storage Options Reconfigured: performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.strict-o-direct: on performance.quick-read: off performance.read-ahead: off performance.io-cache: off performance.low-prio-threads: 32 network.remote-dio: off cluster.eager-lock: enable cluster.quorum-type: auto cluster.server-quorum-type: server cluster.data-self-heal-algorithm: full cluster.locking-scheme: granular cluster.shd-max-threads: 8 cluster.shd-wait-qlength: 10000 features.shard: on user.cifs: off cluster.choose-local: off client.event-threads: 4 server.event-threads: 4 network.ping-timeout: 30 storage.owner-uid: 36 storage.owner-gid: 36 cluster.granular-entry-heal: enab [root@node01:/var/log/glusterfs] # gluster vol status  ssd_storage Status of volume: ssd_storage Gluster process                             TCP Port  RDMA Port  Online Pid ------------------------------------------------------------------------------ Brick node01.company.com:/gluster_br icks/ssd_storage/ssd_storage                49152     0          Y 63488 Brick node02.company.com:/gluster_br icks/ssd_storage/ssd_storage                49152     0          Y 18860 Brick node03.company.com:/gluster_br icks/ssd_storage/ssd_storage                49152     0          Y 15262 Self-heal Daemon on localhost               N/A       N/A        Y 63511 Self-heal Daemon on node03.dc-dus.dalason.n et                                          N/A       N/A        Y 15285 Self-heal Daemon on 10.100.200.12           N/A       N/A        Y 18883 Task Status of Volume ssd_storage ------------------------------------------------------------------------------ There are no active volume tasks [root@node01:/var/log/glusterfs] # gluster vol heal ssd_storage info Brick node01.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 Brick node02.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 Brick node03.company.com:/gluster_bricks/ssd_storage/ssd_storage Status: Connected Number of entries: 0 And everything is mounted where its supposed to. But no VMs start due to IO Error. I checked a gluster-based file (CentOS iso) md5 against a local copy, it matches. One VM at one point managed to start, but failed subsequent starts. The data/disks seem okay, /var/log/glusterfs/"rhev-data-center-mnt-glusterSD-node01.company.com:_ssd__storage.log-20200202" has entries like: [2020-02-01 23:15:15.449902] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-1: remote operation failed. Path: /.shard/86da0289-f74f-4200-9284-678e7bd76195.1405 (00000000-0000-0000-0000-000000000000) [Permission denied] [2020-02-01 23:15:15.484363] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-ssd_storage-client-1: remote operation failed. Path: /.shard/86da0289-f74f-4200-9284-678e7bd76195.1400 (00000000-0000-0000-0000-000000000000) [Permission denied] Before this happened we put one host into maintenance mode, it all started during migration. Any help? We're sweating blood here.

The log appears to indicate that there may be a permissions issue.  What is the ownership and permissions on your gluster brick dirs and mounts?

I checked my HCI cluster and those permissions seem to match what I'm seeing.  Since there's no VMs running currently have you tried restarting the gluster volumes as well as the glusterd service? I'm not sure what would have caused this with one host placed in maintenance.

Check the contents of these directories: [root@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net <http://node01.dc-dus.dalason.net>:_ssd__storage/fec2eb5e-21b5-496b-9ea...] # l total 345K drwxr-xr-x. 46 vdsm kvm 8.0K Feb  2 23:18 . drwxr-xr-x.  5 vdsm kvm   64 Feb  3 00:31 .. drwxr-xr-x.  2 vdsm kvm 8.0K Jan 17 15:54 0b21c949-7133-4b34-b909-a6660ae12800 drwxr-xr-x.  2 vdsm kvm  165 Feb  3 01:48 0dde79ab-d773-4d23-b397-7c39371ccc60 drwxr-xr-x.  2 vdsm kvm 8.0K Jan 17 09:49 1347d489-012b-40fc-acb5-d00a9ea133a4 drwxr-xr-x.  2 vdsm kvm 8.0K Jan 22 15:04 1ccc4db6-f47d-4474-b0fa-a0c1eddb0fa7 drwxr-xr-x.  2 vdsm kvm 8.0K Jan 21 16:28 22cab044-a26d-4266-9af7-a6408eaf140c drwxr-xr-x.  2 vdsm kvm 8.0K Jan 30 06:03 288d061a-6c6c-4536-a594-3bede63c0654 drwxr-xr-x.  2 vdsm kvm 8.0K Jan  9 16:46 40c51753-1533-45ab-b9de-2c51d8a18370 and what version of Ovirt are you running? This looks a bit like a libvirt change/bug that changed ownership on the actual disk image to root.root on shutdown/migrations, preventing later start attempts. This may help if that’s the case: chown -R vdsm.kvm /rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net <http://node01.dc-dus.dalason.net>:_ssd__storage/fec2eb5e-21b5-496b-9ea... > On Feb 2, 2020, at 8:54 PM, Christian Reiss <email(a)christian-reiss.de > <mailto:email@christian-reiss.de>> wrote: > > Hey, > > it was _while_ placing the host _into_ maintenance, to be precise. > I restarted the volumes and even each machine and the entire cluster > to no avail. > > I am currently migrating the disk images out of ovirt into openvz/kvm > to get them running. The copied disk images are flawless and working. > > > On 03/02/2020 03:28, Jayme wrote: >> I checked my HCI cluster and those permissions seem to match what I'm >> seeing.  Since there's no VMs running currently have you tried >> restarting the gluster volumes as well as the glusterd service? I'm >> not sure what would have caused this with one host placed in maintenance. > > -- > with kind regards, > mit freundlichen Gruessen, > > Christian Reiss > _______________________________________________ > Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> > To unsubscribe send an email to users-leave(a)ovirt.org > <mailto:users-leave@ovirt.org> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/FOFNQO4FLJA...

Further findings: - modified data gets written to local node, not across gluster. - vdsm user can create _new_ files on the cluster, this gets synced immediatly. - vdsm can modify, across all nodes newly created files, changes apply immediately. I think vdsm user can not modify already existing files over the gluster. Something selinux? -Chris. On 03/02/2020 11:46, Christian Reiss wrote: > Hey, > > I think I am barking up the right tree with something (else) here; > Note the timestamps & id's: > > > dd'ing a disk image as vdsm user, try 1: > > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd > of=/dev/null > Mon  3 Feb 11:39:13 CET 2020 > uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > denied > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.169465 s, 396 MB/s >    64MiB 0:00:00 [ 376MiB/s] [  <=> >       ] > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.171726 s, 391 MB/s > > > try 2, directly afterward: > > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd > of=/dev/null > Mon  3 Feb 11:39:16 CET 2020 > uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > denied > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.148846 s, 451 MB/s >    64MiB 0:00:00 [ 427MiB/s] [  <=> >       ] > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.149589 s, 449 MB/s > > > try same as root: > > [root@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > # date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd > of=/dev/null > Mon  3 Feb 11:39:33 CET 2020 > uid=0(root) gid=0(root) groups=0(root) > context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >    50GiB 0:03:06 [ 274MiB/s] [                              <=> >       ] > 104857600+0 records in > 104857600+0 records out > 53687091200 bytes (54 GB) copied, 186.501 s, 288 MB/s > 104857600+0 records in > 104857600+0 records out > 53687091200 bytes (54 GB) copied, 186.502 s, 288 MB/s > > > Followed by another vdsm dd test: > > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd > of=/dev/null > Mon  3 Feb 11:42:46 CET 2020 > uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >    50GiB 0:02:56 [ 290MiB/s] [           <=>      ] > 104857600+0 records in > 104857600+0 records out > 53687091200 bytes (54 GB) copied, 176.189 s, 305 MB/s > 104857600+0 records in > 104857600+0 records out > 53687091200 bytes (54 GB) copied, 176.19 s, 305 MB/s > > So it's a permission problem (access denied) unless root loads it first? > Strange: doing things like file & stat work; I can even cat the meta > file (small text file). Seems only the disk images (or large files?) > are affected. > > huh!? >

On February 3, 2020 2:29:55 PM GMT+02:00, Christian Reiss <email(a)christian-reiss.de&gt; wrote: > Ugh, > > disregarding off all previous stamenets: > > new findinds: vdsm user can NOT read files larger than 64mb. Root can. > > [vdsm@node02:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 > 68 > ; do dd if=/dev/urandom of=file-$i bs=1M count=$i ; done > > [vdsm@node03:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 > 68 > ; do echo $i ; dd if=file-$i of=/dev/null ; done > 60 > 122880+0 records in > 122880+0 records out > 62914560 bytes (63 MB) copied, 0.15656 s, 402 MB/s > 62 > 126976+0 records in > 126976+0 records out > 65011712 bytes (65 MB) copied, 0.172463 s, 377 MB/s > 64 > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.180701 s, 371 MB/s > 66 > dd: error reading ‘file-66’: Permission denied > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.105236 s, 638 MB/s > 68 > dd: error reading ‘file-68’: Permission denied > 131072+0 records in > 131072+0 records out > 67108864 bytes (67 MB) copied, 0.17046 s, 394 MB/s > > > The files appeared instantly on all nodes, Writing large files work, > however. Writing large files seem to work. > > I think this is the core issue. > > > On 03/02/2020 12:22, Christian Reiss wrote: >> Further findings: >> >> - modified data gets written to local node, not across gluster. >> - vdsm user can create _new_ files on the cluster, this gets synced >> immediatly. >> - vdsm can modify, across all nodes newly created files, changes > apply >> immediately. >> >> I think vdsm user can not modify already existing files over the >> gluster. Something selinux? >> >> -Chris. >> >> On 03/02/2020 11:46, Christian Reiss wrote: >>> Hey, >>> >>> I think I am barking up the right tree with something (else) here; >>> Note the timestamps & id's: >>> >>> >>> dd'ing a disk image as vdsm user, try 1: >>> >>> > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd >>> of=/dev/null >>> Mon  3 Feb 11:39:13 CET 2020 >>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) >>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >>> dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > >>> denied >>> 131072+0 records in >>> 131072+0 records out >>> 67108864 bytes (67 MB) copied, 0.169465 s, 396 MB/s >>>    64MiB 0:00:00 [ 376MiB/s] [  <=> >>>       ] >>> 131072+0 records in >>> 131072+0 records out >>> 67108864 bytes (67 MB) copied, 0.171726 s, 391 MB/s >>> >>> >>> try 2, directly afterward: >>> >>> > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd >>> of=/dev/null >>> Mon  3 Feb 11:39:16 CET 2020 >>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) >>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >>> dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > >>> denied >>> 131072+0 records in >>> 131072+0 records out >>> 67108864 bytes (67 MB) copied, 0.148846 s, 451 MB/s >>>    64MiB 0:00:00 [ 427MiB/s] [  <=> >>>       ] >>> 131072+0 records in >>> 131072+0 records out >>> 67108864 bytes (67 MB) copied, 0.149589 s, 449 MB/s >>> >>> >>> try same as root: >>> >>> > [root@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >>> # date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd >>> of=/dev/null >>> Mon  3 Feb 11:39:33 CET 2020 >>> uid=0(root) gid=0(root) groups=0(root) >>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >>>    50GiB 0:03:06 [ 274MiB/s] [                              <=> >>>       ] >>> 104857600+0 records in >>> 104857600+0 records out >>> 53687091200 bytes (54 GB) copied, 186.501 s, 288 MB/s >>> 104857600+0 records in >>> 104857600+0 records out >>> 53687091200 bytes (54 GB) copied, 186.502 s, 288 MB/s >>> >>> >>> Followed by another vdsm dd test: >>> >>> > [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net:_ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv  | dd >>> of=/dev/null >>> Mon  3 Feb 11:42:46 CET 2020 >>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) >>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 >>>    50GiB 0:02:56 [ 290MiB/s] [           <=>      ] >>> 104857600+0 records in >>> 104857600+0 records out >>> 53687091200 bytes (54 GB) copied, 176.189 s, 305 MB/s >>> 104857600+0 records in >>> 104857600+0 records out >>> 53687091200 bytes (54 GB) copied, 176.19 s, 305 MB/s >>> >>> So it's a permission problem (access denied) unless root loads it > first? >>> Strange: doing things like file & stat work; I can even cat the meta > >>> file (small text file). Seems only the disk images (or large files?) > >>> are affected. >>> >>> huh!? >>> >> What version of gluster are you running ? Have you tried the solution with : 1. Run fake setfacl ? Or killing brick processes and start the volume with the 'force' option ? I saw in your brick output your selinux context is 'unconfined_u' ... So check for labeling. Still, it looks like my ACL issue. Best Regards, Strahil Nikolov

Appreciate the updates you've been posting. It's concerning to me as a Gluster user as well. It would be nice to figure out what happened here. On Thu, Feb 6, 2020 at 11:43 AM Christian Reiss <email(a)christian-reiss.de&gt; wrote: > Hey, > > For prosperity: Sadly the only way to fix this was to re-init (wipe) > gluster and start from scratch. > > -Chris. > > On 03/02/2020 19:23, Strahil Nikolov wrote: > > On February 3, 2020 2:29:55 PM GMT+02:00, Christian Reiss < > email(a)christian-reiss.de&gt; wrote: > >> Ugh, > >> > >> disregarding off all previous stamenets: > >> > >> new findinds: vdsm user can NOT read files larger than 64mb. Root can. > >> > >> [vdsm@node02:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 > >> 68 > >> ; do dd if=/dev/urandom of=file-$i bs=1M count=$i ; done > >> > >> [vdsm@node03:/rhev/data-cente[...]c51d8a18370] $ for i in 60 62 64 66 > >> 68 > >> ; do echo $i ; dd if=file-$i of=/dev/null ; done > >> 60 > >> 122880+0 records in > >> 122880+0 records out > >> 62914560 bytes (63 MB) copied, 0.15656 s, 402 MB/s > >> 62 > >> 126976+0 records in > >> 126976+0 records out > >> 65011712 bytes (65 MB) copied, 0.172463 s, 377 MB/s > >> 64 > >> 131072+0 records in > >> 131072+0 records out > >> 67108864 bytes (67 MB) copied, 0.180701 s, 371 MB/s > >> 66 > >> dd: error reading ‘file-66’: Permission denied > >> 131072+0 records in > >> 131072+0 records out > >> 67108864 bytes (67 MB) copied, 0.105236 s, 638 MB/s > >> 68 > >> dd: error reading ‘file-68’: Permission denied > >> 131072+0 records in > >> 131072+0 records out > >> 67108864 bytes (67 MB) copied, 0.17046 s, 394 MB/s > >> > >> > >> The files appeared instantly on all nodes, Writing large files work, > >> however. Writing large files seem to work. > >> > >> I think this is the core issue. > >> > >> > >> On 03/02/2020 12:22, Christian Reiss wrote: > >>> Further findings: > >>> > >>> - modified data gets written to local node, not across gluster. > >>> - vdsm user can create _new_ files on the cluster, this gets synced > >>> immediatly. > >>> - vdsm can modify, across all nodes newly created files, changes > >> apply > >>> immediately. > >>> > >>> I think vdsm user can not modify already existing files over the > >>> gluster. Something selinux? > >>> > >>> -Chris. > >>> > >>> On 03/02/2020 11:46, Christian Reiss wrote: > >>>> Hey, > >>>> > >>>> I think I am barking up the right tree with something (else) here; > >>>> Note the timestamps & id's: > >>>> > >>>> > >>>> dd'ing a disk image as vdsm user, try 1: > >>>> > >>>> > >> [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net: > _ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >> > >>>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv | dd > >>>> of=/dev/null > >>>> Mon 3 Feb 11:39:13 CET 2020 > >>>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > >>>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > >>>> dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > >> > >>>> denied > >>>> 131072+0 records in > >>>> 131072+0 records out > >>>> 67108864 bytes (67 MB) copied, 0.169465 s, 396 MB/s > >>>> 64MiB 0:00:00 [ 376MiB/s] [ <=> > >>>> ] > >>>> 131072+0 records in > >>>> 131072+0 records out > >>>> 67108864 bytes (67 MB) copied, 0.171726 s, 391 MB/s > >>>> > >>>> > >>>> try 2, directly afterward: > >>>> > >>>> > >> [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net: > _ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >> > >>>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv | dd > >>>> of=/dev/null > >>>> Mon 3 Feb 11:39:16 CET 2020 > >>>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > >>>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > >>>> dd: error reading ‘5fca6d0e-e320-425b-a89a-f80563461add’: Permission > >> > >>>> denied > >>>> 131072+0 records in > >>>> 131072+0 records out > >>>> 67108864 bytes (67 MB) copied, 0.148846 s, 451 MB/s > >>>> 64MiB 0:00:00 [ 427MiB/s] [ <=> > >>>> ] > >>>> 131072+0 records in > >>>> 131072+0 records out > >>>> 67108864 bytes (67 MB) copied, 0.149589 s, 449 MB/s > >>>> > >>>> > >>>> try same as root: > >>>> > >>>> > >> [root@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net: > _ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >> > >>>> # date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv | dd > >>>> of=/dev/null > >>>> Mon 3 Feb 11:39:33 CET 2020 > >>>> uid=0(root) gid=0(root) groups=0(root) > >>>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > >>>> 50GiB 0:03:06 [ 274MiB/s] [ <=> > >>>> ] > >>>> 104857600+0 records in > >>>> 104857600+0 records out > >>>> 53687091200 bytes (54 GB) copied, 186.501 s, 288 MB/s > >>>> 104857600+0 records in > >>>> 104857600+0 records out > >>>> 53687091200 bytes (54 GB) copied, 186.502 s, 288 MB/s > >>>> > >>>> > >>>> Followed by another vdsm dd test: > >>>> > >>>> > >> [vdsm@node03:/rhev/data-center/mnt/glusterSD/node01.dc-dus.dalason.net: > _ssd__storage/fec2eb5e-21b5-496b-9ea5-f718b2cb5556/images/4a55b9c0-d550-4ecb-8dd1-cc1f24f2c7ac] > >> > >>>> $ date ; id ; dd if=5fca6d0e-e320-425b-a89a-f80563461add | pv | dd > >>>> of=/dev/null > >>>> Mon 3 Feb 11:42:46 CET 2020 > >>>> uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock) > >>>> context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 > >>>> 50GiB 0:02:56 [ 290MiB/s] [ <=> ] > >>>> 104857600+0 records in > >>>> 104857600+0 records out > >>>> 53687091200 bytes (54 GB) copied, 176.189 s, 305 MB/s > >>>> 104857600+0 records in > >>>> 104857600+0 records out > >>>> 53687091200 bytes (54 GB) copied, 176.19 s, 305 MB/s > >>>> > >>>> So it's a permission problem (access denied) unless root loads it > >> first? > >>>> Strange: doing things like file & stat work; I can even cat the meta > >> > >>>> file (small text file). Seems only the disk images (or large files?) > >> > >>>> are affected. > >>>> > >>>> huh!? > >>>> > >>> > > > > What version of gluster are you running ? > > Have you tried the solution with : > > 1. Run fake setfacl ? > > Or killing brick processes and start the volume with the 'force' > option ? > > > > I saw in your brick output your selinux context is 'unconfined_u' ... > So check for labeling. > > Still, it looks like my ACL issue. > > > > Best Regards, > > Strahil Nikolov > > > > -- > with kind regards, > mit freundlichen Gruessen, > > Christian Reiss > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/3ZYVQJQR7SF... >