On April 8, 2020 2:43:01 PM GMT+03:00, Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
On Tue, Apr 7, 2020 at 8:16 PM Strahil Nikolov <hunter86_bg@yahoo.com> wrote:
Hi Gianluca,
The positive thing is that you can reproduce the issue.
I would ask you to check your gluster version and if there are any updates - update the cluster.
I'd prefer to stick on oVirt release version of Gluster if possible
This is what I ment, but if you are using Gluster v6.0 - you should update to latest version.
Also check the gluster's op-version, as this limits some of the features.
What do you mean by thgis?
Check this one: https://docs.gluster.org/en/latest/Upgrade-Guide/op_version/
If there are none - enable trace logs in gluster (
), start volume peofiling, reproduce the issue and then reduce the log level (it's generating a lot of logs) and stop the profiling.
Once that info is collected, some of the Gluster members can check
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/ht... the
situation.
Best Regards, Strahil Nikolov
Ok. I think that the INFO level set on the different layers outlined problem somehow related to shardin.
Related to this, I found no official docs on Gluster web site after 3.7 version... where is it? Only information I found was in Red Hat Gluster Storage 3.5 Administration Guide, but I would expect something more upstream...
Red Hat versioning is different. So far RH documentation for Gluster has never failed me.
In particular in my case where I have only one host and the gluster volumes are single brick based, do you think I can try to disable sharding and verify if using new disks with it disabled and oVirt thin provisioned disks let the problem go away?
Also, I found some information about sharding block size. Apparently the only supported size on Red Hat Gluster Storage is 512MB, but oVirt sets it to 64MB....? I also found a bugzilla about passing from 128MB to 64MB in oVirt 4.1.5: https://bugzilla.redhat.com/show_bug.cgi?id=1469436
Now I see that by default and so also in my environment I have:
features.shard on
features.shard-block-size 64MB
features.shard-lru-limit 16384
features.shard-deletion-rate 100
Not clear how to cross-check for all the values above in docs....
Can I safely set features.shard off
In Red Hat Gluster storage admin guide it is considered only for replicated gluster volumes (and so also ditributed-replicated...) But in my case with single host and single beick for volumes I think it doesn't give any particular benefit, isn't it?
NEVER DISABLE SHARDING ON A VOLUME!!! There is a very good reply from Amar Tumballi in gluster users mailing list about that. The good thing is that you can do storage migration. The only benefits of sharding are: 1. When using distributed-replicated volume, shards will spread on different bricks and thus partially increase read performance 2. When a heal is going on, the shard is locked - so there will be no disruption for the VM In both cases you cannot benefit of that. Shard size is 64 MB and this is default from Gluster , not from oVirt. Maybe there is a reason behind that large shard size, but I have no clue. In your case, you shokuld consider using VDO, as most of the VMs have almost the same system files, which will lead to data reduction at a small cost of write speed. Another option is to create a new volume and disable sharding at all.
I found this interesting post about sharding and connection between image files and shard files: http://opensource-storage.blogspot.com/2016/07/de-mystifying-gluster-shards....
Gianluca
I have one question - you have a single brick in your volume. Why do you use oVirt instead of plain KVM ? Best Regards, Strahil Nikolov