What do you mean Gluster being announced as EOL? Where did you find this information? On Mon, Apr 26, 2021 at 9:34 AM penguin pages <jeremey.wise(a)gmail.com&gt; wrote:

It was on a support ticket / call I was having. I googled around and the only article I found was the one about features being removed.. But not sure if this effects oVirt / HCI. My ticket was about trying to deploy OCP on a full SSD cluster of three nodes and disk performance over 10Gb will too slow and RH support was " We don't support use of gluster for OCP.. and need you to move off gluster for CEPH. So I opened another ticket about CEPH on HCI .. and was told "not supported.. CEPH nodes must be external" So my three server small work office and demo stack, now is rethinking having to go to anther stack / vendor such as VMWare and vSAN, just because I can't get a stack that meets needs for small HCI stack with Linux.

I haven't seen your email on the gluster users' mailing list . What was your problem with the performance ? Best Regards,Strahil Nikolov On Mon, Apr 26, 2021 at 17:30, penguin pages&lt;jeremey.wise(a)gmail.com&gt; wrote: It was on a support ticket / call I was having.  I googled around and the only article I found was the one about features being removed.. But not sure if this effects oVirt / HCI. My ticket was about trying to deploy OCP on a full SSD cluster of three nodes and disk performance over 10Gb will too slow and RH support was " We don't support use of gluster for OCP.. and need you to move off gluster for CEPH. So I opened another ticket about CEPH on HCI .. and was told "not supported.. CEPH nodes must be external"  So my three server small work office and demo stack, now is rethinking having to go to anther stack / vendor such as VMWare and vSAN, just because I can't get a stack that meets needs for small HCI stack with Linux. _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/K4QWNUSG6QD...

Tuning Gluster with VDO bellow is quite difficult and the overhead of using VDO could reduce performance . I would try with VDO compression and dedup disabled.If your SSD has 512 byte physical & logical size, you can skip VDO at all to check performance. Also FS mount options are very important for XFS. Best Regards,Strahil Nikolov On Mon, Apr 26, 2021 at 20:50, penguin pages&lt;jeremey.wise(a)gmail.com&gt; wrote: The response was that when I select the oVirt HCI storage volumes to deploy to (with VDO enabled) which are a single 512GB SSD with only one small IDM VM running.  The IPI OCP 4.7 deployment fails.  RH closed ticket because "gluster volume is too slow". I then tried to create with my other 1TB SSD in each server gluster volume without VDO and see if that worked.. but though I matched all settings / gluster options oVirt set, IPI OCP would not show disk as deployable option.  I then figured I would use GUI to create bricks vs ansible means (trying to be good and stop doing direct shell but build as ansible playbooks).. but that test is on hold because servers for next two weeks are being re-tasked for another POC.  So I figured I would do some rethinking if oVirt HCI on RHEV 4.5 with Gluster is a rat hole that will never work.  Below is the output from fio test OCP team asked me to run to show gluster was too slow. ################## ansible@LT-A0070501:/mnt/c/GitHub/penguinpages_cluster_devops/cluster_devops$ ssh core(a)172.16.100.184 [core@localhost ~]$ journalctl -b -f -u release-image.service -u bootkube.service -- Logs begin at Sun 2021-04-11 19:18:07 UTC. -- Apr 13 11:50:23 localhost bootkube.sh[1276476]: [#404] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2021-04-13T11:50:23Z is after 2021-04-12T19:12:30Z Apr 13 11:50:23 localhost bootkube.sh[1276476]: [#405] failed to fetch discovery: Get "https://localhost:6443/api?timeout=32s": x509: certificate has expired or is not yet valid: current time 2021-04-13T11:50:23Z is after 2021-04-12T19:12:30Z <snip> [core@localhost ~]$ su - Password: su: Authentication failure [core@localhost ~]$ sudo podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf Trying to pull quay.io/openshift-scale/etcd-perf Trying to pull quay.io/openshift-scale/etcd-perf... Getting image source signatures Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 done Copying blob a3ed95caeb02 skipped: already exists Copying blob fcc022b71ae4 done Copying blob a93d706457b7 done Copying blob 763b3f36c462 done Writing manifest to image destination Storing signatures ---------------------------------------------------------------- Running fio --------------------------------------------------------------------------- {   "fio version" : "fio-3.7",   "timestamp" : 1618315279,   "timestamp_ms" : 1618315279798,   "time" : "Tue Apr 13 12:01:19 2021",   "global options" : {     "rw" : "write",     "ioengine" : "sync",     "fdatasync" : "1",     "directory" : "/var/lib/etcd",     "size" : "22m",     "bs" : "2300"   },   "jobs" : [     {       "jobname" : "etcd_perf",       "groupid" : 0,       "error" : 0,       "eta" : 0,       "elapsed" : 507,       "job options" : {         "name" : "etcd_perf"       },       "read" : {         "io_bytes" : 0,         "io_kbytes" : 0,         "bw_bytes" : 0,         "bw" : 0,         "iops" : 0.000000,         "runtime" : 0,         "total_ios" : 0,         "short_ios" : 10029,         "drop_ios" : 0,         "slat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000         },         "clat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000,           "percentile" : {             "1.000000" : 0,             "5.000000" : 0,             "10.000000" : 0,             "20.000000" : 0,             "30.000000" : 0,             "40.000000" : 0,             "50.000000" : 0,             "60.000000" : 0,             "70.000000" : 0,             "80.000000" : 0,             "90.000000" : 0,             "95.000000" : 0,             "99.000000" : 0,             "99.500000" : 0,             "99.900000" : 0,             "99.950000" : 0,             "99.990000" : 0           }         },         "lat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000         },         "bw_min" : 0,         "bw_max" : 0,         "bw_agg" : 0.000000,         "bw_mean" : 0.000000,         "bw_dev" : 0.000000,         "bw_samples" : 0,         "iops_min" : 0,         "iops_max" : 0,         "iops_mean" : 0.000000,         "iops_stddev" : 0.000000,         "iops_samples" : 0       },       "write" : {         "io_bytes" : 23066700,         "io_kbytes" : 22526,         "bw_bytes" : 45589,         "bw" : 44,         "iops" : 19.821372,         "runtime" : 505969,         "total_ios" : 10029,         "short_ios" : 0,         "drop_ios" : 0,         "slat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000         },         "clat_ns" : {           "min" : 12011,           "max" : 680340,           "mean" : 26360.617210,           "stddev" : 15390.749240,           "percentile" : {             "1.000000" : 13632,             "5.000000" : 14528,             "10.000000" : 15808,             "20.000000" : 18304,             "30.000000" : 19584,             "40.000000" : 20864,             "50.000000" : 22656,             "60.000000" : 25472,             "70.000000" : 28800,             "80.000000" : 33536,             "90.000000" : 40192,             "95.000000" : 46848,             "99.000000" : 64768,             "99.500000" : 73216,             "99.900000" : 96768,             "99.950000" : 103936,             "99.990000" : 651264           }         },         "lat_ns" : {           "min" : 13570,           "max" : 682541,           "mean" : 28604.818825,           "stddev" : 15726.819671         },         "bw_min" : 4,         "bw_max" : 89,         "bw_agg" : 100.000000,         "bw_mean" : 46.770285,         "bw_dev" : 15.478852,         "bw_samples" : 949,         "iops_min" : 1,         "iops_max" : 40,         "iops_mean" : 21.048472,         "iops_stddev" : 6.908059,         "iops_samples" : 949       },       "trim" : {         "io_bytes" : 0,         "io_kbytes" : 0,         "bw_bytes" : 0,         "bw" : 0,         "iops" : 0.000000,         "runtime" : 0,         "total_ios" : 0,         "short_ios" : 0,         "drop_ios" : 0,         "slat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000         },         "clat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000,           "percentile" : {             "1.000000" : 0,             "5.000000" : 0,             "10.000000" : 0,             "20.000000" : 0,             "30.000000" : 0,             "40.000000" : 0,             "50.000000" : 0,             "60.000000" : 0,             "70.000000" : 0,             "80.000000" : 0,             "90.000000" : 0,             "95.000000" : 0,             "99.000000" : 0,             "99.500000" : 0,             "99.900000" : 0,             "99.950000" : 0,             "99.990000" : 0           }         },         "lat_ns" : {           "min" : 0,           "max" : 0,           "mean" : 0.000000,           "stddev" : 0.000000         },         "bw_min" : 0,         "bw_max" : 0,         "bw_agg" : 0.000000,         "bw_mean" : 0.000000,         "bw_dev" : 0.000000,         "bw_samples" : 0,         "iops_min" : 0,         "iops_max" : 0,         "iops_mean" : 0.000000,         "iops_stddev" : 0.000000,         "iops_samples" : 0       },       "sync" : {         "lat_ns" : {           "min" : 10082235,           "max" : 6139419269,           "mean" : 50405768.015057,           "stddev" : 121884279.530146,           "percentile" : {             "1.000000" : 11075584,             "5.000000" : 11993088,             "10.000000" : 12779520,             "20.000000" : 15925248,             "30.000000" : 19267584,             "40.000000" : 30539776,             "50.000000" : 42729472,             "60.000000" : 49545216,             "70.000000" : 55836672,             "80.000000" : 64225280,             "90.000000" : 81264640,             "95.000000" : 113770496,             "99.000000" : 212860928,             "99.500000" : 258998272,             "99.900000" : 1535115264,             "99.950000" : 2868903936,             "99.990000" : 5335154688           }         },         "total_ios" : 0       },       "usr_cpu" : 0.045457,       "sys_cpu" : 0.145266,       "ctx" : 40541,       "majf" : 0,       "minf" : 13,       "iodepth_level" : {         "1" : 200.000000,         "2" : 0.000000,         "4" : 0.000000,         "8" : 0.000000,         "16" : 0.000000,         "32" : 0.000000,         ">=64" : 0.000000       },       "latency_ns" : {         "2" : 0.000000,         "4" : 0.000000,         "10" : 0.000000,         "20" : 0.000000,         "50" : 0.000000,         "100" : 0.000000,         "250" : 0.000000,         "500" : 0.000000,         "750" : 0.000000,         "1000" : 0.000000       },       "latency_us" : {         "2" : 0.000000,         "4" : 0.000000,         "10" : 0.000000,         "20" : 33.702263,         "50" : 62.498754,         "100" : 3.709243,         "250" : 0.059827,         "500" : 0.000000,         "750" : 0.029913,         "1000" : 0.000000       },       "latency_ms" : {         "2" : 0.000000,         "4" : 0.000000,         "10" : 0.000000,         "20" : 0.000000,         "50" : 0.000000,         "100" : 0.000000,         "250" : 0.000000,         "500" : 0.000000,         "750" : 0.000000,         "1000" : 0.000000,         "2000" : 0.000000,         ">=2000" : 0.000000       },       "latency_depth" : 1,       "latency_target" : 0,       "latency_percentile" : 100.000000,       "latency_window" : 0     }   ],   "disk_util" : [     {       "name" : "sda",       "read_ios" : 86,       "write_ios" : 34333,       "read_merges" : 0,       "write_merges" : 4065,       "read_ticks" : 1207,       "write_ticks" : 1389885,       "in_queue" : 1372762,       "util" : 95.134115     }   ] } -------------------------------------------------------------------------------------------------------------------------------------------------------- 99th percentile of fsync is 212860928 ns 99th percentile of the fsync is greater than the recommended value which is 10 ms, faster disks are recommended to host etcd for better performance [core@localhost ~]$ _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZIYECHKULWF...

Due to POSIX compliance, oVirt needs 512 byte physical sector size. If your SSD/NVME has the new standard (4096) you will need to use VDO with '--emulate512' flag (or whatever it was named). Yet, if you already got 512 physical sector size - you can skip VDO totally. About the mount options of the bricks, you can use noatime & inode64 Also if you use SELINUX use the 'context=' mount option to tell the kernel "skip looking for SELINUX Label, it's always..." Also, consider setting the SSD I/O scheduler to none (multique should be enabled on EL8 Hypervisors by default) which will reduce reordering of your I/O requests and speed up on fast storage. NVMEs by default use that. Best Regards,Strahil Nikolov On Mon, Apr 26, 2021 at 22:43, penguin pages&lt;jeremey.wise(a)gmail.com&gt; wrote: "...Tuning Gluster with VDO bellow is quite difficult and the overhead of using VDO could reduce performance ...." Yup. hense creation of a dedicated data00 volume from the 1TB SSD each server had.  Matched options listed in oVirt.. but still OCP would not address the drive as target for deployment. That is when I opened ticket with RH and they noted Gluster is not a supported target for OCP. Hense then off to check if we could do CEPH HCI.. nope. "..I would try with VDO compression and dedup disabled.If your SSD has 512 byte physical..& logical size, you can skip VDO at all to check performance....."  Yes.. VDO removed was/ is next test.  But your note about 512 is yes.. Are their tuning parameters for Gluster with this? "...Also FS mount options are very important for XFS...."  - What options do you use / recommend?  Do you have a link to said tuning manual page where I could review and knowing the base HCI volume is  VDO + XFS + Gluster.  But second volume for OCP will be just  XFS + Gluster I would assume this may change recommendations. Thanks, _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/JHRUZV6JS2F...