Ovirt 4.3.10 Glusterfs SSD slow performance over 10GE
Hello all, I am using Ovirt 4.3.10 on Centos 7.8 with glusterfs 6.9 . My Gluster setup is of 3 hosts in replica 3 (2 hosts + 1 arbiter). All the 3 hosts are Dell R720 with Perc Raid Controller H710 mini(that has maximim throughtout 6Gbs) and with 2×1TB samsumg SSD in RAID 0. The volume is partitioned using LVM thin provision and formated XFS. The hosts have separate 10GE network cards for storage traffic. The Gluster Network is connected to this 10GE network cards and is mounted using Fuse Glusterfs(NFS is disabled).Also Migration Network is activated on the same storage network. The problem is that the 10GE network is not used at full potential by the Gluster. If i do live Migration of Vms i can see speeds of 7GB/s ~ 9GB/s. The same network tests using iperf3 reported 9.9GB/s , these exluding the network setup as a bottleneck(i will not paste all the iperf3 tests here for now). I did not enable all the Volume options from "Optimize for Virt Store", because of the bug that cant set volume cluster.granural-heal to enable(this was fixed in vdsm-4 40, but that is working only on Centos 8 with ovirt 4.4 ) . i whould be happy to know what are all these "Optimize for Virt Store" options, so i can set them manually. The speed on the disk inside the host using dd is b etween 1GB/s to 700Mbs. [root@host1 ~]# dd if=/dev/zero of=test bs=100M count=40 cou nt=80 status=progress 8074035200 bytes (8.1 GB) copied, 11.059372 s, 730 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 11.9928 s, 699 MB/s The dd write test on the gluster volme inside the host is poor only ~ 120MB/s . During the dd test, if i look at Networks->Gluster network ->Hosts at Tx and Rx the network speed barerly reaches over 1Gbs (~1073 Mbs) out of maximum of 10000 Mbs. dd if=/dev/zero of=/rhev/data-center/mnt/glu sterSD/gluster1.domain.local\:_data/test bs=100M count=80 status=progress 8283750400 bytes (8.3 GB) copied, 71.297942 s, 116 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 71.9545 s, 117 MB/s I have attached my Gluster volume settings and mount options. Thanks, Emy
Hello , Let me ask some questions: 1. What is the scheduler for your PV ? 2. Have you aligned your PV during the setup 'pvcreate --dataalignment alignment_value device' 3. What is your tuned profile ? Do you use rhgs-random-io from the ftp://ftp.redhat.com/redhat/linux/enterprise/7Server/en/RHS/SRPMS/redhat-storage-server-3.5.0.0-6.el7rhgs.src.rpm ? 4. What is the output of "xfs_info /path/to/your/gluster/brick" ? 5. Are you using Jumbo Frames ? Does your infra support them? Usually MTU of 9k is standard, but some switches and NICs support up to 16k. All the options for "optimize for virt...." are located at /var/lib/glusterd/groups/virt on each gluster node. Best Regards, Strahil Nikolov В неделя, 28 юни 2020 г., 22:13:09 Гринуич+3, jury cat <shadow.emy1@gmail.com> написа: Hello all, I am using Ovirt 4.3.10 on Centos 7.8 with glusterfs 6.9 . My Gluster setup is of 3 hosts in replica 3 (2 hosts + 1 arbiter). All the 3 hosts are Dell R720 with Perc Raid Controller H710 mini(that has maximim throughtout 6Gbs) and with 2×1TB samsumg SSD in RAID 0. The volume is partitioned using LVM thin provision and formated XFS. The hosts have separate 10GE network cards for storage traffic. The Gluster Network is connected to this 10GE network cards and is mounted using Fuse Glusterfs(NFS is disabled).Also Migration Network is activated on the same storage network. The problem is that the 10GE network is not used at full potential by the Gluster. If i do live Migration of Vms i can see speeds of 7GB/s ~ 9GB/s. The same network tests using iperf3 reported 9.9GB/s , these exluding the network setup as a bottleneck(i will not paste all the iperf3 tests here for now). I did not enable all the Volume options from "Optimize for Virt Store", because of the bug that cant set volume cluster.granural-heal to enable(this was fixed in vdsm-4 40, but that is working only on Centos 8 with ovirt 4.4 ) . i whould be happy to know what are all these "Optimize for Virt Store" options, so i can set them manually. The speed on the disk inside the host using dd is b etween 1GB/s to 700Mbs. [root@host1 ~]# dd if=/dev/zero of=test bs=100M count=40 cou nt=80 status=progress 8074035200 bytes (8.1 GB) copied, 11.059372 s, 730 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 11.9928 s, 699 MB/s The dd write test on the gluster volme inside the host is poor only ~ 120MB/s . During the dd test, if i look at Networks->Gluster network ->Hosts at Tx and Rx the network speed barerly reaches over 1Gbs (~1073 Mbs) out of maximum of 10000 Mbs. dd if=/dev/zero of=/rhev/data-center/mnt/glu sterSD/gluster1.domain.local\: _data/test bs=100M count=80 status=progress 8283750400 bytes (8.3 GB) copied, 71.297942 s, 116 MB/s 80+0 records in 80+0 records out 8388608000 bytes (8.4 GB) copied, 71.9545 s, 117 MB/s I have attached my Gluster volume settings and mount options. Thanks, Emy _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/7BR6TZQ4EXS4SI...
Hello ,
Hello and thank you for the reply.Bellow are the answers to your questions.
Let me ask some questions: 1. What is the scheduler for your PV ?
On the Raid Controller device where the SSD disks are in Raid 0 (device sda) it is set to "deadline". But on the lvm volume logical volume dm-7, where the logical block is set for "data" volunr it is set to none.(i think this is ok ) [root@host1 ~]# ls -al /dev/mapper/gluster_vg_sd v_data ter_l lrwxrwxrwx. 1 root root 7 Jun 28 14:14 /dev/mapper/gluster_v g_sda3-gluster_lv_data -> ../dm-7 [root@host1 ~]# cat /sys/block/dm-7/queue/scheduler none root@host1:~[root@host1 ~]# cat /sys/block/dm-7/queue/schedu [root@host1 ~]# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
2. Have you aligned your PV during the setup 'pvcreate --dataalignment alignment_value device'
I did not made other alignment then the default.Bellow are the partitions on /dev/sda. Can i enable partition alignment now, if yes how ? sfdisk -d /dev/sda # partition table of /dev/sda unit: sectors /dev/sda1 : start= 2048, size= 487424, Id=83, bootable /dev/sda2 : start= 489472, size= 95731712, Id=8e /dev/sda3 : start= 96221184, size=3808675840, Id=83 /dev/sda4 : start= 0, size= 0, Id= 0
My tuned active profile is virtual-host Current active profile: virtual-host No i dont use any of the rhgs-random-io profiles
4. What is the output of "xfs_info /path/to/your/gluster/brick" ?
xfs_info /gluster_bricks/data meta-data=/dev/mapper/gluster_vg_sda3-gluster_lv_data isize= 512 agcount=32, agsize=6553600 blks = sectsz=512 attr=2, projid 32bit=1 = crc=1 finobt=0 spino des=0 data = bsize=4096 blocks=2097152 00, imaxpct=25 = sunit=64 swidth=64 blks naming =version 2 bsize=8192 ascii-ci=0 fty pe=1 log =internal bsize=4096 blocks=102400, version=2 = sectsz=512 sunit=64 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtex tents=0
5. Are you using Jumbo Frames ? Does your infra support them? Usually MTU of 9k is standard, but some switches and NICs support up to 16k.
Unfortunately I can not enable MTU to 9000 and Jumbo Frames on these Cisco SG350X switches to specific ports.The switches dont suport Jumbo Frames enable to a single port, only on all ports . I have others devices connected to the switches on the remaining 48 ports that have 1Gb/s.
All the options for "optimize for virt...." are located at /var/lib/glusterd/groups/virt on each gluster node.
I have already looked previously at that file, but not all the volume settings that are set by "Optime for Virt Store" are stored there. For example "Optimize for Virt Store " sets network.remote.dio to disable and in the glusterd/groups/virt is set to enabled.Or cluster.granular-entry-heal: enable is not present there, bit it is set by "Optimize for Virt Store"
I’ve tried various methods to improve gluster performance on similar hardware and never had much luck. Small file workloads were particularly troublesome. I ended up switching high performance vms to nfs storage and performance with nfs improved greatly in my use case. On Sun, Jun 28, 2020 at 6:42 PM shadow emy <shadow.emy1@gmail.com> wrote:
Ovirt is using the default shard file of 64MB and I don't think this is 'small file' at all. There are a lot of tunables to optimize Gluster and I can admit it's not an easy task. Deadline is good for databases, but with SSDs you should try the performance of enabled multiqueue and the 'none' scheduler. By default EL7 is not using multiqueue. PV alignment is important and you can implement it by destroying the brick and recreating it .You need the stripe size and the stripe width (stripe size x data disk in the raid) during the pvcreate. In your case you could consider using the SSDs in JBOD mode, as in case of failiure you will recover only 1 disk. In raid0 the ammount of I/O towards the disks will be the same , which can lead to predictive failiure at the same time. For more details , check RHGS 3.5 Administration Guide. The rhgs-random-io is optimized for Gluster workloads and can bring benefits. In my case, I use a mix of rhgs-random-io and virtual-host. The XFS isize looks OK. You should apply the settings from the virt group, if you wish the optimal settings. WARNING: NEVER disable sharding once enabled -> stays enabled! Best Regards, Strahil Nikolov На 29 юни 2020 г. 1:33:20 GMT+03:00, Jayme <jaymef@gmail.com> написа:
If i destroy the brick, i might upgrade to ovirt 4.4 and Centos 8.2. Do you think upgrade to ovirt 4.4 with glusterfs improves performance or i am better with NFS ? If that partition alignment is so important, can i have an example command how to set it up ? I have upload an image with my current Raid 0 size and strip size. Btw i manage to enable Jumbo Frames with 9k MTU on the Storage Gluster Network and i can also try to enable multique scheduler. Can i use the latest glusterfs version 8 with ovirt 4.3.10 or 4.4 ? if of course has performance benefits. Also can you share the rhgs-random-io.settings you use. Thanks, Emy
На 29 юни 2020 г. 4:14:33 GMT+03:00, jury cat <shadow.emy1@gmail.com> написа:
Actually only you can find out as we cannot know the workload of your VMs. ovirt 4.4 uses gluster v7 , but I have to warn you that several people has reported issues after upgrading from v6.5 to 6.6+ or from 7.0 to 7.1+ . It's still under investigation.
If that partition alignment is so important, can i have an example command how to set it up ?
You are using 64K stripe size , but usually Red Hat recommend either 128k for raid6 or 256k for raid10. In your case 256k sounds nice. Your stripe width will be 64k x 2 data disks = 128k So you should use : pvcreate --dataalignment 128k /dev/raid-device For details , check RHGS Documentation; https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/ht...
Verify that the MTU is the same on egery device. As IP + ICMP stack need 28 bits , you can try: ping -M do -c 10 -s 8972 remote_gluster node Also, you can test lige changing the I/O scheduler.
Also can you share the rhgs-random-io.settings you use.
I can't claim those are universal, but here is mine : [main] summary=Optimize for running KVM guests on Gluster (Random IO) include=throughput-performance [cpu] governor=ondemand|powersave energy_perf_bias=powersave|power [sysctl] #vm.dirty_ratio = 5 #Random io -> 2 , vm host -> 5 #vm.dirty_background_ratio = 4 vm.dirty_background_bytes = 200000000 vm.dirty_bytes = 450000000 # The total time the scheduler will consider a migrated process # "cache hot" and thus less likely to be re-migrated # (system default is 500000, i.e. 0.5 ms) kernel.sched_migration_cost_ns = 5000000 I'm using powersave governor, as I'm chasing better power efficiency than performance . I would recommend you to take a look in the source rpm from the previous e-mail, which contains Red Hat's tuned profile.
Thanks, Emy
Thank you for the information provided. Yeap MTU is working ok with Jumbo Frames, on all gluster nodes. In the next days if i have time, I will try to play with ovirt 4.4 and gluster 7.x vs ovirt 4.4 and NFS to check for performance. I might try even ceph with ovirt 4.4
i found the problem. The kernel version in Centos 7.8 with version 3.x.x is really too old and does not know how to handle fine new SSD disks or RAID Controllers with latest BIOS Updates applied. Booting and Archlinux latest iso image with kernel 5.7.6 or a Centos 8.2 with kernel 4.18 increased the performance at the right values. I run multiple dd tests on the above images using bs of 10, 100 and 1000M and had aconstant write speed of 1.1GB/s.This is the expected value for 2 SSD in RAID 0. I had also enabled cache settings on the Dell Perc 710 Raid controller : Write Cache set to "Write Back", disk cache set to "Enabled", read cache to "Read Ahead".For those who think "Write back" is a problem and the data might be corrupted, this should be ok now with the latest filesystem xfs or ext4 , that can recover in case of power loss.To make data safer, i also have a Raid cache battery and UPS redundancy. Now i know i must run ovirt 4.4 with Centos 8.2 for good performance. I saw that Upgrading from 4.3 to 4.4 is not an easy task, multiple fails and not quite straight forward(i also have hosted engine on the shared Gluster Storage which makes this ipgrade even more difficult), but eventually i think i can get it running. Thanks, Emy
Emy, I was wondering how much if any improvement I'd see with Gluster storage moving to oVirt 4.4/CentOS 8x (but have not made the switch yet myself). You should keep in mind that your Perc controllers aren't supported by CentOS 8 out of the box, they dropped support for many older controllers. You should still be able to get it to work using a driver update disk during install. See: https://forums.centos.org/viewtopic.php?t=71862 Either way, this is good to know ahead of time as to limit surprises! - Jayme On Tue, Jul 7, 2020 at 10:22 AM shadow emy <shadow.emy1@gmail.com> wrote:
Ohh yes is important to know ahead.Not so nice if they drop drivers. Fortunately for now my Perc H710 (LSI MegaRAID SAS 2208) is still supported in megaraid_sas linux module for RHEL 8. Upgrade to ovirt 4.4 is really difficult.I had to have downtimes for it to work correctly. After you deploy a restore from old ovirt 4.3 backup, you cant switch cluster comparibility_version from 4.3 to 4.4 using the web interface, it wont let you and you will have lots of errors. I had to hack into the database and change the cluster : compatibility, cpu type, cpu flags for it to work correctly.On some vms i had to change also the cpu_name in the database. The problem is 4.3 cpu profiles(they where changed in ovirt 4.4) are not supported in ovirt 4.4. Because of this all your hosts with 4.3 will be in NonResponsive state on hosted-engine 4.4 . If you have gluster like me, is even more difficult.Hosts many times failed to activate , because the Storage domains where down. But finally i mange to upgrade it, though was hard.
participants (4)
-
Jayme -
jury cat -
shadow emy
-
Strahil Nikolov