[ovirt-users] Re: Tuning and testing GlusterFS performance

On Fri, 3 Aug 2018 at 3:07 PM, Jayme <jaymef(a)gmail.com&gt; wrote: > Hello, > > The option to optimize for virt store is tough to find (in my opinion) > you have to go to volumes > volume name and then click the two dots to > expand further options in the top right to see it. No one would know to > find it (or that it even exists) if they weren't specifically looking. > > I don't know enough about it but my assumption is that there are reasons > why it's not set by default (as it might or should not need to apply to > ever volume created), however my suggestion would be that it be included in > the cockpit as a selectable option next to each volume you create with a > hint to suggest that for best performance select it for any volume that is > going to be a data volume for VMs > If you have installed via Cockpit, the options are set. Can you provide the “gluster volume info “ output after you optimised for virt? . > > I simply installed using the latest node ISO / default cockpit deployment. > > Hope this helps! > > - Jayme > > On Fri, Aug 3, 2018 at 5:15 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > >> >> >> On Fri, Aug 3, 2018 at 5:25 AM, Jayme <jaymef(a)gmail.com&gt; wrote: >> >>> Bill, >>> >>> I thought I'd let you (and others know this) as it might save you some >>> headaches. I found that my performance problem was resolved by clicking >>> "optimize for virt store" option in the volume settings of the hosted >>> engine (for the data volume). Doing this one change has increased my I/O >>> performance by 10x alone. I don't know why this would not be set or >>> recommended by default but I'm glad I found it! >>> >> >> Thanks for the feedback, Could you log a bug to make it default by >> providing the user flow that you used. >> >> Also, I would be interested to know how you prepared the gluster volume >> for use - if it was using the Cockpit deployment UI, the volume options >> would have been set by default. >> >> >>> - James >>> >>> On Thu, Aug 2, 2018 at 2:32 PM, William Dossett < >>> william.dossett(a)gmail.com&gt; wrote: >>> >>>> Yeah, I am just ramping up here, but this project is mostly on my own >>>> time and money, hence no SSDs for Gluster… I’ve already blown close to $500 >>>> of my own money on 10Gb ethernet cards and SFPs on ebay as my company >>>> frowns on us getting good deals for equipment on ebay and would rather go >>>> to their preferred supplier – where $500 wouldn’t even buy half a 10Gb CNA >>>> ☹ but I believe in this project and it feels like it is getting >>>> ready for showtime – if I can demo this in a few weeks and get some >>>> interest I’ll be asking them to reimburse me, that’s for sure! >>>> >>>> >>>> >>>> Hopefully going to get some of the other work off my plate and work on >>>> this later this afternoon, will let you know any findings. >>>> >>>> >>>> >>>> Regards >>>> >>>> Bill >>>> >>>> >>>> >>>> >>>> >>>> *From:* Jayme <jaymef(a)gmail.com&gt; >>>> *Sent:* Thursday, August 2, 2018 11:07 AM >>>> *To:* William Dossett <william.dossett(a)gmail.com&gt; >>>> *Cc:* users <users(a)ovirt.org&gt; >>>> *Subject:* Re: [ovirt-users] Tuning and testing GlusterFS performance >>>> >>>> >>>> >>>> Bill, >>>> >>>> >>>> >>>> Appreciate the feedback and would be interested to hear some of your >>>> results. I'm a bit worried about what i'm seeing so far on a very stock 3 >>>> node HCI setup. 8mb/sec on that dd test mentioned in the original post >>>> from within a VM (which may be explained by bad testing methods or some >>>> other configuration considerations).. but what is more worrisome to me is >>>> that I tried another dd test to time creating a 32GB file, it was taking a >>>> long time so I exited the process and the VM basically locked up on me, I >>>> couldn't access it or the console and eventually had to do a hard shutdown >>>> of the VM to recover. >>>> >>>> >>>> >>>> I don't plan to host many VMs, probably around 15. They aren't super >>>> demanding servers but some do read/write big directories such as working >>>> with github repos and large node_module folders, rsyncs of fairly large >>>> dirs etc. I'm definitely going to have to do a lot more testing before I >>>> can be assured enough to put any important VMs on this cluster. >>>> >>>> >>>> >>>> - James >>>> >>>> >>>> >>>> On Thu, Aug 2, 2018 at 1:54 PM, William Dossett < >>>> william.dossett(a)gmail.com&gt; wrote: >>>> >>>> I usually look at IOPs using IOMeter… you usually want several workers >>>> running reads and writes in different threads at the same time. You can >>>> run Dynamo on a Linux instance and then connect it to a window GUI running >>>> IOMeter to give you stats. I was getting around 250 IOPs on JBOD sata >>>> 7200rpm drives which isn’t bad for cheap and cheerful sata drives. >>>> >>>> >>>> >>>> As I said, I’ve worked with HCI in VMware now for a couple of years, >>>> intensely this last year when we had some defective Dell hardware and >>>> trying to diagnose the problem. Since then the hardware has been >>>> completely replaced with all flash solution. So when I got the all flash >>>> solution I used IOmeter on it and was only getting around 3000 IOPs on >>>> enterprise flash disks… not exactly stellar, but OK for one VM. The trick >>>> there was the scale out. There is a VMware Fling call HCI Bench. Its very >>>> cool in that you spin up one VM and then it spawns 40 more VMs across the >>>> cluster. I could then use VSAN observer and it showed my hosts were >>>> actually doing 30K IOPs on average which is absolutely stellar >>>> performance. >>>> >>>> >>>> >>>> Anyway, moral of the story there was that your one VM may seem like >>>> its quick, but not what you would expect from flash… but as you add more >>>> VMs in the cluster and they are all doing workloads, it scales out >>>> beautifully and the read/write speed does not slow down as you add more >>>> loads. I’m hoping that’s what we are going to see with Gluster. >>>> >>>> >>>> >>>> Also, you are using mb nomenclature below, is that Mb, or MB? I am >>>> sort of assuming MB megabytes per second… it does not seem very fast. I’m >>>> probably not going to get to work more on my cluster today as I’ve got >>>> other projects that I need to get done on time, but I want to try and get >>>> some templates up and running and do some more testing either tomorrow or >>>> this weekend and see what I get in just basic writing MB/s and let you know. >>>> >>>> >>>> >>>> Regards >>>> >>>> Bill >>>> >>>> >>>> >>>> >>>> >>>> *From:* Jayme <jaymef(a)gmail.com&gt; >>>> *Sent:* Thursday, August 2, 2018 8:12 AM >>>> *To:* users <users(a)ovirt.org&gt; >>>> *Subject:* [ovirt-users] Tuning and testing GlusterFS performance >>>> >>>> >>>> >>>> So I've finally completed my first HCI build using the below >>>> configuration: >>>> >>>> >>>> >>>> 3x >>>> >>>> Dell PowerEdge R720 >>>> >>>> 2x 2.9 GHz 8 Core E5-2690 >>>> >>>> 256GB RAM >>>> >>>> 2x250gb SSD Raid 1 (boot/os) >>>> >>>> 2x2TB SSD jbod passthrough (used for gluster bricks) >>>> >>>> 1Gbe Nic for management 10Gbe nic for Gluster >>>> >>>> >>>> >>>> Using Replica 3 with no arbiter. >>>> >>>> >>>> >>>> Installed the latest version of oVirt available at the time 4.2.5. >>>> Created recommended volumes (with an additional data volume on second SSD). >>>> Not using VDO >>>> >>>> >>>> >>>> First thing I did was setup glusterFS network on 10Gbe and set it to >>>> be used for glusterFS and migration traffic. >>>> >>>> >>>> >>>> I've setup a single test VM using Centos7 minimal on the default >>>> "x-large instance" profile. >>>> >>>> >>>> >>>> Within this VM if I do very basic write test using something like: >>>> >>>> >>>> >>>> dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync >>>> >>>> >>>> >>>> I'm seeing quite slow speeds, only 8mb/sec. >>>> >>>> >>>> >>>> If I do the same from one of the hosts gluster mounts i.e. >>>> >>>> >>>> >>>> host1: /rhev/data-center/mnt/glusterSD/HOST:data >>>> >>>> >>>> >>>> I get about 30mb/sec (which still seems fairly low?) >>>> >>>> >>>> >>>> Am I testing incorrectly here? Is there anything I should be tuning >>>> on the Gluster volumes to increase performance with SSDs? Where can I find >>>> out where the bottle neck is here, or is this expected performance of >>>> Gluster? >>>> >>>> >>>> >>> >>> >>> _______________________________________________ >>> Users mailing list -- users(a)ovirt.org >>> To unsubscribe send an email to users-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/community/about/community- >>> guidelines/ >>> List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/ >>> message/JKUWFXIZOWQ42JFBIJLFJGBKISY7OMPV/ >>> >>> >> >