7:07 p.m.
Bill,
Appreciate the feedback and would be interested to hear some of your
results. I'm a bit worried about what i'm seeing so far on a very stock 3
node HCI setup. 8mb/sec on that dd test mentioned in the original post
from within a VM (which may be explained by bad testing methods or some
other configuration considerations).. but what is more worrisome to me is
that I tried another dd test to time creating a 32GB file, it was taking a
long time so I exited the process and the VM basically locked up on me, I
couldn't access it or the console and eventually had to do a hard shutdown
of the VM to recover.
I don't plan to host many VMs, probably around 15. They aren't super
demanding servers but some do read/write big directories such as working
with github repos and large node_module folders, rsyncs of fairly large
dirs etc. I'm definitely going to have to do a lot more testing before I
can be assured enough to put any important VMs on this cluster.
- James
On Thu, Aug 2, 2018 at 1:54 PM, William Dossett <william.dossett(a)gmail.com>
wrote:
I usually look at IOPs using IOMeter… you usually want several
workers
running reads and writes in different threads at the same time. You can
run Dynamo on a Linux instance and then connect it to a window GUI running
IOMeter to give you stats. I was getting around 250 IOPs on JBOD sata
7200rpm drives which isn’t bad for cheap and cheerful sata drives.
As I said, I’ve worked with HCI in VMware now for a couple of years,
intensely this last year when we had some defective Dell hardware and
trying to diagnose the problem. Since then the hardware has been
completely replaced with all flash solution. So when I got the all flash
solution I used IOmeter on it and was only getting around 3000 IOPs on
enterprise flash disks… not exactly stellar, but OK for one VM. The trick
there was the scale out. There is a VMware Fling call HCI Bench. Its very
cool in that you spin up one VM and then it spawns 40 more VMs across the
cluster. I could then use VSAN observer and it showed my hosts were
actually doing 30K IOPs on average which is absolutely stellar
performance.
Anyway, moral of the story there was that your one VM may seem like its
quick, but not what you would expect from flash… but as you add more VMs
in the cluster and they are all doing workloads, it scales out beautifully
and the read/write speed does not slow down as you add more loads. I’m
hoping that’s what we are going to see with Gluster.
Also, you are using mb nomenclature below, is that Mb, or MB? I am sort
of assuming MB megabytes per second… it does not seem very fast. I’m
probably not going to get to work more on my cluster today as I’ve got
other projects that I need to get done on time, but I want to try and get
some templates up and running and do some more testing either tomorrow or
this weekend and see what I get in just basic writing MB/s and let you know.
Regards
Bill
*From:* Jayme <jaymef(a)gmail.com>
*Sent:* Thursday, August 2, 2018 8:12 AM
*To:* users <users(a)ovirt.org>
*Subject:* [ovirt-users] Tuning and testing GlusterFS performance
So I've finally completed my first HCI build using the below configuration:
3x
Dell PowerEdge R720
2x 2.9 GHz 8 Core E5-2690
256GB RAM
2x250gb SSD Raid 1 (boot/os)
2x2TB SSD jbod passthrough (used for gluster bricks)
1Gbe Nic for management 10Gbe nic for Gluster
Using Replica 3 with no arbiter.
Installed the latest version of oVirt available at the time 4.2.5.
Created recommended volumes (with an additional data volume on second SSD).
Not using VDO
First thing I did was setup glusterFS network on 10Gbe and set it to be
used for glusterFS and migration traffic.
I've setup a single test VM using Centos7 minimal on the default "x-large
instance" profile.
Within this VM if I do very basic write test using something like:
dd bs=1M count=256 if=/dev/zero of=test conv=fdatasync
I'm seeing quite slow speeds, only 8mb/sec.
If I do the same from one of the hosts gluster mounts i.e.
host1: /rhev/data-center/mnt/glusterSD/HOST:data
I get about 30mb/sec (which still seems fairly low?)
Am I testing incorrectly here? Is there anything I should be tuning on
the Gluster volumes to increase performance with SSDs? Where can I find
out where the bottle neck is here, or is this expected performance of
Gluster?