12:52 a.m.
Hi Jim,
I don't have any targeted suggestions, because there isn't much to latch on to. I
can say Gluster replica three (no arbiters) on dedicated servers serving a couple Ovirt
VM clusters here have not had these sorts of issues.
I suspect your long heal times (and the resultant long periods of high load) are at least
partly related to 1G networking. That is just a matter of IO - heals of VMs involve moving
a lot of bits. My cluster uses 10G bonded NICs on the gluster and ovirt boxes for storage
traffic and separate bonded 1G for ovirtmgmt and communication with other machines/people,
and we're occasionally hitting the bandwidth ceiling on the storage network. I'm
starting to think about 40/100G, different ways of splitting up intensive systems, and
considering iSCSI for specific volumes, although I really don't want to go there.
I don't run FreeNAS[1], but I do run FreeBSD as storage servers for their excellent
ZFS implementation, mostly for backups. ZFS will make your `heal` problem go away, but not
your bandwidth problems, which become worse (because of fewer NICS pushing traffic). 10G
hardware is not exactly in the impulse-buy territory, but if you can, I'd recommend
doing some testing using it. I think at least some of your problems are related.
If that's not possible, my next stops would be optimizing everything I could about
sharding, healing and optimizing for serving the shard size to squeeze as much performance
out of 1G as I could, but that will only go so far.
-j
[1] FreeNAS is just a storage-tuned FreeBSD with a GUI.
On Jul 6, 2018, at 1:19 PM, Jim Kusznir <jim(a)palousetech.com>
wrote:
hi all:
Once again my production ovirt cluster is collapsing in on itself. My servers are
intermittently unavailable or degrading, customers are noticing and calling in. This
seems to be yet another gluster failure that I haven't been able to pin down.
I posted about this a while ago, but didn't get anywhere (no replies that I found).
The problem started out as a glusterfsd process consuming large amounts of ram (up to the
point where ram and swap were exhausted and the kernel OOM killer killed off the
glusterfsd process). For reasons not clear to me at this time, that resulted in any VMs
running on that host and that gluster volume to be paused with I/O error (the glusterfs
process is usually unharmed; why it didn't continue I/O with other servers is
confusing to me).
I have 3 servers and a total of 4 gluster volumes (engine, iso, data, and data-hdd). The
first 3 are replica 2+arb; the 4th (data-hdd) is replica 3. The first 3 are backed by an
LVM partition (some thin provisioned) on an SSD; the 4th is on a seagate hybrid disk (hdd
+ some internal flash for acceleration). data-hdd is the only thing on the disk. Servers
are Dell R610 with the PERC/6i raid card, with the disks individually passed through to
the OS (no raid enabled).
The above RAM usage issue came from the data-hdd volume. Yesterday, I cought one of the
glusterfsd high ram usage before the OOM-Killer had to run. I was able to migrate the VMs
off the machine and for good measure, reboot the entire machine (after taking this
opportunity to run the software updates that ovirt said were pending). Upon booting back
up, the necessary volume healing began. However, this time, the healing caused all three
servers to go to very, very high load averages (I saw just under 200 on one server;
typically they've been 40-70) with top reporting IO Wait at 7-20%. Network for this
volume is a dedicated gig network. According to bwm-ng, initially the network bandwidth
would hit 50MB/s (yes, bytes), but tailed off to mostly in the kB/s for a while. All
machines' load averages were still 40+ and gluster volume heal data-hdd info reported
5 items needing healing. Server's were intermittently experiencing IO issues, even on
the 3 gluster volumes that appeared largely unaffected. Even the OS activities on the
hosts itself (logging in, running commands) would often be very delayed. The ovirt engine
was seemingly randomly throwing engine down / engine up / engine failed notifications.
Responsiveness on ANY VM was horrific most of the time, with random VMs being
inaccessible.
I let the gluster heal run overnight. By morning, there were still 5 items needing
healing, all three servers were still experiencing high load, and servers were still
largely unstable.
I've noticed that all of my ovirt outages (and I've had a lot, way more than is
acceptable for a production cluster) have come from gluster. I still have 3 VMs who's
hard disk images have become corrupted by my last gluster crash that I haven't had
time to repair / rebuild yet (I believe this crash was caused by the OOM issue previously
mentioned, but I didn't know it at the time).
Is gluster really ready for production yet? It seems so unstable to me.... I'm
looking at replacing gluster with a dedicated NFS server likely FreeNAS. Any suggestions?
What is the "right" way to do production storage on this (3 node cluster)? Can
I get this gluster volume stable enough to get my VMs to run reliably again until I can
deploy another storage solution?
--Jim
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/YQX3LQFQQPW...