[ovirt-users] gluster self-heal takes cluster offline

[Jim Kusznir]

Hi all:

I'm trying to understand why/how (and most importantly, how to fix) an
substantial issue I had last night.  This happened one other time, but I
didn't know/understand all the parts associated with it until last night.

I have a 3 node hyperconverged (self-hosted engine, Gluster on each node)
cluster.  Gluster is Replica 2 + arbitrar.  Current network configuration
is 2x GigE on load balance ("LAG Group" on switch), plus one GigE from each
server on a separate vlan, intended for Gluster (but not used).  Server
hardware is Dell R610's, each server as an SSD in it.  Server 1 and 2 have
the full replica, server 3 is the arbitrar.

I put server 2 into maintence so I can work on the hardware, including turn
it off and such.  In the course of the work, I found that I needed to
reconfigure the SSD's partitioning somewhat, and it resulted in wiping the
data partition (storing VM images).  I figure, its no big deal, gluster
will rebuild that in short order.  I did take care of the extended attr
settings and the like, and when I booted it up, gluster came up as expected
and began rebuilding the disk.

The problem is that suddenly my entire cluster got very sluggish.  The
entine was marking nodes and VMs failed and unfaling them throughout the
system, fairly randomly.  It didn't matter what node the engine or VM was
on.  At one point, it power cycled server 1 for "non-responsive" (even
though everything was running on it, and the gluster rebuild was working on
it).  As a result of this, about 6 VMs were killed and my entire gluster
system went down hard (suspending all remaining VMs and the engine), as
there were no remaining full copies of the data.  After several minutes
(these are Dell servers, after all...), server 1 came back up, and gluster
resumed the rebuild, and came online on the cluster.  I had to manually
(virtsh command) unpause the engine, and then struggle through trying to
get critical VMs back up.  Everything was super slow, and load averages on
the servers were often seen in excess of 80 (these are 8 core / 16 thread
boxes).  Actual CPU usage (reported by top) was rarely above 40% (inclusive
of all CPUs) for any one server. Glusterfs was often seen using 180%-350%
of a CPU on server 1 and 2.

I ended up putting the cluster in global HA maintence mode and disabling
power fencing on the nodes until the process finished.  It appeared on at
least two occasions a functional node was marked bad and had the fencing
not been disabled, a node would have rebooted, just further exacerbating
the problem.

Its clear that the gluster rebuild overloaded things and caused the
problem.  I don't know why the load was so high (even IOWait was low), but
load averages were definately tied to the glusterfs cpu utilization %.   At
no point did I have any problems pinging any machine (host or VM) unless
the engine decided it was dead and killed it.

Why did my system bite it so hard with the rebuild?  I baby'ed it along
until the rebuild was complete, after which it returned to normal operation.

As of this event, all networking (host/engine management, gluster, and VM
network) were on the same vlan.  I'd love to move things off, but so far
any attempt to do so breaks my cluster.  How can I move my management
interfaces to a separate VLAN/IP Space?  I also want to move Gluster to its
own private space, but it seems if I change anything in the peers file, the
entire gluster cluster goes down.  The dedicated gluster network is listed
as a secondary hostname for all peers already.

Will the above network reconfigurations be enough?  I got the impression
that the issue may not have been purely network based, but possibly server
IO overload.  Is this likely / right?

I appreciate input.  I don't think gluster's recovery is supposed to do as
much damage as it did the last two or three times any healing was required.

Thanks!
--Jim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180315/9509b353/attachment.html>