<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">What version of ovirt and gluster? Sounds like something I just saw with gluster 3.12.x, are you using libgfapi or just fuse mounts?<div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><hr style="border:none;border-top:solid #B5C4DF 1.0pt;padding:0 0 0 0;margin:10px 0 5px 0;" class=""><span style="margin: -1.3px 0.0px 0.0px 0.0px" id="RwhHeaderAttributes" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">From:</b> Sahina Bose <<a href="mailto:sabose@redhat.com" class="">sabose@redhat.com</a>></font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Subject:</b> Re: [ovirt-users] gluster self-heal takes cluster offline</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Date:</b> March 23, 2018 at 1:26:01 AM CDT</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">To:</b> Jim Kusznir</font></span><br class="">
<span style="margin: -1.3px 0.0px 0.0px 0.0px" class=""><font face="Helvetica" size="4" color="#000000" style="font: 13.0px Helvetica; color: #000000" class=""><b class="">Cc:</b> Ravishankar Narayanankutty; users</font></span><br class="">
<br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Mar 16, 2018 at 2:45 AM, Jim Kusznir <span dir="ltr" class=""><<a href="mailto:jim@palousetech.com" target="_blank" class="">jim@palousetech.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Hi all:<div class=""><br class=""></div><div class="">I'm trying to understand why/how (and most importantly, how to fix) an substantial issue I had last night. This happened one other time, but I didn't know/understand all the parts associated with it until last night.</div><div class=""><br class=""></div><div class="">I have a 3 node hyperconverged (self-hosted engine, Gluster on each node) cluster. Gluster is Replica 2 + arbitrar. Current network configuration is 2x GigE on load balance ("LAG Group" on switch), plus one GigE from each server on a separate vlan, intended for Gluster (but not used). Server hardware is Dell R610's, each server as an SSD in it. Server 1 and 2 have the full replica, server 3 is the arbitrar.</div><div class=""><br class=""></div><div class="">I put server 2 into maintence so I can work on the hardware, including turn it off and such. In the course of the work, I found that I needed to reconfigure the SSD's partitioning somewhat, and it resulted in wiping the data partition (storing VM images). I figure, its no big deal, gluster will rebuild that in short order. I did take care of the extended attr settings and the like, and when I booted it up, gluster came up as expected and began rebuilding the disk.</div></div></blockquote><div class=""><br class=""></div><div class="">How big was the data on this partition? What was the shard size set on the gluster volume?</div><div class="">Out of curiosity, how long did it take to heal and come back to operational?</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class=""><br class=""></div><div class="">The problem is that suddenly my entire cluster got very sluggish. The entine was marking nodes and VMs failed and unfaling them throughout the system, fairly randomly. It didn't matter what node the engine or VM was on. At one point, it power cycled server 1 for "non-responsive" (even though everything was running on it, and the gluster rebuild was working on it). As a result of this, about 6 VMs were killed and my entire gluster system went down hard (suspending all remaining VMs and the engine), as there were no remaining full copies of the data. After several minutes (these are Dell servers, after all...), server 1 came back up, and gluster resumed the rebuild, and came online on the cluster. I had to manually (virtsh command) unpause the engine, and then struggle through trying to get critical VMs back up. Everything was super slow, and load averages on the servers were often seen in excess of 80 (these are 8 core / 16 thread boxes). Actual CPU usage (reported by top) was rarely above 40% (inclusive of all CPUs) for any one server. Glusterfs was often seen using 180%-350% of a CPU on server 1 and 2. </div><div class=""><br class=""></div><div class="">I ended up putting the cluster in global HA maintence mode and disabling power fencing on the nodes until the process finished. It appeared on at least two occasions a functional node was marked bad and had the fencing not been disabled, a node would have rebooted, just further exacerbating the problem. </div><div class=""><br class=""></div><div class="">Its clear that the gluster rebuild overloaded things and caused the problem. I don't know why the load was so high (even IOWait was low), but load averages were definately tied to the glusterfs cpu utilization %. At no point did I have any problems pinging any machine (host or VM) unless the engine decided it was dead and killed it.</div><div class=""><br class=""></div><div class="">Why did my system bite it so hard with the rebuild? I baby'ed it along until the rebuild was complete, after which it returned to normal operation.</div><div class=""><br class=""></div><div class="">As of this event, all networking (host/engine management, gluster, and VM network) were on the same vlan. I'd love to move things off, but so far any attempt to do so breaks my cluster. How can I move my management interfaces to a separate VLAN/IP Space? I also want to move Gluster to its own private space, but it seems if I change anything in the peers file, the entire gluster cluster goes down. The dedicated gluster network is listed as a secondary hostname for all peers already.</div><div class=""><br class=""></div><div class="">Will the above network reconfigurations be enough? I got the impression that the issue may not have been purely network based, but possibly server IO overload. Is this likely / right?</div><div class=""><br class=""></div><div class="">I appreciate input. I don't think gluster's recovery is supposed to do as much damage as it did the last two or three times any healing was required.</div><div class=""><br class=""></div><div class="">Thanks!</div><span class="HOEnZb"><font color="#888888" class=""><div class="">--Jim</div></font></span></div>
<br class="">______________________________<wbr class="">_________________<br class="">
Users mailing list<br class="">
<a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank" class="">http://lists.ovirt.org/<wbr class="">mailman/listinfo/users</a><br class="">
<br class=""></blockquote></div><br class=""></div></div>
_______________________________________________<br class="">Users mailing list<br class=""><a href="mailto:Users@ovirt.org" class="">Users@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/users<br class=""></div></blockquote></div><br class=""></div></body></html>