On Sep 29, 2016, at 7:47 AM, Martin Perina <mperina@redhat.com> wrote: =20 Hi, =20 please take a look at my inline comments: =20 On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun = <gervais@demontbrun.com <mailto:gervais@demontbrun.com>> wrote: Hey All, =20 Since updating to 4.0.x of oVirt, I have had an issue with my hosted = engine. After a some poking around, I think I have figured out my issue = and thought I would share to see what others think. The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists = in 4.0.4. =20 Description: When my hosted engine starts it reports that it is in a degraded state = with 7 or 8 services still not started when I run systemctl status. It = takes about 6 or 7 minutes to eventually start all the services and come = online. If I don't set my cluster to Global-Maintenance mode it = eventually thinks that my hosted-engine needs to be rebooted and = restarts it before it can start everything. =20 =E2=80=8BCould you please share with us logs gathered by = ovirt-log-collector? =20 It's just a guess but could you please take a look if you HE VM has = enough entropy? =20 cat /proc/sys/kernel/random/entropy_avail =20 If the value is low (below or around 200), you really need to install = and configure some entropy generator such as haveged =20 =20 Solution: I realized that Apache was the culprit and found that the proxy to the = ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super = long timeout with many retries. I changed the settings and now = everything works for me. =20 -> Before change: <LocationMatch = ^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engi= ne.ssh.key.txt$|rhevm.ssh.key.txt$)> ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3600 retry=3D5 =20 <IfModule deflate_module> AddOutputFilterByType DEFLATE text/javascript text/css = text/html text/xml text/json application/xml application/json = application/x-yaml </IfModule> </LocationMatch> =20 -> After change: <LocationMatch ^/ovirt-engine($|/)> ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D5 retry=3D2 =20 <IfModule deflate_module> AddOutputFilterByType DEFLATE text/javascript text/css = text/html text/xml text/json application/xml application/json = application/x-yaml </IfModule> </LocationMatch> =20 =E2=80=8BThis one is correct for 4.0=E2=80=8B=E2=80=8B, not sure why = it was not updated during upgrade from 3.6. @Simone? =E2=80=8B=20 =20 If I read the timeout settings correctly, it will wait 60 minutes with = 5 retries. 5 hours is way too long for my little server to hold onto all =
--Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Martin, The entropy was super low. Somewhere around 140. I installed and = configured haveged.service to start at bootup, reverted my apache = changes... After a reboot, my systemctl status still says that there are = 7 services queued (note that I erroneously said degraded in my previous = email - the services are, in fact, queued), but the oVirt GUI comes up = almost immediately and everything seems to be great. Thank you for the tip. You solved my issue. Cheers, Gervais those apache processes.
The change I made allows for there to be an error, and also releases = apache's hold on the process. Once everything is ready, apache is ready = to serve requests and everything/everyone is happy. Before making the = change, I just get a whitescreen in my browser and then nothing works = until I restart Apache (or I end up in an endless loop of ovirt-ha = services restarting my hosted-engine. =20 =E2=80=8BWell, if you have an issue with too many apache processes = waiting for engine to respond, then there's some issue in engine. As I = wrote above please share the logs with us and check entropy. =20 Thanks =20 Martin Perina =E2=80=8B=20 =20 I noticed that this setting reverts to the original setting, so oVirt = must be writing this file. Perhaps these number can be changed in oVirt? = If not, I will just setup and ansible play to revert the settings with = working values and restart apache on my engine. :-) =20 Cheers, Gervais =20 =20 =20 =20 _______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users = <http://lists.ovirt.org/mailman/listinfo/users> =20 =20
--Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">Hi Martin,<div class=3D""><br class=3D""></div><div = class=3D"">The entropy was super low. Somewhere around 140. I installed = and configured haveged.service to start at bootup, reverted my = apache changes... After a reboot, my systemctl status still says that = there are 7 services queued (note that I erroneously said degraded in my = previous email - the services are, in fact, queued), but the oVirt GUI = comes up almost immediately and everything seems to be great.</div><div = class=3D""><br class=3D""></div><div class=3D"">Thank you for the tip. = You solved my issue.<br class=3D""><div class=3D""> <div id=3D"signature" class=3D""><br class=3D"">Cheers,<br = class=3D"">Gervais<br class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""><div><blockquote type=3D"cite" class=3D""><div = class=3D"">On Sep 29, 2016, at 7:47 AM, Martin Perina <<a = href=3D"mailto:mperina@redhat.com" class=3D"">mperina@redhat.com</a>> = wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = dir=3D"ltr" class=3D""><div = style=3D"font-family:arial,helvetica,sans-serif" = class=3D"gmail_default">Hi,<br class=3D""><br class=3D""></div><div = style=3D"font-family:arial,helvetica,sans-serif" = class=3D"gmail_default">please take a look at my inline comments:<br = class=3D""></div><div class=3D"gmail_extra"><br class=3D""><div = class=3D"gmail_quote">On Tue, Sep 27, 2016 at 7:23 PM, Gervais de = Montbrun <span dir=3D"ltr" class=3D""><<a target=3D"_blank" = href=3D"mailto:gervais@demontbrun.com" = class=3D"">gervais@demontbrun.com</a>></span> wrote:<br = class=3D""><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px = solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div = style=3D"word-wrap:break-word" class=3D"">Hey All,<div class=3D""><br = class=3D""></div><div class=3D"">Since updating to 4.0.x of oVirt, I = have had an issue with my hosted engine. After a some poking around, I = think I have figured out my issue and thought I would share to see what = others think.</div><div class=3D"">The issue has existed with 4.0, = 4.0.1, 4.0.2, 4.0.3, and still exists in 4.0.4.</div><div class=3D""><br = class=3D""></div><div class=3D"">Description:</div><div class=3D"">When = my hosted engine starts it reports that it is in a degraded state with 7 = or 8 services still not started when I run systemctl status. It takes = about 6 or 7 minutes to eventually start all the services and come = online. If I don't set my cluster to Global-Maintenance mode it = eventually thinks that my hosted-engine needs to be rebooted and = restarts it before it can start everything.</div></div></blockquote><div = class=3D""><br class=3D""><div = style=3D"font-family:arial,helvetica,sans-serif;display:inline" = class=3D"gmail_default">=E2=80=8BCould you please share with us logs = gathered by ovirt-log-collector?<br class=3D""><br class=3D"">It's just = a guess but could you please take a look if you HE VM has enough = entropy?<br class=3D""><br class=3D""> cat = /proc/sys/kernel/random/entropy_avail<br class=3D""><br = class=3D""></div><div = style=3D"font-family:arial,helvetica,sans-serif;display:inline" = class=3D"gmail_default">If the value is low (below or around 200), = you really need to install and configure some entropy generator such as = haveged<br class=3D""><br class=3D""></div></div><blockquote = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div = style=3D"word-wrap:break-word" class=3D""><div class=3D""><br = class=3D""></div><div class=3D"">Solution:</div><div class=3D"">I = realized that Apache was the culprit and found that the proxy to the = ovirt-engine in /etc/httpd/conf.d/z-ovirt-<wbr = class=3D"">engine-proxy.conf has a super long timeout with many retries. = I changed the settings and now everything works for me.</div><div = class=3D""><br class=3D""></div><div class=3D"">-> Before = change:</div><blockquote style=3D"margin:0px 0px 0px = 40px;border-width:medium;border-style:none;border-color:-moz-use-text-colo= r;padding:0px" class=3D""><div class=3D""><div class=3D""> = <LocationMatch ^/(ovirt-engine($|/)|api($|/)|<wbr = class=3D"">RHEVManagerWeb/|<wbr class=3D"">OvirtEngineWeb/|ca.crt$|<wbr = class=3D"">engine.ssh.key.txt$|rhevm.ssh.<wbr = class=3D"">key.txt$)></div><div class=3D""> = ProxyPassMatch <a class=3D"">ajp://127.0.0.1:8702</a> = timeout=3D3600 retry=3D5</div><div class=3D""><br class=3D""></div><div = class=3D""> <IfModule = deflate_module></div><div class=3D""> = AddOutputFilterByType DEFLATE text/javascript text/css = text/html text/xml text/json application/xml application/json = application/x-yaml</div><div class=3D""> = </IfModule></div><div class=3D""> = </LocationMatch></div></div></blockquote><div class=3D""><br = class=3D""></div>-> After change:<blockquote style=3D"margin:0px 0px = 0px = 40px;border-width:medium;border-style:none;border-color:-moz-use-text-colo= r;padding:0px" class=3D""><div class=3D""><div class=3D""> = <LocationMatch ^/ovirt-engine($|/)></div><div class=3D""> = ProxyPassMatch <a class=3D"">ajp://127.0.0.1:8702</a>= timeout=3D5 retry=3D2</div><div class=3D""><br class=3D""></div><div = class=3D""> <IfModule = deflate_module></div><div class=3D""> = AddOutputFilterByType DEFLATE text/javascript text/css = text/html text/xml text/json application/xml application/json = application/x-yaml</div><div class=3D""> = </IfModule></div><div class=3D""> = </LocationMatch></div></div></blockquote></div></blockquote><div = class=3D""><br class=3D""><div class=3D"gmail_default" = style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8BT= his one is correct for 4.0=E2=80=8B</div><div class=3D"gmail_default" = style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8B,= not sure why it was not updated during upgrade from 3.6. @Simone?<br = class=3D"">=E2=80=8B</div> </div><blockquote style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" = class=3D"gmail_quote"><div style=3D"word-wrap:break-word" class=3D""><div = class=3D""><br class=3D""></div>If I read the timeout settings = correctly, it will wait 60 minutes with 5 retries. 5 hours is way too = long for my little server to hold onto all those apache processes. = </div></blockquote><blockquote style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" = class=3D"gmail_quote"><div style=3D"word-wrap:break-word" class=3D"">The = change I made allows for there to be an error, and also releases = apache's hold on the process. Once everything is ready, apache is ready = to serve requests and everything/everyone is happy. Before making the = change, I just get a whitescreen in my browser and then nothing works = until I restart Apache (or I end up in an endless loop of ovirt-ha = services restarting my hosted-engine.<br = class=3D""></div></blockquote><div class=3D""><br class=3D""><div = class=3D"gmail_default" = style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8BW= ell, if you have an issue with too many apache processes waiting for = engine to respond, then there's some issue in engine. As I wrote above = please share the logs with us and check entropy.<br class=3D""><br = class=3D""></div><div class=3D"gmail_default" = style=3D"font-family:arial,helvetica,sans-serif;display:inline">Thanks<br = class=3D""><br class=3D""></div><div class=3D"gmail_default" = style=3D"font-family:arial,helvetica,sans-serif;display:inline">Martin = Perina<br class=3D"">=E2=80=8B</div> </div><blockquote = style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div = style=3D"word-wrap:break-word" class=3D""><div class=3D""><div = class=3D""><div class=3D""><br class=3D""></div><div class=3D"">I = noticed that this setting reverts to the original setting, so oVirt must = be writing this file. Perhaps these number can be changed in oVirt? If = not, I will just setup and ansible play to revert the settings with = working values and restart apache on my engine.</div><div = class=3D"">:-)</div><div class=3D""> <div class=3D""><br class=3D"">Cheers,<br class=3D"">Gervais<br = class=3D""><br class=3D""><br class=3D""></div> </div> <br class=3D""></div></div></div><br = class=3D"">______________________________<wbr = class=3D"">_________________<br class=3D""> Users mailing list<br class=3D""> <a href=3D"mailto:Users@ovirt.org" class=3D"">Users@ovirt.org</a><br = class=3D""> <a target=3D"_blank" rel=3D"noreferrer" = href=3D"http://lists.ovirt.org/mailman/listinfo/users" = class=3D"">http://lists.ovirt.org/<wbr = class=3D"">mailman/listinfo/users</a><br class=3D""> <br class=3D""></blockquote></div><br class=3D""></div></div> </div></blockquote></div><br class=3D""></div></body></html>= --Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4--