3:51 p.m.
--Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Hi Martin,
The entropy was super low. Somewhere around 140. I installed and =
configured haveged.service to start at bootup, reverted my apache =
changes... After a reboot, my systemctl status still says that there are =
7 services queued (note that I erroneously said degraded in my previous =
email - the services are, in fact, queued), but the oVirt GUI comes up =
almost immediately and everything seems to be great.
Thank you for the tip. You solved my issue.
Cheers,
Gervais
On Sep 29, 2016, at 7:47 AM, Martin Perina <mperina(a)redhat.com>
wrote:
=20
Hi,
=20
please take a look at my inline comments:
=20
On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun =
<gervais(a)demontbrun.com
<mailto:gervais@demontbrun.com>> wrote:
Hey All,
=20
Since updating to 4.0.x of oVirt, I have had an issue with my hosted =
engine.
After a some poking around, I think I have figured out my issue =
and thought I would share to see what others think.
The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists
=
in 4.0.4.
=20
Description:
When my hosted engine starts it reports that it is in a degraded state =
with 7 or
8 services still not started when I run systemctl status. It =
takes about 6 or 7 minutes to eventually start all the services and come =
online. If I don't set my cluster to Global-Maintenance mode it =
eventually thinks that my hosted-engine needs to be rebooted and =
restarts it before it can start everything.
=20
=E2=80=8BCould you please share with us logs gathered by =
ovirt-log-collector?
=20
It's just a guess but could you please take a look if you HE VM has =
enough
entropy?
=20
cat /proc/sys/kernel/random/entropy_avail
=20
If the value is low (below or around 200), you really need to install =
and
configure some entropy generator such as haveged
=20
=20
Solution:
I realized that Apache was the culprit and found that the proxy to the =
ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super =
long timeout with many retries. I changed the settings and now =
everything works for me.
=20
-> Before change:
<LocationMatch =
^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engi=
ne.ssh.key.txt$|rhevm.ssh.key.txt$)>
ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3600
retry=3D5
=20
<IfModule deflate_module>
AddOutputFilterByType DEFLATE text/javascript text/css =
text/html
text/xml text/json application/xml application/json =
application/x-yaml
</IfModule>
</LocationMatch>
=20
-> After change:
<LocationMatch ^/ovirt-engine($|/)>
ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D5 retry=3D2
=20
<IfModule deflate_module>
AddOutputFilterByType DEFLATE text/javascript text/css =
text/html
text/xml text/json application/xml application/json =
application/x-yaml
</IfModule>
</LocationMatch>
=20
=E2=80=8BThis one is correct for 4.0=E2=80=8B=E2=80=8B, not sure why =
it was not
updated during upgrade from 3.6. @Simone?
=E2=80=8B=20
=20
If I read the timeout settings correctly, it will wait 60 minutes with =
5 retries.
5 hours is way too long for my little server to hold onto all =
those apache processes.
The change I made allows for there to be an error, and also releases
=
apache's hold on the process. Once everything is ready, apache is ready =
to serve requests and everything/everyone is happy. Before making the =
change, I just get a whitescreen in my browser and then nothing works =
until I restart Apache (or I end up in an endless loop of ovirt-ha =
services restarting my hosted-engine.
=20
=E2=80=8BWell, if you have an issue with too many apache processes =
waiting for
engine to respond, then there's some issue in engine. As I =
wrote above please share the logs with us and check entropy.
=20
Thanks
=20
Martin Perina
=E2=80=8B=20
=20
I noticed that this setting reverts to the original setting, so oVirt =
must be
writing this file. Perhaps these number can be changed in oVirt? =
If not, I will just setup and ansible play to revert the settings with =
working values and restart apache on my engine.
:-)
=20
Cheers,
Gervais
=20
=20
=20
=20
_______________________________________________
Users mailing list
Users(a)ovirt.org <mailto:Users@ovirt.org>
http://lists.ovirt.org/mailman/listinfo/users =
<http://lists.ovirt.org/mailman/listinfo/users>
=20
=20
--Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hi Martin,<div class=3D""><br
class=3D""></div><div =
class=3D"">The entropy was super low. Somewhere around 140. I installed =
and configured haveged.service to start at bootup, reverted my =
apache changes... After a reboot, my systemctl status still says that =
there are 7 services queued (note that I erroneously said degraded in my =
previous email - the services are, in fact, queued), but the oVirt GUI =
comes up almost immediately and everything seems to be great.</div><div =
class=3D""><br class=3D""></div><div
class=3D"">Thank you for the tip. =
You solved my issue.<br class=3D""><div class=3D"">
<div id=3D"signature" class=3D""><br
class=3D"">Cheers,<br =
class=3D"">Gervais<br class=3D""><br
class=3D""><br class=3D""></div>
</div>
<br class=3D""><div><blockquote type=3D"cite"
class=3D""><div =
class=3D"">On Sep 29, 2016, at 7:47 AM, Martin Perina <<a =
href=3D"mailto:mperina@redhat.com"
class=3D"">mperina(a)redhat.com</a>&gt; =
wrote:</div><br class=3D"Apple-interchange-newline"><div
class=3D""><div =
dir=3D"ltr" class=3D""><div =
style=3D"font-family:arial,helvetica,sans-serif" =
class=3D"gmail_default">Hi,<br class=3D""><br
class=3D""></div><div =
style=3D"font-family:arial,helvetica,sans-serif" =
class=3D"gmail_default">please take a look at my inline comments:<br =
class=3D""></div><div class=3D"gmail_extra"><br
class=3D""><div =
class=3D"gmail_quote">On Tue, Sep 27, 2016 at 7:23 PM, Gervais de =
Montbrun <span dir=3D"ltr" class=3D""><<a
target=3D"_blank" =
href=3D"mailto:gervais@demontbrun.com" =
class=3D"">gervais(a)demontbrun.com</a>&gt;</span> wrote:<br
=
class=3D""><blockquote style=3D"margin:0px 0px 0px
0.8ex;border-left:1px =
solid rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div
=
style=3D"word-wrap:break-word" class=3D"">Hey All,<div
class=3D""><br =
class=3D""></div><div class=3D"">Since updating to
4.0.x of oVirt, I =
have had an issue with my hosted engine. After a some poking around, I =
think I have figured out my issue and thought I would share to see what =
others think.</div><div class=3D"">The issue has existed with 4.0,
=
4.0.1, 4.0.2, 4.0.3, and still exists in 4.0.4.</div><div
class=3D""><br =
class=3D""></div><div
class=3D"">Description:</div><div class=3D"">When =
my hosted engine starts it reports that it is in a degraded state with 7 =
or 8 services still not started when I run systemctl status. It takes =
about 6 or 7 minutes to eventually start all the services and come =
online. If I don't set my cluster to Global-Maintenance mode it =
eventually thinks that my hosted-engine needs to be rebooted and =
restarts it before it can start
everything.</div></div></blockquote><div =
class=3D""><br class=3D""><div =
style=3D"font-family:arial,helvetica,sans-serif;display:inline" =
class=3D"gmail_default">=E2=80=8BCould you please share with us logs =
gathered by ovirt-log-collector?<br class=3D""><br
class=3D"">It's just =
a guess but could you please take a look if you HE VM has enough =
entropy?<br class=3D""><br class=3D""> cat =
/proc/sys/kernel/random/entropy_avail<br class=3D""><br =
class=3D""></div><div =
style=3D"font-family:arial,helvetica,sans-serif;display:inline" =
class=3D"gmail_default">If the value is low (below or around 200),
=
you really need to install and configure some entropy generator such as =
haveged<br class=3D""><br
class=3D""></div></div><blockquote =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div =
style=3D"word-wrap:break-word" class=3D""><div
class=3D""><br =
class=3D""></div><div
class=3D"">Solution:</div><div class=3D"">I =
realized that Apache was the culprit and found that the proxy to the =
ovirt-engine in /etc/httpd/conf.d/z-ovirt-<wbr =
class=3D"">engine-proxy.conf has a super long timeout with many retries. =
I changed the settings and now everything works for me.</div><div =
class=3D""><br class=3D""></div><div
class=3D"">-> Before =
change:</div><blockquote style=3D"margin:0px 0px 0px =
40px;border-width:medium;border-style:none;border-color:-moz-use-text-colo=
r;padding:0px" class=3D""><div class=3D""><div
class=3D""> =
<LocationMatch ^/(ovirt-engine($|/)|api($|/)|<wbr =
class=3D"">RHEVManagerWeb/|<wbr
class=3D"">OvirtEngineWeb/|ca.crt$|<wbr =
class=3D"">engine.ssh.key.txt$|rhevm.ssh.<wbr =
class=3D"">key.txt$)></div><div
class=3D""> =
ProxyPassMatch <a class=3D"">ajp://127.0.0.1:8702</a> =
timeout=3D3600 retry=3D5</div><div class=3D""><br
class=3D""></div><div =
class=3D""> <IfModule =
deflate_module></div><div class=3D"">
=
AddOutputFilterByType DEFLATE text/javascript text/css =
text/html text/xml text/json application/xml application/json =
application/x-yaml</div><div class=3D"">
=
</IfModule></div><div class=3D"">
=
</LocationMatch></div></div></blockquote><div
class=3D""><br =
class=3D""></div>-> After change:<blockquote
style=3D"margin:0px 0px =
0px =
40px;border-width:medium;border-style:none;border-color:-moz-use-text-colo=
r;padding:0px" class=3D""><div class=3D""><div
class=3D""> =
<LocationMatch ^/ovirt-engine($|/)></div><div
class=3D""> =
ProxyPassMatch <a
class=3D"">ajp://127.0.0.1:8702</a>=
timeout=3D5 retry=3D2</div><div class=3D""><br
class=3D""></div><div =
class=3D""> <IfModule =
deflate_module></div><div class=3D"">
=
AddOutputFilterByType DEFLATE text/javascript text/css =
text/html text/xml text/json application/xml application/json =
application/x-yaml</div><div class=3D"">
=
</IfModule></div><div class=3D"">
=
</LocationMatch></div></div></blockquote></div></blockquote><div
=
class=3D""><br class=3D""><div
class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8BT=
his one is correct for 4.0=E2=80=8B</div><div class=3D"gmail_default"
=
style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8B,=
not sure why it was not updated during upgrade from 3.6. @Simone?<br =
class=3D"">=E2=80=8B</div> </div><blockquote
style=3D"margin:0px =
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" =
class=3D"gmail_quote"><div style=3D"word-wrap:break-word"
class=3D""><div =
class=3D""><br class=3D""></div>If I read the timeout
settings =
correctly, it will wait 60 minutes with 5 retries. 5 hours is way too =
long for my little server to hold onto all those apache processes. =
</div></blockquote><blockquote style=3D"margin:0px 0px 0px =
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" =
class=3D"gmail_quote"><div style=3D"word-wrap:break-word"
class=3D"">The =
change I made allows for there to be an error, and also releases =
apache's hold on the process. Once everything is ready, apache is ready =
to serve requests and everything/everyone is happy. Before making the =
change, I just get a whitescreen in my browser and then nothing works =
until I restart Apache (or I end up in an endless loop of ovirt-ha =
services restarting my hosted-engine.<br =
class=3D""></div></blockquote><div
class=3D""><br class=3D""><div =
class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;display:inline">=E2=80=8BW=
ell, if you have an issue with too many apache processes waiting for =
engine to respond, then there's some issue in engine. As I wrote above =
please share the logs with us and check entropy.<br class=3D""><br =
class=3D""></div><div class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;display:inline">Thanks<br
=
class=3D""><br class=3D""></div><div
class=3D"gmail_default" =
style=3D"font-family:arial,helvetica,sans-serif;display:inline">Martin =
Perina<br
class=3D"">=E2=80=8B</div> </div><blockquote =
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid =
rgb(204,204,204);padding-left:1ex" class=3D"gmail_quote"><div =
style=3D"word-wrap:break-word" class=3D""><div
class=3D""><div =
class=3D""><div class=3D""><br
class=3D""></div><div class=3D"">I =
noticed that this setting reverts to the original setting, so oVirt must =
be writing this file. Perhaps these number can be changed in oVirt? If =
not, I will just setup and ansible play to revert the settings with =
working values and restart apache on my engine.</div><div =
class=3D"">:-)</div><div class=3D"">
<div class=3D""><br class=3D"">Cheers,<br
class=3D"">Gervais<br =
class=3D""><br class=3D""><br
class=3D""></div>
</div>
<br class=3D""></div></div></div><br =
class=3D"">______________________________<wbr =
class=3D"">_________________<br class=3D"">
Users mailing list<br class=3D"">
<a href=3D"mailto:Users@ovirt.org"
class=3D"">Users(a)ovirt.org</a><br =
class=3D"">
<a target=3D"_blank" rel=3D"noreferrer" =
href=3D"http://lists.ovirt.org/mailman/listinfo/users" =
class=3D"">http://lists.ovirt.org/<wbr =
class=3D"">mailman/listinfo/users</a><br class=3D"">
<br class=3D""></blockquote></div><br
class=3D""></div></div>
</div></blockquote></div><br
class=3D""></div></body></html>=
--Apple-Mail=_B51077DC-F506-4239-9A68-4C1B6F0E0FC4--