From gervais at demontbrun.com Tue Sep 27 13:23:44 2016 Content-Type: multipart/mixed; boundary="===============8369744779128110053==" MIME-Version: 1.0 From: Gervais de Montbrun To: users at ovirt.org Subject: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Tue, 27 Sep 2016 14:23:39 -0300 Message-ID: --===============8369744779128110053== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_36EF5248-AA3D-484E-8A71-55F0C8DF8023 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dus-ascii Hey All, Since updating to 4.0.x of oVirt, I have had an issue with my hosted =3D engine. After a some poking around, I think I have figured out my issue =3D and thought I would share to see what others think. The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in = =3D 4.0.4. Description: When my hosted engine starts it reports that it is in a degraded state =3D with 7 or 8 services still not started when I run systemctl status. It =3D takes about 6 or 7 minutes to eventually start all the services and come = =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start everything. Solution: I realized that Apache was the culprit and found that the proxy to the =3D ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super =3D long timeout with many retries. I changed the settings and now =3D everything works for me. -> Before change: ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D3600 retry=3D3D5 AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml -> After change: ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D5 retry=3D3D2 AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml If I read the timeout settings correctly, it will wait 60 minutes with 5 = =3D retries. 5 hours is way too long for my little server to hold onto all =3D those apache processes. The change I made allows for there to be an =3D error, and also releases apache's hold on the process. Once everything =3D is ready, apache is ready to serve requests and everything/everyone is =3D happy. Before making the change, I just get a whitescreen in my browser =3D and then nothing works until I restart Apache (or I end up in an endless = =3D loop of ovirt-ha services restarting my hosted-engine. I noticed that this setting reverts to the original setting, so oVirt =3D must be writing this file. Perhaps these number can be changed in oVirt? = =3D If not, I will just setup and ansible play to revert the settings with =3D working values and restart apache on my engine. :-) Cheers, Gervais --Apple-Mail=3D_36EF5248-AA3D-484E-8A71-55F0C8DF8023 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dus-ascii Hey All,

Since updating to 4.0.x of oVirt, I have had an issue with my = =3D hosted engine. After a some poking around, I think I have figured out my = =3D issue and thought I would share to see what others think.
The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and =3D still exists in 4.0.4.

Description:
When my hosted engine =3D starts it reports that it is in a degraded state with 7 or 8 services =3D still not started when I run systemctl status. It takes about 6 or 7 =3D minutes to eventually start all the services and come online. If I don't = =3D set my cluster to Global-Maintenance mode it eventually thinks that my =3D hosted-engine needs to be rebooted and restarts it before it can start =3D everything.

Solution:
I realized that Apache was th= e =3D culprit and found that the proxy to the ovirt-engine =3D in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super long =3D timeout with many retries. I changed the settings and now everything =3D works for me.

-> Before change:
    <LocationMatch =3D ^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engi= =3D ne.ssh.key.txt$|rhevm.ssh.key.txt$)>
  =3D       ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D3600 retry=3D3D5

      = =3D   <IfModule deflate_module>
  =3D           AddOutputFilterByType DEFLATE =3D text/javascript text/css text/html text/xml text/json application/xml =3D application/json application/x-yaml
    = =3D     </IfModule>
    =3D </LocationMatch>

-> After change:
    <LocationMatch =3D ^/ovirt-engine($|/)>
        = =3D ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D5 retry=3D3D2

      = =3D   <IfModule deflate_module>
  =3D           AddOutputFilterByType DEFLATE =3D text/javascript text/css text/html text/xml text/json application/xml =3D application/json application/x-yaml
    = =3D     </IfModule>
    =3D </LocationMatch>

If I read the timeout settings correctly, it will wait = =3D 60 minutes with 5 retries. 5 hours is way too long for my little server =3D to hold onto all those apache processes. The change I made allows for =3D there to be an error, and also releases apache's hold on the process. =3D Once everything is ready, apache is ready to serve requests and =3D everything/everyone is happy. Before making the change, I just get a =3D whitescreen in my browser and then nothing works until I restart Apache =3D (or I end up in an endless loop of ovirt-ha services restarting my =3D hosted-engine.

I =3D noticed that this setting reverts to the original setting, so oVirt must = =3D be writing this file. Perhaps these number can be changed in oVirt? If =3D not, I will just setup and ansible play to revert the settings with =3D working values and restart apache on my engine.
:-)

Cheers,
Gervais



=3D --Apple-Mail=3D_36EF5248-AA3D-484E-8A71-55F0C8DF8023-- --===============8369744779128110053== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fMzZFRjUyNDgtQUEzRC00ODRFLThBNzEtNTVGMEM4REY4MDIzCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11cy1hc2NpaQoKSGV5IEFsbCwKClNpbmNlIHVwZGF0aW5nIHRvIDQu MC54IG9mIG9WaXJ0LCBJIGhhdmUgaGFkIGFuIGlzc3VlIHdpdGggbXkgaG9zdGVkID0KZW5naW5l LiBBZnRlciBhIHNvbWUgcG9raW5nIGFyb3VuZCwgSSB0aGluayBJIGhhdmUgZmlndXJlZCBvdXQg bXkgaXNzdWUgPQphbmQgdGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNlZSB3aGF0IG90aGVycyB0 aGluay4KVGhlIGlzc3VlIGhhcyBleGlzdGVkIHdpdGggNC4wLCA0LjAuMSwgNC4wLjIsIDQuMC4z LCBhbmQgc3RpbGwgZXhpc3RzIGluID0KNC4wLjQuCgpEZXNjcmlwdGlvbjoKV2hlbiBteSBob3N0 ZWQgZW5naW5lIHN0YXJ0cyBpdCByZXBvcnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0 ZSA9CndpdGggNyBvciA4IHNlcnZpY2VzIHN0aWxsIG5vdCBzdGFydGVkIHdoZW4gSSBydW4gc3lz dGVtY3RsIHN0YXR1cy4gSXQgPQp0YWtlcyBhYm91dCA2IG9yIDcgbWludXRlcyB0byBldmVudHVh bGx5IHN0YXJ0IGFsbCB0aGUgc2VydmljZXMgYW5kIGNvbWUgPQpvbmxpbmUuIElmIEkgZG9uJ3Qg c2V0IG15IGNsdXN0ZXIgdG8gR2xvYmFsLU1haW50ZW5hbmNlIG1vZGUgaXQgPQpldmVudHVhbGx5 IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1lbmdpbmUgbmVlZHMgdG8gYmUgcmVib290ZWQgYW5kID0K cmVzdGFydHMgaXQgYmVmb3JlIGl0IGNhbiBzdGFydCBldmVyeXRoaW5nLgoKU29sdXRpb246Ckkg cmVhbGl6ZWQgdGhhdCBBcGFjaGUgd2FzIHRoZSBjdWxwcml0IGFuZCBmb3VuZCB0aGF0IHRoZSBw cm94eSB0byB0aGUgPQpvdmlydC1lbmdpbmUgaW4gL2V0Yy9odHRwZC9jb25mLmQvei1vdmlydC1l bmdpbmUtcHJveHkuY29uZiBoYXMgYSBzdXBlciA9CmxvbmcgdGltZW91dCB3aXRoIG1hbnkgcmV0 cmllcy4gSSBjaGFuZ2VkIHRoZSBzZXR0aW5ncyBhbmQgbm93ID0KZXZlcnl0aGluZyB3b3JrcyBm b3IgbWUuCgotPiBCZWZvcmUgY2hhbmdlOgogICAgPExvY2F0aW9uTWF0Y2ggPQpeLyhvdmlydC1l bmdpbmUoJHwvKXxhcGkoJHwvKXxSSEVWTWFuYWdlcldlYi98T3ZpcnRFbmdpbmVXZWIvfGNhLmNy dCR8ZW5naT0KbmUuc3NoLmtleS50eHQkfHJoZXZtLnNzaC5rZXkudHh0JCk+CiAgICAgICAgUHJv eHlQYXNzTWF0Y2ggYWpwOi8vMTI3LjAuMC4xOjg3MDIgdGltZW91dD0zRDM2MDAgcmV0cnk9M0Q1 CgogICAgICAgIDxJZk1vZHVsZSBkZWZsYXRlX21vZHVsZT4KICAgICAgICAgICAgQWRkT3V0cHV0 RmlsdGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzID0KdGV4dC9odG1s IHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiA9CmFw cGxpY2F0aW9uL3gteWFtbAogICAgICAgIDwvSWZNb2R1bGU+CiAgICA8L0xvY2F0aW9uTWF0Y2g+ CgotPiBBZnRlciBjaGFuZ2U6CiAgICA8TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2luZSgkfC8p PgogICAgICAgIFByb3h5UGFzc01hdGNoIGFqcDovLzEyNy4wLjAuMTo4NzAyIHRpbWVvdXQ9M0Q1 IHJldHJ5PTNEMgoKICAgICAgICA8SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGU+CiAgICAgICAgICAg IEFkZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyA9 CnRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFwcGxpY2F0aW9u L2pzb24gPQphcHBsaWNhdGlvbi94LXlhbWwKICAgICAgICA8L0lmTW9kdWxlPgogICAgPC9Mb2Nh dGlvbk1hdGNoPgoKSWYgSSByZWFkIHRoZSB0aW1lb3V0IHNldHRpbmdzIGNvcnJlY3RseSwgaXQg d2lsbCB3YWl0IDYwIG1pbnV0ZXMgd2l0aCA1ID0KcmV0cmllcy4gNSBob3VycyBpcyB3YXkgdG9v IGxvbmcgZm9yIG15IGxpdHRsZSBzZXJ2ZXIgdG8gaG9sZCBvbnRvIGFsbCA9CnRob3NlIGFwYWNo ZSBwcm9jZXNzZXMuIFRoZSBjaGFuZ2UgSSBtYWRlIGFsbG93cyBmb3IgdGhlcmUgdG8gYmUgYW4g PQplcnJvciwgYW5kIGFsc28gcmVsZWFzZXMgYXBhY2hlJ3MgaG9sZCBvbiB0aGUgcHJvY2Vzcy4g T25jZSBldmVyeXRoaW5nID0KaXMgcmVhZHksIGFwYWNoZSBpcyByZWFkeSB0byBzZXJ2ZSByZXF1 ZXN0cyBhbmQgZXZlcnl0aGluZy9ldmVyeW9uZSBpcyA9CmhhcHB5LiBCZWZvcmUgbWFraW5nIHRo ZSBjaGFuZ2UsIEkganVzdCBnZXQgYSB3aGl0ZXNjcmVlbiBpbiBteSBicm93c2VyID0KYW5kIHRo ZW4gbm90aGluZyB3b3JrcyB1bnRpbCBJIHJlc3RhcnQgQXBhY2hlIChvciBJIGVuZCB1cCBpbiBh biBlbmRsZXNzID0KbG9vcCBvZiBvdmlydC1oYSBzZXJ2aWNlcyByZXN0YXJ0aW5nIG15IGhvc3Rl ZC1lbmdpbmUuCgpJIG5vdGljZWQgdGhhdCB0aGlzIHNldHRpbmcgcmV2ZXJ0cyB0byB0aGUgb3Jp Z2luYWwgc2V0dGluZywgc28gb1ZpcnQgPQptdXN0IGJlIHdyaXRpbmcgdGhpcyBmaWxlLiBQZXJo YXBzIHRoZXNlIG51bWJlciBjYW4gYmUgY2hhbmdlZCBpbiBvVmlydD8gPQpJZiBub3QsIEkgd2ls bCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkgdG8gcmV2ZXJ0IHRoZSBzZXR0aW5ncyB3aXRo ID0Kd29ya2luZyB2YWx1ZXMgYW5kIHJlc3RhcnQgYXBhY2hlIG9uIG15IGVuZ2luZS4KOi0pCgpD aGVlcnMsCkdlcnZhaXMKCgoKCi0tQXBwbGUtTWFpbD1fMzZFRjUyNDgtQUEzRC00ODRFLThBNzEt NTVGMEM4REY4MDIzCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUK Q29udGVudC1UeXBlOiB0ZXh0L2h0bWw7CgljaGFyc2V0PXVzLWFzY2lpCgo8aHRtbD48aGVhZD48 bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWwgPQpj aGFyc2V0PTNEdXMtYXNjaWkiPjwvaGVhZD48Ym9keSBzdHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFr LXdvcmQ7ID0KLXdlYmtpdC1uYnNwLW1vZGU6IHNwYWNlOyAtd2Via2l0LWxpbmUtYnJlYWs6IGFm dGVyLXdoaXRlLXNwYWNlOyIgPQpjbGFzcz0zRCIiPkhleSBBbGwsPGRpdiBjbGFzcz0zRCIiPjxi ciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPlNpbmNlIHVwZGF0aW5nIHRvIDQu MC54IG9mIG9WaXJ0LCBJIGhhdmUgaGFkIGFuIGlzc3VlIHdpdGggbXkgPQpob3N0ZWQgZW5naW5l LiBBZnRlciBhIHNvbWUgcG9raW5nIGFyb3VuZCwgSSB0aGluayBJIGhhdmUgZmlndXJlZCBvdXQg bXkgPQppc3N1ZSBhbmQgdGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNlZSB3aGF0IG90aGVycyB0 aGluay48L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj5UaGUgaXNzdWUgaGFzIGV4aXN0ZWQgd2l0aCA0 LjAsIDQuMC4xLCA0LjAuMiwgNC4wLjMsIGFuZCA9CnN0aWxsIGV4aXN0cyBpbiA0LjAuNC48L2Rp dj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+ RGVzY3JpcHRpb246PC9kaXY+PGRpdiBjbGFzcz0zRCIiPldoZW4gbXkgaG9zdGVkIGVuZ2luZSA9 CnN0YXJ0cyBpdCByZXBvcnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0ZSB3aXRoIDcg b3IgOCBzZXJ2aWNlcyA9CnN0aWxsIG5vdCBzdGFydGVkIHdoZW4gSSBydW4gc3lzdGVtY3RsIHN0 YXR1cy4gSXQgdGFrZXMgYWJvdXQgNiBvciA3ID0KbWludXRlcyB0byBldmVudHVhbGx5IHN0YXJ0 IGFsbCB0aGUgc2VydmljZXMgYW5kIGNvbWUgb25saW5lLiBJZiBJIGRvbid0ID0Kc2V0IG15IGNs dXN0ZXIgdG8gR2xvYmFsLU1haW50ZW5hbmNlIG1vZGUgaXQgZXZlbnR1YWxseSB0aGlua3MgdGhh dCBteSA9Cmhvc3RlZC1lbmdpbmUgbmVlZHMgdG8gYmUgcmVib290ZWQgYW5kIHJlc3RhcnRzIGl0 IGJlZm9yZSBpdCBjYW4gc3RhcnQgPQpldmVyeXRoaW5nLjwvZGl2PjxkaXYgY2xhc3M9M0QiIj48 YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj5Tb2x1dGlvbjo8L2Rpdj48ZGl2 IGNsYXNzPTNEIiI+SSByZWFsaXplZCB0aGF0IEFwYWNoZSB3YXMgdGhlID0KY3VscHJpdCBhbmQg Zm91bmQgdGhhdCB0aGUgcHJveHkgdG8gdGhlIG92aXJ0LWVuZ2luZSA9CmluJm5ic3A7L2V0Yy9o dHRwZC9jb25mLmQvei1vdmlydC1lbmdpbmUtcHJveHkuY29uZiBoYXMgYSBzdXBlciBsb25nID0K dGltZW91dCB3aXRoIG1hbnkgcmV0cmllcy4gSSBjaGFuZ2VkIHRoZSBzZXR0aW5ncyBhbmQgbm93 IGV2ZXJ5dGhpbmcgPQp3b3JrcyBmb3IgbWUuPC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxiciBjbGFz cz0zRCIiPjwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPi0mZ3Q7IEJlZm9yZSBjaGFuZ2U6PC9kaXY+ PGJsb2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOiAwIDAgMCA9CjQwcHg7IGJvcmRlcjogbm9uZTsg cGFkZGluZzogMHB4OyIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNE IiI+Jm5ic3A7ICZuYnNwOyAmbHQ7TG9jYXRpb25NYXRjaCA9Cl4vKG92aXJ0LWVuZ2luZSgkfC8p fGFwaSgkfC8pfFJIRVZNYW5hZ2VyV2ViL3xPdmlydEVuZ2luZVdlYi98Y2EuY3J0JHxlbmdpPQpu ZS5zc2gua2V5LnR4dCR8cmhldm0uc3NoLmtleS50eHQkKSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNE IiI+Jm5ic3A7ID0KJm5ic3A7ICZuYnNwOyAmbmJzcDsgUHJveHlQYXNzTWF0Y2ggPGEgaHJlZj0z RCJhanA6Ly8xMjcuMC4wLjE6ODcwMiIgPQpjbGFzcz0zRCIiPmFqcDovLzEyNy4wLjAuMTo4NzAy PC9hPiB0aW1lb3V0PTNEMzYwMCByZXRyeT0zRDU8L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj48YnIg Y2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgPQom bmJzcDsgJmx0O0lmTW9kdWxlIGRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0Qi Ij4mbmJzcDsgPQombmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7IEFkZE91dHB1dEZp bHRlckJ5VHlwZSBERUZMQVRFID0KdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzIHRleHQvaHRtbCB0 ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sID0KYXBwbGljYXRpb24vanNvbiBhcHBs aWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyA9CiZuYnNw OyAmbmJzcDsgJmx0Oy9JZk1vZHVsZSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZu YnNwOyA9CiZsdDsvTG9jYXRpb25NYXRjaCZndDs8L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PGRp diBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+LSZndDsgQWZ0ZXIgY2hhbmdlOjxi bG9ja3F1b3RlIHN0eWxlPTNEIm1hcmdpbjogMCAwIDAgPQo0MHB4OyBib3JkZXI6IG5vbmU7IHBh ZGRpbmc6IDBweDsiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIi PiZuYnNwOyAmbmJzcDsgJmx0O0xvY2F0aW9uTWF0Y2ggPQpeL292aXJ0LWVuZ2luZSgkfC8pJmd0 OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgPQpQcm94 eVBhc3NNYXRjaCA8YSBocmVmPTNEImFqcDovLzEyNy4wLjAuMTo4NzAyIiA9CmNsYXNzPTNEIiI+ YWpwOi8vMTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9M0Q1IHJldHJ5PTNEMjwvZGl2PjxkaXYg PQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsg Jm5ic3A7ICZuYnNwOyA9CiZuYnNwOyAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9k aXY+PGRpdiBjbGFzcz0zRCIiPiZuYnNwOyA9CiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAm bmJzcDsgQWRkT3V0cHV0RmlsdGVyQnlUeXBlIERFRkxBVEUgPQp0ZXh0L2phdmFzY3JpcHQgdGV4 dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgPQphcHBs aWNhdGlvbi9qc29uIGFwcGxpY2F0aW9uL3gteWFtbDwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJz cDsgJm5ic3A7ID0KJm5ic3A7ICZuYnNwOyAmbHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xh c3M9M0QiIj4mbmJzcDsgJm5ic3A7ID0KJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2PjwvZGl2 PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPTNEIiI+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj5JZiBJ IHJlYWQgdGhlIHRpbWVvdXQgc2V0dGluZ3MgY29ycmVjdGx5LCBpdCB3aWxsIHdhaXQgPQo2MCBt aW51dGVzIHdpdGggNSByZXRyaWVzLiA1IGhvdXJzIGlzIHdheSB0b28gbG9uZyBmb3IgbXkgbGl0 dGxlIHNlcnZlciA9CnRvIGhvbGQgb250byBhbGwgdGhvc2UgYXBhY2hlIHByb2Nlc3Nlcy4gVGhl IGNoYW5nZSBJIG1hZGUgYWxsb3dzIGZvciA9CnRoZXJlIHRvIGJlIGFuIGVycm9yLCBhbmQgYWxz byByZWxlYXNlcyBhcGFjaGUncyBob2xkIG9uIHRoZSBwcm9jZXNzLiA9Ck9uY2UgZXZlcnl0aGlu ZyBpcyByZWFkeSwgYXBhY2hlIGlzIHJlYWR5IHRvIHNlcnZlIHJlcXVlc3RzIGFuZCA9CmV2ZXJ5 dGhpbmcvZXZlcnlvbmUgaXMgaGFwcHkuIEJlZm9yZSBtYWtpbmcgdGhlIGNoYW5nZSwgSSBqdXN0 IGdldCBhID0Kd2hpdGVzY3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtz IHVudGlsIEkgcmVzdGFydCBBcGFjaGUgPQoob3IgSSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29w IG9mIG92aXJ0LWhhIHNlcnZpY2VzIHJlc3RhcnRpbmcgbXkgPQpob3N0ZWQtZW5naW5lLjxiciBj bGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNE IiI+PGJyIGNsYXNzPTNEIndlYmtpdC1ibG9jay1wbGFjZWhvbGRlciI+PC9kaXY+PGRpdiBjbGFz cz0zRCIiPkkgPQpub3RpY2VkIHRoYXQgdGhpcyBzZXR0aW5nIHJldmVydHMgdG8gdGhlIG9yaWdp bmFsIHNldHRpbmcsIHNvIG9WaXJ0IG11c3QgPQpiZSB3cml0aW5nIHRoaXMgZmlsZS4gUGVyaGFw cyB0aGVzZSBudW1iZXIgY2FuIGJlIGNoYW5nZWQgaW4gb1ZpcnQ/IElmID0Kbm90LCBJIHdpbGwg anVzdCBzZXR1cCBhbmQgYW5zaWJsZSBwbGF5IHRvIHJldmVydCB0aGUgc2V0dGluZ3Mgd2l0aCA9 CndvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBvbiBteSBlbmdpbmUuPC9kaXY+PGRp diA9CmNsYXNzPTNEIiI+Oi0pPC9kaXY+PGRpdiBjbGFzcz0zRCIiPgo8ZGl2IGlkPTNEInNpZ25h dHVyZSIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj5DaGVlcnMsPGJyID0KY2xhc3M9M0QiIj5H ZXJ2YWlzPGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+ Cgo8L2Rpdj4KCgoKPGJyIGNsYXNzPTNEIiI+PC9kaXY+PC9kaXY+PC9ib2R5PjwvaHRtbD49Cgot LUFwcGxlLU1haWw9XzM2RUY1MjQ4LUFBM0QtNDg0RS04QTcxLTU1RjBDOERGODAyMy0tCg== --===============8369744779128110053==-- From mperina at redhat.com Thu Sep 29 06:47:17 2016 Content-Type: multipart/mixed; boundary="===============7586538229997345221==" MIME-Version: 1.0 From: Martin Perina To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 12:47:15 +0200 Message-ID: In-Reply-To: D08A50DA-A363-4D5A-A111-1D104E579ECE@demontbrun.com --===============7586538229997345221== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, please take a look at my inline comments: On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun wrote: > Hey All, > > Since updating to 4.0.x of oVirt, I have had an issue with my hosted > engine. After a some poking around, I think I have figured out my issue a= nd > thought I would share to see what others think. > The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in > 4.0.4. > > Description: > When my hosted engine starts it reports that it is in a degraded state > with 7 or 8 services still not started when I run systemctl status. It > takes about 6 or 7 minutes to eventually start all the services and come > online. If I don't set my cluster to Global-Maintenance mode it eventually > thinks that my hosted-engine needs to be rebooted and restarts it before = it > can start everything. > =E2=80=8BCould you please share with us logs gathered by ovirt-log-collecto= r? It's just a guess but could you please take a look if you HE VM has enough entropy? cat /proc/sys/kernel/random/entropy_avail If the value is low (below or around 200), you really need to install and configure some entropy generator such as haveged > Solution: > I realized that Apache was the culprit and found that the proxy to the > ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super > long timeout with many retries. I changed the settings and now everything > works for me. > > -> Before change: > > OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$|rhevm.ssh.key.txt$)> > ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 > > > AddOutputFilterByType DEFLATE text/javascript text/css > text/html text/xml text/json application/xml application/json > application/x-yaml > > > > > -> After change: > > > ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 > > > AddOutputFilterByType DEFLATE text/javascript text/css > text/html text/xml text/json application/xml application/json > application/x-yaml > > > > =E2=80=8BThis one is correct for 4.0=E2=80=8B =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Simone? =E2=80=8B > > If I read the timeout settings correctly, it will wait 60 minutes with 5 > retries. 5 hours is way too long for my little server to hold onto all > those apache processes. > The change I made allows for there to be an error, and also releases > apache's hold on the process. Once everything is ready, apache is ready to > serve requests and everything/everyone is happy. Before making the change, > I just get a whitescreen in my browser and then nothing works until I > restart Apache (or I end up in an endless loop of ovirt-ha services > restarting my hosted-engine. > =E2=80=8BWell, if you have an issue with too many apache processes waiting = for engine to respond, then there's some issue in engine. As I wrote above please share the logs with us and check entropy. Thanks Martin Perina =E2=80=8B > > I noticed that this setting reverts to the original setting, so oVirt must > be writing this file. Perhaps these number can be changed in oVirt? If no= t, > I will just setup and ansible play to revert the settings with working > values and restart apache on my engine. > :-) > > Cheers, > Gervais > > > > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============7586538229997345221== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNh bnMtc2VyaWYiIGNsYXNzPSJnbWFpbF9kZWZhdWx0Ij5IaSw8YnI+PGJyPjwvZGl2PjxkaXYgc3R5 bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmIiBjbGFzcz0iZ21haWxf ZGVmYXVsdCI+cGxlYXNlIHRha2UgYSBsb29rIGF0IG15IGlubGluZSBjb21tZW50czo8YnI+PC9k aXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+ T24gVHVlLCBTZXAgMjcsIDIwMTYgYXQgNzoyMyBQTSwgR2VydmFpcyBkZSBNb250YnJ1biA8c3Bh biBkaXI9Imx0ciI+Jmx0OzxhIHRhcmdldD0iX2JsYW5rIiBocmVmPSJtYWlsdG86Z2VydmFpc0Bk ZW1vbnRicnVuLmNvbSI+Z2VydmFpc0BkZW1vbnRicnVuLmNvbTwvYT4mZ3Q7PC9zcGFuPiB3cm90 ZTo8YnI+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXIt bGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0i Z21haWxfcXVvdGUiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj5IZXkgQWxsLDxk aXY+PGJyPjwvZGl2PjxkaXY+U2luY2UgdXBkYXRpbmcgdG8gNC4wLnggb2Ygb1ZpcnQsIEkgaGF2 ZSBoYWQgYW4gaXNzdWUgd2l0aCBteSBob3N0ZWQgZW5naW5lLiBBZnRlciBhIHNvbWUgcG9raW5n IGFyb3VuZCwgSSB0aGluayBJIGhhdmUgZmlndXJlZCBvdXQgbXkgaXNzdWUgYW5kIHRob3VnaHQg SSB3b3VsZCBzaGFyZSB0byBzZWUgd2hhdCBvdGhlcnMgdGhpbmsuPC9kaXY+PGRpdj5UaGUgaXNz dWUgaGFzIGV4aXN0ZWQgd2l0aCA0LjAsIDQuMC4xLCA0LjAuMiwgNC4wLjMsIGFuZCBzdGlsbCBl eGlzdHMgaW4gNC4wLjQuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5EZXNjcmlwdGlvbjo8L2Rp dj48ZGl2PldoZW4gbXkgaG9zdGVkIGVuZ2luZSBzdGFydHMgaXQgcmVwb3J0cyB0aGF0IGl0IGlz IGluIGEgZGVncmFkZWQgc3RhdGUgd2l0aCA3IG9yIDggc2VydmljZXMgc3RpbGwgbm90IHN0YXJ0 ZWQgd2hlbiBJIHJ1biBzeXN0ZW1jdGwgc3RhdHVzLiBJdCB0YWtlcyBhYm91dCA2IG9yIDcgbWlu dXRlcyB0byBldmVudHVhbGx5IHN0YXJ0IGFsbCB0aGUgc2VydmljZXMgYW5kIGNvbWUgb25saW5l LiBJZiBJIGRvbiYjMzk7dCBzZXQgbXkgY2x1c3RlciB0byBHbG9iYWwtTWFpbnRlbmFuY2UgbW9k ZSBpdCBldmVudHVhbGx5IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1lbmdpbmUgbmVlZHMgdG8gYmUg cmVib290ZWQgYW5kIHJlc3RhcnRzIGl0IGJlZm9yZSBpdCBjYW4gc3RhcnQgZXZlcnl0aGluZy48 L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdj48YnI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6 YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiIGNsYXNzPSJnbWFpbF9k ZWZhdWx0Ij7igItDb3VsZCB5b3UgcGxlYXNlIHNoYXJlIHdpdGggdXMgbG9ncyBnYXRoZXJlZCBi eSBvdmlydC1sb2ctY29sbGVjdG9yPzxicj48YnI+SXQmIzM5O3MganVzdCBhIGd1ZXNzIGJ1dCBj b3VsZCB5b3UgcGxlYXNlIHRha2UgYSBsb29rIGlmIHlvdSBIRSBWTSBoYXMgZW5vdWdoIGVudHJv cHk/PGJyPjxicj7CoCBjYXQgL3Byb2Mvc3lzL2tlcm5lbC9yYW5kb20vZW50cm9weV9hdmFpbDxi cj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMt c2VyaWY7ZGlzcGxheTppbmxpbmUiIGNsYXNzPSJnbWFpbF9kZWZhdWx0Ij5JZiB0aGUgdmFsdWUg aXMgbG93IChiZWxvdyBvciBhcm91bmQgMjAwKSzCoCB5b3UgcmVhbGx5IG5lZWQgdG8gaW5zdGFs bCBhbmQgY29uZmlndXJlIHNvbWUgZW50cm9weSBnZW5lcmF0b3Igc3VjaCBhcyBoYXZlZ2VkPGJy Pjxicj48L2Rpdj48L2Rpdj48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAu OGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDox ZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQi PjxkaXY+PGJyPjwvZGl2PjxkaXY+U29sdXRpb246PC9kaXY+PGRpdj5JIHJlYWxpemVkIHRoYXQg QXBhY2hlIHdhcyB0aGUgY3VscHJpdCBhbmQgZm91bmQgdGhhdCB0aGUgcHJveHkgdG8gdGhlIG92 aXJ0LWVuZ2luZSBpbsKgL2V0Yy9odHRwZC9jb25mLmQvei1vdmlydC08d2JyPmVuZ2luZS1wcm94 eS5jb25mIGhhcyBhIHN1cGVyIGxvbmcgdGltZW91dCB3aXRoIG1hbnkgcmV0cmllcy4gSSBjaGFu Z2VkIHRoZSBzZXR0aW5ncyBhbmQgbm93IGV2ZXJ5dGhpbmcgd29ya3MgZm9yIG1lLjwvZGl2Pjxk aXY+PGJyPjwvZGl2PjxkaXY+LSZndDsgQmVmb3JlIGNoYW5nZTo8L2Rpdj48YmxvY2txdW90ZSBz dHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDQwcHg7Ym9yZGVyLXdpZHRoOm1lZGl1bTtib3JkZXIt c3R5bGU6bm9uZTtib3JkZXItY29sb3I6LW1vei11c2UtdGV4dC1jb2xvcjtwYWRkaW5nOjBweCI+ PGRpdj48ZGl2PsKgIMKgICZsdDtMb2NhdGlvbk1hdGNoIF4vKG92aXJ0LWVuZ2luZSgkfC8pfGFw aSgkfC8pfDx3YnI+UkhFVk1hbmFnZXJXZWIvfDx3YnI+T3ZpcnRFbmdpbmVXZWIvfGNhLmNydCR8 PHdicj5lbmdpbmUuc3NoLmtleS50eHQkfHJoZXZtLnNzaC48d2JyPmtleS50eHQkKSZndDs8L2Rp dj48ZGl2PsKgIMKgIMKgIMKgIFByb3h5UGFzc01hdGNoIDxhPmFqcDovLzEyNy4wLjAuMTo4NzAy PC9hPiB0aW1lb3V0PTM2MDAgcmV0cnk9NTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+wqAgwqAg wqAgwqAgJmx0O0lmTW9kdWxlIGRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAg wqAgwqAgwqAgQWRkT3V0cHV0RmlsdGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRl eHQvY3NzIHRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFwcGxp Y2F0aW9uL2pzb24gYXBwbGljYXRpb24veC15YW1sPC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7 L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAgJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2 PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rpdj4tJmd0OyBBZnRlciBjaGFuZ2U6PGJs b2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCA0MHB4O2JvcmRlci13aWR0aDptZWRp dW07Ym9yZGVyLXN0eWxlOm5vbmU7Ym9yZGVyLWNvbG9yOi1tb3otdXNlLXRleHQtY29sb3I7cGFk ZGluZzowcHgiPjxkaXY+PGRpdj7CoCDCoCAmbHQ7TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2lu ZSgkfC8pJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgUHJveHlQYXNzTWF0Y2ggPGE+YWpwOi8v MTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9NSByZXRyeT0yPC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRp dj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0L2ph dmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlv bi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKgIMKg IMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9uTWF0 Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxi cj48ZGl2IGNsYXNzPSJnbWFpbF9kZWZhdWx0IiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVs dmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPuKAi1RoaXMgb25lIGlzIGNvcnJlY3Qg Zm9yIDQuMOKAizwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2RlZmF1bHQiIHN0eWxlPSJmb250LWZh bWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+4oCLLCBub3Qg c3VyZSB3aHkgaXQgd2FzIG5vdCB1cGRhdGVkIGR1cmluZyB1cGdyYWRlIGZyb20gMy42LiBAU2lt b25lPzxicj7igIs8L2Rpdj7CoDwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBw eCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGlu Zy1sZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJl YWstd29yZCI+PGRpdj48YnI+PC9kaXY+SWYgSSByZWFkIHRoZSB0aW1lb3V0IHNldHRpbmdzIGNv cnJlY3RseSwgaXQgd2lsbCB3YWl0IDYwIG1pbnV0ZXMgd2l0aCA1IHJldHJpZXMuIDUgaG91cnMg aXMgd2F5IHRvbyBsb25nIGZvciBteSBsaXR0bGUgc2VydmVyIHRvIGhvbGQgb250byBhbGwgdGhv c2UgYXBhY2hlIHByb2Nlc3Nlcy4gPC9kaXY+PC9ibG9ja3F1b3RlPjxibG9ja3F1b3RlIHN0eWxl PSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQs MjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxl PSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+VGhlIGNoYW5nZSBJIG1hZGUgYWxsb3dzIGZvciB0aGVy ZSB0byBiZSBhbiBlcnJvciwgYW5kIGFsc28gcmVsZWFzZXMgYXBhY2hlJiMzOTtzIGhvbGQgb24g dGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyByZWFkeSwgYXBhY2hlIGlzIHJlYWR5IHRv IHNlcnZlIHJlcXVlc3RzIGFuZCBldmVyeXRoaW5nL2V2ZXJ5b25lIGlzIGhhcHB5LiBCZWZvcmUg bWFraW5nIHRoZSBjaGFuZ2UsIEkganVzdCBnZXQgYSB3aGl0ZXNjcmVlbiBpbiBteSBicm93c2Vy IGFuZCB0aGVuIG5vdGhpbmcgd29ya3MgdW50aWwgSSByZXN0YXJ0IEFwYWNoZSAob3IgSSBlbmQg dXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhhIHNlcnZpY2VzIHJlc3RhcnRpbmcgbXkg aG9zdGVkLWVuZ2luZS48YnI+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjxkaXYgY2xhc3M9 ImdtYWlsX2RlZmF1bHQiIHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1z ZXJpZjtkaXNwbGF5OmlubGluZSI+4oCLV2VsbCwgaWYgeW91IGhhdmUgYW4gaXNzdWUgd2l0aCB0 b28gbWFueSBhcGFjaGUgcHJvY2Vzc2VzIHdhaXRpbmcgZm9yIGVuZ2luZSB0byByZXNwb25kLCB0 aGVuIHRoZXJlJiMzOTtzIHNvbWUgaXNzdWUgaW4gZW5naW5lLiBBcyBJIHdyb3RlIGFib3ZlIHBs ZWFzZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayBlbnRyb3B5Ljxicj48YnI+PC9k aXY+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhl bHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj5UaGFua3M8YnI+PGJyPjwvZGl2Pjxk aXYgY2xhc3M9ImdtYWlsX2RlZmF1bHQiIHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRp Y2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+TWFydGluIFBlcmluYTxicj7igIs8L2Rpdj7C oDwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVy LWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9 ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+PGRpdj48ZGl2 PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBub3RpY2VkIHRoYXQgdGhpcyBzZXR0aW5nIHJldmVydHMg dG8gdGhlIG9yaWdpbmFsIHNldHRpbmcsIHNvIG9WaXJ0IG11c3QgYmUgd3JpdGluZyB0aGlzIGZp bGUuIFBlcmhhcHMgdGhlc2UgbnVtYmVyIGNhbiBiZSBjaGFuZ2VkIGluIG9WaXJ0PyBJZiBub3Qs IEkgd2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkgdG8gcmV2ZXJ0IHRoZSBzZXR0aW5n cyB3aXRoIHdvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBvbiBteSBlbmdpbmUuPC9k aXY+PGRpdj46LSk8L2Rpdj48ZGl2Pgo8ZGl2Pjxicj5DaGVlcnMsPGJyPkdlcnZhaXM8YnI+PGJy Pjxicj48L2Rpdj4KCjwvZGl2PgoKCgo8YnI+PC9kaXY+PC9kaXY+PC9kaXY+PGJyPl9fX19fX19f X19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1h aWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3Zp cnQub3JnPC9hPjxicj4KPGEgdGFyZ2V0PSJfYmxhbmsiIHJlbD0ibm9yZWZlcnJlciIgaHJlZj0i aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Js b2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo= --===============7586538229997345221==-- From alukiano at redhat.com Thu Sep 29 06:55:34 2016 Content-Type: multipart/mixed; boundary="===============3606106474461512155==" MIME-Version: 1.0 From: Artyom Lukianov To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 13:55:33 +0300 Message-ID: In-Reply-To: CAP5iht7RWuTxuJT42eiCmQ_ZtPQc_Xq7WZ7Go3Aj69eHcvGJhw@mail.gmail.com --===============3606106474461512155== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable We have the same configuration under the file /etc/httpd/conf.d/z-ovirt-engine-proxy.conf for the regular engine under 3.6 and 4.0, so I do not sure if it relates to the problem. About entropy level check the bug https://bugzilla.redhat.com/show_bug.cgi?id=3D1357246. Best Regards On Thu, Sep 29, 2016 at 1:47 PM, Martin Perina wrote: > Hi, > > please take a look at my inline comments: > > On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun < > gervais(a)demontbrun.com> wrote: > >> Hey All, >> >> Since updating to 4.0.x of oVirt, I have had an issue with my hosted >> engine. After a some poking around, I think I have figured out my issue = and >> thought I would share to see what others think. >> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in >> 4.0.4. >> >> Description: >> When my hosted engine starts it reports that it is in a degraded state >> with 7 or 8 services still not started when I run systemctl status. It >> takes about 6 or 7 minutes to eventually start all the services and come >> online. If I don't set my cluster to Global-Maintenance mode it eventual= ly >> thinks that my hosted-engine needs to be rebooted and restarts it before= it >> can start everything. >> > > =E2=80=8BCould you please share with us logs gathered by ovirt-log-collec= tor? > > It's just a guess but could you please take a look if you HE VM has enough > entropy? > > cat /proc/sys/kernel/random/entropy_avail > > If the value is low (below or around 200), you really need to install and > configure some entropy generator such as haveged > > >> Solution: >> I realized that Apache was the culprit and found that the proxy to the >> ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super >> long timeout with many retries. I changed the settings and now everything >> works for me. >> >> -> Before change: >> >> > RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$| >> rhevm.ssh.key.txt$)> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> >> -> After change: >> >> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> > =E2=80=8BThis one is correct for 4.0=E2=80=8B > =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Simo= ne? > =E2=80=8B > > >> >> If I read the timeout settings correctly, it will wait 60 minutes with 5 >> retries. 5 hours is way too long for my little server to hold onto all >> those apache processes. >> > The change I made allows for there to be an error, and also releases >> apache's hold on the process. Once everything is ready, apache is ready = to >> serve requests and everything/everyone is happy. Before making the chang= e, >> I just get a whitescreen in my browser and then nothing works until I >> restart Apache (or I end up in an endless loop of ovirt-ha services >> restarting my hosted-engine. >> > > =E2=80=8BWell, if you have an issue with too many apache processes waitin= g for > engine to respond, then there's some issue in engine. As I wrote above > please share the logs with us and check entropy. > > Thanks > > Martin Perina > =E2=80=8B > > >> >> I noticed that this setting reverts to the original setting, so oVirt >> must be writing this file. Perhaps these number can be changed in oVirt?= If >> not, I will just setup and ansible play to revert the settings with work= ing >> values and restart apache on my engine. >> :-) >> >> Cheers, >> Gervais >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > > --===============3606106474461512155== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+V2UgaGF2ZSB0aGUgc2FtZSBjb25maWd1cmF0aW9uIHVuZGVyIHRoZSBm aWxlIC9ldGMvaHR0cGQvY29uZi5kL3otb3ZpcnQtZW5naW5lLXByb3h5LmNvbmYgwqBmb3IgdGhl IHJlZ3VsYXIgZW5naW5lIHVuZGVyIDMuNiBhbmQgNC4wLCBzbyBJIGRvIG5vdCBzdXJlIGlmIGl0 IHJlbGF0ZXMgdG8gdGhlIHByb2JsZW0uPGRpdj5BYm91dCBlbnRyb3B5IGxldmVsIGNoZWNrIHRo ZSBidWfCoDxhIGhyZWY9Imh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1Zy5jZ2k/ aWQ9MTM1NzI0NiI+aHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD0x MzU3MjQ2PC9hPi48L2Rpdj48ZGl2PkJlc3QgUmVnYXJkczwvZGl2PjwvZGl2PjxkaXYgY2xhc3M9 ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPk9uIFRodSwgU2VwIDI5 LCAyMDE2IGF0IDE6NDcgUE0sIE1hcnRpbiBQZXJpbmEgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBo cmVmPSJtYWlsdG86bXBlcmluYUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+bXBlcmluYUBy ZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21h aWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBz b2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250LWZh bWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZiIgY2xhc3M9ImdtYWlsX2RlZmF1bHQiPkhp LDxicj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNh bnMtc2VyaWYiIGNsYXNzPSJnbWFpbF9kZWZhdWx0Ij5wbGVhc2UgdGFrZSBhIGxvb2sgYXQgbXkg aW5saW5lIGNvbW1lbnRzOjxicj48L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPjxk aXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48c3BhbiBjbGFzcz0iIj5PbiBUdWUsIFNlcCAyNywgMjAx NiBhdCA3OjIzIFBNLCBHZXJ2YWlzIGRlIE1vbnRicnVuIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEg aHJlZj0ibWFpbHRvOmdlcnZhaXNAZGVtb250YnJ1bi5jb20iIHRhcmdldD0iX2JsYW5rIj5nZXJ2 YWlzQGRlbW9udGJydW4uY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBz dHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2Io MjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBz dHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPkhleSBBbGwsPGRpdj48YnI+PC9kaXY+PGRpdj5T aW5jZSB1cGRhdGluZyB0byA0LjAueCBvZiBvVmlydCwgSSBoYXZlIGhhZCBhbiBpc3N1ZSB3aXRo IG15IGhvc3RlZCBlbmdpbmUuIEFmdGVyIGEgc29tZSBwb2tpbmcgYXJvdW5kLCBJIHRoaW5rIEkg aGF2ZSBmaWd1cmVkIG91dCBteSBpc3N1ZSBhbmQgdGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNl ZSB3aGF0IG90aGVycyB0aGluay48L2Rpdj48ZGl2PlRoZSBpc3N1ZSBoYXMgZXhpc3RlZCB3aXRo IDQuMCwgNC4wLjEsIDQuMC4yLCA0LjAuMywgYW5kIHN0aWxsIGV4aXN0cyBpbiA0LjAuNC48L2Rp dj48ZGl2Pjxicj48L2Rpdj48ZGl2PkRlc2NyaXB0aW9uOjwvZGl2PjxkaXY+V2hlbiBteSBob3N0 ZWQgZW5naW5lIHN0YXJ0cyBpdCByZXBvcnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0 ZSB3aXRoIDcgb3IgOCBzZXJ2aWNlcyBzdGlsbCBub3Qgc3RhcnRlZCB3aGVuIEkgcnVuIHN5c3Rl bWN0bCBzdGF0dXMuIEl0IHRha2VzIGFib3V0IDYgb3IgNyBtaW51dGVzIHRvIGV2ZW50dWFsbHkg c3RhcnQgYWxsIHRoZSBzZXJ2aWNlcyBhbmQgY29tZSBvbmxpbmUuIElmIEkgZG9uJiMzOTt0IHNl dCBteSBjbHVzdGVyIHRvIEdsb2JhbC1NYWludGVuYW5jZSBtb2RlIGl0IGV2ZW50dWFsbHkgdGhp bmtzIHRoYXQgbXkgaG9zdGVkLWVuZ2luZSBuZWVkcyB0byBiZSByZWJvb3RlZCBhbmQgcmVzdGFy dHMgaXQgYmVmb3JlIGl0IGNhbiBzdGFydCBldmVyeXRoaW5nLjwvZGl2PjwvZGl2PjwvYmxvY2tx dW90ZT48L3NwYW4+PGRpdj48YnI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0 aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiIGNsYXNzPSJnbWFpbF9kZWZhdWx0Ij7igItD b3VsZCB5b3UgcGxlYXNlIHNoYXJlIHdpdGggdXMgbG9ncyBnYXRoZXJlZCBieSBvdmlydC1sb2ct Y29sbGVjdG9yPzxicj48YnI+SXQmIzM5O3MganVzdCBhIGd1ZXNzIGJ1dCBjb3VsZCB5b3UgcGxl YXNlIHRha2UgYSBsb29rIGlmIHlvdSBIRSBWTSBoYXMgZW5vdWdoIGVudHJvcHk/PGJyPjxicj7C oCBjYXQgL3Byb2Mvc3lzL2tlcm5lbC9yYW5kb20vPHdicj5lbnRyb3B5X2F2YWlsPGJyPjxicj48 L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtk aXNwbGF5OmlubGluZSIgY2xhc3M9ImdtYWlsX2RlZmF1bHQiPklmIHRoZSB2YWx1ZSBpcyBsb3cg KGJlbG93IG9yIGFyb3VuZCAyMDApLMKgIHlvdSByZWFsbHkgbmVlZCB0byBpbnN0YWxsIGFuZCBj b25maWd1cmUgc29tZSBlbnRyb3B5IGdlbmVyYXRvciBzdWNoIGFzIGhhdmVnZWQ8YnI+PGJyPjwv ZGl2PjwvZGl2PjxzcGFuIGNsYXNzPSIiPjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBw eCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGlu Zy1sZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJl YWstd29yZCI+PGRpdj48YnI+PC9kaXY+PGRpdj5Tb2x1dGlvbjo8L2Rpdj48ZGl2PkkgcmVhbGl6 ZWQgdGhhdCBBcGFjaGUgd2FzIHRoZSBjdWxwcml0IGFuZCBmb3VuZCB0aGF0IHRoZSBwcm94eSB0 byB0aGUgb3ZpcnQtZW5naW5lIGluwqAvZXRjL2h0dHBkL2NvbmYuZC96LW92aXJ0LWU8d2JyPm5n aW5lLXByb3h5LmNvbmYgaGFzIGEgc3VwZXIgbG9uZyB0aW1lb3V0IHdpdGggbWFueSByZXRyaWVz LiBJIGNoYW5nZWQgdGhlIHNldHRpbmdzIGFuZCBub3cgZXZlcnl0aGluZyB3b3JrcyBmb3IgbWUu PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj4tJmd0OyBCZWZvcmUgY2hhbmdlOjwvZGl2PjxibG9j a3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggNDBweDtib3JkZXItd2lkdGg6bWVkaXVt O2JvcmRlci1zdHlsZTpub25lO2JvcmRlci1jb2xvcjotbW96LXVzZS10ZXh0LWNvbG9yO3BhZGRp bmc6MHB4Ij48ZGl2PjxkaXY+wqAgwqAgJmx0O0xvY2F0aW9uTWF0Y2ggXi8ob3ZpcnQtZW5naW5l KCR8Lyl8YXBpKCR8Lyl8PHdicj5SSEVWTWFuYWdlcldlYi98T3ZpcnRFbmdpbmVXZWI8d2JyPi98 Y2EuY3J0JHxlbmdpbmUuc3NoLmtleS50eHQkfDx3YnI+cmhldm0uc3NoLmtleS50eHQkKSZndDs8 L2Rpdj48ZGl2PsKgIMKgIMKgIMKgIFByb3h5UGFzc01hdGNoIDxhPmFqcDovLzEyNy4wLjAuMTo4 NzAyPC9hPiB0aW1lb3V0PTM2MDAgcmV0cnk9NTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+wqAg wqAgwqAgwqAgJmx0O0lmTW9kdWxlIGRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAg wqAgwqAgwqAgwqAgQWRkT3V0cHV0RmlsdGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0 IHRleHQvY3NzIHRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFw cGxpY2F0aW9uL2pzb24gYXBwbGljYXRpb24veC15YW1sPC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAm bHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAgJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0Ozwv ZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rpdj4tJmd0OyBBZnRlciBjaGFuZ2U6 PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCA0MHB4O2JvcmRlci13aWR0aDpt ZWRpdW07Ym9yZGVyLXN0eWxlOm5vbmU7Ym9yZGVyLWNvbG9yOi1tb3otdXNlLXRleHQtY29sb3I7 cGFkZGluZzowcHgiPjxkaXY+PGRpdj7CoCDCoCAmbHQ7TG9jYXRpb25NYXRjaCBeL292aXJ0LWVu Z2luZSgkfC8pJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgUHJveHlQYXNzTWF0Y2ggPGE+YWpw Oi8vMTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9NSByZXRyeT0yPC9kaXY+PGRpdj48YnI+PC9k aXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+ PGRpdj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0 L2phdmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNh dGlvbi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKg IMKgIMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9u TWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjwvZGl2PjwvYmxvY2txdW90ZT48L3Nw YW4+PGRpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIgc3R5bGU9ImZvbnQtZmFtaWx5 OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igItUaGlzIG9uZSBp cyBjb3JyZWN0IGZvciA0LjDigIs8L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9kZWZhdWx0IiBzdHls ZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUi PuKAiywgbm90IHN1cmUgd2h5IGl0IHdhcyBub3QgdXBkYXRlZCBkdXJpbmcgdXBncmFkZSBmcm9t IDMuNi4gQFNpbW9uZT88YnI+4oCLPC9kaXY+wqA8L2Rpdj48c3BhbiBjbGFzcz0iIj48YmxvY2tx dW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xp ZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+ PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPjxkaXY+PGJyPjwvZGl2PklmIEkgcmVh ZCB0aGUgdGltZW91dCBzZXR0aW5ncyBjb3JyZWN0bHksIGl0IHdpbGwgd2FpdCA2MCBtaW51dGVz IHdpdGggNSByZXRyaWVzLiA1IGhvdXJzIGlzIHdheSB0b28gbG9uZyBmb3IgbXkgbGl0dGxlIHNl cnZlciB0byBob2xkIG9udG8gYWxsIHRob3NlIGFwYWNoZSBwcm9jZXNzZXMuIDwvZGl2PjwvYmxv Y2txdW90ZT48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRl ci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNz PSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPlRoZSBjaGFu Z2UgSSBtYWRlIGFsbG93cyBmb3IgdGhlcmUgdG8gYmUgYW4gZXJyb3IsIGFuZCBhbHNvIHJlbGVh c2VzIGFwYWNoZSYjMzk7cyBob2xkIG9uIHRoZSBwcm9jZXNzLiBPbmNlIGV2ZXJ5dGhpbmcgaXMg cmVhZHksIGFwYWNoZSBpcyByZWFkeSB0byBzZXJ2ZSByZXF1ZXN0cyBhbmQgZXZlcnl0aGluZy9l dmVyeW9uZSBpcyBoYXBweS4gQmVmb3JlIG1ha2luZyB0aGUgY2hhbmdlLCBJIGp1c3QgZ2V0IGEg d2hpdGVzY3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtzIHVudGlsIEkg cmVzdGFydCBBcGFjaGUgKG9yIEkgZW5kIHVwIGluIGFuIGVuZGxlc3MgbG9vcCBvZiBvdmlydC1o YSBzZXJ2aWNlcyByZXN0YXJ0aW5nIG15IGhvc3RlZC1lbmdpbmUuPGJyPjwvZGl2PjwvYmxvY2tx dW90ZT48L3NwYW4+PGRpdj48YnI+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIgc3R5bGU9ImZv bnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igItX ZWxsLCBpZiB5b3UgaGF2ZSBhbiBpc3N1ZSB3aXRoIHRvbyBtYW55IGFwYWNoZSBwcm9jZXNzZXMg d2FpdGluZyBmb3IgZW5naW5lIHRvIHJlc3BvbmQsIHRoZW4gdGhlcmUmIzM5O3Mgc29tZSBpc3N1 ZSBpbiBlbmdpbmUuIEFzIEkgd3JvdGUgYWJvdmUgcGxlYXNlIHNoYXJlIHRoZSBsb2dzIHdpdGgg dXMgYW5kIGNoZWNrIGVudHJvcHkuPGJyPjxicj48L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9kZWZh dWx0IiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxh eTppbmxpbmUiPlRoYW5rczxicj48YnI+PC9kaXY+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIg c3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5s aW5lIj5NYXJ0aW4gUGVyaW5hPGJyPuKAizwvZGl2PsKgPC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9 Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwy MDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxzcGFuIGNsYXNz PSIiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj48ZGl2PjxkaXY+PGRpdj48YnI+ PC9kaXY+PGRpdj5JIG5vdGljZWQgdGhhdCB0aGlzIHNldHRpbmcgcmV2ZXJ0cyB0byB0aGUgb3Jp Z2luYWwgc2V0dGluZywgc28gb1ZpcnQgbXVzdCBiZSB3cml0aW5nIHRoaXMgZmlsZS4gUGVyaGFw cyB0aGVzZSBudW1iZXIgY2FuIGJlIGNoYW5nZWQgaW4gb1ZpcnQ/IElmIG5vdCwgSSB3aWxsIGp1 c3Qgc2V0dXAgYW5kIGFuc2libGUgcGxheSB0byByZXZlcnQgdGhlIHNldHRpbmdzIHdpdGggd29y a2luZyB2YWx1ZXMgYW5kIHJlc3RhcnQgYXBhY2hlIG9uIG15IGVuZ2luZS48L2Rpdj48ZGl2Pjot KTwvZGl2PjxkaXY+CjxkaXY+PGJyPkNoZWVycyw8YnI+R2VydmFpczxicj48YnI+PGJyPjwvZGl2 PgoKPC9kaXY+CgoKCjxicj48L2Rpdj48L2Rpdj48L2Rpdj48YnI+PC9zcGFuPl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxp bmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxh bmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIHJlbD0ibm9yZWZlcnJlciIgaHJlZj0iaHR0 cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiB0YXJnZXQ9Il9ibGFu ayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4vbGlzdGluZm8vdXNlcnM8L2E+ PGJyPgo8YnI+PC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4KPGJyPl9fX19fX19f X19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1h aWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyI+VXNlcnNAb3Zp cnQub3JnPC9hPjxicj4KPGEgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xp c3RpbmZvL3VzZXJzIiByZWw9Im5vcmVmZXJyZXIiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlz dHMub3ZpcnQub3JnLzx3YnI+bWFpbG1hbi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Js b2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pgo= --===============3606106474461512155==-- From gervais at demontbrun.com Thu Sep 29 08:52:02 2016 Content-Type: multipart/mixed; boundary="===============6484840784968530656==" MIME-Version: 1.0 From: Gervais de Montbrun To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 09:51:59 -0300 Message-ID: In-Reply-To: CAP5iht7RWuTxuJT42eiCmQ_ZtPQc_Xq7WZ7Go3Aj69eHcvGJhw@mail.gmail.com --===============6484840784968530656== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_B51077DC-F506-4239-9A68-4C1B6F0E0FC4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 Hi Martin, The entropy was super low. Somewhere around 140. I installed and =3D configured haveged.service to start at bootup, reverted my apache =3D changes... After a reboot, my systemctl status still says that there are = =3D 7 services queued (note that I erroneously said degraded in my previous =3D email - the services are, in fact, queued), but the oVirt GUI comes up =3D almost immediately and everything seems to be great. Thank you for the tip. You solved my issue. Cheers, Gervais > On Sep 29, 2016, at 7:47 AM, Martin Perina wrote: >=3D20 > Hi, >=3D20 > please take a look at my inline comments: >=3D20 > On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun =3D > wrote: > Hey All, >=3D20 > Since updating to 4.0.x of oVirt, I have had an issue with my hosted =3D engine. After a some poking around, I think I have figured out my issue =3D and thought I would share to see what others think. > The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists =3D in 4.0.4. >=3D20 > Description: > When my hosted engine starts it reports that it is in a degraded state = =3D with 7 or 8 services still not started when I run systemctl status. It =3D takes about 6 or 7 minutes to eventually start all the services and come = =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start everything. >=3D20 > =3DE2=3D80=3D8BCould you please share with us logs gathered by =3D ovirt-log-collector? >=3D20 > It's just a guess but could you please take a look if you HE VM has =3D enough entropy? >=3D20 > cat /proc/sys/kernel/random/entropy_avail >=3D20 > If the value is low (below or around 200), you really need to install = =3D and configure some entropy generator such as haveged >=3D20 >=3D20 > Solution: > I realized that Apache was the culprit and found that the proxy to the = =3D ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super =3D long timeout with many retries. I changed the settings and now =3D everything works for me. >=3D20 > -> Before change: > > ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D3600 retry=3D3= D5 >=3D20 > > AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml > > >=3D20 > -> After change: > > ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D5 retry=3D3D2 >=3D20 > > AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml > > >=3D20 > =3DE2=3D80=3D8BThis one is correct for 4.0=3DE2=3D80=3D8B=3DE2=3D80=3D8B,= not sure why =3D it was not updated during upgrade from 3.6. @Simone? > =3DE2=3D80=3D8B=3D20 >=3D20 > If I read the timeout settings correctly, it will wait 60 minutes with = =3D 5 retries. 5 hours is way too long for my little server to hold onto all = =3D those apache processes. > The change I made allows for there to be an error, and also releases =3D apache's hold on the process. Once everything is ready, apache is ready =3D to serve requests and everything/everyone is happy. Before making the =3D change, I just get a whitescreen in my browser and then nothing works =3D until I restart Apache (or I end up in an endless loop of ovirt-ha =3D services restarting my hosted-engine. >=3D20 > =3DE2=3D80=3D8BWell, if you have an issue with too many apache processes = =3D waiting for engine to respond, then there's some issue in engine. As I =3D wrote above please share the logs with us and check entropy. >=3D20 > Thanks >=3D20 > Martin Perina > =3DE2=3D80=3D8B=3D20 >=3D20 > I noticed that this setting reverts to the original setting, so oVirt =3D must be writing this file. Perhaps these number can be changed in oVirt? = =3D If not, I will just setup and ansible play to revert the settings with =3D working values and restart apache on my engine. > :-) >=3D20 > Cheers, > Gervais >=3D20 >=3D20 >=3D20 >=3D20 > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users =3D >=3D20 >=3D20 --Apple-Mail=3D_B51077DC-F506-4239-9A68-4C1B6F0E0FC4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8 Hi Martin,

The entropy was super low. Somewhere around 140. I installed = =3D and configured haveged.service to start at bootup, reverted my =3D apache changes... After a reboot, my systemctl status still says that =3D there are 7 services queued (note that I erroneously said degraded in my = =3D previous email - the services are, in fact, queued), but the oVirt GUI =3D comes up almost immediately and everything seems to be great.

Thank you for the tip= . =3D You solved my issue.

Cheers,
Gervais



On Sep 29, 2016, at 7:47 AM, Martin Perina <mperina(a)redhat.com> =3D wrote:

Hi,

please take a look at my inline comments:

On Tue, Sep 27, 2016 at 7:23 PM, Gervais de =3D Montbrun <gervais(a)demontbrun.com> wrote:
Hey All,
Since updating to 4.0.x of oVirt, I = =3D have had an issue with my hosted engine. After a some poking around, I =3D think I have figured out my issue and thought I would share to see what =3D others think.
The issue has existed with 4.0, =3D 4.0.1, 4.0.2, 4.0.3, and still exists in 4.0.4.

Description:
Wh= en =3D my hosted engine starts it reports that it is in a degraded state with 7 = =3D or 8 services still not started when I run systemctl status. It takes =3D about 6 or 7 minutes to eventually start all the services and come =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start everything.

=3DE2=3D80=3D8BCould you please share with us log= s =3D gathered by ovirt-log-collector?

It's just= =3D a guess but could you please take a look if you HE VM has enough =3D entropy?

  cat =3D /proc/sys/kernel/random/entropy_avail

If the value is low (below or around 200),  = =3D you really need to install and configure some entropy generator such as =3D haveged


Solution:
I =3D realized that Apache was the culprit and found that the proxy to the =3D ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super long timeout with many retries. = =3D I changed the settings and now everything works for me.

-> Before =3D change:
  &nbs= p; =3D <LocationMatch ^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$|rhevm.ssh.key.txt$)>
      =3D   ProxyPassMatch ajp://127.0.0.1:8702 =3D timeout=3D3D3600 retry=3D3D5

=
        <IfModule =3D deflate_module>
        =3D     AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml
        =3D </IfModule>
    =3D </LocationMatch>

-> After change:
  &nbs= p; =3D <LocationMatch ^/ovirt-engine($|/)>

        <IfModule =3D deflate_module>

_______________________________________________
Users mailing list
Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users



=3D --Apple-Mail=3D_B51077DC-F506-4239-9A68-4C1B6F0E0FC4-- --===============6484840784968530656== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fQjUxMDc3REMtRjUwNi00MjM5LTlBNjgtNEMxQjZGMEUwRkM0CkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKSGkgTWFydGluLAoKVGhlIGVudHJvcHkgd2FzIHN1cGVy IGxvdy4gU29tZXdoZXJlIGFyb3VuZCAxNDAuIEkgaW5zdGFsbGVkIGFuZCA9CmNvbmZpZ3VyZWQg aGF2ZWdlZC5zZXJ2aWNlIHRvIHN0YXJ0IGF0IGJvb3R1cCwgcmV2ZXJ0ZWQgbXkgYXBhY2hlID0K Y2hhbmdlcy4uLiBBZnRlciBhIHJlYm9vdCwgbXkgc3lzdGVtY3RsIHN0YXR1cyBzdGlsbCBzYXlz IHRoYXQgdGhlcmUgYXJlID0KNyBzZXJ2aWNlcyBxdWV1ZWQgKG5vdGUgdGhhdCBJIGVycm9uZW91 c2x5IHNhaWQgZGVncmFkZWQgaW4gbXkgcHJldmlvdXMgPQplbWFpbCAtIHRoZSBzZXJ2aWNlcyBh cmUsIGluIGZhY3QsIHF1ZXVlZCksIGJ1dCB0aGUgb1ZpcnQgR1VJIGNvbWVzIHVwID0KYWxtb3N0 IGltbWVkaWF0ZWx5IGFuZCBldmVyeXRoaW5nIHNlZW1zIHRvIGJlIGdyZWF0LgoKVGhhbmsgeW91 IGZvciB0aGUgdGlwLiBZb3Ugc29sdmVkIG15IGlzc3VlLgoKQ2hlZXJzLApHZXJ2YWlzCgoKCj4g T24gU2VwIDI5LCAyMDE2LCBhdCA3OjQ3IEFNLCBNYXJ0aW4gUGVyaW5hIDxtcGVyaW5hQHJlZGhh dC5jb20+IHdyb3RlOgo+PTIwCj4gSGksCj49MjAKPiBwbGVhc2UgdGFrZSBhIGxvb2sgYXQgbXkg aW5saW5lIGNvbW1lbnRzOgo+PTIwCj4gT24gVHVlLCBTZXAgMjcsIDIwMTYgYXQgNzoyMyBQTSwg R2VydmFpcyBkZSBNb250YnJ1biA9CjxnZXJ2YWlzQGRlbW9udGJydW4uY29tIDxtYWlsdG86Z2Vy dmFpc0BkZW1vbnRicnVuLmNvbT4+IHdyb3RlOgo+IEhleSBBbGwsCj49MjAKPiBTaW5jZSB1cGRh dGluZyB0byA0LjAueCBvZiBvVmlydCwgSSBoYXZlIGhhZCBhbiBpc3N1ZSB3aXRoIG15IGhvc3Rl ZCA9CmVuZ2luZS4gQWZ0ZXIgYSBzb21lIHBva2luZyBhcm91bmQsIEkgdGhpbmsgSSBoYXZlIGZp Z3VyZWQgb3V0IG15IGlzc3VlID0KYW5kIHRob3VnaHQgSSB3b3VsZCBzaGFyZSB0byBzZWUgd2hh dCBvdGhlcnMgdGhpbmsuCj4gVGhlIGlzc3VlIGhhcyBleGlzdGVkIHdpdGggNC4wLCA0LjAuMSwg NC4wLjIsIDQuMC4zLCBhbmQgc3RpbGwgZXhpc3RzID0KaW4gNC4wLjQuCj49MjAKPiBEZXNjcmlw dGlvbjoKPiBXaGVuIG15IGhvc3RlZCBlbmdpbmUgc3RhcnRzIGl0IHJlcG9ydHMgdGhhdCBpdCBp cyBpbiBhIGRlZ3JhZGVkIHN0YXRlID0Kd2l0aCA3IG9yIDggc2VydmljZXMgc3RpbGwgbm90IHN0 YXJ0ZWQgd2hlbiBJIHJ1biBzeXN0ZW1jdGwgc3RhdHVzLiBJdCA9CnRha2VzIGFib3V0IDYgb3Ig NyBtaW51dGVzIHRvIGV2ZW50dWFsbHkgc3RhcnQgYWxsIHRoZSBzZXJ2aWNlcyBhbmQgY29tZSA9 Cm9ubGluZS4gSWYgSSBkb24ndCBzZXQgbXkgY2x1c3RlciB0byBHbG9iYWwtTWFpbnRlbmFuY2Ug bW9kZSBpdCA9CmV2ZW50dWFsbHkgdGhpbmtzIHRoYXQgbXkgaG9zdGVkLWVuZ2luZSBuZWVkcyB0 byBiZSByZWJvb3RlZCBhbmQgPQpyZXN0YXJ0cyBpdCBiZWZvcmUgaXQgY2FuIHN0YXJ0IGV2ZXJ5 dGhpbmcuCj49MjAKPiA9RTI9ODA9OEJDb3VsZCB5b3UgcGxlYXNlIHNoYXJlIHdpdGggdXMgbG9n cyBnYXRoZXJlZCBieSA9Cm92aXJ0LWxvZy1jb2xsZWN0b3I/Cj49MjAKPiBJdCdzIGp1c3QgYSBn dWVzcyBidXQgY291bGQgeW91IHBsZWFzZSB0YWtlIGEgbG9vayBpZiB5b3UgSEUgVk0gaGFzID0K ZW5vdWdoIGVudHJvcHk/Cj49MjAKPiAgIGNhdCAvcHJvYy9zeXMva2VybmVsL3JhbmRvbS9lbnRy b3B5X2F2YWlsCj49MjAKPiBJZiB0aGUgdmFsdWUgaXMgbG93IChiZWxvdyBvciBhcm91bmQgMjAw KSwgIHlvdSByZWFsbHkgbmVlZCB0byBpbnN0YWxsID0KYW5kIGNvbmZpZ3VyZSBzb21lIGVudHJv cHkgZ2VuZXJhdG9yIHN1Y2ggYXMgaGF2ZWdlZAo+PTIwCj49MjAKPiBTb2x1dGlvbjoKPiBJIHJl YWxpemVkIHRoYXQgQXBhY2hlIHdhcyB0aGUgY3VscHJpdCBhbmQgZm91bmQgdGhhdCB0aGUgcHJv eHkgdG8gdGhlID0Kb3ZpcnQtZW5naW5lIGluIC9ldGMvaHR0cGQvY29uZi5kL3otb3ZpcnQtZW5n aW5lLXByb3h5LmNvbmYgaGFzIGEgc3VwZXIgPQpsb25nIHRpbWVvdXQgd2l0aCBtYW55IHJldHJp ZXMuIEkgY2hhbmdlZCB0aGUgc2V0dGluZ3MgYW5kIG5vdyA9CmV2ZXJ5dGhpbmcgd29ya3MgZm9y IG1lLgo+PTIwCj4gLT4gQmVmb3JlIGNoYW5nZToKPiAgICAgPExvY2F0aW9uTWF0Y2ggPQpeLyhv dmlydC1lbmdpbmUoJHwvKXxhcGkoJHwvKXxSSEVWTWFuYWdlcldlYi98T3ZpcnRFbmdpbmVXZWIv fGNhLmNydCR8ZW5naT0KbmUuc3NoLmtleS50eHQkfHJoZXZtLnNzaC5rZXkudHh0JCk+Cj4gICAg ICAgICBQcm94eVBhc3NNYXRjaCBhanA6Ly8xMjcuMC4wLjE6ODcwMiA8PiB0aW1lb3V0PTNEMzYw MCByZXRyeT0zRDUKPj0yMAo+ICAgICAgICAgPElmTW9kdWxlIGRlZmxhdGVfbW9kdWxlPgo+ICAg ICAgICAgICAgIEFkZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0 ZXh0L2NzcyA9CnRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFw cGxpY2F0aW9uL2pzb24gPQphcHBsaWNhdGlvbi94LXlhbWwKPiAgICAgICAgIDwvSWZNb2R1bGU+ Cj4gICAgIDwvTG9jYXRpb25NYXRjaD4KPj0yMAo+IC0+IEFmdGVyIGNoYW5nZToKPiAgICAgPExv Y2F0aW9uTWF0Y2ggXi9vdmlydC1lbmdpbmUoJHwvKT4KPiAgICAgICAgIFByb3h5UGFzc01hdGNo IGFqcDovLzEyNy4wLjAuMTo4NzAyIDw+IHRpbWVvdXQ9M0Q1IHJldHJ5PTNEMgo+PTIwCj4gICAg ICAgICA8SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGU+Cj4gICAgICAgICAgICAgQWRkT3V0cHV0Rmls dGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzID0KdGV4dC9odG1sIHRl eHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiA9CmFwcGxp Y2F0aW9uL3gteWFtbAo+ICAgICAgICAgPC9JZk1vZHVsZT4KPiAgICAgPC9Mb2NhdGlvbk1hdGNo Pgo+PTIwCj4gPUUyPTgwPThCVGhpcyBvbmUgaXMgY29ycmVjdCBmb3IgNC4wPUUyPTgwPThCPUUy PTgwPThCLCBub3Qgc3VyZSB3aHkgPQppdCB3YXMgbm90IHVwZGF0ZWQgZHVyaW5nIHVwZ3JhZGUg ZnJvbSAzLjYuIEBTaW1vbmU/Cj4gPUUyPTgwPThCPTIwCj49MjAKPiBJZiBJIHJlYWQgdGhlIHRp bWVvdXQgc2V0dGluZ3MgY29ycmVjdGx5LCBpdCB3aWxsIHdhaXQgNjAgbWludXRlcyB3aXRoID0K NSByZXRyaWVzLiA1IGhvdXJzIGlzIHdheSB0b28gbG9uZyBmb3IgbXkgbGl0dGxlIHNlcnZlciB0 byBob2xkIG9udG8gYWxsID0KdGhvc2UgYXBhY2hlIHByb2Nlc3Nlcy4KPiBUaGUgY2hhbmdlIEkg bWFkZSBhbGxvd3MgZm9yIHRoZXJlIHRvIGJlIGFuIGVycm9yLCBhbmQgYWxzbyByZWxlYXNlcyA9 CmFwYWNoZSdzIGhvbGQgb24gdGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyByZWFkeSwg YXBhY2hlIGlzIHJlYWR5ID0KdG8gc2VydmUgcmVxdWVzdHMgYW5kIGV2ZXJ5dGhpbmcvZXZlcnlv bmUgaXMgaGFwcHkuIEJlZm9yZSBtYWtpbmcgdGhlID0KY2hhbmdlLCBJIGp1c3QgZ2V0IGEgd2hp dGVzY3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtzID0KdW50aWwgSSBy ZXN0YXJ0IEFwYWNoZSAob3IgSSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhh ID0Kc2VydmljZXMgcmVzdGFydGluZyBteSBob3N0ZWQtZW5naW5lLgo+PTIwCj4gPUUyPTgwPThC V2VsbCwgaWYgeW91IGhhdmUgYW4gaXNzdWUgd2l0aCB0b28gbWFueSBhcGFjaGUgcHJvY2Vzc2Vz ID0Kd2FpdGluZyBmb3IgZW5naW5lIHRvIHJlc3BvbmQsIHRoZW4gdGhlcmUncyBzb21lIGlzc3Vl IGluIGVuZ2luZS4gQXMgSSA9Cndyb3RlIGFib3ZlIHBsZWFzZSBzaGFyZSB0aGUgbG9ncyB3aXRo IHVzIGFuZCBjaGVjayBlbnRyb3B5Lgo+PTIwCj4gVGhhbmtzCj49MjAKPiBNYXJ0aW4gUGVyaW5h Cj4gPUUyPTgwPThCPTIwCj49MjAKPiBJIG5vdGljZWQgdGhhdCB0aGlzIHNldHRpbmcgcmV2ZXJ0 cyB0byB0aGUgb3JpZ2luYWwgc2V0dGluZywgc28gb1ZpcnQgPQptdXN0IGJlIHdyaXRpbmcgdGhp cyBmaWxlLiBQZXJoYXBzIHRoZXNlIG51bWJlciBjYW4gYmUgY2hhbmdlZCBpbiBvVmlydD8gPQpJ ZiBub3QsIEkgd2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkgdG8gcmV2ZXJ0IHRoZSBz ZXR0aW5ncyB3aXRoID0Kd29ya2luZyB2YWx1ZXMgYW5kIHJlc3RhcnQgYXBhY2hlIG9uIG15IGVu Z2luZS4KPiA6LSkKPj0yMAo+IENoZWVycywKPiBHZXJ2YWlzCj49MjAKPj0yMAo+PTIwCj49MjAK PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+IFVzZXJz IG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0Lm9yZyA8bWFpbHRvOlVzZXJzQG92aXJ0Lm9yZz4K PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMgPQo8aHR0cDov L2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPgo+PTIwCj49MjAKCgotLUFw cGxlLU1haWw9X0I1MTA3N0RDLUY1MDYtNDIzOS05QTY4LTRDMUI2RjBFMEZDNApDb250ZW50LVRy YW5zZmVyLUVuY29kaW5nOiBxdW90ZWQtcHJpbnRhYmxlCkNvbnRlbnQtVHlwZTogdGV4dC9odG1s OwoJY2hhcnNldD11dGYtOAoKPGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0zRCJDb250ZW50 LVR5cGUiIGNvbnRlbnQ9M0QidGV4dC9odG1sID0KY2hhcnNldD0zRHV0Zi04Ij48L2hlYWQ+PGJv ZHkgc3R5bGU9M0Qid29yZC13cmFwOiBicmVhay13b3JkOyA9Ci13ZWJraXQtbmJzcC1tb2RlOiBz cGFjZTsgLXdlYmtpdC1saW5lLWJyZWFrOiBhZnRlci13aGl0ZS1zcGFjZTsiID0KY2xhc3M9M0Qi Ij5IaSBNYXJ0aW4sPGRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgPQpj bGFzcz0zRCIiPlRoZSBlbnRyb3B5IHdhcyBzdXBlciBsb3cuIFNvbWV3aGVyZSBhcm91bmQgMTQw LiBJIGluc3RhbGxlZCA9CmFuZCBjb25maWd1cmVkJm5ic3A7aGF2ZWdlZC5zZXJ2aWNlIHRvIHN0 YXJ0IGF0IGJvb3R1cCwgcmV2ZXJ0ZWQgbXkgPQphcGFjaGUgY2hhbmdlcy4uLiBBZnRlciBhIHJl Ym9vdCwgbXkgc3lzdGVtY3RsIHN0YXR1cyBzdGlsbCBzYXlzIHRoYXQgPQp0aGVyZSBhcmUgNyBz ZXJ2aWNlcyBxdWV1ZWQgKG5vdGUgdGhhdCBJIGVycm9uZW91c2x5IHNhaWQgZGVncmFkZWQgaW4g bXkgPQpwcmV2aW91cyBlbWFpbCAtIHRoZSBzZXJ2aWNlcyBhcmUsIGluIGZhY3QsIHF1ZXVlZCks IGJ1dCB0aGUgb1ZpcnQgR1VJID0KY29tZXMgdXAgYWxtb3N0IGltbWVkaWF0ZWx5IGFuZCBldmVy eXRoaW5nIHNlZW1zIHRvIGJlIGdyZWF0LjwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPjxiciBjbGFz cz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5UaGFuayB5b3UgZm9yIHRoZSB0aXAuID0KWW91 IHNvbHZlZCBteSBpc3N1ZS48YnIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+CjxkaXYgaWQ9 M0Qic2lnbmF0dXJlIiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPkNoZWVycyw8YnIgPQpjbGFz cz0zRCIiPkdlcnZhaXM8YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0Qi Ij48L2Rpdj4KCjwvZGl2Pgo8YnIgY2xhc3M9M0QiIj48ZGl2PjxibG9ja3F1b3RlIHR5cGU9M0Qi Y2l0ZSIgY2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj5PbiBTZXAgMjksIDIwMTYsIGF0IDc6 NDcgQU0sIE1hcnRpbiBQZXJpbmEgJmx0OzxhID0KaHJlZj0zRCJtYWlsdG86bXBlcmluYUByZWRo YXQuY29tIiBjbGFzcz0zRCIiPm1wZXJpbmFAcmVkaGF0LmNvbTwvYT4mZ3Q7ID0Kd3JvdGU6PC9k aXY+PGJyIGNsYXNzPTNEIkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPjxkaXYgY2xhc3M9M0Qi Ij48ZGl2ID0KZGlyPTNEImx0ciIgY2xhc3M9M0QiIj48ZGl2ID0Kc3R5bGU9M0QiZm9udC1mYW1p bHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWYiID0KY2xhc3M9M0QiZ21haWxfZGVmYXVsdCI+ SGksPGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CnN0eWxlPTNEImZv bnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmIiA9CmNsYXNzPTNEImdtYWlsX2Rl ZmF1bHQiPnBsZWFzZSB0YWtlIGEgbG9vayBhdCBteSBpbmxpbmUgY29tbWVudHM6PGJyID0KY2xh c3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEImdtYWlsX2V4dHJhIj48YnIgY2xhc3M9M0QiIj48 ZGl2ID0KY2xhc3M9M0QiZ21haWxfcXVvdGUiPk9uIFR1ZSwgU2VwIDI3LCAyMDE2IGF0IDc6MjMg UE0sIEdlcnZhaXMgZGUgPQpNb250YnJ1biA8c3BhbiBkaXI9M0QibHRyIiBjbGFzcz0zRCIiPiZs dDs8YSB0YXJnZXQ9M0QiX2JsYW5rIiA9CmhyZWY9M0QibWFpbHRvOmdlcnZhaXNAZGVtb250YnJ1 bi5jb20iID0KY2xhc3M9M0QiIj5nZXJ2YWlzQGRlbW9udGJydW4uY29tPC9hPiZndDs8L3NwYW4+ IHdyb3RlOjxiciA9CmNsYXNzPTNEIiI+PGJsb2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOjBweCAw cHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCA9CnNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFk ZGluZy1sZWZ0OjFleCIgY2xhc3M9M0QiZ21haWxfcXVvdGUiPjxkaXYgPQpzdHlsZT0zRCJ3b3Jk LXdyYXA6YnJlYWstd29yZCIgY2xhc3M9M0QiIj5IZXkgQWxsLDxkaXYgY2xhc3M9M0QiIj48YnIg PQpjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5TaW5jZSB1cGRhdGluZyB0byA0LjAu eCBvZiBvVmlydCwgSSA9CmhhdmUgaGFkIGFuIGlzc3VlIHdpdGggbXkgaG9zdGVkIGVuZ2luZS4g QWZ0ZXIgYSBzb21lIHBva2luZyBhcm91bmQsIEkgPQp0aGluayBJIGhhdmUgZmlndXJlZCBvdXQg bXkgaXNzdWUgYW5kIHRob3VnaHQgSSB3b3VsZCBzaGFyZSB0byBzZWUgd2hhdCA9Cm90aGVycyB0 aGluay48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+VGhlIGlzc3VlIGhhcyBleGlzdGVkIHdpdGggNC4w LCA9CjQuMC4xLCA0LjAuMiwgNC4wLjMsIGFuZCBzdGlsbCBleGlzdHMgaW4gNC4wLjQuPC9kaXY+ PGRpdiBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPkRl c2NyaXB0aW9uOjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5XaGVuID0KbXkgaG9zdGVkIGVuZ2luZSBz dGFydHMgaXQgcmVwb3J0cyB0aGF0IGl0IGlzIGluIGEgZGVncmFkZWQgc3RhdGUgd2l0aCA3ID0K b3IgOCBzZXJ2aWNlcyBzdGlsbCBub3Qgc3RhcnRlZCB3aGVuIEkgcnVuIHN5c3RlbWN0bCBzdGF0 dXMuIEl0IHRha2VzID0KYWJvdXQgNiBvciA3IG1pbnV0ZXMgdG8gZXZlbnR1YWxseSBzdGFydCBh bGwgdGhlIHNlcnZpY2VzIGFuZCBjb21lID0Kb25saW5lLiBJZiBJIGRvbid0IHNldCBteSBjbHVz dGVyIHRvIEdsb2JhbC1NYWludGVuYW5jZSBtb2RlIGl0ID0KZXZlbnR1YWxseSB0aGlua3MgdGhh dCBteSBob3N0ZWQtZW5naW5lIG5lZWRzIHRvIGJlIHJlYm9vdGVkIGFuZCA9CnJlc3RhcnRzIGl0 IGJlZm9yZSBpdCBjYW4gc3RhcnQgZXZlcnl0aGluZy48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+ PGRpdiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdiA9CnN0eWxlPTNEImZvbnQtZmFt aWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIiA9CmNsYXNzPTNE ImdtYWlsX2RlZmF1bHQiPj1FMj04MD04QkNvdWxkIHlvdSBwbGVhc2Ugc2hhcmUgd2l0aCB1cyBs b2dzID0KZ2F0aGVyZWQgYnkgb3ZpcnQtbG9nLWNvbGxlY3Rvcj88YnIgY2xhc3M9M0QiIj48YnIg Y2xhc3M9M0QiIj5JdCdzIGp1c3QgPQphIGd1ZXNzIGJ1dCBjb3VsZCB5b3UgcGxlYXNlIHRha2Ug YSBsb29rIGlmIHlvdSBIRSBWTSBoYXMgZW5vdWdoID0KZW50cm9weT88YnIgY2xhc3M9M0QiIj48 YnIgY2xhc3M9M0QiIj4mbmJzcDsgY2F0ID0KL3Byb2Mvc3lzL2tlcm5lbC9yYW5kb20vZW50cm9w eV9hdmFpbDxiciBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CnN0eWxl PTNEImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5l IiA9CmNsYXNzPTNEImdtYWlsX2RlZmF1bHQiPklmIHRoZSB2YWx1ZSBpcyBsb3cgKGJlbG93IG9y IGFyb3VuZCAyMDApLCZuYnNwOyA9CnlvdSByZWFsbHkgbmVlZCB0byBpbnN0YWxsIGFuZCBjb25m aWd1cmUgc29tZSBlbnRyb3B5IGdlbmVyYXRvciBzdWNoIGFzID0KaGF2ZWdlZDxiciBjbGFzcz0z RCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjxibG9ja3F1b3RlID0Kc3R5bGU9M0QibWFy Z2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCA9CnJnYigyMDQsMjA0 LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9M0QiZ21haWxfcXVvdGUiPjxkaXYgPQpzdHls ZT0zRCJ3b3JkLXdyYXA6YnJlYWstd29yZCIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGJy ID0KY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+U29sdXRpb246PC9kaXY+PGRpdiBj bGFzcz0zRCIiPkkgPQpyZWFsaXplZCB0aGF0IEFwYWNoZSB3YXMgdGhlIGN1bHByaXQgYW5kIGZv dW5kIHRoYXQgdGhlIHByb3h5IHRvIHRoZSA9Cm92aXJ0LWVuZ2luZSBpbiZuYnNwOy9ldGMvaHR0 cGQvY29uZi5kL3otb3ZpcnQtPHdiciA9CmNsYXNzPTNEIiI+ZW5naW5lLXByb3h5LmNvbmYgaGFz IGEgc3VwZXIgbG9uZyB0aW1lb3V0IHdpdGggbWFueSByZXRyaWVzLiA9CkkgY2hhbmdlZCB0aGUg c2V0dGluZ3MgYW5kIG5vdyBldmVyeXRoaW5nIHdvcmtzIGZvciBtZS48L2Rpdj48ZGl2ID0KY2xh c3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+LSZndDsgQmVmb3Jl ID0KY2hhbmdlOjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPTNEIm1hcmdpbjowcHggMHB4IDBweCA9 CjQwcHg7Ym9yZGVyLXdpZHRoOm1lZGl1bTtib3JkZXItc3R5bGU6bm9uZTtib3JkZXItY29sb3I6 LW1vei11c2UtdGV4dC1jb2xvPQpyO3BhZGRpbmc6MHB4IiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9 M0QiIj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyA9CiZsdDtMb2NhdGlvbk1hdGNoIF4v KG92aXJ0LWVuZ2luZSgkfC8pfGFwaSgkfC8pfDx3YnIgPQpjbGFzcz0zRCIiPlJIRVZNYW5hZ2Vy V2ViL3w8d2JyIGNsYXNzPTNEIiI+T3ZpcnRFbmdpbmVXZWIvfGNhLmNydCR8PHdiciA9CmNsYXNz PTNEIiI+ZW5naW5lLnNzaC5rZXkudHh0JHxyaGV2bS5zc2guPHdiciA9CmNsYXNzPTNEIiI+a2V5 LnR4dCQpJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyA9CiZu YnNwOyBQcm94eVBhc3NNYXRjaCA8YSBjbGFzcz0zRCIiPmFqcDovLzEyNy4wLjAuMTo4NzAyPC9h PiA9CnRpbWVvdXQ9M0QzNjAwIHJldHJ5PTNENTwvZGl2PjxkaXYgY2xhc3M9M0QiIj48YnIgY2xh c3M9M0QiIj48L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz cDsgJmx0O0lmTW9kdWxlID0KZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRpdiBjbGFzcz0zRCIi PiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyA9CiZuYnNwOyAmbmJzcDsgQWRkT3V0cHV0Rmls dGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzID0KdGV4dC9odG1sIHRl eHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiA9CmFwcGxp Y2F0aW9uL3gteWFtbDwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAm bmJzcDsgPQombHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5i c3A7ID0KJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2 IGNsYXNzPTNEIiI+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj4tJmd0OyBBZnRlciBjaGFuZ2U6PGJs b2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggPQowcHggPQo0MHB4O2JvcmRlci13aWR0 aDptZWRpdW07Ym9yZGVyLXN0eWxlOm5vbmU7Ym9yZGVyLWNvbG9yOi1tb3otdXNlLXRleHQtY29s bz0KcjtwYWRkaW5nOjBweCIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0z RCIiPiZuYnNwOyAmbmJzcDsgPQombHQ7TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2luZSgkfC8p Jmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgPQombmJzcDsgJm5ic3A7ICZuYnNwOyBQ cm94eVBhc3NNYXRjaCA8YSBjbGFzcz0zRCIiPmFqcDovLzEyNy4wLjAuMTo4NzAyPC9hPj0KIHRp bWVvdXQ9M0Q1IHJldHJ5PTNEMjwvZGl2PjxkaXYgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48 L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0O0lm TW9kdWxlID0KZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRpdiBjbGFzcz0zRCIiPiZuYnNwOyAm bmJzcDsgJm5ic3A7ICZuYnNwOyA9CiZuYnNwOyAmbmJzcDsgQWRkT3V0cHV0RmlsdGVyQnlUeXBl IERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzID0KdGV4dC9odG1sIHRleHQveG1sIHRl eHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiA9CmFwcGxpY2F0aW9uL3gt eWFtbDwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgPQom bHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ID0KJmx0 Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48L2Jsb2Nr cXVvdGU+PGRpdiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCJnbWFp bF9kZWZhdWx0IiA9CnN0eWxlPTNEImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNl cmlmO2Rpc3BsYXk6aW5saW5lIj49RTI9ODA9OEJUPQpoaXMgb25lIGlzIGNvcnJlY3QgZm9yIDQu MD1FMj04MD04QjwvZGl2PjxkaXYgY2xhc3M9M0QiZ21haWxfZGVmYXVsdCIgPQpzdHlsZT0zRCJm b250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+PUUy PTgwPThCLD0KIG5vdCBzdXJlIHdoeSBpdCB3YXMgbm90IHVwZGF0ZWQgZHVyaW5nIHVwZ3JhZGUg ZnJvbSAzLjYuIEBTaW1vbmU/PGJyID0KY2xhc3M9M0QiIj49RTI9ODA9OEI8L2Rpdj4mbmJzcDs8 L2Rpdj48YmxvY2txdW90ZSBzdHlsZT0zRCJtYXJnaW46MHB4ID0KMHB4IDBweCAwLjhleDtib3Jk ZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiA9CmNs YXNzPTNEImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPTNEIndvcmQtd3JhcDpicmVhay13b3JkIiBj bGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PklmIEkgcmVh ZCB0aGUgdGltZW91dCBzZXR0aW5ncyA9CmNvcnJlY3RseSwgaXQgd2lsbCB3YWl0IDYwIG1pbnV0 ZXMgd2l0aCA1IHJldHJpZXMuIDUgaG91cnMgaXMgd2F5IHRvbyA9CmxvbmcgZm9yIG15IGxpdHRs ZSBzZXJ2ZXIgdG8gaG9sZCBvbnRvIGFsbCB0aG9zZSBhcGFjaGUgcHJvY2Vzc2VzLiA9CjwvZGl2 PjwvYmxvY2txdW90ZT48YmxvY2txdW90ZSBzdHlsZT0zRCJtYXJnaW46MHB4IDBweCAwcHggPQow LjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6 MWV4IiA9CmNsYXNzPTNEImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPTNEIndvcmQtd3JhcDpicmVh ay13b3JkIiBjbGFzcz0zRCIiPlRoZSA9CmNoYW5nZSBJIG1hZGUgYWxsb3dzIGZvciB0aGVyZSB0 byBiZSBhbiBlcnJvciwgYW5kIGFsc28gcmVsZWFzZXMgPQphcGFjaGUncyBob2xkIG9uIHRoZSBw cm9jZXNzLiBPbmNlIGV2ZXJ5dGhpbmcgaXMgcmVhZHksIGFwYWNoZSBpcyByZWFkeSA9CnRvIHNl cnZlIHJlcXVlc3RzIGFuZCBldmVyeXRoaW5nL2V2ZXJ5b25lIGlzIGhhcHB5LiBCZWZvcmUgbWFr aW5nIHRoZSA9CmNoYW5nZSwgSSBqdXN0IGdldCBhIHdoaXRlc2NyZWVuIGluIG15IGJyb3dzZXIg YW5kIHRoZW4gbm90aGluZyB3b3JrcyA9CnVudGlsIEkgcmVzdGFydCBBcGFjaGUgKG9yIEkgZW5k IHVwIGluIGFuIGVuZGxlc3MgbG9vcCBvZiBvdmlydC1oYSA9CnNlcnZpY2VzIHJlc3RhcnRpbmcg bXkgaG9zdGVkLWVuZ2luZS48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2 IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEImdtYWlsX2RlZmF1bHQi ID0Kc3R5bGU9M0QiZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxh eTppbmxpbmUiPj1FMj04MD04Qlc9CmVsbCwgaWYgeW91IGhhdmUgYW4gaXNzdWUgd2l0aCB0b28g bWFueSBhcGFjaGUgcHJvY2Vzc2VzIHdhaXRpbmcgZm9yID0KZW5naW5lIHRvIHJlc3BvbmQsIHRo ZW4gdGhlcmUncyBzb21lIGlzc3VlIGluIGVuZ2luZS4gQXMgSSB3cm90ZSBhYm92ZSA9CnBsZWFz ZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayBlbnRyb3B5LjxiciBjbGFzcz0zRCIi PjxiciA9CmNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCJnbWFpbF9kZWZhdWx0IiA9CnN0 eWxlPTNEImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5s aW5lIj5UaGFua3M8YnIgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xh c3M9M0QiZ21haWxfZGVmYXVsdCIgPQpzdHlsZT0zRCJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRp Y2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+TWFydGluID0KUGVyaW5hPGJyIGNsYXNzPTNE IiI+PUUyPTgwPThCPC9kaXY+Jm5ic3A7PC9kaXY+PGJsb2NrcXVvdGUgPQpzdHlsZT0zRCJtYXJn aW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkID0KcmdiKDIwNCwyMDQs MjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+PGRpdiA9CnN0eWxl PTNEIndvcmQtd3JhcDpicmVhay13b3JkIiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48ZGl2 ID0KY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiBj bGFzcz0zRCIiPkkgPQpub3RpY2VkIHRoYXQgdGhpcyBzZXR0aW5nIHJldmVydHMgdG8gdGhlIG9y aWdpbmFsIHNldHRpbmcsIHNvIG9WaXJ0IG11c3QgPQpiZSB3cml0aW5nIHRoaXMgZmlsZS4gUGVy aGFwcyB0aGVzZSBudW1iZXIgY2FuIGJlIGNoYW5nZWQgaW4gb1ZpcnQ/IElmID0Kbm90LCBJIHdp bGwganVzdCBzZXR1cCBhbmQgYW5zaWJsZSBwbGF5IHRvIHJldmVydCB0aGUgc2V0dGluZ3Mgd2l0 aCA9CndvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBvbiBteSBlbmdpbmUuPC9kaXY+ PGRpdiA9CmNsYXNzPTNEIiI+Oi0pPC9kaXY+PGRpdiBjbGFzcz0zRCIiPgo8ZGl2IGNsYXNzPTNE IiI+PGJyIGNsYXNzPTNEIiI+Q2hlZXJzLDxiciBjbGFzcz0zRCIiPkdlcnZhaXM8YnIgPQpjbGFz cz0zRCIiPjxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PgoKPC9kaXY+CgoKCjxi ciBjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvZGl2PjxiciA9CmNsYXNzPTNEIiI+X19fX19fX19f X19fX19fX19fX19fX19fX19fX19fPHdiciA9CmNsYXNzPTNEIiI+X19fX19fX19fX19fX19fX188 YnIgY2xhc3M9M0QiIj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyIGNsYXNzPTNEIiI+CjxhIGhyZWY9 M0QibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgY2xhc3M9M0QiIj5Vc2Vyc0BvdmlydC5vcmc8L2E+ PGJyID0KY2xhc3M9M0QiIj4KPGEgdGFyZ2V0PTNEIl9ibGFuayIgcmVsPTNEIm5vcmVmZXJyZXIi ID0KaHJlZj0zRCJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMi ID0KY2xhc3M9M0QiIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnLzx3YnIgPQpjbGFzcz0zRCIiPm1h aWxtYW4vbGlzdGluZm8vdXNlcnM8L2E+PGJyIGNsYXNzPTNEIiI+CjxiciBjbGFzcz0zRCIiPjwv YmxvY2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48L2Rpdj48L2Rpdj4KPC9kaXY+PC9ibG9j a3F1b3RlPjwvZGl2PjxiciBjbGFzcz0zRCIiPjwvZGl2PjwvYm9keT48L2h0bWw+PQoKLS1BcHBs ZS1NYWlsPV9CNTEwNzdEQy1GNTA2LTQyMzktOUE2OC00QzFCNkYwRTBGQzQtLQo= --===============6484840784968530656==-- From stirabos at redhat.com Thu Sep 29 09:02:05 2016 Content-Type: multipart/mixed; boundary="===============3935525266990280390==" MIME-Version: 1.0 From: Simone Tiraboschi To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 15:01:34 +0200 Message-ID: In-Reply-To: ACF09C6B-4437-442A-B8CB-113EBA99C62A@demontbrun.com --===============3935525266990280390== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 29, 2016 at 2:51 PM, Gervais de Montbrun wrote: > Hi Martin, > > The entropy was super low. Somewhere around 140. I installed and > configured haveged.service to start at bootup, reverted my apache > changes... After a reboot, my systemctl status still says that there are 7 > services queued (note that I erroneously said degraded in my previous ema= il > - the services are, in fact, queued), but the oVirt GUI comes up almost > immediately and everything seems to be great. > > Take care that using havaged on a VM should not be considered a good source of entropy and the oVirt PKi is managed by the engine. http://security.stackexchange.com/questions/34523/is-it- appropriate-to-use-haveged-as-a-source-of-entropy-on-virtual-machines A better approach is the virtio-rng paravirtualised rng driver as for patch https://gerrit.ovirt.org/#/c/62334/ > Thank you for the tip. You solved my issue. > > Cheers, > Gervais > > > > On Sep 29, 2016, at 7:47 AM, Martin Perina wrote: > > Hi, > > please take a look at my inline comments: > > On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun < > gervais(a)demontbrun.com> wrote: > >> Hey All, >> >> Since updating to 4.0.x of oVirt, I have had an issue with my hosted >> engine. After a some poking around, I think I have figured out my issue = and >> thought I would share to see what others think. >> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in >> 4.0.4. >> >> Description: >> When my hosted engine starts it reports that it is in a degraded state >> with 7 or 8 services still not started when I run systemctl status. It >> takes about 6 or 7 minutes to eventually start all the services and come >> online. If I don't set my cluster to Global-Maintenance mode it eventual= ly >> thinks that my hosted-engine needs to be rebooted and restarts it before= it >> can start everything. >> > > =E2=80=8BCould you please share with us logs gathered by ovirt-log-collec= tor? > > It's just a guess but could you please take a look if you HE VM has enough > entropy? > > cat /proc/sys/kernel/random/entropy_avail > > If the value is low (below or around 200), you really need to install and > configure some entropy generator such as haveged > > >> Solution: >> I realized that Apache was the culprit and found that the proxy to the >> ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super >> long timeout with many retries. I changed the settings and now everything >> works for me. >> >> -> Before change: >> >> > RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$| >> rhevm.ssh.key.txt$)> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> >> -> After change: >> >> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> > =E2=80=8BThis one is correct for 4.0=E2=80=8B > =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Simo= ne? > =E2=80=8B > > >> >> If I read the timeout settings correctly, it will wait 60 minutes with 5 >> retries. 5 hours is way too long for my little server to hold onto all >> those apache processes. >> > The change I made allows for there to be an error, and also releases >> apache's hold on the process. Once everything is ready, apache is ready = to >> serve requests and everything/everyone is happy. Before making the chang= e, >> I just get a whitescreen in my browser and then nothing works until I >> restart Apache (or I end up in an endless loop of ovirt-ha services >> restarting my hosted-engine. >> > > =E2=80=8BWell, if you have an issue with too many apache processes waitin= g for > engine to respond, then there's some issue in engine. As I wrote above > please share the logs with us and check entropy. > > Thanks > > Martin Perina > =E2=80=8B > > >> >> I noticed that this setting reverts to the original setting, so oVirt >> must be writing this file. Perhaps these number can be changed in oVirt?= If >> not, I will just setup and ansible play to revert the settings with work= ing >> values and restart apache on my engine. >> :-) >> >> Cheers, >> Gervais >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > > --===============3935525266990280390== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFRodSwgU2VwIDI5LCAyMDE2IGF0IDI6NTEgUE0sIEdlcnZhaXMg ZGUgTW9udGJydW4gPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86Z2VydmFpc0Bk ZW1vbnRicnVuLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmdlcnZhaXNAZGVtb250YnJ1bi5jb208L2E+ Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5 bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIw NCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWst d29yZCI+SGkgTWFydGluLDxkaXY+PGJyPjwvZGl2PjxkaXY+VGhlIGVudHJvcHkgd2FzIHN1cGVy IGxvdy4gU29tZXdoZXJlIGFyb3VuZCAxNDAuIEkgaW5zdGFsbGVkIGFuZCBjb25maWd1cmVkwqBo YXZlZ2VkLnNlcnZpY2UgdG8gc3RhcnQgYXQgYm9vdHVwLCByZXZlcnRlZCBteSBhcGFjaGUgY2hh bmdlcy4uLiBBZnRlciBhIHJlYm9vdCwgbXkgc3lzdGVtY3RsIHN0YXR1cyBzdGlsbCBzYXlzIHRo YXQgdGhlcmUgYXJlIDcgc2VydmljZXMgcXVldWVkIChub3RlIHRoYXQgSSBlcnJvbmVvdXNseSBz YWlkIGRlZ3JhZGVkIGluIG15IHByZXZpb3VzIGVtYWlsIC0gdGhlIHNlcnZpY2VzIGFyZSwgaW4g ZmFjdCwgcXVldWVkKSwgYnV0IHRoZSBvVmlydCBHVUkgY29tZXMgdXAgYWxtb3N0IGltbWVkaWF0 ZWx5IGFuZCBldmVyeXRoaW5nIHNlZW1zIHRvIGJlIGdyZWF0LjwvZGl2PjxkaXY+PGJyPjwvZGl2 PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+PGJyPjwvZGl2PlRh a2UgY2FyZSB0aGF0IHVzaW5nIGhhdmFnZWQgb24gYSBWTSBzaG91bGQgbm90IGJlIGNvbnNpZGVy ZWQgYSBnb29kIHNvdXJjZSBvZiBlbnRyb3B5IGFuZCB0aGUgb1ZpcnQgUEtpIGlzIG1hbmFnZWQg YnkgdGhlIGVuZ2luZS48YnI+PGEgaHJlZj0iaHR0cDovL3NlY3VyaXR5LnN0YWNrZXhjaGFuZ2Uu Y29tL3F1ZXN0aW9ucy8zNDUyMy9pcy1pdC1hcHByb3ByaWF0ZS10by11c2UtaGF2ZWdlZC1hcy1h LXNvdXJjZS1vZi1lbnRyb3B5LW9uLXZpcnR1YWwtbWFjaGluZXMiIHRhcmdldD0iX2JsYW5rIj5o dHRwOi8vc2VjdXJpdHkuc3RhY2tleGNoYW5nZS48d2JyPmNvbS9xdWVzdGlvbnMvMzQ1MjMvaXMt aXQtPHdicj5hcHByb3ByaWF0ZS10by11c2UtaGF2ZWdlZC1hcy08d2JyPmEtc291cmNlLW9mLWVu dHJvcHktb24tPHdicj52aXJ0dWFsLW1hY2hpbmVzPC9hPjxicj48YnI+QSBiZXR0ZXIgYXBwcm9h Y2ggaXMgdGhlIHZpcnRpby1ybmcgcGFyYXZpcnR1YWxpc2VkIHJuZyBkcml2ZXIgYXMgZm9yIHBh dGNowqA8YSBocmVmPSJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzYyMzM0LyI+aHR0cHM6 Ly9nZXJyaXQub3ZpcnQub3JnLyMvYy82MjMzNC88L2E+PC9kaXY+PGRpdiBjbGFzcz0iZ21haWxf cXVvdGUiPjxicj48ZGl2PsKgPC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBz dHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2Io MjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVh ay13b3JkIj48ZGl2PjwvZGl2PjxkaXY+VGhhbmsgeW91IGZvciB0aGUgdGlwLiBZb3Ugc29sdmVk IG15IGlzc3VlLjxicj48ZGl2Pgo8ZGl2Pjxicj5DaGVlcnMsPGJyPkdlcnZhaXM8YnI+PGJyPjxi cj48L2Rpdj4KCjwvZGl2PjxkaXY+PGRpdj4KPGJyPjxkaXY+PGJsb2NrcXVvdGUgdHlwZT0iY2l0 ZSI+PGRpdj5PbiBTZXAgMjksIDIwMTYsIGF0IDc6NDcgQU0sIE1hcnRpbiBQZXJpbmEgJmx0Ozxh IGhyZWY9Im1haWx0bzptcGVyaW5hQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5tcGVyaW5h QHJlZGhhdC5jb208L2E+Jmd0OyB3cm90ZTo8L2Rpdj48YnI+PGRpdj48ZGl2IGRpcj0ibHRyIj48 ZGl2PkhpLDxicj48YnI+PC9kaXY+PGRpdj5wbGVhc2UgdGFrZSBhIGxvb2sgYXQgbXkgaW5saW5l IGNvbW1lbnRzOjxicj48L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPjxkaXYgY2xh c3M9ImdtYWlsX3F1b3RlIj5PbiBUdWUsIFNlcCAyNywgMjAxNiBhdCA3OjIzIFBNLCBHZXJ2YWlz IGRlIE1vbnRicnVuIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmdlcnZhaXNA ZGVtb250YnJ1bi5jb20iIHRhcmdldD0iX2JsYW5rIj5nZXJ2YWlzQGRlbW9udGJydW4uY29tPC9h PiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHgg MHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmct bGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFr LXdvcmQiPkhleSBBbGwsPGRpdj48YnI+PC9kaXY+PGRpdj5TaW5jZSB1cGRhdGluZyB0byA0LjAu eCBvZiBvVmlydCwgSSBoYXZlIGhhZCBhbiBpc3N1ZSB3aXRoIG15IGhvc3RlZCBlbmdpbmUuIEFm dGVyIGEgc29tZSBwb2tpbmcgYXJvdW5kLCBJIHRoaW5rIEkgaGF2ZSBmaWd1cmVkIG91dCBteSBp c3N1ZSBhbmQgdGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNlZSB3aGF0IG90aGVycyB0aGluay48 L2Rpdj48ZGl2PlRoZSBpc3N1ZSBoYXMgZXhpc3RlZCB3aXRoIDQuMCwgNC4wLjEsIDQuMC4yLCA0 LjAuMywgYW5kIHN0aWxsIGV4aXN0cyBpbiA0LjAuNC48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2 PkRlc2NyaXB0aW9uOjwvZGl2PjxkaXY+V2hlbiBteSBob3N0ZWQgZW5naW5lIHN0YXJ0cyBpdCBy ZXBvcnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0ZSB3aXRoIDcgb3IgOCBzZXJ2aWNl cyBzdGlsbCBub3Qgc3RhcnRlZCB3aGVuIEkgcnVuIHN5c3RlbWN0bCBzdGF0dXMuIEl0IHRha2Vz IGFib3V0IDYgb3IgNyBtaW51dGVzIHRvIGV2ZW50dWFsbHkgc3RhcnQgYWxsIHRoZSBzZXJ2aWNl cyBhbmQgY29tZSBvbmxpbmUuIElmIEkgZG9uJiMzOTt0IHNldCBteSBjbHVzdGVyIHRvIEdsb2Jh bC1NYWludGVuYW5jZSBtb2RlIGl0IGV2ZW50dWFsbHkgdGhpbmtzIHRoYXQgbXkgaG9zdGVkLWVu Z2luZSBuZWVkcyB0byBiZSByZWJvb3RlZCBhbmQgcmVzdGFydHMgaXQgYmVmb3JlIGl0IGNhbiBz dGFydCBldmVyeXRoaW5nLjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48ZGl2PuKA i0NvdWxkIHlvdSBwbGVhc2Ugc2hhcmUgd2l0aCB1cyBsb2dzIGdhdGhlcmVkIGJ5IG92aXJ0LWxv Zy1jb2xsZWN0b3I/PGJyPjxicj5JdCYjMzk7cyBqdXN0IGEgZ3Vlc3MgYnV0IGNvdWxkIHlvdSBw bGVhc2UgdGFrZSBhIGxvb2sgaWYgeW91IEhFIFZNIGhhcyBlbm91Z2ggZW50cm9weT88YnI+PGJy PsKgIGNhdCAvcHJvYy9zeXMva2VybmVsL3JhbmRvbS9lbnRyb3A8d2JyPnlfYXZhaWw8YnI+PGJy PjwvZGl2PjxkaXY+SWYgdGhlIHZhbHVlIGlzIGxvdyAoYmVsb3cgb3IgYXJvdW5kIDIwMCkswqAg eW91IHJlYWxseSBuZWVkIHRvIGluc3RhbGwgYW5kIGNvbmZpZ3VyZSBzb21lIGVudHJvcHkgZ2Vu ZXJhdG9yIHN1Y2ggYXMgaGF2ZWdlZDxicj48YnI+PC9kaXY+PC9kaXY+PGJsb2NrcXVvdGUgc3R5 bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIw NCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgc3R5 bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj48ZGl2Pjxicj48L2Rpdj48ZGl2PlNvbHV0aW9uOjwv ZGl2PjxkaXY+SSByZWFsaXplZCB0aGF0IEFwYWNoZSB3YXMgdGhlIGN1bHByaXQgYW5kIGZvdW5k IHRoYXQgdGhlIHByb3h5IHRvIHRoZSBvdmlydC1lbmdpbmUgaW7CoC9ldGMvaHR0cGQvY29uZi5k L3otb3ZpcnQtZTx3YnI+bmdpbmUtcHJveHkuY29uZiBoYXMgYSBzdXBlciBsb25nIHRpbWVvdXQg d2l0aCBtYW55IHJldHJpZXMuIEkgY2hhbmdlZCB0aGUgc2V0dGluZ3MgYW5kIG5vdyBldmVyeXRo aW5nIHdvcmtzIGZvciBtZS48L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pi0mZ3Q7IEJlZm9yZSBj aGFuZ2U6PC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCA0MHB4O2Jv cmRlci13aWR0aDptZWRpdW07Ym9yZGVyLXN0eWxlOm5vbmU7cGFkZGluZzowcHgiPjxkaXY+PGRp dj7CoCDCoCAmbHQ7TG9jYXRpb25NYXRjaCBeLyhvdmlydC1lbmdpbmUoJHwvKXxhcGkoJHwvKXw8 d2JyPlJIRVZNYW5hZ2VyV2ViL3xPdmlydEVuZ2luZVdlYjx3YnI+L3xjYS5jcnQkfGVuZ2luZS5z c2gua2V5LnR4dCR8PHdicj5yaGV2bS5zc2gua2V5LnR4dCQpJmd0OzwvZGl2PjxkaXY+wqAgwqAg wqAgwqAgUHJveHlQYXNzTWF0Y2ggPGE+YWpwOi8vMTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9 MzYwMCByZXRyeT01PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZN b2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRP dXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9o dG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiBh cHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7 PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9uTWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9j a3F1b3RlPjxkaXY+PGJyPjwvZGl2Pi0mZ3Q7IEFmdGVyIGNoYW5nZTo8YmxvY2txdW90ZSBzdHls ZT0ibWFyZ2luOjBweCAwcHggMHB4IDQwcHg7Ym9yZGVyLXdpZHRoOm1lZGl1bTtib3JkZXItc3R5 bGU6bm9uZTtwYWRkaW5nOjBweCI+PGRpdj48ZGl2PsKgIMKgICZsdDtMb2NhdGlvbk1hdGNoIF4v b3ZpcnQtZW5naW5lKCR8LykmZ3Q7PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCBQcm94eVBhc3NNYXRj aCA8YT5hanA6Ly8xMjcuMC4wLjE6ODcwMjwvYT4gdGltZW91dD01IHJldHJ5PTI8L2Rpdj48ZGl2 Pjxicj48L2Rpdj48ZGl2PsKgIMKgIMKgIMKgICZsdDtJZk1vZHVsZSBkZWZsYXRlX21vZHVsZSZn dDs8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgIMKgIMKgIEFkZE91dHB1dEZpbHRlckJ5VHlwZSBERUZM QVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29u IGFwcGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uIGFwcGxpY2F0aW9uL3gteWFtbDwvZGl2 PjxkaXY+wqAgwqAgwqAgwqAgJmx0Oy9JZk1vZHVsZSZndDs8L2Rpdj48ZGl2PsKgIMKgICZsdDsv TG9jYXRpb25NYXRjaCZndDs8L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9kaXY+PC9ibG9ja3F1 b3RlPjxkaXY+PGJyPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5z LXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igItUaGlzIG9uZSBpcyBjb3JyZWN0IGZvciA0LjDigIs8 L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtk aXNwbGF5OmlubGluZSI+4oCLLCBub3Qgc3VyZSB3aHkgaXQgd2FzIG5vdCB1cGRhdGVkIGR1cmlu ZyB1cGdyYWRlIGZyb20gMy42LiBAU2ltb25lPzxicj7igIs8L2Rpdj7CoDwvZGl2PjxibG9ja3F1 b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlk IHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48 ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+PGRpdj48YnI+PC9kaXY+SWYgSSByZWFk IHRoZSB0aW1lb3V0IHNldHRpbmdzIGNvcnJlY3RseSwgaXQgd2lsbCB3YWl0IDYwIG1pbnV0ZXMg d2l0aCA1IHJldHJpZXMuIDUgaG91cnMgaXMgd2F5IHRvbyBsb25nIGZvciBteSBsaXR0bGUgc2Vy dmVyIHRvIGhvbGQgb250byBhbGwgdGhvc2UgYXBhY2hlIHByb2Nlc3Nlcy4gPC9kaXY+PC9ibG9j a3F1b3RlPjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVy LWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9 ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+VGhlIGNoYW5n ZSBJIG1hZGUgYWxsb3dzIGZvciB0aGVyZSB0byBiZSBhbiBlcnJvciwgYW5kIGFsc28gcmVsZWFz ZXMgYXBhY2hlJiMzOTtzIGhvbGQgb24gdGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyBy ZWFkeSwgYXBhY2hlIGlzIHJlYWR5IHRvIHNlcnZlIHJlcXVlc3RzIGFuZCBldmVyeXRoaW5nL2V2 ZXJ5b25lIGlzIGhhcHB5LiBCZWZvcmUgbWFraW5nIHRoZSBjaGFuZ2UsIEkganVzdCBnZXQgYSB3 aGl0ZXNjcmVlbiBpbiBteSBicm93c2VyIGFuZCB0aGVuIG5vdGhpbmcgd29ya3MgdW50aWwgSSBy ZXN0YXJ0IEFwYWNoZSAob3IgSSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhh IHNlcnZpY2VzIHJlc3RhcnRpbmcgbXkgaG9zdGVkLWVuZ2luZS48YnI+PC9kaXY+PC9ibG9ja3F1 b3RlPjxkaXY+PGJyPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5z LXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igItXZWxsLCBpZiB5b3UgaGF2ZSBhbiBpc3N1ZSB3aXRo IHRvbyBtYW55IGFwYWNoZSBwcm9jZXNzZXMgd2FpdGluZyBmb3IgZW5naW5lIHRvIHJlc3BvbmQs IHRoZW4gdGhlcmUmIzM5O3Mgc29tZSBpc3N1ZSBpbiBlbmdpbmUuIEFzIEkgd3JvdGUgYWJvdmUg cGxlYXNlIHNoYXJlIHRoZSBsb2dzIHdpdGggdXMgYW5kIGNoZWNrIGVudHJvcHkuPGJyPjxicj48 L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtk aXNwbGF5OmlubGluZSI+VGhhbmtzPGJyPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWls eTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+TWFydGluIFBlcmlu YTxicj7igIs8L2Rpdj7CoDwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAw cHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1s ZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWst d29yZCI+PGRpdj48ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBub3RpY2VkIHRoYXQgdGhpcyBz ZXR0aW5nIHJldmVydHMgdG8gdGhlIG9yaWdpbmFsIHNldHRpbmcsIHNvIG9WaXJ0IG11c3QgYmUg d3JpdGluZyB0aGlzIGZpbGUuIFBlcmhhcHMgdGhlc2UgbnVtYmVyIGNhbiBiZSBjaGFuZ2VkIGlu IG9WaXJ0PyBJZiBub3QsIEkgd2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkgdG8gcmV2 ZXJ0IHRoZSBzZXR0aW5ncyB3aXRoIHdvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBv biBteSBlbmdpbmUuPC9kaXY+PGRpdj46LSk8L2Rpdj48ZGl2Pgo8ZGl2Pjxicj5DaGVlcnMsPGJy PkdlcnZhaXM8YnI+PGJyPjxicj48L2Rpdj4KCjwvZGl2PgoKCgo8YnI+PC9kaXY+PC9kaXY+PC9k aXY+PGJyPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnI+X19fX19fX19fX19fX19f X188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEgaHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0 Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0Lm9yZzwvYT48YnI+CjxhIHJlbD0ibm9y ZWZlcnJlciIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Vz ZXJzIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4v bGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+PC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rpdj48 L2Rpdj4KPC9kaXY+PC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rp dj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo= --===============3935525266990280390==-- From stirabos at redhat.com Thu Sep 29 09:04:50 2016 Content-Type: multipart/mixed; boundary="===============6782051326175534664==" MIME-Version: 1.0 From: Simone Tiraboschi To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 15:04:18 +0200 Message-ID: In-Reply-To: CAP5iht7RWuTxuJT42eiCmQ_ZtPQc_Xq7WZ7Go3Aj69eHcvGJhw@mail.gmail.com --===============6782051326175534664== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 29, 2016 at 12:47 PM, Martin Perina wrot= e: > Hi, > > please take a look at my inline comments: > > On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun < > gervais(a)demontbrun.com> wrote: > >> Hey All, >> >> Since updating to 4.0.x of oVirt, I have had an issue with my hosted >> engine. After a some poking around, I think I have figured out my issue = and >> thought I would share to see what others think. >> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in >> 4.0.4. >> >> Description: >> When my hosted engine starts it reports that it is in a degraded state >> with 7 or 8 services still not started when I run systemctl status. It >> takes about 6 or 7 minutes to eventually start all the services and come >> online. If I don't set my cluster to Global-Maintenance mode it eventual= ly >> thinks that my hosted-engine needs to be rebooted and restarts it before= it >> can start everything. >> > > =E2=80=8BCould you please share with us logs gathered by ovirt-log-collec= tor? > > It's just a guess but could you please take a look if you HE VM has enough > entropy? > > cat /proc/sys/kernel/random/entropy_avail > > If the value is low (below or around 200), you really need to install and > configure some entropy generator such as haveged > > >> Solution: >> I realized that Apache was the culprit and found that the proxy to the >> ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super >> long timeout with many retries. I changed the settings and now everything >> works for me. >> >> -> Before change: >> >> > RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$| >> rhevm.ssh.key.txt$)> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> >> -> After change: >> >> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> >> > =E2=80=8BThis one is correct for 4.0=E2=80=8B > =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Simo= ne? > =E2=80=8B > Honestly it's ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 AddOutputFilterByType DEFLATE text/javascript text/css text/html text/xml text/json application/xml application/json application/x-yaml also on a fresh 4.0 engine from our latest engine-appliance. > >> If I read the timeout settings correctly, it will wait 60 minutes with 5 >> retries. 5 hours is way too long for my little server to hold onto all >> those apache processes. >> > The change I made allows for there to be an error, and also releases >> apache's hold on the process. Once everything is ready, apache is ready = to >> serve requests and everything/everyone is happy. Before making the chang= e, >> I just get a whitescreen in my browser and then nothing works until I >> restart Apache (or I end up in an endless loop of ovirt-ha services >> restarting my hosted-engine. >> > > =E2=80=8BWell, if you have an issue with too many apache processes waitin= g for > engine to respond, then there's some issue in engine. As I wrote above > please share the logs with us and check entropy. > > Thanks > > Martin Perina > =E2=80=8B > > >> >> I noticed that this setting reverts to the original setting, so oVirt >> must be writing this file. Perhaps these number can be changed in oVirt?= If >> not, I will just setup and ansible play to revert the settings with work= ing >> values and restart apache on my engine. >> :-) >> >> Cheers, >> Gervais >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> > --===============6782051326175534664== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFRodSwgU2VwIDI5LCAyMDE2IGF0IDEyOjQ3IFBNLCBNYXJ0aW4g UGVyaW5hIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOm1wZXJpbmFAcmVkaGF0 LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPm1wZXJpbmFAcmVkaGF0LmNvbTwvYT4mZ3Q7PC9zcGFuPiB3 cm90ZTo8YnI+PGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBw eCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3Bh ZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxkaXY+SGksPGJyPjxicj48L2Rpdj48ZGl2 PnBsZWFzZSB0YWtlIGEgbG9vayBhdCBteSBpbmxpbmUgY29tbWVudHM6PGJyPjwvZGl2PjxkaXYg Y2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxzcGFuIGNs YXNzPSJnbWFpbC0iPk9uIFR1ZSwgU2VwIDI3LCAyMDE2IGF0IDc6MjMgUE0sIEdlcnZhaXMgZGUg TW9udGJydW4gPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86Z2VydmFpc0BkZW1v bnRicnVuLmNvbSIgdGFyZ2V0PSJfYmxhbmsiPmdlcnZhaXNAZGVtb250YnJ1bi5jb208L2E+Jmd0 Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHgg MC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0 OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29y ZCI+SGV5IEFsbCw8ZGl2Pjxicj48L2Rpdj48ZGl2PlNpbmNlIHVwZGF0aW5nIHRvIDQuMC54IG9m IG9WaXJ0LCBJIGhhdmUgaGFkIGFuIGlzc3VlIHdpdGggbXkgaG9zdGVkIGVuZ2luZS4gQWZ0ZXIg YSBzb21lIHBva2luZyBhcm91bmQsIEkgdGhpbmsgSSBoYXZlIGZpZ3VyZWQgb3V0IG15IGlzc3Vl IGFuZCB0aG91Z2h0IEkgd291bGQgc2hhcmUgdG8gc2VlIHdoYXQgb3RoZXJzIHRoaW5rLjwvZGl2 PjxkaXY+VGhlIGlzc3VlIGhhcyBleGlzdGVkIHdpdGggNC4wLCA0LjAuMSwgNC4wLjIsIDQuMC4z LCBhbmQgc3RpbGwgZXhpc3RzIGluIDQuMC40LjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+RGVz Y3JpcHRpb246PC9kaXY+PGRpdj5XaGVuIG15IGhvc3RlZCBlbmdpbmUgc3RhcnRzIGl0IHJlcG9y dHMgdGhhdCBpdCBpcyBpbiBhIGRlZ3JhZGVkIHN0YXRlIHdpdGggNyBvciA4IHNlcnZpY2VzIHN0 aWxsIG5vdCBzdGFydGVkIHdoZW4gSSBydW4gc3lzdGVtY3RsIHN0YXR1cy4gSXQgdGFrZXMgYWJv dXQgNiBvciA3IG1pbnV0ZXMgdG8gZXZlbnR1YWxseSBzdGFydCBhbGwgdGhlIHNlcnZpY2VzIGFu ZCBjb21lIG9ubGluZS4gSWYgSSBkb24mIzM5O3Qgc2V0IG15IGNsdXN0ZXIgdG8gR2xvYmFsLU1h aW50ZW5hbmNlIG1vZGUgaXQgZXZlbnR1YWxseSB0aGlua3MgdGhhdCBteSBob3N0ZWQtZW5naW5l IG5lZWRzIHRvIGJlIHJlYm9vdGVkIGFuZCByZXN0YXJ0cyBpdCBiZWZvcmUgaXQgY2FuIHN0YXJ0 IGV2ZXJ5dGhpbmcuPC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjwvc3Bhbj48ZGl2Pjxicj48ZGl2 PuKAi0NvdWxkIHlvdSBwbGVhc2Ugc2hhcmUgd2l0aCB1cyBsb2dzIGdhdGhlcmVkIGJ5IG92aXJ0 LWxvZy1jb2xsZWN0b3I/PGJyPjxicj5JdCYjMzk7cyBqdXN0IGEgZ3Vlc3MgYnV0IGNvdWxkIHlv dSBwbGVhc2UgdGFrZSBhIGxvb2sgaWYgeW91IEhFIFZNIGhhcyBlbm91Z2ggZW50cm9weT88YnI+ PGJyPsKgIGNhdCAvcHJvYy9zeXMva2VybmVsL3JhbmRvbS88d2JyPmVudHJvcHlfYXZhaWw8YnI+ PGJyPjwvZGl2PjxkaXY+SWYgdGhlIHZhbHVlIGlzIGxvdyAoYmVsb3cgb3IgYXJvdW5kIDIwMCks wqAgeW91IHJlYWxseSBuZWVkIHRvIGluc3RhbGwgYW5kIGNvbmZpZ3VyZSBzb21lIGVudHJvcHkg Z2VuZXJhdG9yIHN1Y2ggYXMgaGF2ZWdlZDxicj48YnI+PC9kaXY+PC9kaXY+PHNwYW4gY2xhc3M9 ImdtYWlsLSI+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3Jk ZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFz cz0iZ21haWxfcXVvdGUiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj48ZGl2Pjxi cj48L2Rpdj48ZGl2PlNvbHV0aW9uOjwvZGl2PjxkaXY+SSByZWFsaXplZCB0aGF0IEFwYWNoZSB3 YXMgdGhlIGN1bHByaXQgYW5kIGZvdW5kIHRoYXQgdGhlIHByb3h5IHRvIHRoZSBvdmlydC1lbmdp bmUgaW7CoC9ldGMvaHR0cGQvY29uZi5kL3otb3ZpcnQtZTx3YnI+bmdpbmUtcHJveHkuY29uZiBo YXMgYSBzdXBlciBsb25nIHRpbWVvdXQgd2l0aCBtYW55IHJldHJpZXMuIEkgY2hhbmdlZCB0aGUg c2V0dGluZ3MgYW5kIG5vdyBldmVyeXRoaW5nIHdvcmtzIGZvciBtZS48L2Rpdj48ZGl2Pjxicj48 L2Rpdj48ZGl2Pi0mZ3Q7IEJlZm9yZSBjaGFuZ2U6PC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1h cmdpbjowcHggMHB4IDBweCA0MHB4O2JvcmRlci13aWR0aDptZWRpdW07Ym9yZGVyLXN0eWxlOm5v bmU7cGFkZGluZzowcHgiPjxkaXY+PGRpdj7CoCDCoCAmbHQ7TG9jYXRpb25NYXRjaCBeLyhvdmly dC1lbmdpbmUoJHwvKXxhcGkoJHwvKXw8d2JyPlJIRVZNYW5hZ2VyV2ViL3xPdmlydEVuZ2luZVdl Yjx3YnI+L3xjYS5jcnQkfGVuZ2luZS5zc2gua2V5LnR4dCR8PHdicj5yaGV2bS5zc2gua2V5LnR4 dCQpJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgUHJveHlQYXNzTWF0Y2ggPGE+YWpwOi8vMTI3 LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9MzYwMCByZXRyeT01PC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRp dj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0L2ph dmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlv bi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKgIMKg IMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9uTWF0 Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+PGJyPjwvZGl2Pi0mZ3Q7IEFmdGVy IGNoYW5nZTo8YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDQwcHg7Ym9yZGVy LXdpZHRoOm1lZGl1bTtib3JkZXItc3R5bGU6bm9uZTtwYWRkaW5nOjBweCI+PGRpdj48ZGl2PsKg IMKgICZsdDtMb2NhdGlvbk1hdGNoIF4vb3ZpcnQtZW5naW5lKCR8LykmZ3Q7PC9kaXY+PGRpdj7C oCDCoCDCoCDCoCBQcm94eVBhc3NNYXRjaCA8YT5hanA6Ly8xMjcuMC4wLjE6ODcwMjwvYT4gdGlt ZW91dD01IHJldHJ5PTI8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PsKgIMKgIMKgIMKgICZsdDtJ Zk1vZHVsZSBkZWZsYXRlX21vZHVsZSZndDs8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgIMKgIMKgIEFk ZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0 L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uIGFwcGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29u IGFwcGxpY2F0aW9uL3gteWFtbDwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgJmx0Oy9JZk1vZHVsZSZn dDs8L2Rpdj48ZGl2PsKgIMKgICZsdDsvTG9jYXRpb25NYXRjaCZndDs8L2Rpdj48L2Rpdj48L2Js b2NrcXVvdGU+PC9kaXY+PC9ibG9ja3F1b3RlPjwvc3Bhbj48ZGl2Pjxicj48ZGl2IHN0eWxlPSJm b250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+4oCL VGhpcyBvbmUgaXMgY29ycmVjdCBmb3IgNC4w4oCLPC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1p bHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPuKAiywgbm90IHN1 cmUgd2h5IGl0IHdhcyBub3QgdXBkYXRlZCBkdXJpbmcgdXBncmFkZSBmcm9tIDMuNi4gQFNpbW9u ZT88YnI+4oCLPC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXY+ PGJyPjwvZGl2PjxkaXY+SG9uZXN0bHkgaXQmIzM5O3M8L2Rpdj48ZGl2PjxkaXY+wqAgwqAgJmx0 O0xvY2F0aW9uTWF0Y2ggXi9vdmlydC1lbmdpbmUoJHwvKSZndDs8L2Rpdj48ZGl2PsKgIMKgIMKg IMKgIFByb3h5UGFzc01hdGNoIGFqcDovLzxhIGhyZWY9Imh0dHA6Ly8xMjcuMC4wLjE6ODcwMiI+ MTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9MzYwMCByZXRyeT01PC9kaXY+PGRpdj48YnI+PC9k aXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+ PGRpdj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0 L2phdmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNh dGlvbi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKg IMKgIMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9u TWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PGRpdj5hbHNvIG9uIGEgZnJlc2ggNC4wIGVuZ2luZSBmcm9t IG91ciBsYXRlc3QgZW5naW5lLWFwcGxpYW5jZS48L2Rpdj48ZGl2PsKgPC9kaXY+PGJsb2NrcXVv dGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2Jv cmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxk aXYgZGlyPSJsdHIiPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48ZGl2IGNsYXNzPSJnbWFpbF9x dW90ZSI+PHNwYW4gY2xhc3M9ImdtYWlsLSI+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHgg MHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRk aW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpi cmVhay13b3JkIj48ZGl2Pjxicj48L2Rpdj5JZiBJIHJlYWQgdGhlIHRpbWVvdXQgc2V0dGluZ3Mg Y29ycmVjdGx5LCBpdCB3aWxsIHdhaXQgNjAgbWludXRlcyB3aXRoIDUgcmV0cmllcy4gNSBob3Vy cyBpcyB3YXkgdG9vIGxvbmcgZm9yIG15IGxpdHRsZSBzZXJ2ZXIgdG8gaG9sZCBvbnRvIGFsbCB0 aG9zZSBhcGFjaGUgcHJvY2Vzc2VzLiA8L2Rpdj48L2Jsb2NrcXVvdGU+PGJsb2NrcXVvdGUgc3R5 bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIw NCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgc3R5 bGU9IndvcmQtd3JhcDpicmVhay13b3JkIj5UaGUgY2hhbmdlIEkgbWFkZSBhbGxvd3MgZm9yIHRo ZXJlIHRvIGJlIGFuIGVycm9yLCBhbmQgYWxzbyByZWxlYXNlcyBhcGFjaGUmIzM5O3MgaG9sZCBv biB0aGUgcHJvY2Vzcy4gT25jZSBldmVyeXRoaW5nIGlzIHJlYWR5LCBhcGFjaGUgaXMgcmVhZHkg dG8gc2VydmUgcmVxdWVzdHMgYW5kIGV2ZXJ5dGhpbmcvZXZlcnlvbmUgaXMgaGFwcHkuIEJlZm9y ZSBtYWtpbmcgdGhlIGNoYW5nZSwgSSBqdXN0IGdldCBhIHdoaXRlc2NyZWVuIGluIG15IGJyb3dz ZXIgYW5kIHRoZW4gbm90aGluZyB3b3JrcyB1bnRpbCBJIHJlc3RhcnQgQXBhY2hlIChvciBJIGVu ZCB1cCBpbiBhbiBlbmRsZXNzIGxvb3Agb2Ygb3ZpcnQtaGEgc2VydmljZXMgcmVzdGFydGluZyBt eSBob3N0ZWQtZW5naW5lLjxicj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9zcGFuPjxkaXY+PGJyPjxk aXYgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6 aW5saW5lIj7igItXZWxsLCBpZiB5b3UgaGF2ZSBhbiBpc3N1ZSB3aXRoIHRvbyBtYW55IGFwYWNo ZSBwcm9jZXNzZXMgd2FpdGluZyBmb3IgZW5naW5lIHRvIHJlc3BvbmQsIHRoZW4gdGhlcmUmIzM5 O3Mgc29tZSBpc3N1ZSBpbiBlbmdpbmUuIEFzIEkgd3JvdGUgYWJvdmUgcGxlYXNlIHNoYXJlIHRo ZSBsb2dzIHdpdGggdXMgYW5kIGNoZWNrIGVudHJvcHkuPGJyPjxicj48L2Rpdj48ZGl2IHN0eWxl PSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+ VGhhbmtzPGJyPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRp Y2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+TWFydGluIFBlcmluYTxicj7igIs8L2Rpdj7C oDwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVy LWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9 ImdtYWlsX3F1b3RlIj48c3BhbiBjbGFzcz0iZ21haWwtIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6 YnJlYWstd29yZCI+PGRpdj48ZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+SSBub3RpY2VkIHRoYXQg dGhpcyBzZXR0aW5nIHJldmVydHMgdG8gdGhlIG9yaWdpbmFsIHNldHRpbmcsIHNvIG9WaXJ0IG11 c3QgYmUgd3JpdGluZyB0aGlzIGZpbGUuIFBlcmhhcHMgdGhlc2UgbnVtYmVyIGNhbiBiZSBjaGFu Z2VkIGluIG9WaXJ0PyBJZiBub3QsIEkgd2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkg dG8gcmV2ZXJ0IHRoZSBzZXR0aW5ncyB3aXRoIHdvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFw YWNoZSBvbiBteSBlbmdpbmUuPC9kaXY+PGRpdj46LSk8L2Rpdj48ZGl2Pgo8ZGl2Pjxicj5DaGVl cnMsPGJyPkdlcnZhaXM8YnI+PGJyPjxicj48L2Rpdj4KCjwvZGl2PgoKCgo8YnI+PC9kaXY+PC9k aXY+PC9kaXY+PGJyPjwvc3Bhbj5fX19fX19fX19fX19fX19fX19fX19fX19fX19fX188d2JyPl9f X19fX19fX19fX19fX19fPGJyPgpVc2VycyBtYWlsaW5nIGxpc3Q8YnI+CjxhIGhyZWY9Im1haWx0 bzpVc2Vyc0BvdmlydC5vcmciIHRhcmdldD0iX2JsYW5rIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJy Pgo8YSByZWw9Im5vcmVmZXJyZXIiIGhyZWY9Imh0dHA6Ly9saXN0cy5vdmlydC5vcmcvbWFpbG1h bi9saXN0aW5mby91c2VycyIgdGFyZ2V0PSJfYmxhbmsiPmh0dHA6Ly9saXN0cy5vdmlydC5vcmcv bWFpbG1hbjx3YnI+L2xpc3RpbmZvL3VzZXJzPC9hPjxicj4KPGJyPjwvYmxvY2txdW90ZT48L2Rp dj48YnI+PC9kaXY+PC9kaXY+CjwvYmxvY2txdW90ZT48L2Rpdj48YnI+PC9kaXY+PC9kaXY+Cg== --===============6782051326175534664==-- From gervais at demontbrun.com Thu Sep 29 09:10:29 2016 Content-Type: multipart/mixed; boundary="===============3067116927732368603==" MIME-Version: 1.0 From: Gervais de Montbrun To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 10:10:23 -0300 Message-ID: <48834371-4D3A-42B8-BD52-666DFB6B5FF2@demontbrun.com> In-Reply-To: CAN8-ONpupxzhgjCCc1Oc1he_qr3OU8r9b6wzPZt2nNS=YL=-Fw@mail.gmail.com --===============3067116927732368603== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_19C0B956-90E3-4499-AC9F-B50879BD6D0A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 Hi Simone, Yes... I guess it was not clear in my original email. I changed the =3D numbers myself to lower the timeout and retries. With them set as they =3D were set by ovirt (timeout=3D3D3600 retry=3D3D5) things were not working fo= r =3D me.=3D20 Cheers, Gervais > On Sep 29, 2016, at 10:04 AM, Simone Tiraboschi = =3D wrote: >=3D20 >=3D20 >=3D20 > On Thu, Sep 29, 2016 at 12:47 PM, Martin Perina > wrote: > Hi, >=3D20 > please take a look at my inline comments: >=3D20 > On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun =3D > wrote: > Hey All, >=3D20 > Since updating to 4.0.x of oVirt, I have had an issue with my hosted =3D engine. After a some poking around, I think I have figured out my issue =3D and thought I would share to see what others think. > The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists =3D in 4.0.4. >=3D20 > Description: > When my hosted engine starts it reports that it is in a degraded state = =3D with 7 or 8 services still not started when I run systemctl status. It =3D takes about 6 or 7 minutes to eventually start all the services and come = =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start everything. >=3D20 > =3DE2=3D80=3D8BCould you please share with us logs gathered by =3D ovirt-log-collector? >=3D20 > It's just a guess but could you please take a look if you HE VM has =3D enough entropy? >=3D20 > cat /proc/sys/kernel/random/entropy_avail >=3D20 > If the value is low (below or around 200), you really need to install = =3D and configure some entropy generator such as haveged >=3D20 >=3D20 > Solution: > I realized that Apache was the culprit and found that the proxy to the = =3D ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super =3D long timeout with many retries. I changed the settings and now =3D everything works for me. >=3D20 > -> Before change: > > ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D3600 retry=3D3= D5 >=3D20 > > AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml > > >=3D20 > -> After change: > > ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D5 retry=3D3D2 >=3D20 > > AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml > > >=3D20 > =3DE2=3D80=3D8BThis one is correct for 4.0=3DE2=3D80=3D8B=3DE2=3D80=3D8B,= not sure why =3D it was not updated during upgrade from 3.6. @Simone? > =3DE2=3D80=3D8B >=3D20 > Honestly it's > > ProxyPassMatch ajp://127.0.0.1:8702 =3D timeout=3D3D3600 retry=3D3D5 >=3D20 > > AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml > > > also on a fresh 4.0 engine from our latest engine-appliance. > =3D20 >=3D20 > If I read the timeout settings correctly, it will wait 60 minutes with = =3D 5 retries. 5 hours is way too long for my little server to hold onto all = =3D those apache processes. > The change I made allows for there to be an error, and also releases =3D apache's hold on the process. Once everything is ready, apache is ready =3D to serve requests and everything/everyone is happy. Before making the =3D change, I just get a whitescreen in my browser and then nothing works =3D until I restart Apache (or I end up in an endless loop of ovirt-ha =3D services restarting my hosted-engine. >=3D20 > =3DE2=3D80=3D8BWell, if you have an issue with too many apache processes = =3D waiting for engine to respond, then there's some issue in engine. As I =3D wrote above please share the logs with us and check entropy. >=3D20 > Thanks >=3D20 > Martin Perina > =3DE2=3D80=3D8B=3D20 >=3D20 > I noticed that this setting reverts to the original setting, so oVirt =3D must be writing this file. Perhaps these number can be changed in oVirt? = =3D If not, I will just setup and ansible play to revert the settings with =3D working values and restart apache on my engine. > :-) >=3D20 > Cheers, > Gervais >=3D20 >=3D20 >=3D20 >=3D20 > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users =3D >=3D20 >=3D20 >=3D20 --Apple-Mail=3D_19C0B956-90E3-4499-AC9F-B50879BD6D0A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8 Hi Simone,

Yes... I guess it was not clear in my original email. I =3D changed the numbers myself to lower the timeout and retries. With them =3D set as they were set by ovirt (timeout=3D3D3600 retry=3D3D5) things were no= t =3D working for me. 

Cheers,
Gervais



On Sep 29, 2016, at 10:04 AM, Simone Tiraboschi <stirabos(a)redhat.com<= /a>>=3D wrote:


On Thu, Sep 29, 2016 at 12:47 PM,= =3D Martin Perina <mperina(a)redhat.com> wrote:
Hi,

please take a look at my inline =3D comments:

On Tue, = Sep =3D 27, 2016 at 7:23 PM, Gervais de Montbrun <gervais(a)demontbrun.com> wrote:
Hey All,
Since updating to 4.0.x of oVirt, I = =3D have had an issue with my hosted engine. After a some poking around, I =3D think I have figured out my issue and thought I would share to see what =3D others think.
The issue has existed with 4.0, =3D 4.0.1, 4.0.2, 4.0.3, and still exists in 4.0.4.

Description:
Wh= en =3D my hosted engine starts it reports that it is in a degraded state with 7 = =3D or 8 services still not started when I run systemctl status. It takes =3D about 6 or 7 minutes to eventually start all the services and come =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start =3D everything.

=3DE2=3D80=3D8BCould you please share with u= s logs =3D gathered by ovirt-log-collector?

It's just= =3D a guess but could you please take a look if you HE VM has enough =3D entropy?

  cat =3D /proc/sys/kernel/random/entropy_avail
If the value is low (below or around = =3D 200),  you really need to install and configure some entropy =3D generator such as haveged

=

Solution:
I realized that Apache was the culprit and found that the =3D proxy to the ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super long timeout with many retries. = =3D I changed the settings and now everything works for me.

-> Before =3D change:
    <LocationMatch =3D ^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$|rhevm.ssh.key.txt$)>
    = =3D     ProxyPassMatch ajp://127.0.0.1:8702 =3D timeout=3D3D3600 retry=3D3D5

=
        <IfModule =3D deflate_module>
        =3D     AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml
        =3D </IfModule>
    =3D </LocationMatch>

-> After change:
    =3D <LocationMatch ^/ovirt-engine($|/)>

        <IfModule =3D deflate_module>
<= =3D div class=3D3D"">
=3DE2=3D80=3D8BThis one is correct for 4.0=3DE2=3D80=3D8B
=3DE2=3D80=3D8B, not sure why it was not updated during upgrad= e =3D from 3.6. @Simone?
=3DE2=3D80=3D8B

    <LocationMatch =3D ^/ovirt-engine($|/)>

      = =3D   <IfModule deflate_module>
  =3D           AddOutputFilterByType DEFLATE =3D text/javascript text/css text/html text/xml text/json application/xml =3D application/json application/x-yaml
    = =3D     </IfModule>
    =3D </LocationMatch>
also on a fresh 4.0 =3D engine from our latest engine-appliance.
 
=

If I read the timeout settings =3D correctly, it will wait 60 minutes with 5 retries. 5 hours is way too =3D long for my little server to hold onto all those apache processes. =3D
= The =3D change I made allows for there to be an error, and also releases =3D apache's hold on the process. Once everything is ready, apache is ready =3D to serve requests and everything/everyone is happy. Before making the =3D change, I just get a whitescreen in my browser and then nothing works =3D until I restart Apache (or I end up in an endless loop of ovirt-ha =3D services restarting my hosted-engine.

<= div =3D style=3D3D"font-family:arial,helvetica,sans-serif;display:inline" =3D class=3D3D"">=3DE2=3D80=3D8BWell, if you have an issue with too many apache= =3D processes waiting for engine to respond, then there's some issue in =3D engine. As I wrote above please share the logs with us and check =3D entropy.

Thanks

Martin Perina
=3DE2=3D80=3D8B
 

I noticed that this setting reverts to the original setting, = =3D so oVirt must be writing this file. Perhaps these number can be changed =3D in oVirt? If not, I will just setup and ansible play to revert the =3D settings with working values and restart apache on my engine.
:-)

Cheers,
Gervais




_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users




=3D --Apple-Mail=3D_19C0B956-90E3-4499-AC9F-B50879BD6D0A-- --===============3067116927732368603== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fMTlDMEI5NTYtOTBFMy00NDk5LUFDOUYtQjUwODc5QkQ2RDBBCkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKSGkgU2ltb25lLAoKWWVzLi4uIEkgZ3Vlc3MgaXQgd2Fz IG5vdCBjbGVhciBpbiBteSBvcmlnaW5hbCBlbWFpbC4gSSBjaGFuZ2VkIHRoZSA9Cm51bWJlcnMg bXlzZWxmIHRvIGxvd2VyIHRoZSB0aW1lb3V0IGFuZCByZXRyaWVzLiBXaXRoIHRoZW0gc2V0IGFz IHRoZXkgPQp3ZXJlIHNldCBieSBvdmlydCAodGltZW91dD0zRDM2MDAgcmV0cnk9M0Q1KSB0aGlu Z3Mgd2VyZSBub3Qgd29ya2luZyBmb3IgPQptZS49MjAKCkNoZWVycywKR2VydmFpcwoKCgo+IE9u IFNlcCAyOSwgMjAxNiwgYXQgMTA6MDQgQU0sIFNpbW9uZSBUaXJhYm9zY2hpIDxzdGlyYWJvc0By ZWRoYXQuY29tPiA9Cndyb3RlOgo+PTIwCj49MjAKPj0yMAo+IE9uIFRodSwgU2VwIDI5LCAyMDE2 IGF0IDEyOjQ3IFBNLCBNYXJ0aW4gUGVyaW5hIDxtcGVyaW5hQHJlZGhhdC5jb20gPQo8bWFpbHRv Om1wZXJpbmFAcmVkaGF0LmNvbT4+IHdyb3RlOgo+IEhpLAo+PTIwCj4gcGxlYXNlIHRha2UgYSBs b29rIGF0IG15IGlubGluZSBjb21tZW50czoKPj0yMAo+IE9uIFR1ZSwgU2VwIDI3LCAyMDE2IGF0 IDc6MjMgUE0sIEdlcnZhaXMgZGUgTW9udGJydW4gPQo8Z2VydmFpc0BkZW1vbnRicnVuLmNvbSA8 bWFpbHRvOmdlcnZhaXNAZGVtb250YnJ1bi5jb20+PiB3cm90ZToKPiBIZXkgQWxsLAo+PTIwCj4g U2luY2UgdXBkYXRpbmcgdG8gNC4wLnggb2Ygb1ZpcnQsIEkgaGF2ZSBoYWQgYW4gaXNzdWUgd2l0 aCBteSBob3N0ZWQgPQplbmdpbmUuIEFmdGVyIGEgc29tZSBwb2tpbmcgYXJvdW5kLCBJIHRoaW5r IEkgaGF2ZSBmaWd1cmVkIG91dCBteSBpc3N1ZSA9CmFuZCB0aG91Z2h0IEkgd291bGQgc2hhcmUg dG8gc2VlIHdoYXQgb3RoZXJzIHRoaW5rLgo+IFRoZSBpc3N1ZSBoYXMgZXhpc3RlZCB3aXRoIDQu MCwgNC4wLjEsIDQuMC4yLCA0LjAuMywgYW5kIHN0aWxsIGV4aXN0cyA9CmluIDQuMC40Lgo+PTIw Cj4gRGVzY3JpcHRpb246Cj4gV2hlbiBteSBob3N0ZWQgZW5naW5lIHN0YXJ0cyBpdCByZXBvcnRz IHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0ZSA9CndpdGggNyBvciA4IHNlcnZpY2VzIHN0 aWxsIG5vdCBzdGFydGVkIHdoZW4gSSBydW4gc3lzdGVtY3RsIHN0YXR1cy4gSXQgPQp0YWtlcyBh Ym91dCA2IG9yIDcgbWludXRlcyB0byBldmVudHVhbGx5IHN0YXJ0IGFsbCB0aGUgc2VydmljZXMg YW5kIGNvbWUgPQpvbmxpbmUuIElmIEkgZG9uJ3Qgc2V0IG15IGNsdXN0ZXIgdG8gR2xvYmFsLU1h aW50ZW5hbmNlIG1vZGUgaXQgPQpldmVudHVhbGx5IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1lbmdp bmUgbmVlZHMgdG8gYmUgcmVib290ZWQgYW5kID0KcmVzdGFydHMgaXQgYmVmb3JlIGl0IGNhbiBz dGFydCBldmVyeXRoaW5nLgo+PTIwCj4gPUUyPTgwPThCQ291bGQgeW91IHBsZWFzZSBzaGFyZSB3 aXRoIHVzIGxvZ3MgZ2F0aGVyZWQgYnkgPQpvdmlydC1sb2ctY29sbGVjdG9yPwo+PTIwCj4gSXQn cyBqdXN0IGEgZ3Vlc3MgYnV0IGNvdWxkIHlvdSBwbGVhc2UgdGFrZSBhIGxvb2sgaWYgeW91IEhF IFZNIGhhcyA9CmVub3VnaCBlbnRyb3B5Pwo+PTIwCj4gICBjYXQgL3Byb2Mvc3lzL2tlcm5lbC9y YW5kb20vZW50cm9weV9hdmFpbAo+PTIwCj4gSWYgdGhlIHZhbHVlIGlzIGxvdyAoYmVsb3cgb3Ig YXJvdW5kIDIwMCksICB5b3UgcmVhbGx5IG5lZWQgdG8gaW5zdGFsbCA9CmFuZCBjb25maWd1cmUg c29tZSBlbnRyb3B5IGdlbmVyYXRvciBzdWNoIGFzIGhhdmVnZWQKPj0yMAo+PTIwCj4gU29sdXRp b246Cj4gSSByZWFsaXplZCB0aGF0IEFwYWNoZSB3YXMgdGhlIGN1bHByaXQgYW5kIGZvdW5kIHRo YXQgdGhlIHByb3h5IHRvIHRoZSA9Cm92aXJ0LWVuZ2luZSBpbiAvZXRjL2h0dHBkL2NvbmYuZC96 LW92aXJ0LWVuZ2luZS1wcm94eS5jb25mIGhhcyBhIHN1cGVyID0KbG9uZyB0aW1lb3V0IHdpdGgg bWFueSByZXRyaWVzLiBJIGNoYW5nZWQgdGhlIHNldHRpbmdzIGFuZCBub3cgPQpldmVyeXRoaW5n IHdvcmtzIGZvciBtZS4KPj0yMAo+IC0+IEJlZm9yZSBjaGFuZ2U6Cj4gICAgIDxMb2NhdGlvbk1h dGNoID0KXi8ob3ZpcnQtZW5naW5lKCR8Lyl8YXBpKCR8Lyl8UkhFVk1hbmFnZXJXZWIvfE92aXJ0 RW5naW5lV2ViL3xjYS5jcnQkfGVuZ2k9Cm5lLnNzaC5rZXkudHh0JHxyaGV2bS5zc2gua2V5LnR4 dCQpPgo+ICAgICAgICAgUHJveHlQYXNzTWF0Y2ggYWpwOi8vMTI3LjAuMC4xOjg3MDIgPD4gdGlt ZW91dD0zRDM2MDAgcmV0cnk9M0Q1Cj49MjAKPiAgICAgICAgIDxJZk1vZHVsZSBkZWZsYXRlX21v ZHVsZT4KPiAgICAgICAgICAgICBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0L2ph dmFzY3JpcHQgdGV4dC9jc3MgPQp0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uIGFwcGxpY2F0 aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uID0KYXBwbGljYXRpb24veC15YW1sCj4gICAgICAgICA8 L0lmTW9kdWxlPgo+ICAgICA8L0xvY2F0aW9uTWF0Y2g+Cj49MjAKPiAtPiBBZnRlciBjaGFuZ2U6 Cj4gICAgIDxMb2NhdGlvbk1hdGNoIF4vb3ZpcnQtZW5naW5lKCR8Lyk+Cj4gICAgICAgICBQcm94 eVBhc3NNYXRjaCBhanA6Ly8xMjcuMC4wLjE6ODcwMiA8PiB0aW1lb3V0PTNENSByZXRyeT0zRDIK Pj0yMAo+ICAgICAgICAgPElmTW9kdWxlIGRlZmxhdGVfbW9kdWxlPgo+ICAgICAgICAgICAgIEFk ZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyA9CnRl eHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFwcGxpY2F0aW9uL2pz b24gPQphcHBsaWNhdGlvbi94LXlhbWwKPiAgICAgICAgIDwvSWZNb2R1bGU+Cj4gICAgIDwvTG9j YXRpb25NYXRjaD4KPj0yMAo+ID1FMj04MD04QlRoaXMgb25lIGlzIGNvcnJlY3QgZm9yIDQuMD1F Mj04MD04Qj1FMj04MD04Qiwgbm90IHN1cmUgd2h5ID0KaXQgd2FzIG5vdCB1cGRhdGVkIGR1cmlu ZyB1cGdyYWRlIGZyb20gMy42LiBAU2ltb25lPwo+ID1FMj04MD04Qgo+PTIwCj4gSG9uZXN0bHkg aXQncwo+ICAgICA8TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2luZSgkfC8pPgo+ICAgICAgICAg UHJveHlQYXNzTWF0Y2ggYWpwOi8vMTI3LjAuMC4xOjg3MDIgPGh0dHA6Ly8xMjcuMC4wLjE6ODcw Mi8+ID0KdGltZW91dD0zRDM2MDAgcmV0cnk9M0Q1Cj49MjAKPiAgICAgICAgIDxJZk1vZHVsZSBk ZWZsYXRlX21vZHVsZT4KPiAgICAgICAgICAgICBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFU RSB0ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgPQp0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29u IGFwcGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uID0KYXBwbGljYXRpb24veC15YW1sCj4g ICAgICAgICA8L0lmTW9kdWxlPgo+ICAgICA8L0xvY2F0aW9uTWF0Y2g+Cj4gYWxzbyBvbiBhIGZy ZXNoIDQuMCBlbmdpbmUgZnJvbSBvdXIgbGF0ZXN0IGVuZ2luZS1hcHBsaWFuY2UuCj4gPTIwCj49 MjAKPiBJZiBJIHJlYWQgdGhlIHRpbWVvdXQgc2V0dGluZ3MgY29ycmVjdGx5LCBpdCB3aWxsIHdh aXQgNjAgbWludXRlcyB3aXRoID0KNSByZXRyaWVzLiA1IGhvdXJzIGlzIHdheSB0b28gbG9uZyBm b3IgbXkgbGl0dGxlIHNlcnZlciB0byBob2xkIG9udG8gYWxsID0KdGhvc2UgYXBhY2hlIHByb2Nl c3Nlcy4KPiBUaGUgY2hhbmdlIEkgbWFkZSBhbGxvd3MgZm9yIHRoZXJlIHRvIGJlIGFuIGVycm9y LCBhbmQgYWxzbyByZWxlYXNlcyA9CmFwYWNoZSdzIGhvbGQgb24gdGhlIHByb2Nlc3MuIE9uY2Ug ZXZlcnl0aGluZyBpcyByZWFkeSwgYXBhY2hlIGlzIHJlYWR5ID0KdG8gc2VydmUgcmVxdWVzdHMg YW5kIGV2ZXJ5dGhpbmcvZXZlcnlvbmUgaXMgaGFwcHkuIEJlZm9yZSBtYWtpbmcgdGhlID0KY2hh bmdlLCBJIGp1c3QgZ2V0IGEgd2hpdGVzY3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3Ro aW5nIHdvcmtzID0KdW50aWwgSSByZXN0YXJ0IEFwYWNoZSAob3IgSSBlbmQgdXAgaW4gYW4gZW5k bGVzcyBsb29wIG9mIG92aXJ0LWhhID0Kc2VydmljZXMgcmVzdGFydGluZyBteSBob3N0ZWQtZW5n aW5lLgo+PTIwCj4gPUUyPTgwPThCV2VsbCwgaWYgeW91IGhhdmUgYW4gaXNzdWUgd2l0aCB0b28g bWFueSBhcGFjaGUgcHJvY2Vzc2VzID0Kd2FpdGluZyBmb3IgZW5naW5lIHRvIHJlc3BvbmQsIHRo ZW4gdGhlcmUncyBzb21lIGlzc3VlIGluIGVuZ2luZS4gQXMgSSA9Cndyb3RlIGFib3ZlIHBsZWFz ZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayBlbnRyb3B5Lgo+PTIwCj4gVGhhbmtz Cj49MjAKPiBNYXJ0aW4gUGVyaW5hCj4gPUUyPTgwPThCPTIwCj49MjAKPiBJIG5vdGljZWQgdGhh dCB0aGlzIHNldHRpbmcgcmV2ZXJ0cyB0byB0aGUgb3JpZ2luYWwgc2V0dGluZywgc28gb1ZpcnQg PQptdXN0IGJlIHdyaXRpbmcgdGhpcyBmaWxlLiBQZXJoYXBzIHRoZXNlIG51bWJlciBjYW4gYmUg Y2hhbmdlZCBpbiBvVmlydD8gPQpJZiBub3QsIEkgd2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxl IHBsYXkgdG8gcmV2ZXJ0IHRoZSBzZXR0aW5ncyB3aXRoID0Kd29ya2luZyB2YWx1ZXMgYW5kIHJl c3RhcnQgYXBhY2hlIG9uIG15IGVuZ2luZS4KPiA6LSkKPj0yMAo+IENoZWVycywKPiBHZXJ2YWlz Cj49MjAKPj0yMAo+PTIwCj49MjAKPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwo+IFVzZXJzIG1haWxpbmcgbGlzdAo+IFVzZXJzQG92aXJ0Lm9yZyA8bWFp bHRvOlVzZXJzQG92aXJ0Lm9yZz4KPiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlz dGluZm8vdXNlcnMgPQo8aHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3Vz ZXJzPgo+PTIwCj49MjAKPj0yMAoKCi0tQXBwbGUtTWFpbD1fMTlDMEI5NTYtOTBFMy00NDk5LUFD OUYtQjUwODc5QkQ2RDBBCkNvbnRlbnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFi bGUKQ29udGVudC1UeXBlOiB0ZXh0L2h0bWw7CgljaGFyc2V0PXV0Zi04Cgo8aHRtbD48aGVhZD48 bWV0YSBodHRwLWVxdWl2PTNEIkNvbnRlbnQtVHlwZSIgY29udGVudD0zRCJ0ZXh0L2h0bWwgPQpj aGFyc2V0PTNEdXRmLTgiPjwvaGVhZD48Ym9keSBzdHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdv cmQ7ID0KLXdlYmtpdC1uYnNwLW1vZGU6IHNwYWNlOyAtd2Via2l0LWxpbmUtYnJlYWs6IGFmdGVy LXdoaXRlLXNwYWNlOyIgPQpjbGFzcz0zRCIiPkhpIFNpbW9uZSw8ZGl2IGNsYXNzPTNEIiI+PGJy IGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+WWVzLi4uIEkgZ3Vlc3MgaXQgd2Fz IG5vdCBjbGVhciBpbiBteSBvcmlnaW5hbCBlbWFpbC4gSSA9CmNoYW5nZWQgdGhlIG51bWJlcnMg bXlzZWxmIHRvIGxvd2VyIHRoZSB0aW1lb3V0IGFuZCByZXRyaWVzLiBXaXRoIHRoZW0gPQpzZXQg YXMgdGhleSB3ZXJlIHNldCBieSBvdmlydCAodGltZW91dD0zRDM2MDAgcmV0cnk9M0Q1KSB0aGlu Z3Mgd2VyZSBub3QgPQp3b3JraW5nIGZvciBtZS4mbmJzcDs8YnIgY2xhc3M9M0QiIj48ZGl2IGNs YXNzPTNEIiI+CjxkaXYgaWQ9M0Qic2lnbmF0dXJlIiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIi PkNoZWVycyw8YnIgPQpjbGFzcz0zRCIiPkdlcnZhaXM8YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9 M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj4KCjwvZGl2Pgo8YnIgY2xhc3M9M0QiIj48ZGl2Pjxi bG9ja3F1b3RlIHR5cGU9M0QiY2l0ZSIgY2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj5PbiBT ZXAgMjksIDIwMTYsIGF0IDEwOjA0IEFNLCBTaW1vbmUgVGlyYWJvc2NoaSAmbHQ7PGEgPQpocmVm PTNEIm1haWx0bzpzdGlyYWJvc0ByZWRoYXQuY29tIiBjbGFzcz0zRCIiPnN0aXJhYm9zQHJlZGhh dC5jb208L2E+Jmd0Oz0KIHdyb3RlOjwvZGl2PjxiciBjbGFzcz0zRCJBcHBsZS1pbnRlcmNoYW5n ZS1uZXdsaW5lIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiA9CmRpcj0zRCJsdHIiIGNsYXNzPTNEIiI+ PGJyIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyID0KY2xhc3M9M0Qi Ij48ZGl2IGNsYXNzPTNEImdtYWlsX3F1b3RlIj5PbiBUaHUsIFNlcCAyOSwgMjAxNiBhdCAxMjo0 NyBQTSwgPQpNYXJ0aW4gUGVyaW5hIDxzcGFuIGRpcj0zRCJsdHIiIGNsYXNzPTNEIiI+Jmx0Ozxh ID0KaHJlZj0zRCJtYWlsdG86bXBlcmluYUByZWRoYXQuY29tIiB0YXJnZXQ9M0QiX2JsYW5rIiA9 CmNsYXNzPTNEIiI+bXBlcmluYUByZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxiciA9 CmNsYXNzPTNEIiI+PGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNEIm1h cmdpbjowcHggMHB4IDBweCA9CjAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIw NCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgPQpkaXI9M0QibHRyIiBjbGFzcz0zRCIiPjxk aXYgY2xhc3M9M0QiIj5IaSw8YnIgY2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2Pjxk aXYgY2xhc3M9M0QiIj5wbGVhc2UgdGFrZSBhIGxvb2sgYXQgbXkgaW5saW5lID0KY29tbWVudHM6 PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGJyID0KY2xh c3M9M0QiIj48ZGl2IGNsYXNzPTNEImdtYWlsX3F1b3RlIj48c3BhbiBjbGFzcz0zRCJnbWFpbC0i Pk9uIFR1ZSwgU2VwID0KMjcsIDIwMTYgYXQgNzoyMyBQTSwgR2VydmFpcyBkZSBNb250YnJ1biA8 c3BhbiBkaXI9M0QibHRyIiA9CmNsYXNzPTNEIiI+Jmx0OzxhIGhyZWY9M0QibWFpbHRvOmdlcnZh aXNAZGVtb250YnJ1bi5jb20iIHRhcmdldD0zRCJfYmxhbmsiPQogY2xhc3M9M0QiIj5nZXJ2YWlz QGRlbW9udGJydW4uY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxiciA9CmNsYXNzPTNEIiI+PGJs b2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFw eCA9CnNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9M0QiZ21h aWxfcXVvdGUiPjxkaXYgPQpzdHlsZT0zRCJ3b3JkLXdyYXA6YnJlYWstd29yZCIgY2xhc3M9M0Qi Ij5IZXkgQWxsLDxkaXYgY2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xh c3M9M0QiIj5TaW5jZSB1cGRhdGluZyB0byA0LjAueCBvZiBvVmlydCwgSSA9CmhhdmUgaGFkIGFu IGlzc3VlIHdpdGggbXkgaG9zdGVkIGVuZ2luZS4gQWZ0ZXIgYSBzb21lIHBva2luZyBhcm91bmQs IEkgPQp0aGluayBJIGhhdmUgZmlndXJlZCBvdXQgbXkgaXNzdWUgYW5kIHRob3VnaHQgSSB3b3Vs ZCBzaGFyZSB0byBzZWUgd2hhdCA9Cm90aGVycyB0aGluay48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+ VGhlIGlzc3VlIGhhcyBleGlzdGVkIHdpdGggNC4wLCA9CjQuMC4xLCA0LjAuMiwgNC4wLjMsIGFu ZCBzdGlsbCBleGlzdHMgaW4gNC4wLjQuPC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxiciA9CmNsYXNz PTNEIiI+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPkRlc2NyaXB0aW9uOjwvZGl2PjxkaXYgY2xhc3M9 M0QiIj5XaGVuID0KbXkgaG9zdGVkIGVuZ2luZSBzdGFydHMgaXQgcmVwb3J0cyB0aGF0IGl0IGlz IGluIGEgZGVncmFkZWQgc3RhdGUgd2l0aCA3ID0Kb3IgOCBzZXJ2aWNlcyBzdGlsbCBub3Qgc3Rh cnRlZCB3aGVuIEkgcnVuIHN5c3RlbWN0bCBzdGF0dXMuIEl0IHRha2VzID0KYWJvdXQgNiBvciA3 IG1pbnV0ZXMgdG8gZXZlbnR1YWxseSBzdGFydCBhbGwgdGhlIHNlcnZpY2VzIGFuZCBjb21lID0K b25saW5lLiBJZiBJIGRvbid0IHNldCBteSBjbHVzdGVyIHRvIEdsb2JhbC1NYWludGVuYW5jZSBt b2RlIGl0ID0KZXZlbnR1YWxseSB0aGlua3MgdGhhdCBteSBob3N0ZWQtZW5naW5lIG5lZWRzIHRv IGJlIHJlYm9vdGVkIGFuZCA9CnJlc3RhcnRzIGl0IGJlZm9yZSBpdCBjYW4gc3RhcnQgPQpldmVy eXRoaW5nLjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L3NwYW4+PGRpdiBjbGFzcz0zRCIiPjxi ciA9CmNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPj1FMj04MD04QkNvdWxkIHlvdSBwbGVhc2Ug c2hhcmUgd2l0aCB1cyBsb2dzID0KZ2F0aGVyZWQgYnkgb3ZpcnQtbG9nLWNvbGxlY3Rvcj88YnIg Y2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj5JdCdzIGp1c3QgPQphIGd1ZXNzIGJ1dCBjb3VsZCB5 b3UgcGxlYXNlIHRha2UgYSBsb29rIGlmIHlvdSBIRSBWTSBoYXMgZW5vdWdoID0KZW50cm9weT88 YnIgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj4mbmJzcDsgY2F0ID0KL3Byb2Mvc3lzL2tlcm5l bC9yYW5kb20vPHdiciBjbGFzcz0zRCIiPmVudHJvcHlfYXZhaWw8YnIgY2xhc3M9M0QiIj48YnIg PQpjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5JZiB0aGUgdmFsdWUgaXMgbG93IChi ZWxvdyBvciBhcm91bmQgPQoyMDApLCZuYnNwOyB5b3UgcmVhbGx5IG5lZWQgdG8gaW5zdGFsbCBh bmQgY29uZmlndXJlIHNvbWUgZW50cm9weSA9CmdlbmVyYXRvciBzdWNoIGFzIGhhdmVnZWQ8YnIg Y2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48L2Rpdj48c3BhbiA9CmNsYXNzPTNEImdt YWlsLSI+PGJsb2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggMHB4ID0KMC44ZXg7Ym9y ZGVyLWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgPQpj bGFzcz0zRCJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0zRCJ3b3JkLXdyYXA6YnJlYWstd29yZCIg Y2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNs YXNzPTNEIiI+U29sdXRpb246PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+SSByZWFsaXplZCB0aGF0 IEFwYWNoZSB3YXMgdGhlIGN1bHByaXQgYW5kIGZvdW5kIHRoYXQgdGhlID0KcHJveHkgdG8gdGhl IG92aXJ0LWVuZ2luZSBpbiZuYnNwOy9ldGMvaHR0cGQvY29uZi5kL3otb3ZpcnQtZTx3YnIgPQpj bGFzcz0zRCIiPm5naW5lLXByb3h5LmNvbmYgaGFzIGEgc3VwZXIgbG9uZyB0aW1lb3V0IHdpdGgg bWFueSByZXRyaWVzLiA9CkkgY2hhbmdlZCB0aGUgc2V0dGluZ3MgYW5kIG5vdyBldmVyeXRoaW5n IHdvcmtzIGZvciBtZS48L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rp dj48ZGl2IGNsYXNzPTNEIiI+LSZndDsgQmVmb3JlID0KY2hhbmdlOjwvZGl2PjxibG9ja3F1b3Rl IHN0eWxlPTNEIm1hcmdpbjowcHggMHB4IDBweCA9CjQwcHg7Ym9yZGVyLXdpZHRoOm1lZGl1bTti b3JkZXItc3R5bGU6bm9uZTtwYWRkaW5nOjBweCIgY2xhc3M9M0QiIj48ZGl2ID0KY2xhc3M9M0Qi Ij48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbHQ7TG9jYXRpb25NYXRjaCA9Cl4vKG92 aXJ0LWVuZ2luZSgkfC8pfGFwaSgkfC8pfDx3YnIgPQpjbGFzcz0zRCIiPlJIRVZNYW5hZ2VyV2Vi L3xPdmlydEVuZ2luZVdlYjx3YnIgPQpjbGFzcz0zRCIiPi98Y2EuY3J0JHxlbmdpbmUuc3NoLmtl eS50eHQkfDx3YnIgPQpjbGFzcz0zRCIiPnJoZXZtLnNzaC5rZXkudHh0JCkmZ3Q7PC9kaXY+PGRp diBjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgPQombmJzcDsgJm5ic3A7IFByb3h5UGFzc01hdGNo IDxhIGNsYXNzPTNEIiI+YWpwOi8vMTI3LjAuMC4xOjg3MDI8L2E+ID0KdGltZW91dD0zRDM2MDAg cmV0cnk9M0Q1PC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYg PQpjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyAmbHQ7SWZNb2R1bGUgPQpk ZWZsYXRlX21vZHVsZSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbmJz cDsgJm5ic3A7ID0KJm5ic3A7ICZuYnNwOyBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0 ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgPQp0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uIGFw cGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uID0KYXBwbGljYXRpb24veC15YW1sPC9kaXY+ PGRpdiBjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgJm5ic3A7ICZuYnNwOyA9CiZsdDsvSWZNb2R1 bGUmZ3Q7PC9kaXY+PGRpdiBjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgPQombHQ7L0xvY2F0aW9u TWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9ja3F1b3RlPjxkaXYgY2xhc3M9M0QiIj48YnIgPQpj bGFzcz0zRCIiPjwvZGl2Pi0mZ3Q7IEFmdGVyIGNoYW5nZTo8YmxvY2txdW90ZSBzdHlsZT0zRCJt YXJnaW46MHB4IDBweCA9CjBweCA0MHB4O2JvcmRlci13aWR0aDptZWRpdW07Ym9yZGVyLXN0eWxl Om5vbmU7cGFkZGluZzowcHgiID0KY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGRpdiBjbGFz cz0zRCIiPiZuYnNwOyAmbmJzcDsgPQombHQ7TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2luZSgk fC8pJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgPQombmJzcDsgJm5ic3A7ICZuYnNw OyBQcm94eVBhc3NNYXRjaCA8YSBjbGFzcz0zRCIiPmFqcDovLzEyNy4wLjAuMTo4NzAyPC9hPj0K IHRpbWVvdXQ9M0Q1IHJldHJ5PTNEMjwvZGl2PjxkaXYgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0Qi Ij48L2Rpdj48ZGl2ID0KY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgJmx0 O0lmTW9kdWxlID0KZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRpdiBjbGFzcz0zRCIiPiZuYnNw OyAmbmJzcDsgJm5ic3A7ICZuYnNwOyA9CiZuYnNwOyAmbmJzcDsgQWRkT3V0cHV0RmlsdGVyQnlU eXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzID0KdGV4dC9odG1sIHRleHQveG1s IHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiA9CmFwcGxpY2F0aW9u L3gteWFtbDwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsg PQombHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ID0K Jmx0Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48L2Js b2NrcXVvdGU+PC9zcGFuPjw9CmRpdiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxkaXYgPQpz dHlsZT0zRCJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5Omlu bGluZSIgPQpjbGFzcz0zRCIiPj1FMj04MD04QlRoaXMgb25lIGlzIGNvcnJlY3QgZm9yIDQuMD1F Mj04MD04QjwvZGl2PjxkaXYgPQpzdHlsZT0zRCJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Es c2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSIgPQpjbGFzcz0zRCIiPj1FMj04MD04Qiwgbm90IHN1 cmUgd2h5IGl0IHdhcyBub3QgdXBkYXRlZCBkdXJpbmcgdXBncmFkZSA9CmZyb20gMy42LiBAU2lt b25lPzxiciA9CmNsYXNzPTNEIiI+PUUyPTgwPThCPC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9k aXY+PC9ibG9ja3F1b3RlPjxkaXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2Pjxk aXYgY2xhc3M9M0QiIj5Ib25lc3RseSBpdCdzPC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+PGRpdiBj bGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgJmx0O0xvY2F0aW9uTWF0Y2ggPQpeL292aXJ0LWVuZ2lu ZSgkfC8pJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJz cDsgPQpQcm94eVBhc3NNYXRjaCBhanA6Ly88YSBocmVmPTNEImh0dHA6Ly8xMjcuMC4wLjE6ODcw Mi8iID0KY2xhc3M9M0QiIj4xMjcuMC4wLjE6ODcwMjwvYT4gdGltZW91dD0zRDM2MDAgcmV0cnk9 M0Q1PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiBjbGFz cz0zRCIiPiZuYnNwOyAmbmJzcDsgJm5ic3A7ID0KJm5ic3A7ICZsdDtJZk1vZHVsZSBkZWZsYXRl X21vZHVsZSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ID0KJm5ic3A7ICZuYnNwOyAm bmJzcDsgJm5ic3A7ICZuYnNwOyBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFURSA9CnRleHQv amF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uIGFwcGxpY2F0 aW9uL3htbCA9CmFwcGxpY2F0aW9uL2pzb24gYXBwbGljYXRpb24veC15YW1sPC9kaXY+PGRpdiBj bGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgPQombmJzcDsgJm5ic3A7ICZsdDsvSWZNb2R1bGUmZ3Q7 PC9kaXY+PGRpdiBjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDsgPQombHQ7L0xvY2F0aW9uTWF0Y2gm Z3Q7PC9kaXY+PC9kaXY+PGRpdiBjbGFzcz0zRCIiPmFsc28gb24gYSBmcmVzaCA0LjAgPQplbmdp bmUgZnJvbSBvdXIgbGF0ZXN0IGVuZ2luZS1hcHBsaWFuY2UuPC9kaXY+PGRpdiA9CmNsYXNzPTNE IiI+Jm5ic3A7PC9kaXY+PGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiID0Kc3R5bGU9 M0QibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCA9CnJnYigy MDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBkaXI9M0QibHRyIiBjbGFzcz0zRCIi PjxkaXYgPQpjbGFzcz0zRCJnbWFpbF9leHRyYSI+PGRpdiBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+ PHNwYW4gPQpjbGFzcz0zRCJnbWFpbC0iPjxibG9ja3F1b3RlIHN0eWxlPTNEIm1hcmdpbjowcHgg MHB4IDBweCA9CjAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3Bh ZGRpbmctbGVmdDoxZXgiID0KY2xhc3M9M0QiZ21haWxfcXVvdGUiPjxkaXYgc3R5bGU9M0Qid29y ZC13cmFwOmJyZWFrLXdvcmQiIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEIiI+PGJyIGNsYXNz PTNEIiI+PC9kaXY+SWYgSSByZWFkIHRoZSB0aW1lb3V0IHNldHRpbmdzID0KY29ycmVjdGx5LCBp dCB3aWxsIHdhaXQgNjAgbWludXRlcyB3aXRoIDUgcmV0cmllcy4gNSBob3VycyBpcyB3YXkgdG9v ID0KbG9uZyBmb3IgbXkgbGl0dGxlIHNlcnZlciB0byBob2xkIG9udG8gYWxsIHRob3NlIGFwYWNo ZSBwcm9jZXNzZXMuID0KPC9kaXY+PC9ibG9ja3F1b3RlPjxibG9ja3F1b3RlIHN0eWxlPTNEIm1h cmdpbjowcHggMHB4IDBweCA9CjAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIw NCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiID0KY2xhc3M9M0QiZ21haWxfcXVvdGUiPjxkaXYgc3R5 bGU9M0Qid29yZC13cmFwOmJyZWFrLXdvcmQiIGNsYXNzPTNEIiI+VGhlID0KY2hhbmdlIEkgbWFk ZSBhbGxvd3MgZm9yIHRoZXJlIHRvIGJlIGFuIGVycm9yLCBhbmQgYWxzbyByZWxlYXNlcyA9CmFw YWNoZSdzIGhvbGQgb24gdGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyByZWFkeSwgYXBh Y2hlIGlzIHJlYWR5ID0KdG8gc2VydmUgcmVxdWVzdHMgYW5kIGV2ZXJ5dGhpbmcvZXZlcnlvbmUg aXMgaGFwcHkuIEJlZm9yZSBtYWtpbmcgdGhlID0KY2hhbmdlLCBJIGp1c3QgZ2V0IGEgd2hpdGVz Y3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtzID0KdW50aWwgSSByZXN0 YXJ0IEFwYWNoZSAob3IgSSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhhID0K c2VydmljZXMgcmVzdGFydGluZyBteSBob3N0ZWQtZW5naW5lLjxiciA9CmNsYXNzPTNEIiI+PC9k aXY+PC9ibG9ja3F1b3RlPjwvc3Bhbj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRp diA9CnN0eWxlPTNEImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3Bs YXk6aW5saW5lIiA9CmNsYXNzPTNEIiI+PUUyPTgwPThCV2VsbCwgaWYgeW91IGhhdmUgYW4gaXNz dWUgd2l0aCB0b28gbWFueSBhcGFjaGUgPQpwcm9jZXNzZXMgd2FpdGluZyBmb3IgZW5naW5lIHRv IHJlc3BvbmQsIHRoZW4gdGhlcmUncyBzb21lIGlzc3VlIGluID0KZW5naW5lLiBBcyBJIHdyb3Rl IGFib3ZlIHBsZWFzZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayA9CmVudHJvcHku PGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CnN0eWxlPTNEImZvbnQt ZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIiA9CmNsYXNz PTNEIiI+VGhhbmtzPGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9CnN0 eWxlPTNEImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5s aW5lIiA9CmNsYXNzPTNEIiI+TWFydGluIFBlcmluYTxiciBjbGFzcz0zRCIiPj1FMj04MD04Qjwv ZGl2PiZuYnNwOzwvZGl2PjxibG9ja3F1PQpvdGUgc3R5bGU9M0QibWFyZ2luOjBweCAwcHggMHB4 IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCA9CnJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1s ZWZ0OjFleCIgY2xhc3M9M0QiZ21haWxfcXVvdGUiPjxzcGFuID0KY2xhc3M9M0QiZ21haWwtIj48 ZGl2IHN0eWxlPTNEIndvcmQtd3JhcDpicmVhay13b3JkIiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFz cz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9k aXY+PGRpdiA9CmNsYXNzPTNEIiI+SSBub3RpY2VkIHRoYXQgdGhpcyBzZXR0aW5nIHJldmVydHMg dG8gdGhlIG9yaWdpbmFsIHNldHRpbmcsID0Kc28gb1ZpcnQgbXVzdCBiZSB3cml0aW5nIHRoaXMg ZmlsZS4gUGVyaGFwcyB0aGVzZSBudW1iZXIgY2FuIGJlIGNoYW5nZWQgPQppbiBvVmlydD8gSWYg bm90LCBJIHdpbGwganVzdCBzZXR1cCBhbmQgYW5zaWJsZSBwbGF5IHRvIHJldmVydCB0aGUgPQpz ZXR0aW5ncyB3aXRoIHdvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBvbiBteSBlbmdp bmUuPC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+Oi0pPC9kaXY+PGRpdiBjbGFzcz0zRCIiPgo8ZGl2 IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+Q2hlZXJzLDxiciBjbGFzcz0zRCIiPkdlcnZhaXM8 YnIgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PgoKPC9k aXY+CgoKCjxiciBjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvZGl2PjxiciA9CmNsYXNzPTNEIiI+ PC9zcGFuPl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fXzx3YnIgPQpjbGFzcz0zRCIiPl9f X19fX19fX19fX19fX19fPGJyIGNsYXNzPTNEIiI+ClVzZXJzIG1haWxpbmcgbGlzdDxiciBjbGFz cz0zRCIiPgo8YSBocmVmPTNEIm1haWx0bzpVc2Vyc0BvdmlydC5vcmciIHRhcmdldD0zRCJfYmxh bmsiID0KY2xhc3M9M0QiIj5Vc2Vyc0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPTNEIiI+CjxhIHJl bD0zRCJub3JlZmVycmVyIiA9CmhyZWY9M0QiaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFu L2xpc3RpbmZvL3VzZXJzIiB0YXJnZXQ9M0QiX2JsYW5rIiA9CmNsYXNzPTNEIiI+aHR0cDovL2xp c3RzLm92aXJ0Lm9yZy9tYWlsbWFuPHdiciA9CmNsYXNzPTNEIiI+L2xpc3RpbmZvL3VzZXJzPC9h PjxiciBjbGFzcz0zRCIiPgo8YnIgY2xhc3M9M0QiIj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyIGNs YXNzPTNEIiI+PC9kaXY+PC9kaXY+CjwvYmxvY2txdW90ZT48L2Rpdj48YnIgY2xhc3M9M0QiIj48 L2Rpdj48L2Rpdj4KPC9kaXY+PC9ibG9ja3F1b3RlPjwvZGl2PjxiciBjbGFzcz0zRCIiPjwvZGl2 PjwvYm9keT48L2h0bWw+PQoKLS1BcHBsZS1NYWlsPV8xOUMwQjk1Ni05MEUzLTQ0OTktQUM5Ri1C NTA4NzlCRDZEMEEtLQo= --===============3067116927732368603==-- From mperina at redhat.com Thu Sep 29 09:11:09 2016 Content-Type: multipart/mixed; boundary="===============5598247186357199401==" MIME-Version: 1.0 From: Martin Perina To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 15:11:07 +0200 Message-ID: In-Reply-To: CAN8-ONpupxzhgjCCc1Oc1he_qr3OU8r9b6wzPZt2nNS=YL=-Fw@mail.gmail.com --===============5598247186357199401== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 29, 2016 at 3:04 PM, Simone Tiraboschi wrote: > > > On Thu, Sep 29, 2016 at 12:47 PM, Martin Perina > wrote: > >> Hi, >> >> please take a look at my inline comments: >> >> On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun < >> gervais(a)demontbrun.com> wrote: >> >>> Hey All, >>> >>> Since updating to 4.0.x of oVirt, I have had an issue with my hosted >>> engine. After a some poking around, I think I have figured out my issue= and >>> thought I would share to see what others think. >>> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists in >>> 4.0.4. >>> >>> Description: >>> When my hosted engine starts it reports that it is in a degraded state >>> with 7 or 8 services still not started when I run systemctl status. It >>> takes about 6 or 7 minutes to eventually start all the services and come >>> online. If I don't set my cluster to Global-Maintenance mode it eventua= lly >>> thinks that my hosted-engine needs to be rebooted and restarts it befor= e it >>> can start everything. >>> >> >> =E2=80=8BCould you please share with us logs gathered by ovirt-log-colle= ctor? >> >> It's just a guess but could you please take a look if you HE VM has >> enough entropy? >> >> cat /proc/sys/kernel/random/entropy_avail >> >> If the value is low (below or around 200), you really need to install >> and configure some entropy generator such as haveged >> >> >>> Solution: >>> I realized that Apache was the culprit and found that the proxy to the >>> ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super >>> long timeout with many retries. I changed the settings and now everythi= ng >>> works for me. >>> >>> -> Before change: >>> >>> >> RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$| >>> rhevm.ssh.key.txt$)> >>> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >>> >>> >>> AddOutputFilterByType DEFLATE text/javascript text/css >>> text/html text/xml text/json application/xml application/json >>> application/x-yaml >>> >>> >>> >>> >>> -> After change: >>> >>> >>> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 >>> >>> >>> AddOutputFilterByType DEFLATE text/javascript text/css >>> text/html text/xml text/json application/xml application/json >>> application/x-yaml >>> >>> >>> >>> >> =E2=80=8BThis one is correct for 4.0=E2=80=8B >> =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Sim= one? >> =E2=80=8B >> > > Honestly it's > > ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 > > > AddOutputFilterByType DEFLATE text/javascript text/css > text/html text/xml text/json application/xml application/json > application/x-yaml > > > also on a fresh 4.0 engine from our latest engine-appliance. > =E2=80=8BRight, I missed the timeout=E2=80=8B/retry option changes. But the= important part is why old configuration (with different LocationMatch) was not overwritten during upgrade. > >> >>> If I read the timeout settings correctly, it will wait 60 minutes with 5 >>> retries. 5 hours is way too long for my little server to hold onto all >>> those apache processes. >>> >> The change I made allows for there to be an error, and also releases >>> apache's hold on the process. Once everything is ready, apache is ready= to >>> serve requests and everything/everyone is happy. Before making the chan= ge, >>> I just get a whitescreen in my browser and then nothing works until I >>> restart Apache (or I end up in an endless loop of ovirt-ha services >>> restarting my hosted-engine. >>> >> >> =E2=80=8BWell, if you have an issue with too many apache processes waiti= ng for >> engine to respond, then there's some issue in engine. As I wrote above >> please share the logs with us and check entropy. >> >> Thanks >> >> Martin Perina >> =E2=80=8B >> >> >>> >>> I noticed that this setting reverts to the original setting, so oVirt >>> must be writing this file. Perhaps these number can be changed in oVirt= ? If >>> not, I will just setup and ansible play to revert the settings with wor= king >>> values and restart apache on my engine. >>> :-) >>> >>> Cheers, >>> Gervais >>> >>> >>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> > --===============5598247186357199401== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGRpdiBjbGFzcz0iZ21haWxfZGVmYXVsdCIgc3R5bGU9ImZvbnQtZmFt aWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmIj48YnI+PC9kaXY+PGRpdiBjbGFzcz0iZ21h aWxfZXh0cmEiPjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1LCBTZXAgMjksIDIw MTYgYXQgMzowNCBQTSwgU2ltb25lIFRpcmFib3NjaGkgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBo cmVmPSJtYWlsdG86c3RpcmFib3NAcmVkaGF0LmNvbSIgdGFyZ2V0PSJfYmxhbmsiPnN0aXJhYm9z QHJlZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJn bWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2Nj IHNvbGlkO3BhZGRpbmctbGVmdDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxicj48ZGl2IGNsYXNzPSJn bWFpbF9leHRyYSI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj5PbiBUaHUsIFNlcCAyOSwg MjAxNiBhdCAxMjo0NyBQTSwgTWFydGluIFBlcmluYSA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhy ZWY9Im1haWx0bzptcGVyaW5hQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5tcGVyaW5hQHJl ZGhhdC5jb208L2E+Jmd0Ozwvc3Bhbj4gd3JvdGU6PGJyPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFp bF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHgg c29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48 ZGl2PkhpLDxicj48YnI+PC9kaXY+PGRpdj5wbGVhc2UgdGFrZSBhIGxvb2sgYXQgbXkgaW5saW5l IGNvbW1lbnRzOjxicj48L2Rpdj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGJyPjxkaXYgY2xh c3M9ImdtYWlsX3F1b3RlIj48c3Bhbj5PbiBUdWUsIFNlcCAyNywgMjAxNiBhdCA3OjIzIFBNLCBH ZXJ2YWlzIGRlIE1vbnRicnVuIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0ibWFpbHRvOmdl cnZhaXNAZGVtb250YnJ1bi5jb20iIHRhcmdldD0iX2JsYW5rIj5nZXJ2YWlzQGRlbW9udGJydW4u Y29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBw eCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3Bh ZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFw OmJyZWFrLXdvcmQiPkhleSBBbGwsPGRpdj48YnI+PC9kaXY+PGRpdj5TaW5jZSB1cGRhdGluZyB0 byA0LjAueCBvZiBvVmlydCwgSSBoYXZlIGhhZCBhbiBpc3N1ZSB3aXRoIG15IGhvc3RlZCBlbmdp bmUuIEFmdGVyIGEgc29tZSBwb2tpbmcgYXJvdW5kLCBJIHRoaW5rIEkgaGF2ZSBmaWd1cmVkIG91 dCBteSBpc3N1ZSBhbmQgdGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNlZSB3aGF0IG90aGVycyB0 aGluay48L2Rpdj48ZGl2PlRoZSBpc3N1ZSBoYXMgZXhpc3RlZCB3aXRoIDQuMCwgNC4wLjEsIDQu MC4yLCA0LjAuMywgYW5kIHN0aWxsIGV4aXN0cyBpbiA0LjAuNC48L2Rpdj48ZGl2Pjxicj48L2Rp dj48ZGl2PkRlc2NyaXB0aW9uOjwvZGl2PjxkaXY+V2hlbiBteSBob3N0ZWQgZW5naW5lIHN0YXJ0 cyBpdCByZXBvcnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0ZSB3aXRoIDcgb3IgOCBz ZXJ2aWNlcyBzdGlsbCBub3Qgc3RhcnRlZCB3aGVuIEkgcnVuIHN5c3RlbWN0bCBzdGF0dXMuIEl0 IHRha2VzIGFib3V0IDYgb3IgNyBtaW51dGVzIHRvIGV2ZW50dWFsbHkgc3RhcnQgYWxsIHRoZSBz ZXJ2aWNlcyBhbmQgY29tZSBvbmxpbmUuIElmIEkgZG9uJiMzOTt0IHNldCBteSBjbHVzdGVyIHRv IEdsb2JhbC1NYWludGVuYW5jZSBtb2RlIGl0IGV2ZW50dWFsbHkgdGhpbmtzIHRoYXQgbXkgaG9z dGVkLWVuZ2luZSBuZWVkcyB0byBiZSByZWJvb3RlZCBhbmQgcmVzdGFydHMgaXQgYmVmb3JlIGl0 IGNhbiBzdGFydCBldmVyeXRoaW5nLjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48L3NwYW4+PGRp dj48YnI+PGRpdj7igItDb3VsZCB5b3UgcGxlYXNlIHNoYXJlIHdpdGggdXMgbG9ncyBnYXRoZXJl ZCBieSBvdmlydC1sb2ctY29sbGVjdG9yPzxicj48YnI+SXQmIzM5O3MganVzdCBhIGd1ZXNzIGJ1 dCBjb3VsZCB5b3UgcGxlYXNlIHRha2UgYSBsb29rIGlmIHlvdSBIRSBWTSBoYXMgZW5vdWdoIGVu dHJvcHk/PGJyPjxicj7CoCBjYXQgL3Byb2Mvc3lzL2tlcm5lbC9yYW5kb20vZW50cm9wPHdicj55 X2F2YWlsPGJyPjxicj48L2Rpdj48ZGl2PklmIHRoZSB2YWx1ZSBpcyBsb3cgKGJlbG93IG9yIGFy b3VuZCAyMDApLMKgIHlvdSByZWFsbHkgbmVlZCB0byBpbnN0YWxsIGFuZCBjb25maWd1cmUgc29t ZSBlbnRyb3B5IGdlbmVyYXRvciBzdWNoIGFzIGhhdmVnZWQ8YnI+PGJyPjwvZGl2PjwvZGl2Pjxz cGFuPjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxl ZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9Imdt YWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+PGRpdj48YnI+PC9k aXY+PGRpdj5Tb2x1dGlvbjo8L2Rpdj48ZGl2PkkgcmVhbGl6ZWQgdGhhdCBBcGFjaGUgd2FzIHRo ZSBjdWxwcml0IGFuZCBmb3VuZCB0aGF0IHRoZSBwcm94eSB0byB0aGUgb3ZpcnQtZW5naW5lIGlu wqAvZXRjL2h0dHBkL2NvbmYuZC96LW92aXJ0LWU8d2JyPm5naW5lLXByb3h5LmNvbmYgaGFzIGEg c3VwZXIgbG9uZyB0aW1lb3V0IHdpdGggbWFueSByZXRyaWVzLiBJIGNoYW5nZWQgdGhlIHNldHRp bmdzIGFuZCBub3cgZXZlcnl0aGluZyB3b3JrcyBmb3IgbWUuPC9kaXY+PGRpdj48YnI+PC9kaXY+ PGRpdj4tJmd0OyBCZWZvcmUgY2hhbmdlOjwvZGl2PjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46 MHB4IDBweCAwcHggNDBweDtib3JkZXItd2lkdGg6bWVkaXVtO2JvcmRlci1zdHlsZTpub25lO3Bh ZGRpbmc6MHB4Ij48ZGl2PjxkaXY+wqAgwqAgJmx0O0xvY2F0aW9uTWF0Y2ggXi8ob3ZpcnQtZW5n aW5lKCR8Lyl8YXBpKCR8Lyl8PHdicj5SSEVWTWFuYWdlcldlYi98T3ZpcnRFbmdpbmVXZWI8d2Jy Pi98Y2EuY3J0JHxlbmdpbmUuc3NoLmtleS50eHQkfDx3YnI+cmhldm0uc3NoLmtleS50eHQkKSZn dDs8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgIFByb3h5UGFzc01hdGNoIDxhPmFqcDovLzEyNy4wLjAu MTo4NzAyPC9hPiB0aW1lb3V0PTM2MDAgcmV0cnk9NTwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ wqAgwqAgwqAgwqAgJmx0O0lmTW9kdWxlIGRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAg wqAgwqAgwqAgwqAgwqAgQWRkT3V0cHV0RmlsdGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2Ny aXB0IHRleHQvY3NzIHRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1s IGFwcGxpY2F0aW9uL2pzb24gYXBwbGljYXRpb24veC15YW1sPC9kaXY+PGRpdj7CoCDCoCDCoCDC oCAmbHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAgJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0 OzwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48L2Rpdj4tJmd0OyBBZnRlciBjaGFu Z2U6PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCA0MHB4O2JvcmRlci13aWR0 aDptZWRpdW07Ym9yZGVyLXN0eWxlOm5vbmU7cGFkZGluZzowcHgiPjxkaXY+PGRpdj7CoCDCoCAm bHQ7TG9jYXRpb25NYXRjaCBeL292aXJ0LWVuZ2luZSgkfC8pJmd0OzwvZGl2PjxkaXY+wqAgwqAg wqAgwqAgUHJveHlQYXNzTWF0Y2ggPGE+YWpwOi8vMTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9 NSByZXRyeT0yPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1 bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRw dXRGaWx0ZXJCeVR5cGUgREVGTEFURSB0ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1s IHRleHQveG1sIHRleHQvanNvbiBhcHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBs aWNhdGlvbi94LXlhbWw8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9k aXY+PGRpdj7CoCDCoCAmbHQ7L0xvY2F0aW9uTWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PC9ibG9ja3F1 b3RlPjwvZGl2PjwvYmxvY2txdW90ZT48L3NwYW4+PGRpdj48YnI+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPuKAi1RoaXMg b25lIGlzIGNvcnJlY3QgZm9yIDQuMOKAizwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OmFy aWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igIssIG5vdCBzdXJlIHdo eSBpdCB3YXMgbm90IHVwZGF0ZWQgZHVyaW5nIHVwZ3JhZGUgZnJvbSAzLjYuIEBTaW1vbmU/PGJy PuKAizwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48 L2Rpdj48ZGl2PkhvbmVzdGx5IGl0JiMzOTtzPC9kaXY+PGRpdj48ZGl2PsKgIMKgICZsdDtMb2Nh dGlvbk1hdGNoIF4vb3ZpcnQtZW5naW5lKCR8LykmZ3Q7PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCBQ cm94eVBhc3NNYXRjaCBhanA6Ly88YSBocmVmPSJodHRwOi8vMTI3LjAuMC4xOjg3MDIiIHRhcmdl dD0iX2JsYW5rIj4xMjcuMC4wLjE6ODcwMjwvYT4gdGltZW91dD0zNjAwIHJldHJ5PTU8L2Rpdj48 ZGl2Pjxicj48L2Rpdj48ZGl2PsKgIMKgIMKgIMKgICZsdDtJZk1vZHVsZSBkZWZsYXRlX21vZHVs ZSZndDs8L2Rpdj48ZGl2PsKgIMKgIMKgIMKgIMKgIMKgIEFkZE91dHB1dEZpbHRlckJ5VHlwZSBE RUZMQVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9q c29uIGFwcGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uIGFwcGxpY2F0aW9uL3gteWFtbDwv ZGl2PjxkaXY+wqAgwqAgwqAgwqAgJmx0Oy9JZk1vZHVsZSZndDs8L2Rpdj48ZGl2PsKgIMKgICZs dDsvTG9jYXRpb25NYXRjaCZndDs8L2Rpdj48L2Rpdj48ZGl2PmFsc28gb24gYSBmcmVzaCA0LjAg ZW5naW5lIGZyb20gb3VyIGxhdGVzdCBlbmdpbmUtYXBwbGlhbmNlLjwvZGl2PjwvZGl2PjwvZGl2 PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2Pjxicj48ZGl2IGNsYXNzPSJnbWFpbF9kZWZhdWx0IiBz dHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxp bmUiPuKAi1JpZ2h0LCBJIG1pc3NlZCB0aGUgdGltZW91dOKAiy9yZXRyeSBvcHRpb24gY2hhbmdl cy4gQnV0IHRoZSBpbXBvcnRhbnQgcGFydCBpcyB3aHkgb2xkIGNvbmZpZ3VyYXRpb24gKHdpdGgg ZGlmZmVyZW50IExvY2F0aW9uTWF0Y2gpIHdhcyBub3Qgb3ZlcndyaXR0ZW4gZHVyaW5nIHVwZ3Jh ZGUuPGJyPjxicj48L2Rpdj48L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0 eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5n LWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2IGNsYXNzPSJnbWFpbF9leHRyYSI+PGRpdiBj bGFzcz0iZ21haWxfcXVvdGUiPjxkaXY+wqA8L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxf cXVvdGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNv bGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBkaXI9Imx0ciI+PGRp diBjbGFzcz0iZ21haWxfZXh0cmEiPjxkaXYgY2xhc3M9ImdtYWlsX3F1b3RlIj48c3Bhbj48Ymxv Y2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBz b2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90 ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPjxkaXY+PGJyPjwvZGl2PklmIEkg cmVhZCB0aGUgdGltZW91dCBzZXR0aW5ncyBjb3JyZWN0bHksIGl0IHdpbGwgd2FpdCA2MCBtaW51 dGVzIHdpdGggNSByZXRyaWVzLiA1IGhvdXJzIGlzIHdheSB0b28gbG9uZyBmb3IgbXkgbGl0dGxl IHNlcnZlciB0byBob2xkIG9udG8gYWxsIHRob3NlIGFwYWNoZSBwcm9jZXNzZXMuIDwvZGl2Pjwv YmxvY2txdW90ZT48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2Jv cmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNs YXNzPSJnbWFpbF9xdW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPlRoZSBj aGFuZ2UgSSBtYWRlIGFsbG93cyBmb3IgdGhlcmUgdG8gYmUgYW4gZXJyb3IsIGFuZCBhbHNvIHJl bGVhc2VzIGFwYWNoZSYjMzk7cyBob2xkIG9uIHRoZSBwcm9jZXNzLiBPbmNlIGV2ZXJ5dGhpbmcg aXMgcmVhZHksIGFwYWNoZSBpcyByZWFkeSB0byBzZXJ2ZSByZXF1ZXN0cyBhbmQgZXZlcnl0aGlu Zy9ldmVyeW9uZSBpcyBoYXBweS4gQmVmb3JlIG1ha2luZyB0aGUgY2hhbmdlLCBJIGp1c3QgZ2V0 IGEgd2hpdGVzY3JlZW4gaW4gbXkgYnJvd3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtzIHVudGls IEkgcmVzdGFydCBBcGFjaGUgKG9yIEkgZW5kIHVwIGluIGFuIGVuZGxlc3MgbG9vcCBvZiBvdmly dC1oYSBzZXJ2aWNlcyByZXN0YXJ0aW5nIG15IGhvc3RlZC1lbmdpbmUuPGJyPjwvZGl2PjwvYmxv Y2txdW90ZT48L3NwYW4+PGRpdj48YnI+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVs dmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPuKAi1dlbGwsIGlmIHlvdSBoYXZlIGFu IGlzc3VlIHdpdGggdG9vIG1hbnkgYXBhY2hlIHByb2Nlc3NlcyB3YWl0aW5nIGZvciBlbmdpbmUg dG8gcmVzcG9uZCwgdGhlbiB0aGVyZSYjMzk7cyBzb21lIGlzc3VlIGluIGVuZ2luZS4gQXMgSSB3 cm90ZSBhYm92ZSBwbGVhc2Ugc2hhcmUgdGhlIGxvZ3Mgd2l0aCB1cyBhbmQgY2hlY2sgZW50cm9w eS48YnI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxz YW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj5UaGFua3M8YnI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj5N YXJ0aW4gUGVyaW5hPGJyPuKAizwvZGl2PsKgPC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdp bjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0 KTtwYWRkaW5nLWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxzcGFuPjxkaXYgc3R5bGU9 IndvcmQtd3JhcDpicmVhay13b3JkIj48ZGl2PjxkaXY+PGRpdj48YnI+PC9kaXY+PGRpdj5JIG5v dGljZWQgdGhhdCB0aGlzIHNldHRpbmcgcmV2ZXJ0cyB0byB0aGUgb3JpZ2luYWwgc2V0dGluZywg c28gb1ZpcnQgbXVzdCBiZSB3cml0aW5nIHRoaXMgZmlsZS4gUGVyaGFwcyB0aGVzZSBudW1iZXIg Y2FuIGJlIGNoYW5nZWQgaW4gb1ZpcnQ/IElmIG5vdCwgSSB3aWxsIGp1c3Qgc2V0dXAgYW5kIGFu c2libGUgcGxheSB0byByZXZlcnQgdGhlIHNldHRpbmdzIHdpdGggd29ya2luZyB2YWx1ZXMgYW5k IHJlc3RhcnQgYXBhY2hlIG9uIG15IGVuZ2luZS48L2Rpdj48ZGl2PjotKTwvZGl2PjxkaXY+Cjxk aXY+PGJyPkNoZWVycyw8YnI+R2VydmFpczxicj48YnI+PGJyPjwvZGl2PgoKPC9kaXY+CgoKCjxi cj48L2Rpdj48L2Rpdj48L2Rpdj48YnI+PC9zcGFuPl9fX19fX19fX19fX19fX19fX19fX19fX19f X19fXzx3YnI+X19fX19fX19fX19fX19fX188YnI+ClVzZXJzIG1haWxpbmcgbGlzdDxicj4KPGEg aHJlZj0ibWFpbHRvOlVzZXJzQG92aXJ0Lm9yZyIgdGFyZ2V0PSJfYmxhbmsiPlVzZXJzQG92aXJ0 Lm9yZzwvYT48YnI+CjxhIHJlbD0ibm9yZWZlcnJlciIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0 Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzIiB0YXJnZXQ9Il9ibGFuayI+aHR0cDovL2xpc3Rz Lm92aXJ0Lm9yZy9tYWlsbWFuPHdicj4vbGlzdGluZm8vdXNlcnM8L2E+PGJyPgo8YnI+PC9ibG9j a3F1b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4KPC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rp dj48L2Rpdj4KPC9ibG9ja3F1b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4K --===============5598247186357199401==-- From gervais at demontbrun.com Thu Sep 29 09:11:48 2016 Content-Type: multipart/mixed; boundary="===============4932116376986761683==" MIME-Version: 1.0 From: Gervais de Montbrun To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 10:11:43 -0300 Message-ID: <8960A4D2-C484-4D1F-B8DD-ABF3E3D0FF28@demontbrun.com> In-Reply-To: CAN8-ONrm3XjZnq9u9K=hmTkKUfP7cD2_7dObQbmxPaU9JNKeRg@mail.gmail.com --===============4932116376986761683== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable --Apple-Mail=3D_7BF15C59-FE49-4792-B9A3-069F1A828C8D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=3Dutf-8 Hi Simone, Thanks for the info. I'll look at the solution that you suggested. Cheers, Gervais > On Sep 29, 2016, at 10:01 AM, Simone Tiraboschi = =3D wrote: >=3D20 >=3D20 >=3D20 > On Thu, Sep 29, 2016 at 2:51 PM, Gervais de Montbrun =3D > wrote: > Hi Martin, >=3D20 > The entropy was super low. Somewhere around 140. I installed and =3D configured haveged.service to start at bootup, reverted my apache =3D changes... After a reboot, my systemctl status still says that there are = =3D 7 services queued (note that I erroneously said degraded in my previous =3D email - the services are, in fact, queued), but the oVirt GUI comes up =3D almost immediately and everything seems to be great. >=3D20 >=3D20 > Take care that using havaged on a VM should not be considered a good =3D source of entropy and the oVirt PKi is managed by the engine. > =3D http://security.stackexchange.com/questions/34523/is-it-appropriate-to-use= =3D -haveged-as-a-source-of-entropy-on-virtual-machines =3D >=3D20 > A better approach is the virtio-rng paravirtualised rng driver as for =3D patch https://gerrit.ovirt.org/#/c/62334/ =3D >=3D20 > =3D20 > Thank you for the tip. You solved my issue. >=3D20 > Cheers, > Gervais >=3D20 >=3D20 >=3D20 >> On Sep 29, 2016, at 7:47 AM, Martin Perina > wrote: >>=3D20 >> Hi, >>=3D20 >> please take a look at my inline comments: >>=3D20 >> On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun =3D > wrote: >> Hey All, >>=3D20 >> Since updating to 4.0.x of oVirt, I have had an issue with my hosted =3D engine. After a some poking around, I think I have figured out my issue =3D and thought I would share to see what others think. >> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists = =3D in 4.0.4. >>=3D20 >> Description: >> When my hosted engine starts it reports that it is in a degraded =3D state with 7 or 8 services still not started when I run systemctl =3D status. It takes about 6 or 7 minutes to eventually start all the =3D services and come online. If I don't set my cluster to =3D Global-Maintenance mode it eventually thinks that my hosted-engine needs = =3D to be rebooted and restarts it before it can start everything. >>=3D20 >> =3DE2=3D80=3D8BCould you please share with us logs gathered by =3D ovirt-log-collector? >>=3D20 >> It's just a guess but could you please take a look if you HE VM has =3D enough entropy? >>=3D20 >> cat /proc/sys/kernel/random/entropy_avail >>=3D20 >> If the value is low (below or around 200), you really need to =3D install and configure some entropy generator such as haveged >>=3D20 >>=3D20 >> Solution: >> I realized that Apache was the culprit and found that the proxy to =3D the ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a =3D super long timeout with many retries. I changed the settings and now =3D everything works for me. >>=3D20 >> -> Before change: >> >> ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D3600 retry=3D= 3D5=3D >>=3D20 >> >> AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml >> >> >>=3D20 >> -> After change: >> >> ProxyPassMatch ajp://127.0.0.1:8702 <> timeout=3D3D5 retry=3D3D2 >>=3D20 >> >> AddOutputFilterByType DEFLATE text/javascript text/css =3D text/html text/xml text/json application/xml application/json =3D application/x-yaml >> >> >>=3D20 >> =3DE2=3D80=3D8BThis one is correct for 4.0=3DE2=3D80=3D8B=3DE2=3D80=3D8B= , not sure why =3D it was not updated during upgrade from 3.6. @Simone? >> =3DE2=3D80=3D8B=3D20 >>=3D20 >> If I read the timeout settings correctly, it will wait 60 minutes =3D with 5 retries. 5 hours is way too long for my little server to hold =3D onto all those apache processes. >> The change I made allows for there to be an error, and also releases =3D apache's hold on the process. Once everything is ready, apache is ready =3D to serve requests and everything/everyone is happy. Before making the =3D change, I just get a whitescreen in my browser and then nothing works =3D until I restart Apache (or I end up in an endless loop of ovirt-ha =3D services restarting my hosted-engine. >>=3D20 >> =3DE2=3D80=3D8BWell, if you have an issue with too many apache processes= =3D waiting for engine to respond, then there's some issue in engine. As I =3D wrote above please share the logs with us and check entropy. >>=3D20 >> Thanks >>=3D20 >> Martin Perina >> =3DE2=3D80=3D8B=3D20 >>=3D20 >> I noticed that this setting reverts to the original setting, so oVirt = =3D must be writing this file. Perhaps these number can be changed in oVirt? = =3D If not, I will just setup and ansible play to revert the settings with =3D working values and restart apache on my engine. >> :-) >>=3D20 >> Cheers, >> Gervais >>=3D20 >>=3D20 >>=3D20 >>=3D20 >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users =3D --Apple-Mail=3D_7BF15C59-FE49-4792-B9A3-069F1A828C8D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=3Dutf-8 Hi Simone,

Thanks for the info. I'll look at the solution that you =3D suggested.

Cheers,
Gervais



On Sep 29, 2016, at 10:01 AM, Simone Tiraboschi <stirabos(a)redhat.com<= /a>>=3D wrote:


On =3D Thu, Sep 29, 2016 at 2:51 PM, Gervais de Montbrun <gervais(a)demontbrun.com> wrote:
Hi Martin,

The entropy was super low. Somewhere = =3D around 140. I installed and configured haveged.service to start at =3D bootup, reverted my apache changes... After a reboot, my systemctl =3D status still says that there are 7 services queued (note that I =3D erroneously said degraded in my previous email - the services are, in =3D fact, queued), but the oVirt GUI comes up almost immediately and =3D everything seems to be great.


Take care that using havaged on a VM should not be =3D considered a good source of entropy and the oVirt PKi is managed by the =3D engine.
http://security.stackexchange.com/questions/34523/is-it-appropriate-to-use-haveged-as-a-source-of-entropy-on-virtual-machines<= br =3D class=3D3D"">
A better approach is the virtio-rng =3D paravirtualised rng driver as for patch https://gerrit.ovirt.org/#/c/62334/

 
Thank you for the tip. You solved my issue.

Cheers,
Ger= vais


On Sep 29, 2016, at 7:47 AM, Martin Perina = =3D <mperina(a)redhat.com> wrote:

Hi,

please take a look at= my =3D inline comments:

On Tue, Sep 27, 2016 at 7:23 PM, = =3D Gervais de Montbrun <gervais(a)demontbrun.com> wrote:
Hey All,
<= br =3D class=3D3D"">
Since updating to 4.0.x of oVirt, I = =3D have had an issue with my hosted engine. After a some poking around, I =3D think I have figured out my issue and thought I would share to see what =3D others think.
The issue has existed with 4.0, =3D 4.0.1, 4.0.2, 4.0.3, and still exists in 4.0.4.

Description:
Wh= en =3D my hosted engine starts it reports that it is in a degraded state with 7 = =3D or 8 services still not started when I run systemctl status. It takes =3D about 6 or 7 minutes to eventually start all the services and come =3D online. If I don't set my cluster to Global-Maintenance mode it =3D eventually thinks that my hosted-engine needs to be rebooted and =3D restarts it before it can start everything.

=3DE2=3D80=3D8BCould you pl= ease =3D share with us logs gathered by ovirt-log-collector?

It's just a guess but could you please take a look if you HE = =3D VM has enough entropy?

  cat =3D /proc/sys/kernel/random/entropy_avail
If the value is low (below or around = =3D 200),  you really need to install and configure some entropy =3D generator such as haveged


Solution:
I =3D realized that Apache was the culprit and found that the proxy to the =3D ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a super long timeout with many retries. = =3D I changed the settings and now everything works for me.

-> Before =3D change:
    <LocationMatch =3D ^/(ovirt-engine($|/)|api($|/)|RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$|rhevm.ssh.key.txt$)>
    = =3D     ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D3600 =3D retry=3D3D5

 =3D       <IfModule =3D deflate_module>
        =3D     AddOutputFilterByType =3D DEFLATE text/javascript text/css text/html text/xml text/json =3D application/xml application/json application/x-yaml
        </IfModule>
    </LocationMatch>
= <=3D /div>

-> After =3D change:
<= div=3D class=3D3D"">    <LocationMatch =3D ^/ovirt-engine($|/)>
      =3D   ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3D5 =3D retry=3D3D2

 =3D       <IfModule =3D deflate_module>
        =3D     AddOutputFilterByType =3D DEFLATE text/javascript text/css text/html text/xml text/json =3D application/xml application/json application/x-yaml
        </IfModule>
    </LocationMatch>
= <=3D /div>

=3DE2=3D80=3D8BThis one is correct for 4.0=3DE2=3D80=3D8B
=3DE2=3D80=3D8B, not sure why it was not updated during upgrad= e =3D from 3.6. @Simone?
=3DE2=3D80=3D8B
 

If I rea= d =3D the timeout settings correctly, it will wait 60 minutes with 5 retries. =3D 5 hours is way too long for my little server to hold onto all those =3D apache processes.
Th= e =3D change I made allows for there to be an error, and also releases =3D apache's hold on the process. Once everything is ready, apache is ready =3D to serve requests and everything/everyone is happy. Before making the =3D change, I just get a whitescreen in my browser and then nothing works =3D until I restart Apache (or I end up in an endless loop of ovirt-ha =3D services restarting my hosted-engine.

=3DE2=3D80=3D8BWell, if you have an issue with too many apache= =3D processes waiting for engine to respond, then there's some issue in =3D engine. As I wrote above please share the logs with us and check =3D entropy.

Thanks

Martin Perina
=3DE2=3D80=3D8B
 

I noticed that this setting reverts to the original setting, = =3D so oVirt must be writing this file. Perhaps these number can be changed =3D in oVirt? If not, I will just setup and ansible play to revert the =3D settings with working values and restart apache on my engine.
:-)

Cheers,
Gervais




_______________________________________________
Users mailing list
Users(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
= =3D
=3D --Apple-Mail=3D_7BF15C59-FE49-4792-B9A3-069F1A828C8D-- --===============4932116376986761683== Content-Type: multipart/alternative MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.bin" Ci0tQXBwbGUtTWFpbD1fN0JGMTVDNTktRkU0OS00NzkyLUI5QTMtMDY5RjFBODI4QzhECkNvbnRl bnQtVHJhbnNmZXItRW5jb2Rpbmc6IHF1b3RlZC1wcmludGFibGUKQ29udGVudC1UeXBlOiB0ZXh0 L3BsYWluOwoJY2hhcnNldD11dGYtOAoKSGkgU2ltb25lLAoKVGhhbmtzIGZvciB0aGUgaW5mby4g SSdsbCBsb29rIGF0IHRoZSBzb2x1dGlvbiB0aGF0IHlvdSBzdWdnZXN0ZWQuCgpDaGVlcnMsCkdl cnZhaXMKCgoKPiBPbiBTZXAgMjksIDIwMTYsIGF0IDEwOjAxIEFNLCBTaW1vbmUgVGlyYWJvc2No aSA8c3RpcmFib3NAcmVkaGF0LmNvbT4gPQp3cm90ZToKPj0yMAo+PTIwCj49MjAKPiBPbiBUaHUs IFNlcCAyOSwgMjAxNiBhdCAyOjUxIFBNLCBHZXJ2YWlzIGRlIE1vbnRicnVuID0KPGdlcnZhaXNA ZGVtb250YnJ1bi5jb20gPG1haWx0bzpnZXJ2YWlzQGRlbW9udGJydW4uY29tPj4gd3JvdGU6Cj4g SGkgTWFydGluLAo+PTIwCj4gVGhlIGVudHJvcHkgd2FzIHN1cGVyIGxvdy4gU29tZXdoZXJlIGFy b3VuZCAxNDAuIEkgaW5zdGFsbGVkIGFuZCA9CmNvbmZpZ3VyZWQgaGF2ZWdlZC5zZXJ2aWNlIHRv IHN0YXJ0IGF0IGJvb3R1cCwgcmV2ZXJ0ZWQgbXkgYXBhY2hlID0KY2hhbmdlcy4uLiBBZnRlciBh IHJlYm9vdCwgbXkgc3lzdGVtY3RsIHN0YXR1cyBzdGlsbCBzYXlzIHRoYXQgdGhlcmUgYXJlID0K NyBzZXJ2aWNlcyBxdWV1ZWQgKG5vdGUgdGhhdCBJIGVycm9uZW91c2x5IHNhaWQgZGVncmFkZWQg aW4gbXkgcHJldmlvdXMgPQplbWFpbCAtIHRoZSBzZXJ2aWNlcyBhcmUsIGluIGZhY3QsIHF1ZXVl ZCksIGJ1dCB0aGUgb1ZpcnQgR1VJIGNvbWVzIHVwID0KYWxtb3N0IGltbWVkaWF0ZWx5IGFuZCBl dmVyeXRoaW5nIHNlZW1zIHRvIGJlIGdyZWF0Lgo+PTIwCj49MjAKPiBUYWtlIGNhcmUgdGhhdCB1 c2luZyBoYXZhZ2VkIG9uIGEgVk0gc2hvdWxkIG5vdCBiZSBjb25zaWRlcmVkIGEgZ29vZCA9CnNv dXJjZSBvZiBlbnRyb3B5IGFuZCB0aGUgb1ZpcnQgUEtpIGlzIG1hbmFnZWQgYnkgdGhlIGVuZ2lu ZS4KPiA9Cmh0dHA6Ly9zZWN1cml0eS5zdGFja2V4Y2hhbmdlLmNvbS9xdWVzdGlvbnMvMzQ1MjMv aXMtaXQtYXBwcm9wcmlhdGUtdG8tdXNlPQotaGF2ZWdlZC1hcy1hLXNvdXJjZS1vZi1lbnRyb3B5 LW9uLXZpcnR1YWwtbWFjaGluZXMgPQo8aHR0cDovL3NlY3VyaXR5LnN0YWNrZXhjaGFuZ2UuY29t L3F1ZXN0aW9ucy8zNDUyMy9pcy1pdC1hcHByb3ByaWF0ZS10by11cz0KZS1oYXZlZ2VkLWFzLWEt c291cmNlLW9mLWVudHJvcHktb24tdmlydHVhbC1tYWNoaW5lcz4KPj0yMAo+IEEgYmV0dGVyIGFw cHJvYWNoIGlzIHRoZSB2aXJ0aW8tcm5nIHBhcmF2aXJ0dWFsaXNlZCBybmcgZHJpdmVyIGFzIGZv ciA9CnBhdGNoIGh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvNjIzMzQvID0KPGh0dHBzOi8v Z2Vycml0Lm92aXJ0Lm9yZy8jL2MvNjIzMzQvPgo+PTIwCj4gPTIwCj4gVGhhbmsgeW91IGZvciB0 aGUgdGlwLiBZb3Ugc29sdmVkIG15IGlzc3VlLgo+PTIwCj4gQ2hlZXJzLAo+IEdlcnZhaXMKPj0y MAo+PTIwCj49MjAKPj4gT24gU2VwIDI5LCAyMDE2LCBhdCA3OjQ3IEFNLCBNYXJ0aW4gUGVyaW5h IDxtcGVyaW5hQHJlZGhhdC5jb20gPQo8bWFpbHRvOm1wZXJpbmFAcmVkaGF0LmNvbT4+IHdyb3Rl Ogo+Pj0yMAo+PiBIaSwKPj49MjAKPj4gcGxlYXNlIHRha2UgYSBsb29rIGF0IG15IGlubGluZSBj b21tZW50czoKPj49MjAKPj4gT24gVHVlLCBTZXAgMjcsIDIwMTYgYXQgNzoyMyBQTSwgR2VydmFp cyBkZSBNb250YnJ1biA9CjxnZXJ2YWlzQGRlbW9udGJydW4uY29tIDxtYWlsdG86Z2VydmFpc0Bk ZW1vbnRicnVuLmNvbT4+IHdyb3RlOgo+PiBIZXkgQWxsLAo+Pj0yMAo+PiBTaW5jZSB1cGRhdGlu ZyB0byA0LjAueCBvZiBvVmlydCwgSSBoYXZlIGhhZCBhbiBpc3N1ZSB3aXRoIG15IGhvc3RlZCA9 CmVuZ2luZS4gQWZ0ZXIgYSBzb21lIHBva2luZyBhcm91bmQsIEkgdGhpbmsgSSBoYXZlIGZpZ3Vy ZWQgb3V0IG15IGlzc3VlID0KYW5kIHRob3VnaHQgSSB3b3VsZCBzaGFyZSB0byBzZWUgd2hhdCBv dGhlcnMgdGhpbmsuCj4+IFRoZSBpc3N1ZSBoYXMgZXhpc3RlZCB3aXRoIDQuMCwgNC4wLjEsIDQu MC4yLCA0LjAuMywgYW5kIHN0aWxsIGV4aXN0cyA9CmluIDQuMC40Lgo+Pj0yMAo+PiBEZXNjcmlw dGlvbjoKPj4gV2hlbiBteSBob3N0ZWQgZW5naW5lIHN0YXJ0cyBpdCByZXBvcnRzIHRoYXQgaXQg aXMgaW4gYSBkZWdyYWRlZCA9CnN0YXRlIHdpdGggNyBvciA4IHNlcnZpY2VzIHN0aWxsIG5vdCBz dGFydGVkIHdoZW4gSSBydW4gc3lzdGVtY3RsID0Kc3RhdHVzLiBJdCB0YWtlcyBhYm91dCA2IG9y IDcgbWludXRlcyB0byBldmVudHVhbGx5IHN0YXJ0IGFsbCB0aGUgPQpzZXJ2aWNlcyBhbmQgY29t ZSBvbmxpbmUuIElmIEkgZG9uJ3Qgc2V0IG15IGNsdXN0ZXIgdG8gPQpHbG9iYWwtTWFpbnRlbmFu Y2UgbW9kZSBpdCBldmVudHVhbGx5IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1lbmdpbmUgbmVlZHMg PQp0byBiZSByZWJvb3RlZCBhbmQgcmVzdGFydHMgaXQgYmVmb3JlIGl0IGNhbiBzdGFydCBldmVy eXRoaW5nLgo+Pj0yMAo+PiA9RTI9ODA9OEJDb3VsZCB5b3UgcGxlYXNlIHNoYXJlIHdpdGggdXMg bG9ncyBnYXRoZXJlZCBieSA9Cm92aXJ0LWxvZy1jb2xsZWN0b3I/Cj4+PTIwCj4+IEl0J3MganVz dCBhIGd1ZXNzIGJ1dCBjb3VsZCB5b3UgcGxlYXNlIHRha2UgYSBsb29rIGlmIHlvdSBIRSBWTSBo YXMgPQplbm91Z2ggZW50cm9weT8KPj49MjAKPj4gICBjYXQgL3Byb2Mvc3lzL2tlcm5lbC9yYW5k b20vZW50cm9weV9hdmFpbAo+Pj0yMAo+PiBJZiB0aGUgdmFsdWUgaXMgbG93IChiZWxvdyBvciBh cm91bmQgMjAwKSwgIHlvdSByZWFsbHkgbmVlZCB0byA9Cmluc3RhbGwgYW5kIGNvbmZpZ3VyZSBz b21lIGVudHJvcHkgZ2VuZXJhdG9yIHN1Y2ggYXMgaGF2ZWdlZAo+Pj0yMAo+Pj0yMAo+PiBTb2x1 dGlvbjoKPj4gSSByZWFsaXplZCB0aGF0IEFwYWNoZSB3YXMgdGhlIGN1bHByaXQgYW5kIGZvdW5k IHRoYXQgdGhlIHByb3h5IHRvID0KdGhlIG92aXJ0LWVuZ2luZSBpbiAvZXRjL2h0dHBkL2NvbmYu ZC96LW92aXJ0LWVuZ2luZS1wcm94eS5jb25mIGhhcyBhID0Kc3VwZXIgbG9uZyB0aW1lb3V0IHdp dGggbWFueSByZXRyaWVzLiBJIGNoYW5nZWQgdGhlIHNldHRpbmdzIGFuZCBub3cgPQpldmVyeXRo aW5nIHdvcmtzIGZvciBtZS4KPj49MjAKPj4gLT4gQmVmb3JlIGNoYW5nZToKPj4gICAgIDxMb2Nh dGlvbk1hdGNoID0KXi8ob3ZpcnQtZW5naW5lKCR8Lyl8YXBpKCR8Lyl8UkhFVk1hbmFnZXJXZWIv fE92aXJ0RW5naW5lV2ViL3xjYS5jcnQkfGVuZ2k9Cm5lLnNzaC5rZXkudHh0JHxyaGV2bS5zc2gu a2V5LnR4dCQpPgo+PiAgICAgICAgIFByb3h5UGFzc01hdGNoIGFqcDovLzEyNy4wLjAuMTo4NzAy IDw+IHRpbWVvdXQ9M0QzNjAwIHJldHJ5PTNENT0KCj4+PTIwCj4+ICAgICAgICAgPElmTW9kdWxl IGRlZmxhdGVfbW9kdWxlPgo+PiAgICAgICAgICAgICBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVG TEFURSB0ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgPQp0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9q c29uIGFwcGxpY2F0aW9uL3htbCBhcHBsaWNhdGlvbi9qc29uID0KYXBwbGljYXRpb24veC15YW1s Cj4+ICAgICAgICAgPC9JZk1vZHVsZT4KPj4gICAgIDwvTG9jYXRpb25NYXRjaD4KPj49MjAKPj4g LT4gQWZ0ZXIgY2hhbmdlOgo+PiAgICAgPExvY2F0aW9uTWF0Y2ggXi9vdmlydC1lbmdpbmUoJHwv KT4KPj4gICAgICAgICBQcm94eVBhc3NNYXRjaCBhanA6Ly8xMjcuMC4wLjE6ODcwMiA8PiB0aW1l b3V0PTNENSByZXRyeT0zRDIKPj49MjAKPj4gICAgICAgICA8SWZNb2R1bGUgZGVmbGF0ZV9tb2R1 bGU+Cj4+ICAgICAgICAgICAgIEFkZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2 YXNjcmlwdCB0ZXh0L2NzcyA9CnRleHQvaHRtbCB0ZXh0L3htbCB0ZXh0L2pzb24gYXBwbGljYXRp b24veG1sIGFwcGxpY2F0aW9uL2pzb24gPQphcHBsaWNhdGlvbi94LXlhbWwKPj4gICAgICAgICA8 L0lmTW9kdWxlPgo+PiAgICAgPC9Mb2NhdGlvbk1hdGNoPgo+Pj0yMAo+PiA9RTI9ODA9OEJUaGlz IG9uZSBpcyBjb3JyZWN0IGZvciA0LjA9RTI9ODA9OEI9RTI9ODA9OEIsIG5vdCBzdXJlIHdoeSA9 Cml0IHdhcyBub3QgdXBkYXRlZCBkdXJpbmcgdXBncmFkZSBmcm9tIDMuNi4gQFNpbW9uZT8KPj4g PUUyPTgwPThCPTIwCj4+PTIwCj4+IElmIEkgcmVhZCB0aGUgdGltZW91dCBzZXR0aW5ncyBjb3Jy ZWN0bHksIGl0IHdpbGwgd2FpdCA2MCBtaW51dGVzID0Kd2l0aCA1IHJldHJpZXMuIDUgaG91cnMg aXMgd2F5IHRvbyBsb25nIGZvciBteSBsaXR0bGUgc2VydmVyIHRvIGhvbGQgPQpvbnRvIGFsbCB0 aG9zZSBhcGFjaGUgcHJvY2Vzc2VzLgo+PiBUaGUgY2hhbmdlIEkgbWFkZSBhbGxvd3MgZm9yIHRo ZXJlIHRvIGJlIGFuIGVycm9yLCBhbmQgYWxzbyByZWxlYXNlcyA9CmFwYWNoZSdzIGhvbGQgb24g dGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyByZWFkeSwgYXBhY2hlIGlzIHJlYWR5ID0K dG8gc2VydmUgcmVxdWVzdHMgYW5kIGV2ZXJ5dGhpbmcvZXZlcnlvbmUgaXMgaGFwcHkuIEJlZm9y ZSBtYWtpbmcgdGhlID0KY2hhbmdlLCBJIGp1c3QgZ2V0IGEgd2hpdGVzY3JlZW4gaW4gbXkgYnJv d3NlciBhbmQgdGhlbiBub3RoaW5nIHdvcmtzID0KdW50aWwgSSByZXN0YXJ0IEFwYWNoZSAob3Ig SSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhhID0Kc2VydmljZXMgcmVzdGFy dGluZyBteSBob3N0ZWQtZW5naW5lLgo+Pj0yMAo+PiA9RTI9ODA9OEJXZWxsLCBpZiB5b3UgaGF2 ZSBhbiBpc3N1ZSB3aXRoIHRvbyBtYW55IGFwYWNoZSBwcm9jZXNzZXMgPQp3YWl0aW5nIGZvciBl bmdpbmUgdG8gcmVzcG9uZCwgdGhlbiB0aGVyZSdzIHNvbWUgaXNzdWUgaW4gZW5naW5lLiBBcyBJ ID0Kd3JvdGUgYWJvdmUgcGxlYXNlIHNoYXJlIHRoZSBsb2dzIHdpdGggdXMgYW5kIGNoZWNrIGVu dHJvcHkuCj4+PTIwCj4+IFRoYW5rcwo+Pj0yMAo+PiBNYXJ0aW4gUGVyaW5hCj4+ID1FMj04MD04 Qj0yMAo+Pj0yMAo+PiBJIG5vdGljZWQgdGhhdCB0aGlzIHNldHRpbmcgcmV2ZXJ0cyB0byB0aGUg b3JpZ2luYWwgc2V0dGluZywgc28gb1ZpcnQgPQptdXN0IGJlIHdyaXRpbmcgdGhpcyBmaWxlLiBQ ZXJoYXBzIHRoZXNlIG51bWJlciBjYW4gYmUgY2hhbmdlZCBpbiBvVmlydD8gPQpJZiBub3QsIEkg d2lsbCBqdXN0IHNldHVwIGFuZCBhbnNpYmxlIHBsYXkgdG8gcmV2ZXJ0IHRoZSBzZXR0aW5ncyB3 aXRoID0Kd29ya2luZyB2YWx1ZXMgYW5kIHJlc3RhcnQgYXBhY2hlIG9uIG15IGVuZ2luZS4KPj4g Oi0pCj4+PTIwCj4+IENoZWVycywKPj4gR2VydmFpcwo+Pj0yMAo+Pj0yMAo+Pj0yMAo+Pj0yMAo+ PiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+PiBVc2Vy cyBtYWlsaW5nIGxpc3QKPj4gVXNlcnNAb3ZpcnQub3JnIDxtYWlsdG86VXNlcnNAb3ZpcnQub3Jn Pgo+PiBodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMgPQo8aHR0 cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3RpbmZvL3VzZXJzPgoKLS1BcHBsZS1NYWls PV83QkYxNUM1OS1GRTQ5LTQ3OTItQjlBMy0wNjlGMUE4MjhDOEQKQ29udGVudC1UcmFuc2Zlci1F bmNvZGluZzogcXVvdGVkLXByaW50YWJsZQpDb250ZW50LVR5cGU6IHRleHQvaHRtbDsKCWNoYXJz ZXQ9dXRmLTgKCjxodG1sPjxoZWFkPjxtZXRhIGh0dHAtZXF1aXY9M0QiQ29udGVudC1UeXBlIiBj b250ZW50PTNEInRleHQvaHRtbCA9CmNoYXJzZXQ9M0R1dGYtOCI+PC9oZWFkPjxib2R5IHN0eWxl PTNEIndvcmQtd3JhcDogYnJlYWstd29yZDsgPQotd2Via2l0LW5ic3AtbW9kZTogc3BhY2U7IC13 ZWJraXQtbGluZS1icmVhazogYWZ0ZXItd2hpdGUtc3BhY2U7IiA9CmNsYXNzPTNEIiI+SGkgU2lt b25lLDxkaXYgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2ID0KY2xhc3M9M0Qi Ij5UaGFua3MgZm9yIHRoZSBpbmZvLiBJJ2xsIGxvb2sgYXQgdGhlIHNvbHV0aW9uIHRoYXQgeW91 ID0Kc3VnZ2VzdGVkLjxiciBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj4KPGRpdiBpZD0zRCJz aWduYXR1cmUiIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+Q2hlZXJzLDxiciA9CmNsYXNzPTNE IiI+R2VydmFpczxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwv ZGl2PgoKPC9kaXY+CjxiciBjbGFzcz0zRCIiPjxkaXY+PGJsb2NrcXVvdGUgdHlwZT0zRCJjaXRl IiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPk9uIFNlcCAyOSwgMjAxNiwgYXQgMTA6MDEg QU0sIFNpbW9uZSBUaXJhYm9zY2hpICZsdDs8YSA9CmhyZWY9M0QibWFpbHRvOnN0aXJhYm9zQHJl ZGhhdC5jb20iIGNsYXNzPTNEIiI+c3RpcmFib3NAcmVkaGF0LmNvbTwvYT4mZ3Q7PQogd3JvdGU6 PC9kaXY+PGJyIGNsYXNzPTNEIkFwcGxlLWludGVyY2hhbmdlLW5ld2xpbmUiPjxkaXYgY2xhc3M9 M0QiIj48YnIgPQpjbGFzcz0zRCJBcHBsZS1pbnRlcmNoYW5nZS1uZXdsaW5lIj48YnIgc3R5bGU9 M0QiZm9udC1mYW1pbHk6IEhlbHZldGljYTsgPQpmb250LXNpemU6IDEycHg7IGZvbnQtc3R5bGU6 IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgPQpmb250LXdlaWdodDogbm9ybWFs OyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBoYW5zOiBhdXRvOyB0ZXh0LWFsaWduOiA9CnN0 YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6 IG5vcm1hbDsgPQp3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQt c3Ryb2tlLXdpZHRoOiAwcHg7IiA9CmNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCJnbWFpbF9xdW90 ZSIgc3R5bGU9M0QiZm9udC1mYW1pbHk6IEhlbHZldGljYTsgPQpmb250LXNpemU6IDEycHg7IGZv bnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5vcm1hbDsgPQpmb250LXdlaWdo dDogbm9ybWFsOyBsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBoYW5zOiBhdXRvOyB0ZXh0LWFs aWduOiA9CnN0YXJ0OyB0ZXh0LWluZGVudDogMHB4OyB0ZXh0LXRyYW5zZm9ybTogbm9uZTsgd2hp dGUtc3BhY2U6IG5vcm1hbDsgPQp3aWRvd3M6IGF1dG87IHdvcmQtc3BhY2luZzogMHB4OyAtd2Vi a2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7Ij5PbiA9ClRodSwgU2VwIDI5LCAyMDE2IGF0IDI6 NTEgUE0sIEdlcnZhaXMgZGUgTW9udGJydW48c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRl ZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPjxzcGFuIGRpcj0zRCJsdHIiID0KY2xhc3M9M0QiIj4mbHQ7 PGEgaHJlZj0zRCJtYWlsdG86Z2VydmFpc0BkZW1vbnRicnVuLmNvbSIgdGFyZ2V0PTNEIl9ibGFu ayI9CiBjbGFzcz0zRCIiPmdlcnZhaXNAZGVtb250YnJ1bi5jb208L2E+Jmd0Ozwvc3Bhbj48c3Bh biA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPndyb3RlOjxi ciA9CmNsYXNzPTNEIiI+PGJsb2NrcXVvdGUgY2xhc3M9M0QiZ21haWxfcXVvdGUiIHN0eWxlPTNE Im1hcmdpbjogMHB4IDBweCA9CjBweCAwLjhleDsgYm9yZGVyLWxlZnQtd2lkdGg6IDFweDsgYm9y ZGVyLWxlZnQtc3R5bGU6IHNvbGlkOyA9CmJvcmRlci1sZWZ0LWNvbG9yOiByZ2IoMjA0LCAyMDQs IDIwNCk7IHBhZGRpbmctbGVmdDogMWV4OyI+PGRpdiA9CnN0eWxlPTNEIndvcmQtd3JhcDogYnJl YWstd29yZDsiIGNsYXNzPTNEIiI+SGkgTWFydGluLDxkaXYgY2xhc3M9M0QiIj48YnIgPQpjbGFz cz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5UaGUgZW50cm9weSB3YXMgc3VwZXIgbG93LiBT b21ld2hlcmUgPQphcm91bmQgMTQwLiBJIGluc3RhbGxlZCBhbmQgY29uZmlndXJlZCZuYnNwO2hh dmVnZWQuc2VydmljZSB0byBzdGFydCBhdCA9CmJvb3R1cCwgcmV2ZXJ0ZWQgbXkgYXBhY2hlIGNo YW5nZXMuLi4gQWZ0ZXIgYSByZWJvb3QsIG15IHN5c3RlbWN0bCA9CnN0YXR1cyBzdGlsbCBzYXlz IHRoYXQgdGhlcmUgYXJlIDcgc2VydmljZXMgcXVldWVkIChub3RlIHRoYXQgSSA9CmVycm9uZW91 c2x5IHNhaWQgZGVncmFkZWQgaW4gbXkgcHJldmlvdXMgZW1haWwgLSB0aGUgc2VydmljZXMgYXJl LCBpbiA9CmZhY3QsIHF1ZXVlZCksIGJ1dCB0aGUgb1ZpcnQgR1VJIGNvbWVzIHVwIGFsbW9zdCBp bW1lZGlhdGVseSBhbmQgPQpldmVyeXRoaW5nIHNlZW1zIHRvIGJlIGdyZWF0LjwvZGl2PjxkaXYg Y2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2 IGNsYXNzPTNEImdtYWlsX3F1b3RlIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PlRha2UgY2FyZSB0 aGF0IHVzaW5nIGhhdmFnZWQgb24gYSBWTSBzaG91bGQgbm90IGJlID0KY29uc2lkZXJlZCBhIGdv b2Qgc291cmNlIG9mIGVudHJvcHkgYW5kIHRoZSBvVmlydCBQS2kgaXMgbWFuYWdlZCBieSB0aGUg PQplbmdpbmUuPGJyIGNsYXNzPTNEIiI+PGEgPQpocmVmPTNEImh0dHA6Ly9zZWN1cml0eS5zdGFj a2V4Y2hhbmdlLmNvbS9xdWVzdGlvbnMvMzQ1MjMvaXMtaXQtYXBwcm9wcmlhdD0KZS10by11c2Ut aGF2ZWdlZC1hcy1hLXNvdXJjZS1vZi1lbnRyb3B5LW9uLXZpcnR1YWwtbWFjaGluZXMiID0KdGFy Z2V0PTNEIl9ibGFuayIgY2xhc3M9M0QiIj5odHRwOi8vc2VjdXJpdHkuc3RhY2tleGNoYW5nZS48 d2JyID0KY2xhc3M9M0QiIj5jb20vcXVlc3Rpb25zLzM0NTIzL2lzLWl0LTx3YnIgPQpjbGFzcz0z RCIiPmFwcHJvcHJpYXRlLXRvLXVzZS1oYXZlZ2VkLWFzLTx3YnIgPQpjbGFzcz0zRCIiPmEtc291 cmNlLW9mLWVudHJvcHktb24tPHdiciBjbGFzcz0zRCIiPnZpcnR1YWwtbWFjaGluZXM8L2E+PGJy ID0KY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj5BIGJldHRlciBhcHByb2FjaCBpcyB0aGUgdmly dGlvLXJuZyA9CnBhcmF2aXJ0dWFsaXNlZCBybmcgZHJpdmVyIGFzIGZvciBwYXRjaCZuYnNwOzxh ID0KaHJlZj0zRCJodHRwczovL2dlcnJpdC5vdmlydC5vcmcvIy9jLzYyMzM0LyIgPQpjbGFzcz0z RCIiPmh0dHBzOi8vZ2Vycml0Lm92aXJ0Lm9yZy8jL2MvNjIzMzQvPC9hPjwvZGl2PjxkaXYgPQpj bGFzcz0zRCJnbWFpbF9xdW90ZSIgc3R5bGU9M0QiZm9udC1mYW1pbHk6IEhlbHZldGljYTsgZm9u dC1zaXplOiAxMnB4OyA9CmZvbnQtc3R5bGU6IG5vcm1hbDsgZm9udC12YXJpYW50LWNhcHM6IG5v cm1hbDsgZm9udC13ZWlnaHQ6IG5vcm1hbDsgPQpsZXR0ZXItc3BhY2luZzogbm9ybWFsOyBvcnBo YW5zOiBhdXRvOyB0ZXh0LWFsaWduOiBzdGFydDsgdGV4dC1pbmRlbnQ6ID0KMHB4OyB0ZXh0LXRy YW5zZm9ybTogbm9uZTsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgd2lkb3dzOiBhdXRvOyA9CndvcmQt c3BhY2luZzogMHB4OyAtd2Via2l0LXRleHQtc3Ryb2tlLXdpZHRoOiAwcHg7Ij48YnIgY2xhc3M9 M0QiIj48ZGl2ID0KY2xhc3M9M0QiIj4mbmJzcDs8L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0zRCJn bWFpbF9xdW90ZSIgc3R5bGU9M0QibWFyZ2luOiA9CjBweCAwcHggMHB4IDAuOGV4OyBib3JkZXIt bGVmdC13aWR0aDogMXB4OyBib3JkZXItbGVmdC1zdHlsZTogc29saWQ7ID0KYm9yZGVyLWxlZnQt Y29sb3I6IHJnYigyMDQsIDIwNCwgMjA0KTsgcGFkZGluZy1sZWZ0OiAxZXg7Ij48ZGl2ID0Kc3R5 bGU9M0Qid29yZC13cmFwOiBicmVhay13b3JkOyIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+ PC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+VGhhbmsgeW91IGZvciB0aGUgdGlwLiBZb3Ugc29sdmVk IG15IGlzc3VlLjxiciBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0Qi Ij48YnIgY2xhc3M9M0QiIj5DaGVlcnMsPGJyIGNsYXNzPTNEIiI+R2VydmFpczxicj0KIGNsYXNz PTNEIiI+PGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PC9kaXY+PGRpdiBjbGFz cz0zRCIiPjxkaXY9CiBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0QiIj48 YmxvY2txdW90ZSB0eXBlPTNEImNpdGUiID0KY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+T24g U2VwIDI5LCAyMDE2LCBhdCA3OjQ3IEFNLCBNYXJ0aW4gUGVyaW5hID0KJmx0OzxhIGhyZWY9M0Qi bWFpbHRvOm1wZXJpbmFAcmVkaGF0LmNvbSIgdGFyZ2V0PTNEIl9ibGFuayIgPQpjbGFzcz0zRCIi Pm1wZXJpbmFAcmVkaGF0LmNvbTwvYT4mZ3Q7IHdyb3RlOjwvZGl2PjxiciBjbGFzcz0zRCIiPjxk aXYgPQpjbGFzcz0zRCIiPjxkaXYgZGlyPTNEImx0ciIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNE IiI+SGksPGJyID0KY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNE IiI+cGxlYXNlIHRha2UgYSBsb29rIGF0IG15ID0KaW5saW5lIGNvbW1lbnRzOjxiciBjbGFzcz0z RCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiZ21haWxfZXh0cmEiPjxiciA9CmNsYXNzPTNEIiI+PGRp diBjbGFzcz0zRCJnbWFpbF9xdW90ZSI+T24gVHVlLCBTZXAgMjcsIDIwMTYgYXQgNzoyMyBQTSwg PQpHZXJ2YWlzIGRlIE1vbnRicnVuPHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3Bh Y2UiPiZuYnNwOzwvc3Bhbj48c3BhbiBkaXI9M0QibHRyIiA9CmNsYXNzPTNEIiI+Jmx0OzxhIGhy ZWY9M0QibWFpbHRvOmdlcnZhaXNAZGVtb250YnJ1bi5jb20iIHRhcmdldD0zRCJfYmxhbmsiPQog Y2xhc3M9M0QiIj5nZXJ2YWlzQGRlbW9udGJydW4uY29tPC9hPiZndDs8L3NwYW4+PHNwYW4gPQpj bGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj53cm90ZTo8YnIgPQpj bGFzcz0zRCIiPjxibG9ja3F1b3RlIGNsYXNzPTNEImdtYWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJn aW46IDBweCAwcHggPQowcHggMC44ZXg7IGJvcmRlci1sZWZ0LXdpZHRoOiAxcHg7IGJvcmRlci1s ZWZ0LXN0eWxlOiBzb2xpZDsgPQpib3JkZXItbGVmdC1jb2xvcjogcmdiKDIwNCwgMjA0LCAyMDQp OyBwYWRkaW5nLWxlZnQ6IDFleDsiPjxkaXYgPQpzdHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdv cmQ7IiBjbGFzcz0zRCIiPkhleSBBbGwsPGRpdiBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+ PC9kaXY+PGRpdiBjbGFzcz0zRCIiPlNpbmNlIHVwZGF0aW5nIHRvIDQuMC54IG9mIG9WaXJ0LCBJ ID0KaGF2ZSBoYWQgYW4gaXNzdWUgd2l0aCBteSBob3N0ZWQgZW5naW5lLiBBZnRlciBhIHNvbWUg cG9raW5nIGFyb3VuZCwgSSA9CnRoaW5rIEkgaGF2ZSBmaWd1cmVkIG91dCBteSBpc3N1ZSBhbmQg dGhvdWdodCBJIHdvdWxkIHNoYXJlIHRvIHNlZSB3aGF0ID0Kb3RoZXJzIHRoaW5rLjwvZGl2Pjxk aXYgY2xhc3M9M0QiIj5UaGUgaXNzdWUgaGFzIGV4aXN0ZWQgd2l0aCA0LjAsID0KNC4wLjEsIDQu MC4yLCA0LjAuMywgYW5kIHN0aWxsIGV4aXN0cyBpbiA0LjAuNC48L2Rpdj48ZGl2IGNsYXNzPTNE IiI+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+RGVzY3JpcHRpb246PC9k aXY+PGRpdiBjbGFzcz0zRCIiPldoZW4gPQpteSBob3N0ZWQgZW5naW5lIHN0YXJ0cyBpdCByZXBv cnRzIHRoYXQgaXQgaXMgaW4gYSBkZWdyYWRlZCBzdGF0ZSB3aXRoIDcgPQpvciA4IHNlcnZpY2Vz IHN0aWxsIG5vdCBzdGFydGVkIHdoZW4gSSBydW4gc3lzdGVtY3RsIHN0YXR1cy4gSXQgdGFrZXMg PQphYm91dCA2IG9yIDcgbWludXRlcyB0byBldmVudHVhbGx5IHN0YXJ0IGFsbCB0aGUgc2Vydmlj ZXMgYW5kIGNvbWUgPQpvbmxpbmUuIElmIEkgZG9uJ3Qgc2V0IG15IGNsdXN0ZXIgdG8gR2xvYmFs LU1haW50ZW5hbmNlIG1vZGUgaXQgPQpldmVudHVhbGx5IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1l bmdpbmUgbmVlZHMgdG8gYmUgcmVib290ZWQgYW5kID0KcmVzdGFydHMgaXQgYmVmb3JlIGl0IGNh biBzdGFydCBldmVyeXRoaW5nLjwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2ID0KY2xhc3M9 M0QiIj48YnIgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PUUyPTgwPThCQ291bGQgeW91IHBs ZWFzZSA9CnNoYXJlIHdpdGggdXMgbG9ncyBnYXRoZXJlZCBieSBvdmlydC1sb2ctY29sbGVjdG9y PzxiciBjbGFzcz0zRCIiPjxiciA9CmNsYXNzPTNEIiI+SXQncyBqdXN0IGEgZ3Vlc3MgYnV0IGNv dWxkIHlvdSBwbGVhc2UgdGFrZSBhIGxvb2sgaWYgeW91IEhFID0KVk0gaGFzIGVub3VnaCBlbnRy b3B5PzxiciBjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPiZuYnNwOzxzcGFuID0KY2xhc3M9M0Qi QXBwbGUtY29udmVydGVkLXNwYWNlIj4mbmJzcDs8L3NwYW4+Y2F0ID0KL3Byb2Mvc3lzL2tlcm5l bC9yYW5kb20vZW50cm9wPHdiciBjbGFzcz0zRCIiPnlfYXZhaWw8YnIgY2xhc3M9M0QiIj48YnIg PQpjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5JZiB0aGUgdmFsdWUgaXMgbG93IChi ZWxvdyBvciBhcm91bmQgPQoyMDApLCZuYnNwOyB5b3UgcmVhbGx5IG5lZWQgdG8gaW5zdGFsbCBh bmQgY29uZmlndXJlIHNvbWUgZW50cm9weSA9CmdlbmVyYXRvciBzdWNoIGFzIGhhdmVnZWQ8YnIg Y2xhc3M9M0QiIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjwvZGl2PjxibG9ja3F1b3RlIGNsYXNz PTNEImdtYWlsX3F1b3RlIiBzdHlsZT0zRCJtYXJnaW46ID0KMHB4IDBweCAwcHggMC44ZXg7IGJv cmRlci1sZWZ0LXdpZHRoOiAxcHg7IGJvcmRlci1sZWZ0LXN0eWxlOiBzb2xpZDsgPQpib3JkZXIt bGVmdC1jb2xvcjogcmdiKDIwNCwgMjA0LCAyMDQpOyBwYWRkaW5nLWxlZnQ6IDFleDsiPjxkaXYg PQpzdHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdvcmQ7IiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9 M0QiIj48YnIgPQpjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj5Tb2x1dGlvbjo8L2Rp dj48ZGl2IGNsYXNzPTNEIiI+SSA9CnJlYWxpemVkIHRoYXQgQXBhY2hlIHdhcyB0aGUgY3VscHJp dCBhbmQgZm91bmQgdGhhdCB0aGUgcHJveHkgdG8gdGhlID0Kb3ZpcnQtZW5naW5lIGluJm5ic3A7 L2V0Yy9odHRwZC9jb25mLmQvei1vdmlydC1lPHdiciA9CmNsYXNzPTNEIiI+bmdpbmUtcHJveHku Y29uZiBoYXMgYSBzdXBlciBsb25nIHRpbWVvdXQgd2l0aCBtYW55IHJldHJpZXMuID0KSSBjaGFu Z2VkIHRoZSBzZXR0aW5ncyBhbmQgbm93IGV2ZXJ5dGhpbmcgd29ya3MgZm9yIG1lLjwvZGl2Pjxk aXYgPQpjbGFzcz0zRCIiPjxiciBjbGFzcz0zRCIiPjwvZGl2PjxkaXYgY2xhc3M9M0QiIj4tJmd0 OyBCZWZvcmUgPQpjaGFuZ2U6PC9kaXY+PGJsb2NrcXVvdGUgc3R5bGU9M0QibWFyZ2luOiAwcHgg MHB4IDBweCA0MHB4OyA9CmJvcmRlci13aWR0aDogbWVkaXVtOyBib3JkZXItc3R5bGU6IG5vbmU7 IHBhZGRpbmc6IDBweDsiIGNsYXNzPTNEIiI+PGRpdiA9CmNsYXNzPTNEIiI+PGRpdiBjbGFzcz0z RCIiPiZuYnNwOyAmbmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+ Jm5ic3A7PC9zcGFuPiZsdDtMb2NhdGlvbk1hdGNoID0KXi8ob3ZpcnQtZW5naW5lKCR8Lyl8YXBp KCR8Lyl8PHdiciA9CmNsYXNzPTNEIiI+UkhFVk1hbmFnZXJXZWIvfE92aXJ0RW5naW5lV2ViPHdi ciA9CmNsYXNzPTNEIiI+L3xjYS5jcnQkfGVuZ2luZS5zc2gua2V5LnR4dCR8PHdiciA9CmNsYXNz PTNEIiI+cmhldm0uc3NoLmtleS50eHQkKSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7 ICZuYnNwOyA9CiZuYnNwOyAmbmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1z cGFjZSI+Jm5ic3A7PC9zcGFuPlByb3h5UGFzc01hdGNoPHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1j b252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj48YSA9CmNsYXNzPTNEIiI+YWpwOi8vMTI3LjAu MC4xOjg3MDI8L2E+PHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNw Ozwvc3Bhbj50aW1lb3V0PTNEMzYwMCA9CnJldHJ5PTNENTwvZGl2PjxkaXYgY2xhc3M9M0QiIj48 YnIgY2xhc3M9M0QiIj48L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7PQogJm5ic3A7ICZuYnNw OyAmbmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9z cGFuPiZsdDtJZk1vZHVsZSA9CmRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0Qi Ij4mbmJzcDsgJm5ic3A7ICZuYnNwOyAmbmJzcDsgPQombmJzcDsgJm5ic3A7PHNwYW4gPQpjbGFz cz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5BZGRPdXRwdXRGaWx0ZXJC eVR5cGUgPQpERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94 bWwgdGV4dC9qc29uID0KYXBwbGljYXRpb24veG1sIGFwcGxpY2F0aW9uL2pzb24gYXBwbGljYXRp b24veC15YW1sPC9kaXY+PGRpdiA9CmNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5i c3A7PHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj4m bHQ7L0lmTW9kdWxlJmd0OzwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDs8c3Bh biA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPiZsdDsvTG9j YXRpb25NYXRjaCZndDs8L2Rpdj48PQovZGl2PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPTNEIiI+ PGJyIGNsYXNzPTNEIiI+PC9kaXY+LSZndDsgQWZ0ZXIgPQpjaGFuZ2U6PGJsb2NrcXVvdGUgc3R5 bGU9M0QibWFyZ2luOiAwcHggMHB4IDBweCA0MHB4OyBib3JkZXItd2lkdGg6ID0KbWVkaXVtOyBi b3JkZXItc3R5bGU6IG5vbmU7IHBhZGRpbmc6IDBweDsiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0z RCIiPjxkaXY9CiBjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFwcGxl LWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPiZsdDtMb2NhdGlvbk1hdGNoID0KXi9vdmly dC1lbmdpbmUoJHwvKSZndDs8L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbmJz cDsgPQombmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7 PC9zcGFuPlByb3h5UGFzc01hdGNoPHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3Bh Y2UiPiZuYnNwOzwvc3Bhbj48YSA9CmNsYXNzPTNEIiI+YWpwOi8vMTI3LjAuMC4xOjg3MDI8L2E+ PHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj50aW1l b3V0PTNENSA9CnJldHJ5PTNEMjwvZGl2PjxkaXYgY2xhc3M9M0QiIj48YnIgY2xhc3M9M0QiIj48 L2Rpdj48ZGl2IGNsYXNzPTNEIiI+Jm5ic3A7PQogJm5ic3A7ICZuYnNwOyAmbmJzcDs8c3BhbiA9 CmNsYXNzPTNEIkFwcGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPiZsdDtJZk1vZHVs ZSA9CmRlZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXYgY2xhc3M9M0QiIj4mbmJzcDsgJm5ic3A7 ICZuYnNwOyAmbmJzcDsgPQombmJzcDsgJm5ic3A7PHNwYW4gPQpjbGFzcz0zRCJBcHBsZS1jb252 ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj5BZGRPdXRwdXRGaWx0ZXJCeVR5cGUgPQpERUZMQVRF IHRleHQvamF2YXNjcmlwdCB0ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uID0K YXBwbGljYXRpb24veG1sIGFwcGxpY2F0aW9uL2pzb24gYXBwbGljYXRpb24veC15YW1sPC9kaXY+ PGRpdiA9CmNsYXNzPTNEIiI+Jm5ic3A7ICZuYnNwOyAmbmJzcDsgJm5ic3A7PHNwYW4gPQpjbGFz cz0zRCJBcHBsZS1jb252ZXJ0ZWQtc3BhY2UiPiZuYnNwOzwvc3Bhbj4mbHQ7L0lmTW9kdWxlJmd0 OzwvZGl2PjxkaXYgPQpjbGFzcz0zRCIiPiZuYnNwOyAmbmJzcDs8c3BhbiA9CmNsYXNzPTNEIkFw cGxlLWNvbnZlcnRlZC1zcGFjZSI+Jm5ic3A7PC9zcGFuPiZsdDsvTG9jYXRpb25NYXRjaCZndDs8 L2Rpdj48PQovZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdiBjbGFzcz0z RCIiPjxiciBjbGFzcz0zRCIiPjxkaXYgPQpzdHlsZT0zRCJmb250LWZhbWlseTogYXJpYWwsIGhl bHZldGljYSwgc2Fucy1zZXJpZjsgZGlzcGxheTogaW5saW5lOyIgPQpjbGFzcz0zRCIiPj1FMj04 MD04QlRoaXMgb25lIGlzIGNvcnJlY3QgZm9yIDQuMD1FMj04MD04QjwvZGl2PjxkaXYgPQpzdHls ZT0zRCJmb250LWZhbWlseTogYXJpYWwsIGhlbHZldGljYSwgc2Fucy1zZXJpZjsgZGlzcGxheTog aW5saW5lOyIgPQpjbGFzcz0zRCIiPj1FMj04MD04Qiwgbm90IHN1cmUgd2h5IGl0IHdhcyBub3Qg dXBkYXRlZCBkdXJpbmcgdXBncmFkZSA9CmZyb20gMy42LiBAU2ltb25lPzxiciBjbGFzcz0zRCIi Pj1FMj04MD04QjwvZGl2PiZuYnNwOzwvZGl2PjxibG9ja3F1b3RlID0KY2xhc3M9M0QiZ21haWxf cXVvdGUiIHN0eWxlPTNEIm1hcmdpbjogMHB4IDBweCAwcHggMC44ZXg7ID0KYm9yZGVyLWxlZnQt d2lkdGg6IDFweDsgYm9yZGVyLWxlZnQtc3R5bGU6IHNvbGlkOyBib3JkZXItbGVmdC1jb2xvcjog PQpyZ2IoMjA0LCAyMDQsIDIwNCk7IHBhZGRpbmctbGVmdDogMWV4OyI+PGRpdiBzdHlsZT0zRCJ3 b3JkLXdyYXA6ID0KYnJlYWstd29yZDsiIGNsYXNzPTNEIiI+PGRpdiBjbGFzcz0zRCIiPjxiciBj bGFzcz0zRCIiPjwvZGl2PklmIEkgcmVhZCA9CnRoZSB0aW1lb3V0IHNldHRpbmdzIGNvcnJlY3Rs eSwgaXQgd2lsbCB3YWl0IDYwIG1pbnV0ZXMgd2l0aCA1IHJldHJpZXMuID0KNSBob3VycyBpcyB3 YXkgdG9vIGxvbmcgZm9yIG15IGxpdHRsZSBzZXJ2ZXIgdG8gaG9sZCBvbnRvIGFsbCB0aG9zZSA9 CmFwYWNoZSBwcm9jZXNzZXMuPC9kaXY+PC9ibG9ja3F1b3RlPjxibG9ja3F1b3RlIGNsYXNzPTNE ImdtYWlsX3F1b3RlIiA9CnN0eWxlPTNEIm1hcmdpbjogMHB4IDBweCAwcHggMC44ZXg7IGJvcmRl ci1sZWZ0LXdpZHRoOiAxcHg7ID0KYm9yZGVyLWxlZnQtc3R5bGU6IHNvbGlkOyBib3JkZXItbGVm dC1jb2xvcjogcmdiKDIwNCwgMjA0LCAyMDQpOyA9CnBhZGRpbmctbGVmdDogMWV4OyI+PGRpdiBz dHlsZT0zRCJ3b3JkLXdyYXA6IGJyZWFrLXdvcmQ7IiBjbGFzcz0zRCIiPlRoZSA9CmNoYW5nZSBJ IG1hZGUgYWxsb3dzIGZvciB0aGVyZSB0byBiZSBhbiBlcnJvciwgYW5kIGFsc28gcmVsZWFzZXMg PQphcGFjaGUncyBob2xkIG9uIHRoZSBwcm9jZXNzLiBPbmNlIGV2ZXJ5dGhpbmcgaXMgcmVhZHks IGFwYWNoZSBpcyByZWFkeSA9CnRvIHNlcnZlIHJlcXVlc3RzIGFuZCBldmVyeXRoaW5nL2V2ZXJ5 b25lIGlzIGhhcHB5LiBCZWZvcmUgbWFraW5nIHRoZSA9CmNoYW5nZSwgSSBqdXN0IGdldCBhIHdo aXRlc2NyZWVuIGluIG15IGJyb3dzZXIgYW5kIHRoZW4gbm90aGluZyB3b3JrcyA9CnVudGlsIEkg cmVzdGFydCBBcGFjaGUgKG9yIEkgZW5kIHVwIGluIGFuIGVuZGxlc3MgbG9vcCBvZiBvdmlydC1o YSA9CnNlcnZpY2VzIHJlc3RhcnRpbmcgbXkgaG9zdGVkLWVuZ2luZS48YnIgPQpjbGFzcz0zRCIi PjwvZGl2PjwvYmxvY2txdW90ZT48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PGRpdiA9 CnN0eWxlPTNEImZvbnQtZmFtaWx5OiBhcmlhbCwgaGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBkaXNw bGF5OiBpbmxpbmU7IiA9CmNsYXNzPTNEIiI+PUUyPTgwPThCV2VsbCwgaWYgeW91IGhhdmUgYW4g aXNzdWUgd2l0aCB0b28gbWFueSBhcGFjaGUgPQpwcm9jZXNzZXMgd2FpdGluZyBmb3IgZW5naW5l IHRvIHJlc3BvbmQsIHRoZW4gdGhlcmUncyBzb21lIGlzc3VlIGluID0KZW5naW5lLiBBcyBJIHdy b3RlIGFib3ZlIHBsZWFzZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayA9CmVudHJv cHkuPGJyIGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiBzdHlsZT0zRCJmb250 LWZhbWlseTogPQphcmlhbCwgaGVsdmV0aWNhLCBzYW5zLXNlcmlmOyBkaXNwbGF5OiBpbmxpbmU7 IiBjbGFzcz0zRCIiPlRoYW5rczxiciA9CmNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+ PGRpdiBzdHlsZT0zRCJmb250LWZhbWlseTogYXJpYWwsID0KaGVsdmV0aWNhLCBzYW5zLXNlcmlm OyBkaXNwbGF5OiBpbmxpbmU7IiBjbGFzcz0zRCIiPk1hcnRpbiBQZXJpbmE8YnIgPQpjbGFzcz0z RCIiPj1FMj04MD04QjwvZGl2PiZuYnNwOzwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPTNEImdtYWls X3F1b3RlIiA9CnN0eWxlPTNEIm1hcmdpbjogMHB4IDBweCAwcHggMC44ZXg7IGJvcmRlci1sZWZ0 LXdpZHRoOiAxcHg7ID0KYm9yZGVyLWxlZnQtc3R5bGU6IHNvbGlkOyBib3JkZXItbGVmdC1jb2xv cjogcmdiKDIwNCwgMjA0LCAyMDQpOyA9CnBhZGRpbmctbGVmdDogMWV4OyI+PGRpdiBzdHlsZT0z RCJ3b3JkLXdyYXA6IGJyZWFrLXdvcmQ7IiBjbGFzcz0zRCIiPjxkaXYgPQpjbGFzcz0zRCIiPjxk aXYgY2xhc3M9M0QiIj48ZGl2IGNsYXNzPTNEIiI+PGJyIGNsYXNzPTNEIiI+PC9kaXY+PGRpdiA9 CmNsYXNzPTNEIiI+SSBub3RpY2VkIHRoYXQgdGhpcyBzZXR0aW5nIHJldmVydHMgdG8gdGhlIG9y aWdpbmFsIHNldHRpbmcsID0Kc28gb1ZpcnQgbXVzdCBiZSB3cml0aW5nIHRoaXMgZmlsZS4gUGVy aGFwcyB0aGVzZSBudW1iZXIgY2FuIGJlIGNoYW5nZWQgPQppbiBvVmlydD8gSWYgbm90LCBJIHdp bGwganVzdCBzZXR1cCBhbmQgYW5zaWJsZSBwbGF5IHRvIHJldmVydCB0aGUgPQpzZXR0aW5ncyB3 aXRoIHdvcmtpbmcgdmFsdWVzIGFuZCByZXN0YXJ0IGFwYWNoZSBvbiBteSBlbmdpbmUuPC9kaXY+ PGRpdiA9CmNsYXNzPTNEIiI+Oi0pPC9kaXY+PGRpdiBjbGFzcz0zRCIiPjxkaXYgY2xhc3M9M0Qi Ij48YnIgPQpjbGFzcz0zRCIiPkNoZWVycyw8YnIgY2xhc3M9M0QiIj5HZXJ2YWlzPGJyIGNsYXNz PTNEIiI+PGJyIGNsYXNzPTNEIiI+PGJyID0KY2xhc3M9M0QiIj48L2Rpdj48L2Rpdj48YnIgY2xh c3M9M0QiIj48L2Rpdj48L2Rpdj48L2Rpdj48YnIgPQpjbGFzcz0zRCIiPl9fX19fX19fX19fX19f X19fX19fX19fX19fX19fXzx3YnIgPQpjbGFzcz0zRCIiPl9fX19fX19fX19fX19fX19fPGJyIGNs YXNzPTNEIiI+VXNlcnMgbWFpbGluZyBsaXN0PGJyID0KY2xhc3M9M0QiIj48YSBocmVmPTNEIm1h aWx0bzpVc2Vyc0BvdmlydC5vcmciIHRhcmdldD0zRCJfYmxhbmsiID0KY2xhc3M9M0QiIj5Vc2Vy c0BvdmlydC5vcmc8L2E+PGJyIGNsYXNzPTNEIiI+PGEgcmVsPTNEIm5vcmVmZXJyZXIiID0KaHJl Zj0zRCJodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRhcmdl dD0zRCJfYmxhbmsiID0KY2xhc3M9M0QiIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW48 d2JyID0KY2xhc3M9M0QiIj4vbGlzdGluZm8vdXNlcnM8L2E+PC9ibG9ja3F1b3RlPjwvZGl2Pjwv ZGl2PjwvZGl2PjwvZGl2PjwvYmxvY2s9CnF1b3RlPjwvZGl2PjwvZGl2PjwvZGl2PjwvZGl2Pjwv ZGl2PjwvYmxvY2txdW90ZT48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PQo8L2Rpdj48YnIgY2xh c3M9M0QiIj48L2Rpdj48L2JvZHk+PC9odG1sPj0KCi0tQXBwbGUtTWFpbD1fN0JGMTVDNTktRkU0 OS00NzkyLUI5QTMtMDY5RjFBODI4QzhELS0K --===============4932116376986761683==-- From stirabos at redhat.com Thu Sep 29 09:16:06 2016 Content-Type: multipart/mixed; boundary="===============1959234228042195510==" MIME-Version: 1.0 From: Simone Tiraboschi To: users at ovirt.org Subject: Re: [ovirt-users] oVirt 4.0.x - hosted-engine was not starting properly Date: Thu, 29 Sep 2016 15:15:35 +0200 Message-ID: In-Reply-To: CAP5iht5o77fhobBauSx_sUwThxmGqT8jXdwRUV1+ZOnH4CUcZw@mail.gmail.com --===============1959234228042195510== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Thu, Sep 29, 2016 at 3:11 PM, Martin Perina wrote: > > > On Thu, Sep 29, 2016 at 3:04 PM, Simone Tiraboschi > wrote: > >> >> >> On Thu, Sep 29, 2016 at 12:47 PM, Martin Perina >> wrote: >> >>> Hi, >>> >>> please take a look at my inline comments: >>> >>> On Tue, Sep 27, 2016 at 7:23 PM, Gervais de Montbrun < >>> gervais(a)demontbrun.com> wrote: >>> >>>> Hey All, >>>> >>>> Since updating to 4.0.x of oVirt, I have had an issue with my hosted >>>> engine. After a some poking around, I think I have figured out my issu= e and >>>> thought I would share to see what others think. >>>> The issue has existed with 4.0, 4.0.1, 4.0.2, 4.0.3, and still exists >>>> in 4.0.4. >>>> >>>> Description: >>>> When my hosted engine starts it reports that it is in a degraded state >>>> with 7 or 8 services still not started when I run systemctl status. It >>>> takes about 6 or 7 minutes to eventually start all the services and co= me >>>> online. If I don't set my cluster to Global-Maintenance mode it eventu= ally >>>> thinks that my hosted-engine needs to be rebooted and restarts it befo= re it >>>> can start everything. >>>> >>> >>> =E2=80=8BCould you please share with us logs gathered by ovirt-log-coll= ector? >>> >>> It's just a guess but could you please take a look if you HE VM has >>> enough entropy? >>> >>> cat /proc/sys/kernel/random/entropy_avail >>> >>> If the value is low (below or around 200), you really need to install >>> and configure some entropy generator such as haveged >>> >>> >>>> Solution: >>>> I realized that Apache was the culprit and found that the proxy to the >>>> ovirt-engine in /etc/httpd/conf.d/z-ovirt-engine-proxy.conf has a >>>> super long timeout with many retries. I changed the settings and now >>>> everything works for me. >>>> >>>> -> Before change: >>>> >>>> >>> RHEVManagerWeb/|OvirtEngineWeb/|ca.crt$|engine.ssh.key.txt$| >>>> rhevm.ssh.key.txt$)> >>>> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >>>> >>>> >>>> AddOutputFilterByType DEFLATE text/javascript text/css >>>> text/html text/xml text/json application/xml application/json >>>> application/x-yaml >>>> >>>> >>>> >>>> >>>> -> After change: >>>> >>>> >>>> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D5 retry=3D2 >>>> >>>> >>>> AddOutputFilterByType DEFLATE text/javascript text/css >>>> text/html text/xml text/json application/xml application/json >>>> application/x-yaml >>>> >>>> >>>> >>>> >>> =E2=80=8BThis one is correct for 4.0=E2=80=8B >>> =E2=80=8B, not sure why it was not updated during upgrade from 3.6. @Si= mone? >>> =E2=80=8B >>> >> >> Honestly it's >> >> ProxyPassMatch ajp://127.0.0.1:8702 timeout=3D3600 retry=3D5 >> >> >> AddOutputFilterByType DEFLATE text/javascript text/css >> text/html text/xml text/json application/xml application/json >> application/x-yaml >> >> >> also on a fresh 4.0 engine from our latest engine-appliance. >> > > =E2=80=8BRight, I missed the timeout=E2=80=8B/retry option changes. But t= he important part > is why old configuration (with different LocationMatch) was not overwritt= en > during upgrade. > > I suspect that it could got overwritten a second time to its 3.6 value in our backup/restore procedure. Adding Didi here. > >> >>> >>>> If I read the timeout settings correctly, it will wait 60 minutes with >>>> 5 retries. 5 hours is way too long for my little server to hold onto a= ll >>>> those apache processes. >>>> >>> The change I made allows for there to be an error, and also releases >>>> apache's hold on the process. Once everything is ready, apache is read= y to >>>> serve requests and everything/everyone is happy. Before making the cha= nge, >>>> I just get a whitescreen in my browser and then nothing works until I >>>> restart Apache (or I end up in an endless loop of ovirt-ha services >>>> restarting my hosted-engine. >>>> >>> >>> =E2=80=8BWell, if you have an issue with too many apache processes wait= ing for >>> engine to respond, then there's some issue in engine. As I wrote above >>> please share the logs with us and check entropy. >>> >>> Thanks >>> >>> Martin Perina >>> =E2=80=8B >>> >>> >>>> >>>> I noticed that this setting reverts to the original setting, so oVirt >>>> must be writing this file. Perhaps these number can be changed in oVir= t? If >>>> not, I will just setup and ansible play to revert the settings with wo= rking >>>> values and restart apache on my engine. >>>> :-) >>>> >>>> Cheers, >>>> Gervais >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>>> >>> >> > --===============1959234228042195510== Content-Type: text/html MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="attachment.html" PGRpdiBkaXI9Imx0ciI+PGJyPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFz cz0iZ21haWxfcXVvdGUiPk9uIFRodSwgU2VwIDI5LCAyMDE2IGF0IDM6MTEgUE0sIE1hcnRpbiBQ ZXJpbmEgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJtYWlsdG86bXBlcmluYUByZWRoYXQu Y29tIiB0YXJnZXQ9Il9ibGFuayI+bXBlcmluYUByZWRoYXQuY29tPC9hPiZndDs8L3NwYW4+IHdy b3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAw IDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2 IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRpY2Esc2Fucy1z ZXJpZiI+PGJyPjwvZGl2PjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48YnI+PGRpdiBjbGFzcz0i Z21haWxfcXVvdGUiPjxkaXY+PGRpdiBjbGFzcz0iaDUiPk9uIFRodSwgU2VwIDI5LCAyMDE2IGF0 IDM6MDQgUE0sIFNpbW9uZSBUaXJhYm9zY2hpIDxzcGFuIGRpcj0ibHRyIj4mbHQ7PGEgaHJlZj0i bWFpbHRvOnN0aXJhYm9zQHJlZGhhdC5jb20iIHRhcmdldD0iX2JsYW5rIj5zdGlyYWJvc0ByZWRo YXQuY29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxf cXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2NjYyBzb2xp ZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48YnI+PGRpdiBjbGFzcz0iZ21haWxf ZXh0cmEiPjxicj48ZGl2IGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gVGh1LCBTZXAgMjksIDIwMTYg YXQgMTI6NDcgUE0sIE1hcnRpbiBQZXJpbmEgPHNwYW4gZGlyPSJsdHIiPiZsdDs8YSBocmVmPSJt YWlsdG86bXBlcmluYUByZWRoYXQuY29tIiB0YXJnZXQ9Il9ibGFuayI+bXBlcmluYUByZWRoYXQu Y29tPC9hPiZndDs8L3NwYW4+IHdyb3RlOjxicj48YmxvY2txdW90ZSBjbGFzcz0iZ21haWxfcXVv dGUiIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlk IHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCI+PGRpdiBkaXI9Imx0ciI+PGRpdj5I aSw8YnI+PGJyPjwvZGl2PjxkaXY+cGxlYXNlIHRha2UgYSBsb29rIGF0IG15IGlubGluZSBjb21t ZW50czo8YnI+PC9kaXY+PGRpdiBjbGFzcz0iZ21haWxfZXh0cmEiPjxicj48ZGl2IGNsYXNzPSJn bWFpbF9xdW90ZSI+PHNwYW4+T24gVHVlLCBTZXAgMjcsIDIwMTYgYXQgNzoyMyBQTSwgR2VydmFp cyBkZSBNb250YnJ1biA8c3BhbiBkaXI9Imx0ciI+Jmx0OzxhIGhyZWY9Im1haWx0bzpnZXJ2YWlz QGRlbW9udGJydW4uY29tIiB0YXJnZXQ9Il9ibGFuayI+Z2VydmFpc0BkZW1vbnRicnVuLmNvbTwv YT4mZ3Q7PC9zcGFuPiB3cm90ZTo8YnI+PGJsb2NrcXVvdGUgc3R5bGU9Im1hcmdpbjowcHggMHB4 IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5n LWxlZnQ6MWV4IiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgc3R5bGU9IndvcmQtd3JhcDpicmVh ay13b3JkIj5IZXkgQWxsLDxkaXY+PGJyPjwvZGl2PjxkaXY+U2luY2UgdXBkYXRpbmcgdG8gNC4w Lnggb2Ygb1ZpcnQsIEkgaGF2ZSBoYWQgYW4gaXNzdWUgd2l0aCBteSBob3N0ZWQgZW5naW5lLiBB ZnRlciBhIHNvbWUgcG9raW5nIGFyb3VuZCwgSSB0aGluayBJIGhhdmUgZmlndXJlZCBvdXQgbXkg aXNzdWUgYW5kIHRob3VnaHQgSSB3b3VsZCBzaGFyZSB0byBzZWUgd2hhdCBvdGhlcnMgdGhpbmsu PC9kaXY+PGRpdj5UaGUgaXNzdWUgaGFzIGV4aXN0ZWQgd2l0aCA0LjAsIDQuMC4xLCA0LjAuMiwg NC4wLjMsIGFuZCBzdGlsbCBleGlzdHMgaW4gNC4wLjQuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRp dj5EZXNjcmlwdGlvbjo8L2Rpdj48ZGl2PldoZW4gbXkgaG9zdGVkIGVuZ2luZSBzdGFydHMgaXQg cmVwb3J0cyB0aGF0IGl0IGlzIGluIGEgZGVncmFkZWQgc3RhdGUgd2l0aCA3IG9yIDggc2Vydmlj ZXMgc3RpbGwgbm90IHN0YXJ0ZWQgd2hlbiBJIHJ1biBzeXN0ZW1jdGwgc3RhdHVzLiBJdCB0YWtl cyBhYm91dCA2IG9yIDcgbWludXRlcyB0byBldmVudHVhbGx5IHN0YXJ0IGFsbCB0aGUgc2Vydmlj ZXMgYW5kIGNvbWUgb25saW5lLiBJZiBJIGRvbiYjMzk7dCBzZXQgbXkgY2x1c3RlciB0byBHbG9i YWwtTWFpbnRlbmFuY2UgbW9kZSBpdCBldmVudHVhbGx5IHRoaW5rcyB0aGF0IG15IGhvc3RlZC1l bmdpbmUgbmVlZHMgdG8gYmUgcmVib290ZWQgYW5kIHJlc3RhcnRzIGl0IGJlZm9yZSBpdCBjYW4g c3RhcnQgZXZlcnl0aGluZy48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PC9zcGFuPjxkaXY+PGJy PjxkaXY+4oCLQ291bGQgeW91IHBsZWFzZSBzaGFyZSB3aXRoIHVzIGxvZ3MgZ2F0aGVyZWQgYnkg b3ZpcnQtbG9nLWNvbGxlY3Rvcj88YnI+PGJyPkl0JiMzOTtzIGp1c3QgYSBndWVzcyBidXQgY291 bGQgeW91IHBsZWFzZSB0YWtlIGEgbG9vayBpZiB5b3UgSEUgVk0gaGFzIGVub3VnaCBlbnRyb3B5 Pzxicj48YnI+wqAgY2F0IC9wcm9jL3N5cy9rZXJuZWwvcmFuZG9tL2VudHJvcDx3YnI+eV9hdmFp bDxicj48YnI+PC9kaXY+PGRpdj5JZiB0aGUgdmFsdWUgaXMgbG93IChiZWxvdyBvciBhcm91bmQg MjAwKSzCoCB5b3UgcmVhbGx5IG5lZWQgdG8gaW5zdGFsbCBhbmQgY29uZmlndXJlIHNvbWUgZW50 cm9weSBnZW5lcmF0b3Igc3VjaCBhcyBoYXZlZ2VkPGJyPjxicj48L2Rpdj48L2Rpdj48c3Bhbj48 YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFw eCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3BhZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9x dW90ZSI+PGRpdiBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQiPjxkaXY+PGJyPjwvZGl2Pjxk aXY+U29sdXRpb246PC9kaXY+PGRpdj5JIHJlYWxpemVkIHRoYXQgQXBhY2hlIHdhcyB0aGUgY3Vs cHJpdCBhbmQgZm91bmQgdGhhdCB0aGUgcHJveHkgdG8gdGhlIG92aXJ0LWVuZ2luZSBpbsKgL2V0 Yy9odHRwZC9jb25mLmQvei1vdmlydC1lPHdicj5uZ2luZS1wcm94eS5jb25mIGhhcyBhIHN1cGVy IGxvbmcgdGltZW91dCB3aXRoIG1hbnkgcmV0cmllcy4gSSBjaGFuZ2VkIHRoZSBzZXR0aW5ncyBh bmQgbm93IGV2ZXJ5dGhpbmcgd29ya3MgZm9yIG1lLjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+ LSZndDsgQmVmb3JlIGNoYW5nZTo8L2Rpdj48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBweCAw cHggMHB4IDQwcHg7Ym9yZGVyLXdpZHRoOm1lZGl1bTtib3JkZXItc3R5bGU6bm9uZTtwYWRkaW5n OjBweCI+PGRpdj48ZGl2PsKgIMKgICZsdDtMb2NhdGlvbk1hdGNoIF4vKG92aXJ0LWVuZ2luZSgk fC8pfGFwaSgkfC8pfDx3YnI+UkhFVk1hbmFnZXJXZWIvfE92aXJ0RW5naW5lV2ViPHdicj4vfGNh LmNydCR8ZW5naW5lLnNzaC5rZXkudHh0JHw8d2JyPnJoZXZtLnNzaC5rZXkudHh0JCkmZ3Q7PC9k aXY+PGRpdj7CoCDCoCDCoCDCoCBQcm94eVBhc3NNYXRjaCA8YT5hanA6Ly8xMjcuMC4wLjE6ODcw MjwvYT4gdGltZW91dD0zNjAwIHJldHJ5PTU8L2Rpdj48ZGl2Pjxicj48L2Rpdj48ZGl2PsKgIMKg IMKgIMKgICZsdDtJZk1vZHVsZSBkZWZsYXRlX21vZHVsZSZndDs8L2Rpdj48ZGl2PsKgIMKgIMKg IMKgIMKgIMKgIEFkZE91dHB1dEZpbHRlckJ5VHlwZSBERUZMQVRFIHRleHQvamF2YXNjcmlwdCB0 ZXh0L2NzcyB0ZXh0L2h0bWwgdGV4dC94bWwgdGV4dC9qc29uIGFwcGxpY2F0aW9uL3htbCBhcHBs aWNhdGlvbi9qc29uIGFwcGxpY2F0aW9uL3gteWFtbDwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgJmx0 Oy9JZk1vZHVsZSZndDs8L2Rpdj48ZGl2PsKgIMKgICZsdDsvTG9jYXRpb25NYXRjaCZndDs8L2Rp dj48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdj48YnI+PC9kaXY+LSZndDsgQWZ0ZXIgY2hhbmdlOjxi bG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggNDBweDtib3JkZXItd2lkdGg6bWVk aXVtO2JvcmRlci1zdHlsZTpub25lO3BhZGRpbmc6MHB4Ij48ZGl2PjxkaXY+wqAgwqAgJmx0O0xv Y2F0aW9uTWF0Y2ggXi9vdmlydC1lbmdpbmUoJHwvKSZndDs8L2Rpdj48ZGl2PsKgIMKgIMKgIMKg IFByb3h5UGFzc01hdGNoIDxhPmFqcDovLzEyNy4wLjAuMTo4NzAyPC9hPiB0aW1lb3V0PTUgcmV0 cnk9MjwvZGl2PjxkaXY+PGJyPjwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgJmx0O0lmTW9kdWxlIGRl ZmxhdGVfbW9kdWxlJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgwqAgwqAgQWRkT3V0cHV0Rmls dGVyQnlUeXBlIERFRkxBVEUgdGV4dC9qYXZhc2NyaXB0IHRleHQvY3NzIHRleHQvaHRtbCB0ZXh0 L3htbCB0ZXh0L2pzb24gYXBwbGljYXRpb24veG1sIGFwcGxpY2F0aW9uL2pzb24gYXBwbGljYXRp b24veC15YW1sPC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7L0lmTW9kdWxlJmd0OzwvZGl2Pjxk aXY+wqAgwqAgJmx0Oy9Mb2NhdGlvbk1hdGNoJmd0OzwvZGl2PjwvZGl2PjwvYmxvY2txdW90ZT48 L2Rpdj48L2Jsb2NrcXVvdGU+PC9zcGFuPjxkaXY+PGJyPjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5 OmFyaWFsLGhlbHZldGljYSxzYW5zLXNlcmlmO2Rpc3BsYXk6aW5saW5lIj7igItUaGlzIG9uZSBp cyBjb3JyZWN0IGZvciA0LjDigIs8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxo ZWx2ZXRpY2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+4oCLLCBub3Qgc3VyZSB3aHkgaXQg d2FzIG5vdCB1cGRhdGVkIGR1cmluZyB1cGdyYWRlIGZyb20gMy42LiBAU2ltb25lPzxicj7igIs8 L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdj48YnI+PC9kaXY+ PGRpdj5Ib25lc3RseSBpdCYjMzk7czwvZGl2PjxkaXY+PGRpdj7CoCDCoCAmbHQ7TG9jYXRpb25N YXRjaCBeL292aXJ0LWVuZ2luZSgkfC8pJmd0OzwvZGl2PjxkaXY+wqAgwqAgwqAgwqAgUHJveHlQ YXNzTWF0Y2ggYWpwOi8vPGEgaHJlZj0iaHR0cDovLzEyNy4wLjAuMTo4NzAyIiB0YXJnZXQ9Il9i bGFuayI+MTI3LjAuMC4xOjg3MDI8L2E+IHRpbWVvdXQ9MzYwMCByZXRyeT01PC9kaXY+PGRpdj48 YnI+PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCAmbHQ7SWZNb2R1bGUgZGVmbGF0ZV9tb2R1bGUmZ3Q7 PC9kaXY+PGRpdj7CoCDCoCDCoCDCoCDCoCDCoCBBZGRPdXRwdXRGaWx0ZXJCeVR5cGUgREVGTEFU RSB0ZXh0L2phdmFzY3JpcHQgdGV4dC9jc3MgdGV4dC9odG1sIHRleHQveG1sIHRleHQvanNvbiBh cHBsaWNhdGlvbi94bWwgYXBwbGljYXRpb24vanNvbiBhcHBsaWNhdGlvbi94LXlhbWw8L2Rpdj48 ZGl2PsKgIMKgIMKgIMKgICZsdDsvSWZNb2R1bGUmZ3Q7PC9kaXY+PGRpdj7CoCDCoCAmbHQ7L0xv Y2F0aW9uTWF0Y2gmZ3Q7PC9kaXY+PC9kaXY+PGRpdj5hbHNvIG9uIGEgZnJlc2ggNC4wIGVuZ2lu ZSBmcm9tIG91ciBsYXRlc3QgZW5naW5lLWFwcGxpYW5jZS48L2Rpdj48L2Rpdj48L2Rpdj48L2Rp dj48L2Jsb2NrcXVvdGU+PC9kaXY+PC9kaXY+PGRpdj48YnI+PGRpdiBzdHlsZT0iZm9udC1mYW1p bHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPuKAi1JpZ2h0LCBJ IG1pc3NlZCB0aGUgdGltZW91dOKAiy9yZXRyeSBvcHRpb24gY2hhbmdlcy4gQnV0IHRoZSBpbXBv cnRhbnQgcGFydCBpcyB3aHkgb2xkIGNvbmZpZ3VyYXRpb24gKHdpdGggZGlmZmVyZW50IExvY2F0 aW9uTWF0Y2gpIHdhcyBub3Qgb3ZlcndyaXR0ZW4gZHVyaW5nIHVwZ3JhZGUuPGJyPjxicj48L2Rp dj48L2Rpdj48L2Rpdj48L2Rpdj48L2Rpdj48L2Jsb2NrcXVvdGU+PGRpdj48YnI+PC9kaXY+PGRp dj5JIHN1c3BlY3QgdGhhdCBpdCBjb3VsZCBnb3Qgb3ZlcndyaXR0ZW4gYSBzZWNvbmQgdGltZSB0 byBpdHMgMy42IHZhbHVlIGluIG91ciBiYWNrdXAvcmVzdG9yZSBwcm9jZWR1cmUuPC9kaXY+PGRp dj5BZGRpbmcgRGlkaSBoZXJlLjwvZGl2PjxkaXY+wqA8L2Rpdj48YmxvY2txdW90ZSBjbGFzcz0i Z21haWxfcXVvdGUiIHN0eWxlPSJtYXJnaW46MCAwIDAgLjhleDtib3JkZXItbGVmdDoxcHggI2Nj YyBzb2xpZDtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2IGNsYXNzPSJnbWFp bF9leHRyYSI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXY+PGRpdiBzdHlsZT0iZm9udC1m YW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPjwvZGl2Pjwv ZGl2PjxzcGFuIGNsYXNzPSIiPjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9 Im1hcmdpbjowIDAgMCAuOGV4O2JvcmRlci1sZWZ0OjFweCAjY2NjIHNvbGlkO3BhZGRpbmctbGVm dDoxZXgiPjxkaXYgZGlyPSJsdHIiPjxkaXYgY2xhc3M9ImdtYWlsX2V4dHJhIj48ZGl2IGNsYXNz PSJnbWFpbF9xdW90ZSI+PGRpdj7CoDwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90 ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQg cmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj48ZGl2IGNs YXNzPSJnbWFpbF9leHRyYSI+PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxzcGFuPjxibG9ja3F1 b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVyLWxlZnQ6MXB4IHNvbGlk IHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9ImdtYWlsX3F1b3RlIj48 ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+PGRpdj48YnI+PC9kaXY+SWYgSSByZWFk IHRoZSB0aW1lb3V0IHNldHRpbmdzIGNvcnJlY3RseSwgaXQgd2lsbCB3YWl0IDYwIG1pbnV0ZXMg d2l0aCA1IHJldHJpZXMuIDUgaG91cnMgaXMgd2F5IHRvbyBsb25nIGZvciBteSBsaXR0bGUgc2Vy dmVyIHRvIGhvbGQgb250byBhbGwgdGhvc2UgYXBhY2hlIHByb2Nlc3Nlcy4gPC9kaXY+PC9ibG9j a3F1b3RlPjxibG9ja3F1b3RlIHN0eWxlPSJtYXJnaW46MHB4IDBweCAwcHggMC44ZXg7Ym9yZGVy LWxlZnQ6MXB4IHNvbGlkIHJnYigyMDQsMjA0LDIwNCk7cGFkZGluZy1sZWZ0OjFleCIgY2xhc3M9 ImdtYWlsX3F1b3RlIj48ZGl2IHN0eWxlPSJ3b3JkLXdyYXA6YnJlYWstd29yZCI+VGhlIGNoYW5n ZSBJIG1hZGUgYWxsb3dzIGZvciB0aGVyZSB0byBiZSBhbiBlcnJvciwgYW5kIGFsc28gcmVsZWFz ZXMgYXBhY2hlJiMzOTtzIGhvbGQgb24gdGhlIHByb2Nlc3MuIE9uY2UgZXZlcnl0aGluZyBpcyBy ZWFkeSwgYXBhY2hlIGlzIHJlYWR5IHRvIHNlcnZlIHJlcXVlc3RzIGFuZCBldmVyeXRoaW5nL2V2 ZXJ5b25lIGlzIGhhcHB5LiBCZWZvcmUgbWFraW5nIHRoZSBjaGFuZ2UsIEkganVzdCBnZXQgYSB3 aGl0ZXNjcmVlbiBpbiBteSBicm93c2VyIGFuZCB0aGVuIG5vdGhpbmcgd29ya3MgdW50aWwgSSBy ZXN0YXJ0IEFwYWNoZSAob3IgSSBlbmQgdXAgaW4gYW4gZW5kbGVzcyBsb29wIG9mIG92aXJ0LWhh IHNlcnZpY2VzIHJlc3RhcnRpbmcgbXkgaG9zdGVkLWVuZ2luZS48YnI+PC9kaXY+PC9ibG9ja3F1 b3RlPjwvc3Bhbj48ZGl2Pjxicj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTphcmlhbCxoZWx2ZXRp Y2Esc2Fucy1zZXJpZjtkaXNwbGF5OmlubGluZSI+4oCLV2VsbCwgaWYgeW91IGhhdmUgYW4gaXNz dWUgd2l0aCB0b28gbWFueSBhcGFjaGUgcHJvY2Vzc2VzIHdhaXRpbmcgZm9yIGVuZ2luZSB0byBy ZXNwb25kLCB0aGVuIHRoZXJlJiMzOTtzIHNvbWUgaXNzdWUgaW4gZW5naW5lLiBBcyBJIHdyb3Rl IGFib3ZlIHBsZWFzZSBzaGFyZSB0aGUgbG9ncyB3aXRoIHVzIGFuZCBjaGVjayBlbnRyb3B5Ljxi cj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMt c2VyaWY7ZGlzcGxheTppbmxpbmUiPlRoYW5rczxicj48YnI+PC9kaXY+PGRpdiBzdHlsZT0iZm9u dC1mYW1pbHk6YXJpYWwsaGVsdmV0aWNhLHNhbnMtc2VyaWY7ZGlzcGxheTppbmxpbmUiPk1hcnRp biBQZXJpbmE8YnI+4oCLPC9kaXY+wqA8L2Rpdj48YmxvY2txdW90ZSBzdHlsZT0ibWFyZ2luOjBw eCAwcHggMHB4IDAuOGV4O2JvcmRlci1sZWZ0OjFweCBzb2xpZCByZ2IoMjA0LDIwNCwyMDQpO3Bh ZGRpbmctbGVmdDoxZXgiIGNsYXNzPSJnbWFpbF9xdW90ZSI+PHNwYW4+PGRpdiBzdHlsZT0id29y ZC13cmFwOmJyZWFrLXdvcmQiPjxkaXY+PGRpdj48ZGl2Pjxicj48L2Rpdj48ZGl2Pkkgbm90aWNl ZCB0aGF0IHRoaXMgc2V0dGluZyByZXZlcnRzIHRvIHRoZSBvcmlnaW5hbCBzZXR0aW5nLCBzbyBv VmlydCBtdXN0IGJlIHdyaXRpbmcgdGhpcyBmaWxlLiBQZXJoYXBzIHRoZXNlIG51bWJlciBjYW4g YmUgY2hhbmdlZCBpbiBvVmlydD8gSWYgbm90LCBJIHdpbGwganVzdCBzZXR1cCBhbmQgYW5zaWJs ZSBwbGF5IHRvIHJldmVydCB0aGUgc2V0dGluZ3Mgd2l0aCB3b3JraW5nIHZhbHVlcyBhbmQgcmVz dGFydCBhcGFjaGUgb24gbXkgZW5naW5lLjwvZGl2PjxkaXY+Oi0pPC9kaXY+PGRpdj4KPGRpdj48 YnI+Q2hlZXJzLDxicj5HZXJ2YWlzPGJyPjxicj48YnI+PC9kaXY+Cgo8L2Rpdj4KCgoKPGJyPjwv ZGl2PjwvZGl2PjwvZGl2Pjxicj48L3NwYW4+X19fX19fX19fX19fX19fX19fX19fX19fX19fX19f PHdicj5fX19fX19fX19fX19fX19fXzxicj4KVXNlcnMgbWFpbGluZyBsaXN0PGJyPgo8YSBocmVm PSJtYWlsdG86VXNlcnNAb3ZpcnQub3JnIiB0YXJnZXQ9Il9ibGFuayI+VXNlcnNAb3ZpcnQub3Jn PC9hPjxicj4KPGEgcmVsPSJub3JlZmVycmVyIiBocmVmPSJodHRwOi8vbGlzdHMub3ZpcnQub3Jn L21haWxtYW4vbGlzdGluZm8vdXNlcnMiIHRhcmdldD0iX2JsYW5rIj5odHRwOi8vbGlzdHMub3Zp cnQub3JnL21haWxtYW48d2JyPi9saXN0aW5mby91c2VyczwvYT48YnI+Cjxicj48L2Jsb2NrcXVv dGU+PC9kaXY+PGJyPjwvZGl2PjwvZGl2Pgo8L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjwvZGl2Pjwv ZGl2Pgo8L2Jsb2NrcXVvdGU+PC9zcGFuPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4KPC9ibG9ja3F1 b3RlPjwvZGl2Pjxicj48L2Rpdj48L2Rpdj4K --===============1959234228042195510==--