Error 500 on Hosted Engine admin portal!!!
Hi everyone! Did the oVirt team check the 4.5 update before release? I've update my production latest 4.4.10 version and get Error 500 on admin page. Clean installation on another bare-metal node using Ovirt Node 4.5 and hosted-engine console setup (installing through the web installer several versions of Ovirt traditionally makes the connection break and disconnects access to the host by means of the IP address) get an error 500 after healthcheck. What I will do in this situation? Httpd log: [Sat Apr 23 15:54:59.826076 2022] [core:notice] [pid 1526:tid 140201844767040] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Sat Apr 23 15:55:05.685872 2022] [proxy:error] [pid 2125:tid 140201102243584] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 15:55:05.685924 2022] [proxy_ajp:error] [pid 2125:tid 140201102243584] [client 192.168.0.14:43194] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 15:55:15.831754 2022] [proxy:error] [pid 2125:tid 140201110636288] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 15:55:15.831811 2022] [proxy_ajp:error] [pid 2125:tid 140201110636288] [client 192.168.0.14:43196] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:21:57.720678 2022] [proxy:error] [pid 7761:tid 140200894203648] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:21:57.720745 2022] [proxy_ajp:error] [pid 7761:tid 140200894203648] [client 192.168.0.14:43516] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:30:57.738971 2022] [proxy:error] [pid 7761:tid 140200877418240] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:30:57.739043 2022] [proxy_ajp:error] [pid 7761:tid 140200877418240] [client 192.168.0.14:43640] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:32:27.838795 2022] [proxy:error] [pid 7761:tid 140200978130688] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:32:27.838881 2022] [proxy_ajp:error] [pid 7761:tid 140200978130688] [client 192.168.0.14:43658] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:37:49.357585 2022] [core:notice] [pid 1497:tid 139855753558336] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Sat Apr 23 16:37:49.361573 2022] [suexec:notice] [pid 1497:tid 139855753558336] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sat Apr 23 16:37:49.388188 2022] [so:warn] [pid 1497:tid 139855753558336] AH01574: module proxy_module is already loaded, skipping [Sat Apr 23 16:37:49.391763 2022] [lbmethod_heartbeat:notice] [pid 1497:tid 139855753558336] AH02282: No slotmem from mod_heartmonitor [Sat Apr 23 16:37:49.399281 2022] [mpm_event:notice] [pid 1497:tid 139855753558336] AH00489: Apache/2.4.37 (centos) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 mod_wsgi/4.6.4 Python/3.6 configured -- resuming normal operations [Sat Apr 23 16:37:49.399320 2022] [core:notice] [pid 1497:tid 139855753558336] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Sat Apr 23 23:20:37.254289 2022] [proxy:error] [pid 6183:tid 139854109198080] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 23:20:37.254354 2022] [proxy_ajp:error] [pid 6183:tid 139854109198080] [client 192.168.0.13:52982] AH00896: failed to make connection to backend: 127.0.0.1 server.log: 2022-04-23 16:38:20,655+03 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 16:38:20,752+03 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 24.0.1.Final (WildFly Core 16.0.1.Final) started (with errors) in 22155ms - Started 1670 of 1890 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) 2022-04-23 23:20:50,622+03 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 47) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 23:20:50,630+03 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 23:20:50,706+03 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 24.0.1.Final (WildFly Core 16.0.1.Final) started (with errors) in 15267ms - Started 1670 of 1890 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand)
Hi Patrick, Most probably you hit postgresl-jdbc version problem: https://lists.ovirt.org/archives/list/users@ovirt.org/thread/SBCWNXLFLJBKTA3... Please let us know if above workaround won't help. Thanks, Martin On Sat, 23 Apr 2022, 22:32 Patrick Lomakin, <patrick.lomakin@gmail.com> wrote:
Hi everyone! Did the oVirt team check the 4.5 update before release? I've update my production latest 4.4.10 version and get Error 500 on admin page. Clean installation on another bare-metal node using Ovirt Node 4.5 and hosted-engine console setup (installing through the web installer several versions of Ovirt traditionally makes the connection break and disconnects access to the host by means of the IP address) get an error 500 after healthcheck. What I will do in this situation? Httpd log: [Sat Apr 23 15:54:59.826076 2022] [core:notice] [pid 1526:tid 140201844767040] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Sat Apr 23 15:55:05.685872 2022] [proxy:error] [pid 2125:tid 140201102243584] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 15:55:05.685924 2022] [proxy_ajp:error] [pid 2125:tid 140201102243584] [client 192.168.0.14:43194] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 15:55:15.831754 2022] [proxy:error] [pid 2125:tid 140201110636288] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 15:55:15.831811 2022] [proxy_ajp:error] [pid 2125:tid 140201110636288] [client 192.168.0.14:43196] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:21:57.720678 2022] [proxy:error] [pid 7761:tid 140200894203648] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:21:57.720745 2022] [proxy_ajp:error] [pid 7761:tid 140200894203648] [client 192.168.0.14:43516] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:30:57.738971 2022] [proxy:error] [pid 7761:tid 140200877418240] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:30:57.739043 2022] [proxy_ajp:error] [pid 7761:tid 140200877418240] [client 192.168.0.14:43640] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:32:27.838795 2022] [proxy:error] [pid 7761:tid 140200978130688] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 16:32:27.838881 2022] [proxy_ajp:error] [pid 7761:tid 140200978130688] [client 192.168.0.14:43658] AH00896: failed to make connection to backend: 127.0.0.1 [Sat Apr 23 16:37:49.357585 2022] [core:notice] [pid 1497:tid 139855753558336] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Sat Apr 23 16:37:49.361573 2022] [suexec:notice] [pid 1497:tid 139855753558336] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Sat Apr 23 16:37:49.388188 2022] [so:warn] [pid 1497:tid 139855753558336] AH01574: module proxy_module is already loaded, skipping [Sat Apr 23 16:37:49.391763 2022] [lbmethod_heartbeat:notice] [pid 1497:tid 139855753558336] AH02282: No slotmem from mod_heartmonitor [Sat Apr 23 16:37:49.399281 2022] [mpm_event:notice] [pid 1497:tid 139855753558336] AH00489: Apache/2.4.37 (centos) OpenSSL/1.1.1k mod_auth_gssapi/1.6.1 mod_wsgi/4.6.4 Python/3.6 configured -- resuming normal operations [Sat Apr 23 16:37:49.399320 2022] [core:notice] [pid 1497:tid 139855753558336] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Sat Apr 23 23:20:37.254289 2022] [proxy:error] [pid 6183:tid 139854109198080] (111)Connection refused: AH00957: AJP: attempt to connect to 127.0.0.1:8702 (127.0.0.1) failed [Sat Apr 23 23:20:37.254354 2022] [proxy_ajp:error] [pid 6183:tid 139854109198080] [client 192.168.0.13:52982] AH00896: failed to make connection to backend: 127.0.0.1
server.log: 2022-04-23 16:38:20,655+03 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 16:38:20,752+03 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 24.0.1.Final (WildFly Core 16.0.1.Final) started (with errors) in 22155ms - Started 1670 of 1890 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) 2022-04-23 23:20:50,622+03 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 47) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 23:20:50,630+03 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-23 23:20:50,706+03 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 24.0.1.Final (WildFly Core 16.0.1.Final) started (with errors) in 15267ms - Started 1670 of 1890 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/MYK4VDWKCM5YWW...
This is the cause, yes Martin - but it wont help since the Installer/deployment does an update while running, so "almost" no way to bypass that WORKAROUND: preparation: - ssh login to the node, prepare ssh root@ - copy paste buffer ready: echo exclude=postgresql-jdbc >> /etc/dnf/dnf.conf when the webinstaller is running, the moment when the ip of the hosted engine vm is displayed (something like 192.168.222.208) immediately ssh to that vm from the host and execute the copy paste buffer before the dnf update takes place uggly workaround, maybe there's another better option, but hey, it describes the problem and works :) Cheers Luis
Hi, I’m not sure if it works with the web installer, but if you deploy with CLI you can use the following to ask the installer to pause before running engine-setup: hosted-engine --deploy --ansible-extra-vars=he_pause_before_engine_setup=true This gives you the time to ssh and exclude the offending package before the installer attempts to upgrade any package. Cheers, Alessandro
Il giorno 25 apr 2022, alle ore 09:27, lpatsch@siriusag.com ha scritto:
This is the cause, yes Martin - but it wont help since the Installer/deployment does an update while running, so "almost" no way to bypass that
WORKAROUND: preparation: - ssh login to the node, prepare ssh root@ - copy paste buffer ready: echo exclude=postgresql-jdbc >> /etc/dnf/dnf.conf
when the webinstaller is running, the moment when the ip of the hosted engine vm is displayed (something like 192.168.222.208) immediately ssh to that vm from the host and execute the copy paste buffer before the dnf update takes place
uggly workaround, maybe there's another better option, but hey, it describes the problem and works :)
Cheers Luis _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ULUGCT73FOPPAY...
The exclude option takes not only a package but package with version, so you can block a specific version of package without blocking future updates. Best Regards,Strahil Nikolov On Mon, Apr 25, 2022 at 11:24, Alessandro De Salvo<Alessandro.DeSalvo@roma1.infn.it> wrote: Hi, I’m not sure if it works with the web installer, but if you deploy with CLI you can use the following to ask the installer to pause before running engine-setup: hosted-engine --deploy --ansible-extra-vars=he_pause_before_engine_setup=true This gives you the time to ssh and exclude the offending package before the installer attempts to upgrade any package. Cheers, Alessandro
Il giorno 25 apr 2022, alle ore 09:27, lpatsch@siriusag.com ha scritto:
This is the cause, yes Martin - but it wont help since the Installer/deployment does an update while running, so "almost" no way to bypass that
WORKAROUND: preparation: - ssh login to the node, prepare ssh root@ - copy paste buffer ready: echo exclude=postgresql-jdbc >> /etc/dnf/dnf.conf
when the webinstaller is running, the moment when the ip of the hosted engine vm is displayed (something like 192.168.222.208) immediately ssh to that vm from the host and execute the copy paste buffer before the dnf update takes place
uggly workaround, maybe there's another better option, but hey, it describes the problem and works :)
Cheers Luis _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/ULUGCT73FOPPAY...
Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/UZALUJRVR4E7RA...
Thank you all for the important pieces of information! I'm deploying a new self-hosted engine and I was hit by this problem. I fact I lost a few hours during the weekend on this.. Finally I was able to ssh into the VM during setup and lock the package using "dnf versionlock add".
The procedure to downgrade postgresql-jdbc and restart ovirt-engine worked for me.
participants (7)
-
Alessandro De Salvo -
atlasz＠gmail.com -
Jason Beard -
lpatsch＠siriusag.com -
Martin Perina -
Patrick Lomakin -
Strahil Nikolov