[ovirt-users] Warning: CentOS Upgrade with Host Engine - 503 Service Temporarily Unavailable
Andrei Verovski
andreil1 at starlett.lv
Thu Mar 15 15:39:48 UTC 2018
Hi !
I have upgraded CentOS 7.4 with oVirt 4.2.1 host engine (with yum upgrade), and its resulted in broken system - "503 Service Temporarily Unavailable" when connecting to the host engine via web.
Service ovirt-engine failed to starts (logs attached at the bottom of this email), other ovirt services seem to run fine.
yum update "ovirt-*-setup*” (upgrade 4.2 -> 4.2.1)
engine-setup
yum upgrade (OS upgrade)
Is this issue somehow related to JDK as described here?
https://bugzilla.redhat.com/show_bug.cgi?id=1217023
New packages:
java-1.8.0-openjdk x86_64 1:1.8.0.161-0.b14.el7_4
java-1.8.0-openjdk-devel x86_64 1:1.8.0.161-0.b14.el7_4
java-1.8.0-openjdk-headless x86_64 1:1.8.0.161-0.b14.el7_4
installed packages (seem to be also 1.8x):
[root at node00 ~]# rpm -qa | grep jdk
java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64
java-1.8.0-openjdk-devel-1.8.0.151-5.b12.el7_4.x86_64
java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64
copy-jdk-configs-2.2-5.el7_4.noarch
Since my host engine is actually a KVM appliance under another SuSE server, I simply discarded it and reverted back old qcow2 image.
So this is a warning to anyone - don’t upgrade CentOS, or at least keep a copy of disk image before ANY upgrade !
********** LOGS *********
******************************
[root at node00 ~]# service ovirt-engine status -l
Redirecting to /bin/systemctl status -l ovirt-engine.service
● ovirt-engine.service - oVirt Engine
Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: disabled)
Active: failed (Result: timeout) since Thu 2018-03-15 15:44:08 EET; 2min 18s ago
Process: 1474 ExecStart=/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify $EXTRA_ARGS start (code=killed, signal=TERM)
Main PID: 1474 (code=killed, signal=TERM)
CGroup: /system.slice/ovirt-engine.service
Mar 15 15:42:37 node00.myhost.lv systemd[1]: Starting oVirt Engine...
Mar 15 15:42:50 node00.myhost.lv ovirt-engine.py[1474]: 2018-03-15 15:42:50,869+0200 ovirt-engine: INFO _detectJBossVersion:187 Detecting JBoss version. Running: /usr/lib/jvm/jre/bin/java ['ovirt-engine-version', '-server', '-XX:+TieredCompilation', '-Xms1024M', '-Xmx1024M', '-Djava.awt.headless=true', '-Dsun.rmi.dgc.client.gcInterval=3600000', '-Dsun.rmi.dgc.server.gcInterval=3600000', '-Djsse.enableSNIExtension=false', '-XX:+HeapDumpOnOutOfMemoryError', '-XX:HeapDumpPath=/var/log/ovirt-engine/dump', '-Djava.util.logging.manager=org.jboss.logmanager', '-Dlogging.configuration=file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-logging.properties', '-Dorg.jboss.resolver.warning=true', '-Djboss.modules.system.pkgs=org.jboss.byteman', '-Djboss.server.default.config=ovirt-engine', '-Djboss.home.dir=/usr/share/ovirt-engine-wildfly', '-Djboss.server.base.dir=/usr/share/ovirt-engine', '-Djboss.server.data.dir=/var/lib/ovirt-engine', '-Djboss.server.log.dir=/var/log/ovirt-engine', '-Djboss.server.config.dir=/var/lib/ovirt-engine/jboss_runtime/config', '-Djboss.server.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp', '-Djboss.controller.temp.dir=/var/lib/ovirt-engine/jboss_runtime/tmp', '-jar', '/usr/share/ovirt-engine-wildfly/jboss-modules.jar', '-mp', '/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/modules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/share/ovirt-engine-wildfly/modules', '-jaxpmodule', 'javax.xml.jaxp-provider', 'org.jboss.as.standalone', '-v']
Mar 15 15:44:08 node00.myhost.lv systemd[1]: ovirt-engine.service start operation timed out. Terminating.
Mar 15 15:44:08 node00.myhost.lv systemd[1]: Failed to start oVirt Engine.
Mar 15 15:44:08 node00.myhost.lv systemd[1]: Unit ovirt-engine.service entered failed state.
Mar 15 15:44:08 node00.myhost.lv systemd[1]: ovirt-engine.service failed.
[root at node00 ~]#
[root at node00 ~]# tail -n 20 /var/log/ovirt-engine/server.log
2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutting down.
2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED paused.
2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping
2018-03-15 15:39:03,202+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutdown complete.
2018-03-15 15:39:03,213+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:/DWHDataSource]
2018-03-15 15:39:03,213+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:/ENGINEDataSource]
2018-03-15 15:39:03,226+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped timeout-base cache from ovirt-engine container
2018-03-15 15:39:03,228+02 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = postgresql
2018-03-15 15:39:03,233+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: root.war) in 271ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: enginesso.war) in 279ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: welcome.war) in 279ms
2018-03-15 15:39:03,235+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped inventory cache from ovirt-engine container
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped dashboard cache from ovirt-engine container
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: webadmin.war) in 281ms
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: docs.war) in 289ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: services.war) in 267ms
2018-03-15 15:39:03,239+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: bll.jar) in 290ms
2018-03-15 15:39:03,242+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment engine.ear (runtime-name: engine.ear) in 286ms
2018-03-15 15:39:03,265+02 INFO [org.jboss.as] (MSC service thread 1-5) WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly Core 3.0.8.Final) stopped in 302ms
[root at node00 ~]#
[root at node00 ~]# tail -n 30 /var/log/ovirt-engine/server.log
2018-03-15 15:39:03,070+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 67) WFLYUT0022: Unregistered web context: '/ovirt-engine/sso' from server 'default-server'
2018-03-15 15:39:03,070+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 12) WFLYUT0022: Unregistered web context: '/ovirt-engine/webadmin' from server 'default-server'
2018-03-15 15:39:03,071+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) WFLYUT0022: Unregistered web context: '/ovirt-engine/docs' from server 'default-server'
2018-03-15 15:39:03,132+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 69) WFLYUT0022: Unregistered web context: '/ovirt-engine/services' from server 'default-server'
2018-03-15 15:39:03,187+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2018-03-15 15:39:03,187+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2018-03-15 15:39:03,194+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment ovirt-web-ui.war (runtime-name: ovirt-web-ui.war) in 233ms
2018-03-15 15:39:03,195+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0028: Stopped deployment apidoc.war (runtime-name: apidoc.war) in 233ms
2018-03-15 15:39:03,195+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment restapi.war (runtime-name: restapi.war) in 247ms
2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutting down.
2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED paused.
2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0019: Host default-host stopping
2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping
2018-03-15 15:39:03,202+02 INFO [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutdown complete.
2018-03-15 15:39:03,213+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:/DWHDataSource]
2018-03-15 15:39:03,213+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:/ENGINEDataSource]
2018-03-15 15:39:03,226+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped timeout-base cache from ovirt-engine container
2018-03-15 15:39:03,228+02 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = postgresql
2018-03-15 15:39:03,233+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: root.war) in 271ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: enginesso.war) in 279ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: welcome.war) in 279ms
2018-03-15 15:39:03,235+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped inventory cache from ovirt-engine container
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) WFLYCLINF0003: Stopped dashboard cache from ovirt-engine container
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: webadmin.war) in 281ms
2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: docs.war) in 289ms
2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: services.war) in 267ms
2018-03-15 15:39:03,239+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: bll.jar) in 290ms
2018-03-15 15:39:03,242+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0028: Stopped deployment engine.ear (runtime-name: engine.ear) in 286ms
2018-03-15 15:39:03,265+02 INFO [org.jboss.as] (MSC service thread 1-5) WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly Core 3.0.8.Final) stopped in 302ms
[root at node00 ~]#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20180315/46c5f9fb/attachment.html>
More information about the Users
mailing list