Warning: CentOS Upgrade with Host Engine - 503 Service Temporarily Unavailable

--Apple-Mail=_3E424712-A2E5-435F-9BB1-B5C8DE1FC44A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi ! I have upgraded CentOS 7.4 with oVirt 4.2.1 host engine (with yum = upgrade), and its resulted in broken system - "503 Service Temporarily = Unavailable" when connecting to the host engine via web. Service ovirt-engine failed to starts (logs attached at the bottom of = this email), other ovirt services seem to run fine. yum update "ovirt-*-setup*=E2=80=9D (upgrade 4.2 -> 4.2.1) engine-setup yum upgrade (OS upgrade) Is this issue somehow related to JDK as described here? https://bugzilla.redhat.com/show_bug.cgi?id=3D1217023 New packages: java-1.8.0-openjdk x86_64 1:1.8.0.161-0.b14.el7_4 java-1.8.0-openjdk-devel x86_64 1:1.8.0.161-0.b14.el7_4=20= java-1.8.0-openjdk-headless x86_64 1:1.8.0.161-0.b14.el7_4=20= installed packages (seem to be also 1.8x): [root@node00 ~]# rpm -qa | grep jdk java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64 java-1.8.0-openjdk-devel-1.8.0.151-5.b12.el7_4.x86_64 java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64 copy-jdk-configs-2.2-5.el7_4.noarch Since my host engine is actually a KVM appliance under another SuSE = server, I simply discarded it and reverted back old qcow2 image. So this is a warning to anyone - don=E2=80=99t upgrade CentOS, or at = least keep a copy of disk image before ANY upgrade ! ********** LOGS ********* ****************************** [root@node00 ~]# service ovirt-engine status -l Redirecting to /bin/systemctl status -l ovirt-engine.service =E2=97=8F ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; = enabled; vendor preset: disabled) Active: failed (Result: timeout) since Thu 2018-03-15 15:44:08 EET; = 2min 18s ago Process: 1474 = ExecStart=3D/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py = --redirect-output --systemd=3Dnotify $EXTRA_ARGS start (code=3Dkilled, = signal=3DTERM) Main PID: 1474 (code=3Dkilled, signal=3DTERM) CGroup: /system.slice/ovirt-engine.service Mar 15 15:42:37 node00.myhost.lv systemd[1]: Starting oVirt Engine... Mar 15 15:42:50 node00.myhost.lv ovirt-engine.py[1474]: 2018-03-15 = 15:42:50,869+0200 ovirt-engine: INFO _detectJBossVersion:187 Detecting = JBoss version. Running: /usr/lib/jvm/jre/bin/java = ['ovirt-engine-version', '-server', '-XX:+TieredCompilation', = '-Xms1024M', '-Xmx1024M', '-Djava.awt.headless=3Dtrue', = '-Dsun.rmi.dgc.client.gcInterval=3D3600000', = '-Dsun.rmi.dgc.server.gcInterval=3D3600000', = '-Djsse.enableSNIExtension=3Dfalse', '-XX:+HeapDumpOnOutOfMemoryError', = '-XX:HeapDumpPath=3D/var/log/ovirt-engine/dump', = '-Djava.util.logging.manager=3Dorg.jboss.logmanager', = '-Dlogging.configuration=3Dfile:///var/lib/ovirt-engine/jboss_runtime/conf= ig/ovirt-engine-logging.properties', = '-Dorg.jboss.resolver.warning=3Dtrue', = '-Djboss.modules.system.pkgs=3Dorg.jboss.byteman', = '-Djboss.server.default.config=3Dovirt-engine', = '-Djboss.home.dir=3D/usr/share/ovirt-engine-wildfly', = '-Djboss.server.base.dir=3D/usr/share/ovirt-engine', = '-Djboss.server.data.dir=3D/var/lib/ovirt-engine', = '-Djboss.server.log.dir=3D/var/log/ovirt-engine', = '-Djboss.server.config.dir=3D/var/lib/ovirt-engine/jboss_runtime/config', = '-Djboss.server.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp', = '-Djboss.controller.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp', = '-jar', '/usr/share/ovirt-engine-wildfly/jboss-modules.jar', '-mp', = '/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/m= odules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/shar= e/ovirt-engine-wildfly/modules', '-jaxpmodule', = 'javax.xml.jaxp-provider', 'org.jboss.as.standalone', '-v'] Mar 15 15:44:08 node00.myhost.lv systemd[1]: ovirt-engine.service start = operation timed out. Terminating. Mar 15 15:44:08 node00.myhost.lv systemd[1]: Failed to start oVirt = Engine. Mar 15 15:44:08 node00.myhost.lv systemd[1]: Unit ovirt-engine.service = entered failed state. Mar 15 15:44:08 node00.myhost.lv systemd[1]: ovirt-engine.service = failed. [root@node00 ~]#=20 [root@node00 ~]# tail -n 20 /var/log/ovirt-engine/server.log=20 2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = shutting down. 2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = paused. 2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC = service thread 1-2) WFLYUT0019: Host default-host stopping 2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC = service thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping 2018-03-15 15:39:03,202+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = shutdown complete. 2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) = WFLYJCA0010: Unbound data source [java:/DWHDataSource] 2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) = WFLYJCA0010: Unbound data source [java:/ENGINEDataSource] 2018-03-15 15:39:03,226+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped timeout-base cache from = ovirt-engine container 2018-03-15 15:39:03,228+02 INFO [org.jboss.as.connector.deployers.jdbc] = (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with = driver-name =3D postgresql 2018-03-15 15:39:03,233+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: = root.war) in 271ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = enginesso.war) in 279ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = welcome.war) in 279ms 2018-03-15 15:39:03,235+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped inventory cache from = ovirt-engine container 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped dashboard cache from = ovirt-engine container 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = webadmin.war) in 281ms 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: = docs.war) in 289ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: = services.war) in 267ms 2018-03-15 15:39:03,239+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: = bll.jar) in 290ms 2018-03-15 15:39:03,242+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-5) WFLYSRV0028: Stopped deployment engine.ear = (runtime-name: engine.ear) in 286ms 2018-03-15 15:39:03,265+02 INFO [org.jboss.as] (MSC service thread 1-5) = WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly Core 3.0.8.Final) = stopped in 302ms [root@node00 ~]#=20 [root@node00 ~]# tail -n 30 /var/log/ovirt-engine/server.log=20 2018-03-15 15:39:03,070+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 67) WFLYUT0022: Unregistered web context: = '/ovirt-engine/sso' from server 'default-server' 2018-03-15 15:39:03,070+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 12) WFLYUT0022: Unregistered web context: = '/ovirt-engine/webadmin' from server 'default-server' 2018-03-15 15:39:03,071+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 66) WFLYUT0022: Unregistered web context: = '/ovirt-engine/docs' from server 'default-server' 2018-03-15 15:39:03,132+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 69) WFLYUT0022: Unregistered web context: = '/ovirt-engine/services' from server 'default-server' 2018-03-15 15:39:03,187+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED = shutting down. 2018-03-15 15:39:03,187+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED = paused. 2018-03-15 15:39:03,194+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-5) WFLYSRV0028: Stopped deployment ovirt-web-ui.war = (runtime-name: ovirt-web-ui.war) in 233ms 2018-03-15 15:39:03,195+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-6) WFLYSRV0028: Stopped deployment apidoc.war = (runtime-name: apidoc.war) in 233ms 2018-03-15 15:39:03,195+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0028: Stopped deployment restapi.war = (runtime-name: restapi.war) in 247ms 2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED = shutdown complete. 2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = shutting down. 2018-03-15 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = paused. 2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC = service thread 1-2) WFLYUT0019: Host default-host stopping 2018-03-15 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC = service thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping 2018-03-15 15:39:03,202+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = shutdown complete. 2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) = WFLYJCA0010: Unbound data source [java:/DWHDataSource] 2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) = WFLYJCA0010: Unbound data source [java:/ENGINEDataSource] 2018-03-15 15:39:03,226+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped timeout-base cache from = ovirt-engine container 2018-03-15 15:39:03,228+02 INFO [org.jboss.as.connector.deployers.jdbc] = (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with = driver-name =3D postgresql 2018-03-15 15:39:03,233+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: = root.war) in 271ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = enginesso.war) in 279ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = welcome.war) in 279ms 2018-03-15 15:39:03,235+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped inventory cache from = ovirt-engine container 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.clustering.infinispan] = (MSC service thread 1-4) WFLYCLINF0003: Stopped dashboard cache from = ovirt-engine container 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-7) WFLYSRV0208: Stopped subdeployment (runtime-name: = webadmin.war) in 281ms 2018-03-15 15:39:03,237+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-5) WFLYSRV0208: Stopped subdeployment (runtime-name: = docs.war) in 289ms 2018-03-15 15:39:03,234+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-3) WFLYSRV0208: Stopped subdeployment (runtime-name: = services.war) in 267ms 2018-03-15 15:39:03,239+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: = bll.jar) in 290ms 2018-03-15 15:39:03,242+02 INFO [org.jboss.as.server.deployment] (MSC = service thread 1-5) WFLYSRV0028: Stopped deployment engine.ear = (runtime-name: engine.ear) in 286ms 2018-03-15 15:39:03,265+02 INFO [org.jboss.as] (MSC service thread 1-5) = WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly Core 3.0.8.Final) = stopped in 302ms [root@node00 ~]#=20 --Apple-Mail=_3E424712-A2E5-435F-9BB1-B5C8DE1FC44A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;" class=3D"">Hi !<br class=3D""><br = class=3D"">I have upgraded CentOS 7.4 with oVirt 4.2.1 host engine (with = yum upgrade), and its resulted in broken system - "503 Service = Temporarily Unavailable" when connecting to the host engine via web.<br = class=3D""><div class=3D"">Service ovirt-engine failed to starts (logs = attached at the bottom of this email), other ovirt services seem to run = fine.<br class=3D""><br class=3D"">yum update "ovirt-*-setup*=E2=80=9D = (upgrade 4.2 -> 4.2.1)<br class=3D"">engine-setup<div class=3D"">yum = upgrade (OS upgrade)<br class=3D""><br class=3D"">Is this = issue somehow related to JDK as described here?<br class=3D""><a = href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1217023" = class=3D"">https://bugzilla.redhat.com/show_bug.cgi?id=3D1217023</a><br = class=3D""><br class=3D"">New packages:<br class=3D"">java-1.8.0-openjdk = = x86_64 1:1.8.0.161-0.b14.el7_4<br = class=3D"">java-1.8.0-openjdk-devel = x86_64 1:1.8.0.161-0.b14.el7_4 <br = class=3D"">java-1.8.0-openjdk-headless = x86_64 1:1.8.0.161-0.b14.el7_4 </div><div = class=3D""><br class=3D""></div><div class=3D"">installed packages (seem = to be also 1.8x):</div><div class=3D"">[root@node00 ~]# rpm -qa | grep = jdk<br class=3D"">java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64<br = class=3D"">java-1.8.0-openjdk-devel-1.8.0.151-5.b12.el7_4.x86_64<br = class=3D"">java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64<br = class=3D"">copy-jdk-configs-2.2-5.el7_4.noarch</div><div class=3D""><br = class=3D""></div><div class=3D"">Since my host engine is actually a KVM = appliance under another SuSE server, I simply discarded it and reverted = back old qcow2 image.</div><div class=3D""><br class=3D""></div><div = class=3D""><b class=3D""><font color=3D"#ff2600" class=3D"">So this is a = warning to anyone - don=E2=80=99t upgrade CentOS, or at least keep a = copy of disk image before ANY upgrade !</font></b></div><div = class=3D""><br class=3D""><br class=3D"">********** LOGS = *********<br class=3D"">******************************<br = class=3D"">[root@node00 ~]# service ovirt-engine status -l<br = class=3D"">Redirecting to /bin/systemctl status -l = ovirt-engine.service<br class=3D"">=E2=97=8F ovirt-engine.service - = oVirt Engine<br class=3D""> Loaded: loaded = (/usr/lib/systemd/system/ovirt-engine.service; enabled; vendor preset: = disabled)<br class=3D""> Active: failed (Result: timeout) = since Thu 2018-03-15 15:44:08 EET; 2min 18s ago<br class=3D""> = Process: 1474 = ExecStart=3D/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py = --redirect-output --systemd=3Dnotify $EXTRA_ARGS start (code=3Dkilled, = signal=3DTERM)<br class=3D""> Main PID: 1474 (code=3Dkilled, = signal=3DTERM)<br class=3D""> CGroup: = /system.slice/ovirt-engine.service<br class=3D""><br class=3D"">Mar 15 = 15:42:37 node00.myhost.lv systemd[1]: Starting oVirt Engine...<br = class=3D"">Mar 15 15:42:50 node00.myhost.lv ovirt-engine.py[1474]: = 2018-03-15 15:42:50,869+0200 ovirt-engine: INFO _detectJBossVersion:187 = Detecting JBoss version. Running: /usr/lib/jvm/jre/bin/java = ['ovirt-engine-version', '-server', '-XX:+TieredCompilation', = '-Xms1024M', '-Xmx1024M', '-Djava.awt.headless=3Dtrue', = '-Dsun.rmi.dgc.client.gcInterval=3D3600000', = '-Dsun.rmi.dgc.server.gcInterval=3D3600000', = '-Djsse.enableSNIExtension=3Dfalse', '-XX:+HeapDumpOnOutOfMemoryError', = '-XX:HeapDumpPath=3D/var/log/ovirt-engine/dump', = '-Djava.util.logging.manager=3Dorg.jboss.logmanager', = '-Dlogging.configuration=3D<a = href=3D"file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-log= ging.properties'" = class=3D"">file:///var/lib/ovirt-engine/jboss_runtime/config/ovirt-engine-= logging.properties'</a>, '-Dorg.jboss.resolver.warning=3Dtrue', = '-Djboss.modules.system.pkgs=3Dorg.jboss.byteman', = '-Djboss.server.default.config=3Dovirt-engine', = '-Djboss.home.dir=3D/usr/share/ovirt-engine-wildfly', = '-Djboss.server.base.dir=3D/usr/share/ovirt-engine', = '-Djboss.server.data.dir=3D/var/lib/ovirt-engine', = '-Djboss.server.log.dir=3D/var/log/ovirt-engine', = '-Djboss.server.config.dir=3D/var/lib/ovirt-engine/jboss_runtime/config', = '-Djboss.server.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp', = '-Djboss.controller.temp.dir=3D/var/lib/ovirt-engine/jboss_runtime/tmp', = '-jar', '/usr/share/ovirt-engine-wildfly/jboss-modules.jar', '-mp', = '/usr/share/ovirt-engine-wildfly-overlay/modules:/usr/share/ovirt-engine/m= odules/common:/usr/share/ovirt-engine-extension-aaa-jdbc/modules:/usr/shar= e/ovirt-engine-wildfly/modules', '-jaxpmodule', = 'javax.xml.jaxp-provider', 'org.jboss.as.standalone', '-v']<br = class=3D"">Mar 15 15:44:08 node00.myhost.lv systemd[1]: = ovirt-engine.service start operation timed out. Terminating.<br = class=3D"">Mar 15 15:44:08 node00.myhost.lv systemd[1]: Failed to start = oVirt Engine.<br class=3D"">Mar 15 15:44:08 node00.myhost.lv systemd[1]: = Unit ovirt-engine.service entered failed state.<br class=3D"">Mar 15 = 15:44:08 node00.myhost.lv systemd[1]: ovirt-engine.service failed.<br = class=3D"">[root@node00 ~]# <br class=3D""><br class=3D""><br = class=3D"">[root@node00 ~]# tail -n 20 = /var/log/ovirt-engine/server.log <br class=3D"">2018-03-15 = 15:39:03,196+02 INFO [org.quartz.core.QuartzScheduler] (MSC = service thread 1-4) Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED = shutting down.<br class=3D"">2018-03-15 15:39:03,196+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED paused.<br = class=3D"">2018-03-15 15:39:03,201+02 INFO = [org.wildfly.extension.undertow] (MSC service thread 1-2) = WFLYUT0019: Host default-host stopping<br class=3D"">2018-03-15 = 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service = thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping<br = class=3D"">2018-03-15 15:39:03,202+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutdown complete.<br = class=3D"">2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service = thread 1-2) WFLYJCA0010: Unbound data source [java:/DWHDataSource]<br = class=3D"">2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service = thread 1-2) WFLYJCA0010: Unbound data source [java:/ENGINEDataSource]<br = class=3D"">2018-03-15 15:39:03,226+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped timeout-base cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,228+02 INFO = [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) = WFLYJCA0019: Stopped Driver service with driver-name =3D postgresql<br = class=3D"">2018-03-15 15:39:03,233+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-3) = WFLYSRV0208: Stopped subdeployment (runtime-name: root.war) in 271ms<br = class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: enginesso.war) in = 279ms<br class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: welcome.war) in = 279ms<br class=3D"">2018-03-15 15:39:03,235+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped inventory cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped dashboard cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: webadmin.war) in = 281ms<br class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-5) = WFLYSRV0208: Stopped subdeployment (runtime-name: docs.war) in 289ms<br = class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-3) = WFLYSRV0208: Stopped subdeployment (runtime-name: services.war) in = 267ms<br class=3D"">2018-03-15 15:39:03,239+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-2) = WFLYSRV0208: Stopped subdeployment (runtime-name: bll.jar) in 290ms<br = class=3D"">2018-03-15 15:39:03,242+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-5) = WFLYSRV0028: Stopped deployment engine.ear (runtime-name: engine.ear) in = 286ms<br class=3D"">2018-03-15 15:39:03,265+02 INFO [org.jboss.as] = (MSC service thread 1-5) WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly = Core 3.0.8.Final) stopped in 302ms<br class=3D"">[root@node00 = ~]# <br class=3D"">[root@node00 ~]# tail -n 30 = /var/log/ovirt-engine/server.log <br class=3D"">2018-03-15 = 15:39:03,070+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 67) WFLYUT0022: Unregistered web context: = '/ovirt-engine/sso' from server 'default-server'<br class=3D"">2018-03-15 = 15:39:03,070+02 INFO [org.wildfly.extension.undertow] = (ServerService Thread Pool -- 12) WFLYUT0022: Unregistered web context: = '/ovirt-engine/webadmin' from server 'default-server'<br = class=3D"">2018-03-15 15:39:03,071+02 INFO = [org.wildfly.extension.undertow] (ServerService Thread Pool -- 66) = WFLYUT0022: Unregistered web context: '/ovirt-engine/docs' from server = 'default-server'<br class=3D"">2018-03-15 15:39:03,132+02 INFO = [org.wildfly.extension.undertow] (ServerService Thread Pool -- 69) = WFLYUT0022: Unregistered web context: '/ovirt-engine/services' from = server 'default-server'<br class=3D"">2018-03-15 15:39:03,187+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.<br = class=3D"">2018-03-15 15:39:03,187+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.<br = class=3D"">2018-03-15 15:39:03,194+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-5) = WFLYSRV0028: Stopped deployment ovirt-web-ui.war (runtime-name: = ovirt-web-ui.war) in 233ms<br class=3D"">2018-03-15 15:39:03,195+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-6) = WFLYSRV0028: Stopped deployment apidoc.war (runtime-name: apidoc.war) in = 233ms<br class=3D"">2018-03-15 15:39:03,195+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0028: Stopped deployment restapi.war (runtime-name: restapi.war) = in 247ms<br class=3D"">2018-03-15 15:39:03,196+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.<br = class=3D"">2018-03-15 15:39:03,196+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutting down.<br = class=3D"">2018-03-15 15:39:03,196+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED paused.<br = class=3D"">2018-03-15 15:39:03,201+02 INFO = [org.wildfly.extension.undertow] (MSC service thread 1-2) = WFLYUT0019: Host default-host stopping<br class=3D"">2018-03-15 = 15:39:03,201+02 INFO [org.wildfly.extension.undertow] (MSC service = thread 1-2) WFLYUT0004: Undertow 1.4.18.Final stopping<br = class=3D"">2018-03-15 15:39:03,202+02 INFO = [org.quartz.core.QuartzScheduler] (MSC service thread 1-4) = Scheduler QuartzOvirtDBScheduler_$_NON_CLUSTERED shutdown complete.<br = class=3D"">2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service = thread 1-2) WFLYJCA0010: Unbound data source [java:/DWHDataSource]<br = class=3D"">2018-03-15 15:39:03,213+02 INFO = [org.jboss.as.connector.subsystems.datasources] (MSC service = thread 1-2) WFLYJCA0010: Unbound data source [java:/ENGINEDataSource]<br = class=3D"">2018-03-15 15:39:03,226+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped timeout-base cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,228+02 INFO = [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) = WFLYJCA0019: Stopped Driver service with driver-name =3D postgresql<br = class=3D"">2018-03-15 15:39:03,233+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-3) = WFLYSRV0208: Stopped subdeployment (runtime-name: root.war) in 271ms<br = class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: enginesso.war) in = 279ms<br class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: welcome.war) in = 279ms<br class=3D"">2018-03-15 15:39:03,235+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped inventory cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) = WFLYCLINF0003: Stopped dashboard cache from ovirt-engine container<br = class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-7) = WFLYSRV0208: Stopped subdeployment (runtime-name: webadmin.war) in = 281ms<br class=3D"">2018-03-15 15:39:03,237+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-5) = WFLYSRV0208: Stopped subdeployment (runtime-name: docs.war) in 289ms<br = class=3D"">2018-03-15 15:39:03,234+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-3) = WFLYSRV0208: Stopped subdeployment (runtime-name: services.war) in = 267ms<br class=3D"">2018-03-15 15:39:03,239+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-2) = WFLYSRV0208: Stopped subdeployment (runtime-name: bll.jar) in 290ms<br = class=3D"">2018-03-15 15:39:03,242+02 INFO = [org.jboss.as.server.deployment] (MSC service thread 1-5) = WFLYSRV0028: Stopped deployment engine.ear (runtime-name: engine.ear) in = 286ms<br class=3D"">2018-03-15 15:39:03,265+02 INFO [org.jboss.as] = (MSC service thread 1-5) WFLYSRV0050: WildFly Full 11.0.0.Final (WildFly = Core 3.0.8.Final) stopped in 302ms<br class=3D"">[root@node00 = ~]# <br class=3D""><br class=3D""></div></div></body></html>= --Apple-Mail=_3E424712-A2E5-435F-9BB1-B5C8DE1FC44A--

On Thu, Mar 15, 2018 at 5:39 PM, Andrei Verovski <andreil1@starlett.lv> wrote:
Hi !
I have upgraded CentOS 7.4 with oVirt 4.2.1 host engine (with yum upgrade), and its resulted in broken system - "503 Service Temporarily Unavailable" when connecting to the host engine via web. Service ovirt-engine failed to starts (logs attached at the bottom of this email), other ovirt services seem to run fine.
yum update "ovirt-*-setup*” (upgrade 4.2 -> 4.2.1) engine-setup yum upgrade (OS upgrade)
Is this issue somehow related to JDK as described here? https://bugzilla.redhat.com/show_bug.cgi?id=1217023
Not very likely. This is a very old bug.
New packages: java-1.8.0-openjdk x86_64 1:1.8.0.161-0.b14.el7_4 java-1.8.0-openjdk-devel x86_64 1:1.8.0.161-0.b14.el7_4 java-1.8.0-openjdk-headless x86_64 1:1.8.0.161-0.b14.el7_4
installed packages (seem to be also 1.8x): [root@node00 ~]# rpm -qa | grep jdk java-1.8.0-openjdk-1.8.0.151-5.b12.el7_4.x86_64 java-1.8.0-openjdk-devel-1.8.0.151-5.b12.el7_4.x86_64 java-1.8.0-openjdk-headless-1.8.0.151-5.b12.el7_4.x86_64 copy-jdk-configs-2.2-5.el7_4.noarch
Indeed. Some versions of the engine could work with both openjdk 1.7 and 1.8, but recent versions require 1.8.
Since my host engine is actually a KVM appliance under another SuSE server, I simply discarded it and reverted back old qcow2 image.
OK. Any chance you kept a copy, or at least more logs, for further debugging?
So this is a warning to anyone - don’t upgrade CentOS, or at least keep a copy of disk image before ANY upgrade !
Having regular backups, both before significant changes like upgrades and also routine, on-going, is always a good idea :-) If you have all of server.log, we can try to understand the cause for the failure. If not, and you try again to upgrade and if fails again, please share all relevant logs before reverting to the backup. Thanks! Best regards, -- Didi
participants (2)
-
Andrei Verovski
-
Yedidyah Bar David