[ovirt-users] Re: upgrade ovirt-engine 4.3 to 4.4. engine.ear problems?

Hi, On Wed, Apr 27, 2022 at 8:38 AM ingvar.jungenas--- via Users <users(a)ovirt.org&gt; wrote: > > - having a 4,3 ovirt-engine up and running on centos 7 > - Create a new engine machine with centos 8 stream, latests updates > - installing packages, restoring ovirt-engine backup from previous host and running ovirt-setup, I suppose you mean you used engine-backup for backup/restore. Otherwise, please provide more details. Also, which exact 4.3 version did you have? > - When started, no access to the admin console at https://<fqdn>/ovirt-engine. The web proxy will return: 500 Internal error"" > - I can access the Grafana login > > - ovirt-engine.service is started without any systemd errors > > Any Ideas how to proceed? > > Log files: > /var/log/ovirt-engine/boot.log # No Errors > /var/log/ovirt-engine/engine.log # No Errors > /var/log/ovirt-engine/server.log # Errors found > If I just grep for "ERROR" in server.log > > 2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance > 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance > 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) > > The complete starting sequence from server.log > > 2022-04-26 13:52:23,322+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "restapi.war" (runtime-name: "restapi.war") > 2022-04-26 13:52:24,074+02 INFO [org.jboss.as.remoting] (MSC service thread 1-2) WFLYRMT0001: Listening on 127.0.0.1:8707 > 2022-04-26 13:52:24,247+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0098: Bound non-transactional data source: java:/ENGINEDataSourceNoJTA > 2022-04-26 13:52:24,901+02 INFO [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete > 2022-04-26 13:52:25,194+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/ENGINEDataSource] > 2022-04-26 13:52:25,195+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/DWHDataSource] > 2022-04-26 13:52:25,530+02 INFO [org.wildfly.security] (MSC service thread 1-3) ELY00001: WildFly Elytron version 1.15.3.Final > 2022-04-26 13:52:25,741+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment restapi.war > 2022-04-26 13:52:25,999+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "services.war") > 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "root.war") > 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "docs.war") > 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "welcome.war") > 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "bll.jar") > 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "enginesso.war") > 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "webadmin.war") > 2022-04-26 13:52:26,368+02 INFO [org.hibernate.validator.internal.util.Version] (MSC service thread 1-2) HV000001: Hibernate Validator 6.0.22.Final > 2022-04-26 13:52:26,752+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 40) WFLYUT0021: Registered web context: '/ovirt-engine/apidoc' for server 'default-server' > 2022-04-26 13:52:26,751+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 41) WFLYUT0021: Registered web context: '/ovirt-engine/web-ui' for server 'default-server' > 2022-04-26 13:52:26,983+02 INFO [org.jboss.weld.Version] (MSC service thread 1-2) WELD-000900: 3.1.6 (Final) > 2022-04-26 13:52:28,596+02 WARN [org.jboss.as.dependency.deprecated] (MSC service thread 1-1) WFLYSRV0221: Deployment "deployment.engine.ear.bll.jar" is using a deprecated module ("sun.jdk") which may be removed in future versions without notice. > 2022-04-26 13:52:28,604+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment engine.ear > 2022-04-26 13:52:28,713+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment root.war > 2022-04-26 13:52:28,747+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment docs.war > 2022-04-26 13:52:28,863+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment enginesso.war > 2022-04-26 13:52:28,929+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment services.war > 2022-04-26 13:52:28,978+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment webadmin.war > 2022-04-26 13:52:29,007+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment welcome.war > 2022-04-26 13:52:29,042+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment bll.jar > 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'LockManager' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'Backend' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'OvirtGlusterSchedulingService' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'InitBackendServicesOnStartupBean' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'ManagedBlockStorageDiskUtil' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'MacPoolPerCluster' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: > 2022-04-26 13:52:30,878+02 INFO [org.infinispan.CONTAINER] (ServerService Thread Pool -- 51) ISPN000128: Infinispan version: Infinispan 'Corona Extra' 11.0.9.Final > 2022-04-26 13:52:31,074+02 INFO [org.infinispan.PERSISTENCE] (ServerService Thread Pool -- 41) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' > 2022-04-26 13:52:31,462+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0002: Started dashboard cache from ovirt-engine container > 2022-04-26 13:52:31,465+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 51) WFLYCLINF0002: Started inventory cache from ovirt-engine container > 2022-04-26 13:52:35,426+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.JsonExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002220: Adding singleton resource org.ovirt.engine.api.restapi.resource.BackendApiResource from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MappingExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.IOExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MalformedIdExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,437+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication > 2022-04-26 13:52:35,615+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 45) WFLYUT0021: Registered web context: '/ovirt-engine/api' for server 'default-server' > 2022-04-26 13:52:40,024+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type org.codehaus.jackson.map.deser.std.StdScalarDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer > 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.JsonDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer > 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.deser.std.StdDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer > 2022-04-26 13:52:40,350+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.DefaultGenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type class org.ovirt.engine.core.dao.DefaultReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ModificationDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.SearchDao<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,352+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.GenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl > 2022-04-26 13:52:40,424+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.springframework.jdbc.core.RowMapper<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] private static class org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper > 2022-04-26 13:52:41,564+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/' for server 'default-server' > 2022-04-26 13:52:41,603+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 51) WFLYUT0021: Registered web context: '/ovirt-engine/docs' for server 'default-server' > 2022-04-26 13:52:41,659+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for ENGINEDataSource > 2022-04-26 13:52:41,673+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/ovirt-engine/services' for server 'default-server' > 2022-04-26 13:52:41,687+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 56) WFLYUT0021: Registered web context: '/ovirt-engine' for server 'default-server' > 2022-04-26 13:52:41,825+02 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] (ServerService Thread Pool -- 51) Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml] > 2022-04-26 13:52:41,889+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for DWHDataSource > 2022-04-26 13:52:41,934+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) WFLYUT0021: Registered web context: '/ovirt-engine/webadmin' for server 'default-server' > 2022-04-26 13:52:41,938+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 44) WFLYUT0021: Registered web context: '/ovirt-engine/sso' for server 'default-server' > 2022-04-26 13:52:42,093+02 INFO [org.springframework.jdbc.support.SQLErrorCodesFactory] (ServerService Thread Pool -- 51) SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase, Hana] > 2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance > 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance > 2022-04-26 13:52:42,240+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "ovirt-web-ui.war" (runtime-name : "ovirt-web-ui.war") > 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "apidoc.war" (runtime-name : "apidoc.war") > 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war") > 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear") > 2022-04-26 13:52:42,254+02 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report > 2022-04-26 13:52:42,564+02 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server > 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) > 2022-04-26 13:52:42,574+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:8706/management > 2022-04-26 13:52:42,575+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:8706 I quickly skimmed through the above log snippet, and do not know why it fails. Some of your errors/warnings do appear in google. Did you try that? Not sure any I noticed are relevant to your case, though. Please note that oVirt 4.5.0 was released recently and 4.4 is EOL. Please try again with 4.5. It should be possible to upgrade via backup/restore directly from 4.3 to 4.5, no need to go through 4.4: https://github.com/oVirt/ovirt-engine/pull/244 Last but not least, thanks for a great report, including all relevant details right on the start. Best regards, -- Didi _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/6DZLI2ATFH3...