[ovirt-users] Re: upgrade ovirt-engine 4.3 to 4.4. engine.ear problems?

- having a 4,3 ovirt-engine up and running on centos 7 - Create a new engine machine with centos 8 stream, latests updates - installing packages, restoring ovirt-engine backup from previous host and running ovirt-setup,

- When started, no access to the admin console at https://<fqdn>/ovirt-engine. The web proxy will return: 500 Internal error"" - I can access the Grafana login - ovirt-engine.service is started without any systemd errors Any Ideas how to proceed? Log files: /var/log/ovirt-engine/boot.log # No Errors /var/log/ovirt-engine/engine.log # No Errors /var/log/ovirt-engine/server.log # Errors found If I just grep for "ERROR" in server.log 2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) The complete starting sequence from server.log 2022-04-26 13:52:23,322+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "restapi.war" (runtime-name: "restapi.war") 2022-04-26 13:52:24,074+02 INFO [org.jboss.as.remoting] (MSC service thread 1-2) WFLYRMT0001: Listening on 127.0.0.1:8707 2022-04-26 13:52:24,247+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0098: Bound non-transactional data source: java:/ENGINEDataSourceNoJTA 2022-04-26 13:52:24,901+02 INFO [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete 2022-04-26 13:52:25,194+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/ENGINEDataSource] 2022-04-26 13:52:25,195+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/DWHDataSource] 2022-04-26 13:52:25,530+02 INFO [org.wildfly.security] (MSC service thread 1-3) ELY00001: WildFly Elytron version 1.15.3.Final 2022-04-26 13:52:25,741+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment restapi.war 2022-04-26 13:52:25,999+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "services.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "root.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "docs.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "welcome.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "bll.jar") 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "enginesso.war") 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "webadmin.war") 2022-04-26 13:52:26,368+02 INFO [org.hibernate.validator.internal.util.Version] (MSC service thread 1-2) HV000001: Hibernate Validator 6.0.22.Final 2022-04-26 13:52:26,752+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 40) WFLYUT0021: Registered web context: '/ovirt-engine/apidoc' for server 'default-server' 2022-04-26 13:52:26,751+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 41) WFLYUT0021: Registered web context: '/ovirt-engine/web-ui' for server 'default-server' 2022-04-26 13:52:26,983+02 INFO [org.jboss.weld.Version] (MSC service thread 1-2) WELD-000900: 3.1.6 (Final) 2022-04-26 13:52:28,596+02 WARN [org.jboss.as.dependency.deprecated] (MSC service thread 1-1) WFLYSRV0221: Deployment "deployment.engine.ear.bll.jar" is using a deprecated module ("sun.jdk") which may be removed in future versions without notice. 2022-04-26 13:52:28,604+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment engine.ear 2022-04-26 13:52:28,713+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment root.war 2022-04-26 13:52:28,747+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment docs.war 2022-04-26 13:52:28,863+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment enginesso.war 2022-04-26 13:52:28,929+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment services.war 2022-04-26 13:52:28,978+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment webadmin.war 2022-04-26 13:52:29,007+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment welcome.war 2022-04-26 13:52:29,042+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment bll.jar 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'LockManager' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'Backend' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'OvirtGlusterSchedulingService' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'InitBackendServicesOnStartupBean' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'ManagedBlockStorageDiskUtil' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'MacPoolPerCluster' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:30,878+02 INFO [org.infinispan.CONTAINER] (ServerService Thread Pool -- 51) ISPN000128: Infinispan version: Infinispan 'Corona Extra' 11.0.9.Final 2022-04-26 13:52:31,074+02 INFO [org.infinispan.PERSISTENCE] (ServerService Thread Pool -- 41) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 2022-04-26 13:52:31,462+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0002: Started dashboard cache from ovirt-engine container 2022-04-26 13:52:31,465+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 51) WFLYCLINF0002: Started inventory cache from ovirt-engine container 2022-04-26 13:52:35,426+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.JsonExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002220: Adding singleton resource org.ovirt.engine.api.restapi.resource.BackendApiResource from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MappingExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.IOExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MalformedIdExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,437+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,615+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 45) WFLYUT0021: Registered web context: '/ovirt-engine/api' for server 'default-server' 2022-04-26 13:52:40,024+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type org.codehaus.jackson.map.deser.std.StdScalarDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.JsonDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.deser.std.StdDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,350+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.DefaultGenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type class org.ovirt.engine.core.dao.DefaultReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ModificationDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.SearchDao<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,352+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.GenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,424+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.springframework.jdbc.core.RowMapper<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] private static class org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper 2022-04-26 13:52:41,564+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/' for server 'default-server' 2022-04-26 13:52:41,603+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 51) WFLYUT0021: Registered web context: '/ovirt-engine/docs' for server 'default-server' 2022-04-26 13:52:41,659+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for ENGINEDataSource 2022-04-26 13:52:41,673+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/ovirt-engine/services' for server 'default-server' 2022-04-26 13:52:41,687+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 56) WFLYUT0021: Registered web context: '/ovirt-engine' for server 'default-server' 2022-04-26 13:52:41,825+02 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] (ServerService Thread Pool -- 51) Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml] 2022-04-26 13:52:41,889+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for DWHDataSource 2022-04-26 13:52:41,934+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) WFLYUT0021: Registered web context: '/ovirt-engine/webadmin' for server 'default-server' 2022-04-26 13:52:41,938+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 44) WFLYUT0021: Registered web context: '/ovirt-engine/sso' for server 'default-server' 2022-04-26 13:52:42,093+02 INFO [org.springframework.jdbc.support.SQLErrorCodesFactory] (ServerService Thread Pool -- 51) SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase, Hana] 2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,240+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "ovirt-web-ui.war" (runtime-name : "ovirt-web-ui.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "apidoc.war" (runtime-name : "apidoc.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear") 2022-04-26 13:52:42,254+02 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report 2022-04-26 13:52:42,564+02 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) 2022-04-26 13:52:42,574+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:8706/management 2022-04-26 13:52:42,575+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:8706