Hi, On Wed, Apr 27, 2022 at 8:38 AM ingvar.jungenas--- via Users <users@ovirt.org> wrote:
- having a 4,3 ovirt-engine up and running on centos 7 - Create a new engine machine with centos 8 stream, latests updates - installing packages, restoring ovirt-engine backup from previous host and running ovirt-setup,
I suppose you mean you used engine-backup for backup/restore. Otherwise, please provide more details. Also, which exact 4.3 version did you have?
- When started, no access to the admin console at https://<fqdn>/ovirt-engine. The web proxy will return: 500 Internal error"" - I can access the Grafana login
- ovirt-engine.service is started without any systemd errors
Any Ideas how to proceed?
Log files: /var/log/ovirt-engine/boot.log # No Errors /var/log/ovirt-engine/engine.log # No Errors /var/log/ovirt-engine/server.log # Errors found If I just grep for "ERROR" in server.log
2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand)
The complete starting sequence from server.log
2022-04-26 13:52:23,322+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "restapi.war" (runtime-name: "restapi.war") 2022-04-26 13:52:24,074+02 INFO [org.jboss.as.remoting] (MSC service thread 1-2) WFLYRMT0001: Listening on 127.0.0.1:8707 2022-04-26 13:52:24,247+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0098: Bound non-transactional data source: java:/ENGINEDataSourceNoJTA 2022-04-26 13:52:24,901+02 INFO [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete 2022-04-26 13:52:25,194+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/ENGINEDataSource] 2022-04-26 13:52:25,195+02 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0001: Bound data source [java:/DWHDataSource] 2022-04-26 13:52:25,530+02 INFO [org.wildfly.security] (MSC service thread 1-3) ELY00001: WildFly Elytron version 1.15.3.Final 2022-04-26 13:52:25,741+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment restapi.war 2022-04-26 13:52:25,999+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "services.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "root.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "docs.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "welcome.war") 2022-04-26 13:52:26,000+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "bll.jar") 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "enginesso.war") 2022-04-26 13:52:26,001+02 INFO [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "webadmin.war") 2022-04-26 13:52:26,368+02 INFO [org.hibernate.validator.internal.util.Version] (MSC service thread 1-2) HV000001: Hibernate Validator 6.0.22.Final 2022-04-26 13:52:26,752+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 40) WFLYUT0021: Registered web context: '/ovirt-engine/apidoc' for server 'default-server' 2022-04-26 13:52:26,751+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 41) WFLYUT0021: Registered web context: '/ovirt-engine/web-ui' for server 'default-server' 2022-04-26 13:52:26,983+02 INFO [org.jboss.weld.Version] (MSC service thread 1-2) WELD-000900: 3.1.6 (Final) 2022-04-26 13:52:28,596+02 WARN [org.jboss.as.dependency.deprecated] (MSC service thread 1-1) WFLYSRV0221: Deployment "deployment.engine.ear.bll.jar" is using a deprecated module ("sun.jdk") which may be removed in future versions without notice. 2022-04-26 13:52:28,604+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment engine.ear 2022-04-26 13:52:28,713+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment root.war 2022-04-26 13:52:28,747+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment docs.war 2022-04-26 13:52:28,863+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment enginesso.war 2022-04-26 13:52:28,929+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0003: Processing weld deployment services.war 2022-04-26 13:52:28,978+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment webadmin.war 2022-04-26 13:52:29,007+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment welcome.war 2022-04-26 13:52:29,042+02 INFO [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment bll.jar 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'LockManager' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'Backend' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'OvirtGlusterSchedulingService' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,067+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'InitBackendServicesOnStartupBean' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'ManagedBlockStorageDiskUtil' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:29,068+02 INFO [org.jboss.as.ejb3.deployment] (MSC service thread 1-2) WFLYEJB0473: JNDI bindings for session bean named 'MacPoolPerCluster' in deployment unit 'subdeployment "bll.jar" of deployment "engine.ear"' are as follows: 2022-04-26 13:52:30,878+02 INFO [org.infinispan.CONTAINER] (ServerService Thread Pool -- 51) ISPN000128: Infinispan version: Infinispan 'Corona Extra' 11.0.9.Final 2022-04-26 13:52:31,074+02 INFO [org.infinispan.PERSISTENCE] (ServerService Thread Pool -- 41) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.marshalling.jboss.JBossMarshaller' 2022-04-26 13:52:31,462+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 41) WFLYCLINF0002: Started dashboard cache from ovirt-engine container 2022-04-26 13:52:31,465+02 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 51) WFLYCLINF0002: Started inventory cache from ovirt-engine container 2022-04-26 13:52:35,426+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002225: Deploying javax.ws.rs.core.Application: class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.JsonExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002220: Adding singleton resource org.ovirt.engine.api.restapi.resource.BackendApiResource from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MappingExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.IOExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,436+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.MalformedIdExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,437+02 INFO [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 45) RESTEASY002210: Adding provider singleton org.ovirt.engine.api.restapi.resource.validation.ValidationExceptionMapper from Application class org.ovirt.engine.api.restapi.BackendApplication 2022-04-26 13:52:35,615+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 45) WFLYUT0021: Registered web context: '/ovirt-engine/api' for server 'default-server' 2022-04-26 13:52:40,024+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type org.codehaus.jackson.map.deser.std.StdScalarDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.JsonDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,031+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 1) WELD-001125: Illegal bean type class org.codehaus.jackson.map.deser.std.StdDeserializer<java.lang.Enum<?>> ignored on [EnhancedAnnotatedTypeImpl] public class org.ovirt.engine.core.utils.serialization.json.JsonEnumDeserializer 2022-04-26 13:52:40,350+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.DefaultGenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type class org.ovirt.engine.core.dao.DefaultReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ModificationDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.SearchDao<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,351+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type interface org.ovirt.engine.core.dao.ReadDao<org.ovirt.engine.core.common.businessentities.Provider<?>,class org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,352+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.ovirt.engine.core.dao.GenericDao<org.ovirt.engine.core.common.businessentities.Provider<?>, org.ovirt.engine.core.compat.Guid> ignored on [EnhancedAnnotatedTypeImpl] public @Named @Singleton class org.ovirt.engine.core.dao.provider.ProviderDaoImpl 2022-04-26 13:52:40,424+02 INFO [org.jboss.weld.Bootstrap] (Weld Thread Pool -- 2) WELD-001125: Illegal bean type org.springframework.jdbc.core.RowMapper<org.ovirt.engine.core.common.businessentities.Provider<?>> ignored on [EnhancedAnnotatedTypeImpl] private static class org.ovirt.engine.core.dao.provider.ProviderDaoImpl$ProviderRowMapper 2022-04-26 13:52:41,564+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/' for server 'default-server' 2022-04-26 13:52:41,603+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 51) WFLYUT0021: Registered web context: '/ovirt-engine/docs' for server 'default-server' 2022-04-26 13:52:41,659+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for ENGINEDataSource 2022-04-26 13:52:41,673+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 58) WFLYUT0021: Registered web context: '/ovirt-engine/services' for server 'default-server' 2022-04-26 13:52:41,687+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 56) WFLYUT0021: Registered web context: '/ovirt-engine' for server 'default-server' 2022-04-26 13:52:41,825+02 INFO [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] (ServerService Thread Pool -- 51) Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml] 2022-04-26 13:52:41,889+02 WARN [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 47) IJ000407: No lazy enlistment available for DWHDataSource 2022-04-26 13:52:41,934+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) WFLYUT0021: Registered web context: '/ovirt-engine/webadmin' for server 'default-server' 2022-04-26 13:52:41,938+02 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 44) WFLYUT0021: Registered web context: '/ovirt-engine/sso' for server 'default-server' 2022-04-26 13:52:42,093+02 INFO [org.springframework.jdbc.support.SQLErrorCodesFactory] (ServerService Thread Pool -- 51) SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase, Hana] 2022-04-26 13:52:42,220+02 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 51) MSC000001: Failed to start service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: org.jboss.msc.service.StartException in service jboss.deployment.subunit."engine.ear"."bll.jar".component.Backend.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,231+02 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "engine.ear")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.deployment.subunit.\"engine.ear\".\"bll.jar\".component.Backend.START" => "java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance 2022-04-26 13:52:42,240+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "ovirt-web-ui.war" (runtime-name : "ovirt-web-ui.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "apidoc.war" (runtime-name : "apidoc.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "restapi.war" (runtime-name : "restapi.war") 2022-04-26 13:52:42,241+02 INFO [org.jboss.as.server] (ServerService Thread Pool -- 25) WFLYSRV0010: Deployed "engine.ear" (runtime-name : "engine.ear") 2022-04-26 13:52:42,254+02 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report 2022-04-26 13:52:42,564+02 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server 2022-04-26 13:52:42,568+02 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: WildFly Full 23.0.2.Final (WildFly Core 15.0.1.Final) started (with errors) in 24096ms - Started 1668 of 1888 services (6 services failed or missing dependencies, 393 services are lazy, passive or on-demand) 2022-04-26 13:52:42,574+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:8706/management 2022-04-26 13:52:42,575+02 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:8706
I quickly skimmed through the above log snippet, and do not know why it fails. Some of your errors/warnings do appear in google. Did you try that? Not sure any I noticed are relevant to your case, though. Please note that oVirt 4.5.0 was released recently and 4.4 is EOL. Please try again with 4.5. It should be possible to upgrade via backup/restore directly from 4.3 to 4.5, no need to go through 4.4: https://github.com/oVirt/ovirt-engine/pull/244 Last but not least, thanks for a great report, including all relevant details right on the start. Best regards, -- Didi