<div dir="ltr"><div><div>Hi Martin,<br><br></div> Storage isn't an issue, that was me trying to force fix another problem (removing 'dead' hosted engine hosts)<br><br></div> This just popped up in engine.log:<br><br><font face="monospace,monospace">2017-10-10 06:53:09,022+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Exception: java.lang.IllegalArgumentException: VdsFenceOptionMapping has no value for version: general<br> at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:65) [dal.jar:]<br> at org.ovirt.engine.core.common.config.Config.getValue(Config.java:28) [common.jar:]<br> at org.ovirt.engine.core.utils.pm.FenceConfigHelper.getFenceConfigurationValue(FenceConfigHelper.java:96) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.cacheFenceAgentsOptionMapping(VdsFenceOptions.java:78) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.initCache(VdsFenceOptions.java:236) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:63) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:47) [utils.jar:]<br> at org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery.executeQueryCommand(GetAgentFenceOptionsQuery.java:18) [bll.jar:]<br> at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:]<br> at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]<br> at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14) [bll.jar:]<br> at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:570) [bll.jar:]<br> at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:539) [bll.jar:]<br> at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]<br> at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)<br> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]<br> at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)<br> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)<br> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)<br> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)<br> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)<br> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)<br> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)<br> at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:]<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runQuery(GenericApiGWTServiceImpl.java:90)<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runMultipleQueries(GenericApiGWTServiceImpl.java:121)<br> at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)<br> at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78)<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]<br> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)<br> at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)<br> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)<br> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)<br> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)<br> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)<br> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)<br> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)<br> at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)<br> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)<br> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)<br> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)<br> at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)<br> at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)<br> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)<br> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)<br> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)<br> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)<br> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)<br> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)<br> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)<br> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)<br> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_144]<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_144]<br> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_144]<br><br>2017-10-10 06:53:09,032+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Query 'GetAgentFenceOptionsQuery' failed: VdsFenceOptionMapping has no value for version: general<br>2017-10-10 06:53:09,032+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Exception: java.lang.IllegalArgumentException: VdsFenceOptionMapping has no value for version: general<br> at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:65) [dal.jar:]<br> at org.ovirt.engine.core.common.config.Config.getValue(Config.java:28) [common.jar:]<br> at org.ovirt.engine.core.utils.pm.FenceConfigHelper.getFenceConfigurationValue(FenceConfigHelper.java:96) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.cacheFenceAgentsOptionMapping(VdsFenceOptions.java:78) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.initCache(VdsFenceOptions.java:236) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:63) [utils.jar:]<br> at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:47) [utils.jar:]<br> at org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery.executeQueryCommand(GetAgentFenceOptionsQuery.java:18) [bll.jar:]<br> at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:]<br> at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]<br> at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor.execute(DefaultBackendQueryExecutor.java:14) [bll.jar:]<br> at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:570) [bll.jar:]<br> at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:539) [bll.jar:]<br> at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:]<br> at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.interceptor.proxy.WeldInvocationContext.proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.as.weld.ejb.DelegatingInterceptorInvocationContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport.delegateInterception(EjbComponentInterceptorSupport.java:60)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.delegateInterception(Jsr299BindingsInterceptor.java:76)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.doMethodInterception(Jsr299BindingsInterceptor.java:88)<br> at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:101)<br> at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]<br> at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:89)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:40)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)<br> at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:52)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:509)<br> at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final]<br> at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1]<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:60)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:438)<br> at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:609)<br> at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:57)<br> at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)<br> at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)<br> at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:198)<br> at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)<br> at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:81)<br> at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:]<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runQuery(GenericApiGWTServiceImpl.java:90)<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.runMultipleQueries(GenericApiGWTServiceImpl.java:121)<br> at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) [:1.8.0_144]<br> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144]<br> at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144]<br> at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:587)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:333)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:303)<br> at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:373)<br> at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]<br> at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.service(GenericApiGWTServiceImpl.java:78)<br> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]<br> at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)<br> at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132)<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:73) [branding.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:]<br> at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)<br> at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)<br> at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)<br> at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)<br> at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)<br> at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)<br> at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:53)<br> at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)<br> at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)<br> at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:59)<br> at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)<br> at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)<br> at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)<br> at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)<br> at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)<br> at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)<br> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)<br> at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)<br> at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)<br> at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)<br> at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1508)<br> at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)<br> at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)<br> at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)<br> at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)<br> at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)<br> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_144]<br> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_144]<br> at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_144]<br></font><br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 October 2017 at 19:21, Martin Perina <span dir="ltr"><<a href="mailto:mperina@redhat.com" target="_blank">mperina@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Oct 9, 2017 at 6:00 PM, Maton, Brett <span dir="ltr"><<a href="mailto:matonb@ltresources.co.uk" target="_blank">matonb@ltresources.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hope this is enough of the log, no idea why it's saying invalid fence agents:<br><br><font face="monospace,monospace">2017-10-09 16:01:01,535+01 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-184) [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,<wbr>802), VDSM <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a> command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=47d12516-a41c-41a7-9d<wbr>a4-320f08fda147, msdUUID=a28ebb26-98be-4850-8e8<wbr>8-0ac2c3d6037a'<br>2017-10-09 16:01:01,535+01 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.ConnectStoragePoo<wbr>lVDSCommand] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-184) [] HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a><br>2017-10-09 16:01:01,535+01 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.ConnectStoragePoo<wbr>lVDSCommand] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-184) [] Command 'ConnectStoragePoolVDSCommand(<wbr>HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a>, ConnectStoragePoolVDSCommandPa<wbr>rameters:{hostId='9390df92-5c7<wbr>9-4a25-9e94-1ca2d4aedc74', vdsId='9390df92-5c79-4a25-9e94<wbr>-1ca2d4aedc74', storagePoolId='47d12516-a41c-4<wbr>1a7-9da4-320f08fda147', masterVersion='2'})' execution failed: IRSGenericException: IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: u'spUUID=47d12516-a41c-41a7-9d<wbr>a4-320f08fda147, msdUUID=a28ebb26-98be-4850-8e8<wbr>8-0ac2c3d6037a'<br>2017-10-09 16:01:01,535+01 INFO [org.ovirt.engine.core.bll.Ini<wbr>tVdsOnUpCommand] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-184) [] Could not connect host '<a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a>' to pool 'GHDC', as the master domain is in inactive/unknown status - not failing the operation<br></font></div></blockquote></span><div><br><div style="font-family:arial,helvetica,sans-serif" class="gmail_default"><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">This seems like severe storage issue</div>, if your master storage domain is inactive/unknown, then your VMs cannot run. You would need to check VDSM logs to find the real cause of the issue. Are you sure there are no storage or network issues on your setup?<br></div><br> </div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font face="monospace,monospace">2017-10-09 16:01:01,565+01 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.SetMOMPolicyParam<wbr>etersVDSCommand] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-1) [594458d4] START, SetMOMPolicyParametersVDSComma<wbr>nd(HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a>, MomPolicyVDSParameters:{hostId<wbr>='9390df92-5c79-4a25-9e94-<wbr>1ca2d4aedc74'}), log id: 43532a7a<br>2017-10-09 16:01:01,595+01 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.SpmStopVDSCommand<wbr>] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-210) [6eeda756] START, SpmStopVDSCommand(HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a>, SpmStopVDSCommandParameters:{h<wbr>ostId='9390df92-5c79-4a25-9e94<wbr>-1ca2d4aedc74', storagePoolId='47d12516-a41c-4<wbr>1a7-9da4-320f08fda147'}), log id: 3861bb24<br>2017-10-09 16:01:01,602+01 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-210) [6eeda756] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,<wbr>802), VDSM <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a> command HSMGetAllTasksStatusesVDS failed: Not SPM: ()<br>2017-10-09 16:01:01,602+01 INFO [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.HSMGetAllTasksSta<wbr>tusesVDSCommand] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-210) [6eeda756] HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a><br>2017-10-09 16:01:01,602+01 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.HSMGetAllTasksSta<wbr>tusesVDSCommand] (org.ovirt.thread.EE-ManagedTh<wbr>readFactory-engine-Thread-210) [6eeda756] Command 'HSMGetAllTasksStatusesVDSComm<wbr>and(HostName = <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a>, VdsIdVDSCommandParametersBase:<wbr>{hostId='9390df92-5c79-4a25-9e<wbr>94-1ca2d4aedc74'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: ()<br>2017-10-09 16:01:01,653+01 INFO [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-1) [2e03c9cf] EVENT_ID: VDS_DETECTED(13), Status of host <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a> was set to Up.<br>2017-10-09 16:01:01,656+01 WARN [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-1) [2e03c9cf] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,<wbr>001), Power Management test failed for Host ov_host.example.com.Invalid fence agents defined for host '192.168.1.12'.<br>2017-10-09 16:01:01,659+01 ERROR [org.ovirt.engine.core.dal.dbb<wbr>roker.auditloghandling.AuditLo<wbr>gDirector] (EE-ManagedThreadFactory-engin<wbr>eScheduled-Thread-1) [2e03c9cf] EVENT_ID: VDS_FENCE_STATUS_FAILED(497), Failed to verify Host <a href="http://ov_host.example.com" target="_blank">ov_host.example.com</a> power management.</font></div></blockquote></span><div><br><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">We are trying to get power management status of the host, when activating the host. In that case there was an error trying when getting power management status, more details should be in VDSM logs of the host that acted as fence proxy.<br></div> </div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 October 2017 at 14:41, Maton, Brett <span dir="ltr"><<a href="mailto:matonb@ltresources.co.uk" target="_blank">matonb@ltresources.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>Hi Martin,<br><br></div> Sure is there a way to trigger the check (I cleared the alerts), or do I just need to have a poke around the engine log next time it happens?<br></div></div></div></blockquote></div></div></blockquote></span><div><br><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">Yes, please take a look at engine-config option PMHealthCheckIntervalInSec:<br><br></div><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default"> engine-config -l | grep</div> <div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">PMHealthCheckIntervalInSec<br><br></div><div style="font-family:arial,helvetica,sans-serif;display:inline" class="gmail_default">After updating this value you need to restart ovirt-engine service.<br></div></div><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><br></div>Regards,<br></div>Brett<br></div><div class="m_-2415001188524693181gmail-m_-529701472995525469HOEnZb"><div class="m_-2415001188524693181gmail-m_-529701472995525469h5"><div class="gmail_extra"><br><div class="gmail_quote">On 9 October 2017 at 14:26, Martin Perina <span dir="ltr"><<a href="mailto:mperina@redhat.com" target="_blank">mperina@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, Oct 9, 2017 at 2:40 PM, Maton, Brett <span dir="ltr"><<a href="mailto:matonb@ltresources.co.uk" target="_blank">matonb@ltresources.co.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Hi,<br><br></div> I recently upgraded my testlab to oVirt 4.2-pre and have been having some issues.<br><br></div> One of which is fencing with Dell Drac 8 remote access cards.<br></div> The configuration I'm using works fine on my other (4.1.6) cluster...<br><br></div> Error:<br><span style="font-family:monospace,monospace"><span class="m_-2415001188524693181gmail-m_-529701472995525469m_-462825152066870607m_-8110768790926482745m_-3397316486640587687m_-9187427046456573381gmail-drawer-pf-notification-message">Health check on Host <a href="http://host.example.com" target="_blank">host.example.com</a> indicates that future attempts to Start this
host using Power-Management are expected to fail.</span></span></div></div></blockquote></span><div><br><div>Hi, </div><br><div>Could you please share engine.log around above message? We are executing get power management status call to check, so if this get PM status failed, the above error is raised.<br><br></div><div>Thanks<br><br></div><div>Martin<br></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div dir="ltr"><div><br><br></div> Fence agent setup for the host appears to be working though:<br><br> <img src="cid:ii_j8k5zvzu0_15f0125c52b42821" width="510" height="508"><br><br><div><br></div><div> Which logs would be helpful in debugging this issue ?</div><div><br></div><div>Regards,</div><div>Brett<br></div></div>
<br></span>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br>
<br></blockquote></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div>