Re: [ovirt-users] Ovirt 4.2 Fencing problems

Hi Martin, Storage isn't an issue, that was me trying to force fix another problem (removing 'dead' hosted engine hosts) This just popped up in engine.log: 2017-10-10 06:53:09,022+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Exception: java.lang.IllegalArgumentException: VdsFenceOptionMapping has no value for version: general at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:65) [dal.jar:] at org.ovirt.engine.core.common.config.Config.getValue(Config.java:28) [common.jar:] at org.ovirt.engine.core.utils.pm.FenceConfigHelper. getFenceConfigurationValue(FenceConfigHelper.java:96) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions. cacheFenceAgentsOptionMapping(VdsFenceOptions.java:78) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.initCache(VdsFenceOptions.java:236) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:63) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:47) [utils.jar:] at org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery. executeQueryCommand(GetAgentFenceOptionsQuery.java:18) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor. execute(DefaultBackendQueryExecutor.java:14) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:570) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:539) [bll.jar:] at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.as.ee.component.ManagedReferenceMethodIntercep tor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocatio nContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final. jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor. LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$ SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final. jar:2.4.3.Final] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocatio nContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport .delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. delegateInterception(Jsr299BindingsInterceptor.java:76) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1. processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors. CorrelationIdTrackerInterceptor.aroundInvoke( CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.as.ee.component.ManagedReferenceLifecycleMetho dInterceptor.processInvocation(ManagedReferenceLifecycleMetho dInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics. ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor. processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation( InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors. ComponentDispatcherInterceptor.processInvocation( ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton. SingletonComponentInstanceAssociationInterceptor.processInvocation( SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat ionInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter ceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. CurrentInvocationContextInterceptor.processInvocation( CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.CR1.jar: 11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor. processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor. processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor. processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor. processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor. processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor. processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation( AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke( ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation( ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke( ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. runQuery(GenericApiGWTServiceImpl.java:90) at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. runMultipleQueries(GenericApiGWTServiceImpl.java:121) at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC. java:587) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall( RemoteServiceServlet.java:333) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall( RemoteServiceServlet.java:303) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost( RemoteServiceServlet.java:373) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost( AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. service(GenericApiGWTServiceImpl.java:78) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest( ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter. doFilter(GwtCachingFilter.java:132) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.branding.BrandingFilter. doFilter(BrandingFilter.java:73) [branding.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest( FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler. handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler. handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security. SecurityContextAssociationHandler.handleRequest( SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.security. SSLInformationAssociationHandler.handleRequest( SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security. ServletAuthenticationCallHandler.handleRequest( ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandle r.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler .handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security. ServletConfidentialityConstraintHandler.handleRequest( ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security. ServletSecurityConstraintHandler.handleRequest( ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandle r.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security. CachedAuthenticatedSessionHandler.handleRequest( CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler. handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssocia tionHandler.handleRequest(AbstractSecurityContextAssocia tionHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler. handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler. handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSet upAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call( ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security. SecurityContextThreadSetupAction.lambda$create$0( SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest( ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors. java:326) at io.undertow.server.HttpServerExchange$1.run( HttpServerExchange.java:812) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_144] 2017-10-10 06:53:09,032+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Query 'GetAgentFenceOptionsQuery' failed: VdsFenceOptionMapping has no value for version: general 2017-10-10 06:53:09,032+01 ERROR [org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery] (default task-62) [3bfc7ef1-bede-40b4-8a97-a8cd1cf2a0f6] Exception: java.lang.IllegalArgumentException: VdsFenceOptionMapping has no value for version: general at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:65) [dal.jar:] at org.ovirt.engine.core.common.config.Config.getValue(Config.java:28) [common.jar:] at org.ovirt.engine.core.utils.pm.FenceConfigHelper. getFenceConfigurationValue(FenceConfigHelper.java:96) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions. cacheFenceAgentsOptionMapping(VdsFenceOptions.java:78) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.initCache(VdsFenceOptions.java:236) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:63) [utils.jar:] at org.ovirt.engine.core.utils.pm.VdsFenceOptions.<init>(VdsFenceOptions.java:47) [utils.jar:] at org.ovirt.engine.core.bll.GetAgentFenceOptionsQuery. executeQueryCommand(GetAgentFenceOptionsQuery.java:18) [bll.jar:] at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCommandBase.java:106) [bll.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendQueryExecutor. execute(DefaultBackendQueryExecutor.java:14) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:570) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:539) [bll.jar:] at sun.reflect.GeneratedMethodAccessor154.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.as.ee.component.ManagedReferenceMethodIntercep tor.processInvocation(ManagedReferenceMethodInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.as.weld.ejb.DelegatingInterceptorInvocatio nContext.proceed(DelegatingInterceptorInvocationContext.java:92) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. interceptorChainCompleted(WeldInvocationContext.java:98) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. proceed(WeldInvocationContext.java:117) [weld-core-impl-2.4.3.Final. jar:2.4.3.Final] at org.ovirt.engine.core.common.di.interceptor. LoggingInterceptor.apply(LoggingInterceptor.java:12) [common.jar:] at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$ SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext.invokeNext(WeldInvocationContext.java:83) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.weld.interceptor.proxy.WeldInvocationContext. proceed(WeldInvocationContext.java:115) [weld-core-impl-2.4.3.Final. jar:2.4.3.Final] at org.jboss.weld.bean.InterceptorImpl.intercept(InterceptorImpl.java:108) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.DelegatingInterceptorInvocatio nContext.proceed(DelegatingInterceptorInvocationContext.java:82) [wildfly-weld-ejb-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.weld.interceptors.EjbComponentInterceptorSupport .delegateInterception(EjbComponentInterceptorSupport.java:60) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. delegateInterception(Jsr299BindingsInterceptor.java:76) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. doMethodInterception(Jsr299BindingsInterceptor.java:88) at org.jboss.as.weld.interceptors.Jsr299BindingsInterceptor. processInvocation(Jsr299BindingsInterceptor.java:101) at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1. processInvocation(UserInterceptorFactory.java:63) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.ovirt.engine.core.bll.interceptors. CorrelationIdTrackerInterceptor.aroundInvoke( CorrelationIdTrackerInterceptor.java:13) [bll.jar:] at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at org.jboss.as.ee.component.ManagedReferenceLifecycleMetho dInterceptor.processInvocation(ManagedReferenceLifecycleMetho dInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics. ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor. processInvocation(ConcurrentContextInterceptor.java:45) [wildfly-ee-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InitialInterceptor.processInvocation( InitialInterceptor.java:40) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.interceptors. ComponentDispatcherInterceptor.processInvocation( ComponentDispatcherInterceptor.java:52) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.singleton. SingletonComponentInstanceAssociationInterceptor.processInvocation( SingletonComponentInstanceAssociationInterceptor.java:53) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:264) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:379) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:244) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext$Invocation. proceed(InterceptorContext.java:509) at org.jboss.weld.ejb.AbstractEJBRequestScopeActivat ionInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73) [weld-core-impl-2.4.3.Final.jar:2.4.3.Final] at org.jboss.as.weld.ejb.EjbRequestScopeActivationInter ceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:89) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. CurrentInvocationContextInterceptor.processInvocation( CurrentInvocationContextInterceptor.java:41) [wildfly-ejb3-11.0.0.CR1.jar: 11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor. processInvocation(WaitTimeInterceptor.java:47) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.security.SecurityContextInterceptor. processInvocation(SecurityContextInterceptor.java:100) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor. processInvocation(StartupAwaitInterceptor.java:22) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors. ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor. processInvocation(LoggingInterceptor.java:67) [wildfly-ejb3-11.0.0.CR1.jar:11.0.0.CR1] at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.as.ee.component.NamespaceContextInterceptor. processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ContextClassLoaderInterceptor. processInvocation(ContextClassLoaderInterceptor.java:60) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.InterceptorContext.run( InterceptorContext.java:438) at org.wildfly.security.manager.WildFlySecurityManager.doChecked( WildFlySecurityManager.java:609) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation( AccessCheckingInterceptor.java:57) at org.jboss.invocation.InterceptorContext.proceed( InterceptorContext.java:422) at org.jboss.invocation.ChainedInterceptor.processInvocation( ChainedInterceptor.java:53) at org.jboss.as.ee.component.ViewService$View.invoke( ViewService.java:198) at org.jboss.as.ee.component.ViewDescription$1.processInvocation( ViewDescription.java:185) at org.jboss.as.ee.component.ProxyInvocationHandler.invoke( ProxyInvocationHandler.java:81) at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view3.runQuery(Unknown Source) [common.jar:] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. runQuery(GenericApiGWTServiceImpl.java:90) at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. runMultipleQueries(GenericApiGWTServiceImpl.java:121) at sun.reflect.GeneratedMethodAccessor161.invoke(Unknown Source) [:1.8.0_144] at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_144] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_144] at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC. java:587) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall( RemoteServiceServlet.java:333) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall( RemoteServiceServlet.java:303) at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost( RemoteServiceServlet.java:373) at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost( AbstractRemoteServiceServlet.java:62) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl. service(GenericApiGWTServiceImpl.java:78) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final] at io.undertow.servlet.handlers.ServletHandler.handleRequest( ServletHandler.java:85) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:129) at org.ovirt.engine.core.utils.servlet.HeaderFilter.doFilter(HeaderFilter.java:94) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter. doFilter(GwtCachingFilter.java:132) at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.branding.BrandingFilter. doFilter(BrandingFilter.java:73) [branding.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:66) [utils.jar:] at io.undertow.servlet.core.ManagedFilter.doFilter( ManagedFilter.java:61) at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl. doFilter(FilterHandler.java:131) at io.undertow.servlet.handlers.FilterHandler.handleRequest( FilterHandler.java:84) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler. handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet.handlers.ServletDispatchingHandler. handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security. SecurityContextAssociationHandler.handleRequest( SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.security. SSLInformationAssociationHandler.handleRequest( SSLInformationAssociationHandler.java:131) at io.undertow.servlet.handlers.security. ServletAuthenticationCallHandler.handleRequest( ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.security.handlers.AuthenticationConstraintHandle r.handleRequest(AuthenticationConstraintHandler.java:53) at io.undertow.security.handlers.AbstractConfidentialityHandler .handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security. ServletConfidentialityConstraintHandler.handleRequest( ServletConfidentialityConstraintHandler.java:64) at io.undertow.servlet.handlers.security. ServletSecurityConstraintHandler.handleRequest( ServletSecurityConstraintHandler.java:59) at io.undertow.security.handlers.AuthenticationMechanismsHandle r.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security. CachedAuthenticatedSessionHandler.handleRequest( CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler. handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssocia tionHandler.handleRequest(AbstractSecurityContextAssocia tionHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler. handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest( PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler. handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 100(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:138) at io.undertow.servlet.handlers.ServletInitialHandler$2.call( ServletInitialHandler.java:135) at io.undertow.servlet.core.ServletRequestContextThreadSet upAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call( ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow.security. SecurityContextThreadSetupAction.lambda$create$0( SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at org.wildfly.extension.undertow.deployment. UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0( UndertowDeploymentInfoService.java:1508) at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest( ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$ 000(ServletInitialHandler.java:81) at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest( ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors. java:326) at io.undertow.server.HttpServerExchange$1.run( HttpServerExchange.java:812) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_144] at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_144] On 9 October 2017 at 19:21, Martin Perina <mperina(a)redhat.com&gt; wrote: > > > On Mon, Oct 9, 2017 at 6:00 PM, Maton, Brett <matonb(a)ltresources.co.uk&gt; > wrote: > >> Hope this is enough of the log, no idea why it's saying invalid fence >> agents: >> >> 2017-10-09 16:01:01,535+01 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-184) >> [] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM >> ov_host.example.com command ConnectStoragePoolVDS failed: Cannot find >> master domain: u'spUUID=47d12516-a41c-41a7-9da4-320f08fda147, >> msdUUID=a28ebb26-98be-4850-8e88-0ac2c3d6037a' >> 2017-10-09 16:01:01,535+01 INFO [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.ConnectStoragePoolVDSCommand] >> (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-184) [] >> HostName = ov_host.example.com >> 2017-10-09 16:01:01,535+01 ERROR [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.ConnectStoragePoolVDSCommand] >> (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-184) [] Command >> 'ConnectStoragePoolVDSCommand(HostName = ov_host.example.com, >> ConnectStoragePoolVDSCommandParameters:{hostId='9390df92-5c79-4a25-9e94-1ca2d4aedc74', >> vdsId='9390df92-5c79-4a25-9e94-1ca2d4aedc74', >> storagePoolId='47d12516-a41c-41a7-9da4-320f08fda147', >> masterVersion='2'})' execution failed: IRSGenericException: >> IRSErrorException: IRSNoMasterDomainException: Cannot find master domain: >> u'spUUID=47d12516-a41c-41a7-9da4-320f08fda147, >> msdUUID=a28ebb26-98be-4850-8e88-0ac2c3d6037a' >> 2017-10-09 16:01:01,535+01 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] >> (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-184) [] Could >> not connect host 'ov_host.example.com' to pool 'GHDC', as the master >> domain is in inactive/unknown status - not failing the operation >> > > ​ > ​This seems like severe storage issue​ > , if your master storage domain is inactive/unknown, then your VMs cannot > ​run. You would need to check VDSM logs to find the real cause of the > issue. Are you sure there are no storage or network issues on your setup? > > > >> 2017-10-09 16:01:01,565+01 INFO [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.SetMOMPolicyParametersVDSCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-1) [594458d4] START, >> SetMOMPolicyParametersVDSCommand(HostName = ov_host.example.com, >> MomPolicyVDSParameters:{hostId='9390df92-5c79-4a25-9e94-1ca2d4aedc74'}), >> log id: 43532a7a >> 2017-10-09 16:01:01,595+01 INFO [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.SpmStopVDSCommand] (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-210) >> [6eeda756] START, SpmStopVDSCommand(HostName = ov_host.example.com, >> SpmStopVDSCommandParameters:{hostId='9390df92-5c79-4a25-9e94-1ca2d4aedc74', >> storagePoolId='47d12516-a41c-41a7-9da4-320f08fda147'}), log id: 3861bb24 >> 2017-10-09 16:01:01,602+01 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-210) >> [6eeda756] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM >> ov_host.example.com command HSMGetAllTasksStatusesVDS failed: Not SPM: >> () >> 2017-10-09 16:01:01,602+01 INFO [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >> (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-210) [6eeda756] >> HostName = ov_host.example.com >> 2017-10-09 16:01:01,602+01 ERROR [org.ovirt.engine.core.vdsbrok >> er.vdsbroker.HSMGetAllTasksStatusesVDSCommand] >> (org.ovirt.thread.EE-ManagedThreadFactory-engine-Thread-210) [6eeda756] >> Command 'HSMGetAllTasksStatusesVDSCommand(HostName = ov_host.example.com, >> VdsIdVDSCommandParametersBase:{hostId='9390df92-5c79-4a25-9e94-1ca2d4aedc74'})' >> execution failed: IRSGenericException: IRSErrorException: >> IRSNonOperationalException: Not SPM: () >> 2017-10-09 16:01:01,653+01 INFO [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-1) >> [2e03c9cf] EVENT_ID: VDS_DETECTED(13), Status of host >> ov_host.example.com was set to Up. >> 2017-10-09 16:01:01,656+01 WARN [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-1) >> [2e03c9cf] EVENT_ID: VDS_ALERT_FENCE_TEST_FAILED(9,001), Power >> Management test failed for Host ov_host.example.com.Invalid fence agents >> defined for host '192.168.1.12'. >> 2017-10-09 16:01:01,659+01 ERROR [org.ovirt.engine.core.dal.dbb >> roker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engineScheduled-Thread-1) >> [2e03c9cf] EVENT_ID: VDS_FENCE_STATUS_FAILED(497), Failed to verify Host >> ov_host.example.com power management. >> > > ​We are trying to get power management status of the host, when > activating the host. In that case there was an error trying when getting > power management status, more details should be in VDSM logs of the host > that acted as fence proxy. > ​ > > >> >> >> >> >> On 9 October 2017 at 14:41, Maton, Brett <matonb(a)ltresources.co.uk&gt; >> wrote: >> >>> Hi Martin, >>> >>> Sure is there a way to trigger the check (I cleared the alerts), or >>> do I just need to have a poke around the engine log next time it happens? >>> >> > ​Yes, please take a look at engine-config option > PMHealthCheckIntervalInSec: > > engine-config -l | grep​ > > ​PMHealthCheckIntervalInSec > > After updating this value you need to restart ovirt-engine service. > ​ > >> >>> Regards, >>> Brett >>> >>> On 9 October 2017 at 14:26, Martin Perina <mperina(a)redhat.com&gt; wrote: >>> >>>> >>>> >>>> On Mon, Oct 9, 2017 at 2:40 PM, Maton, Brett <matonb(a)ltresources.co.uk >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> I recently upgraded my testlab to oVirt 4.2-pre and have been >>>>> having some issues. >>>>> >>>>> One of which is fencing with Dell Drac 8 remote access cards. >>>>> The configuration I'm using works fine on my other (4.1.6) >>>>> cluster... >>>>> >>>>> Error: >>>>> Health check on Host host.example.com indicates that future attempts >>>>> to Start this host using Power-Management are expected to fail. >>>>> >>>> >>>> ​Hi, ​ >>>> >>>> ​Could you please share engine.log around above message? We are >>>> executing get power management status call to check, so if this get PM >>>> status failed, the above error is raised. >>>> >>>> Thanks >>>> >>>> Martin >>>> ​ >>>> >>>> >>>>> >>>>> >>>>> Fence agent setup for the host appears to be working though: >>>>> >>>>> >>>>> ​ >>>>> >>>>> Which logs would be helpful in debugging this issue ? >>>>> >>>>> Regards, >>>>> Brett >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>> >> >