[ovirt-devel] Re: OST failing

Vojtech Juranek <vjuranek(a)redhat.com&gt; writes: > Hi, > OST is constantly failing today in test_verify_add_hosts with > > 19:34:33 E assert find_up_host() -&gt; False != True after 1800 seconds > 19:34:33 E + where find_up_host() -&gt; False != True after 1800 > seconds = &lt;function true_within at 0x7fa62ac860d0&gt;(&lt;function > test_verify_add_hosts.&lt;locals&gt;.find_up_host at > 0x7fa628f97940&gt;, timeout=1800) > 19:34:33 E + where &lt;function true_within at 0x7fa62ac860d0&gt; = assert_utils.true_within > 19:34:33 E + and 1800 = constants.ADD_HOST_TIMEOUT > > See e.g. https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/23661 > > Any idea what's wrong? No. From another failed run: The host installation succeeded: 2021-12-16 21:23:45,073+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID: ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host ost-basic-suite-master-host-0. restart libvirtd. ... 2021-12-16 21:23:48,105+01 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-1) [5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID: VDS_ANSIBLE_INSTALL_FINISHED(561), Ansible host-deploy playbook execution has successfully finished on host ost-basic-suite-master-host-0. And then start repeating tracebacks in engine.log, e.g.: 2021-12-16 21:52:48,301+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connecting to ost-basic-suite-master-host-0/192.168.206.3 2021-12-16 21:52:48,301+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connected to ost-basic-suite-master-host-0/192.168.206.3:54321 2021-12-16 21:52:48,301+01 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Unable to RefreshCapabilities: ConnectException: Connection refused 2021-12-16 21:52:48,301+01 DEBUG

[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring] (EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Exception: org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection issues during send request at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:317) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand.executeVdsBrokerCommand(GetCapabilitiesAsyncVDSCommand.java:30) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65) at org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:409) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor238.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) at org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12) at jdk.internal.reflect.GeneratedMethodAccessor55.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79) at org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown Source) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:782) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshCapabilities(HostMonitoring.java:187) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:146) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:103) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshImpl(VdsManager.java:302) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refresh(VdsManager.java:273) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227) Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed at org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:146) at org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:137) at org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:105) at deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:315) ... 38 more Caused by: java.net.ConnectException: Connection refused at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777) at org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:120) ... 41 more On the host side, Vdsm is restarted 2021-12-16 21:23:43,151+0100 INFO (MainThread) [vds] Received signal 15, shutting down (vdsmd:66) 2021-12-16 21:23:54,048+0100 INFO (MainThread) [vds] (PID: 34647) I am the actual vdsm 4.50.0.3.32.git467747f7a ost-basic-suite-master-host-0 (4.18.0-348.2.1.el8_5.x86_64) (vdsmd:153) and then it looks like normally running.