10:34 p.m.
Hi,
OST is constantly failing today in test_verify_add_hosts with
19:34:33 E assert find_up_host() -> False != True after 1800 seconds
19:34:33 E + where find_up_host() -> False != True after 1800 seconds =
<function true_within at 0x7fa62ac860d0>(<function
test_verify_add_hosts.<locals>.find_up_host at 0x7fa628f97940>,
timeout=1800)
19:34:33 E + where <function true_within at 0x7fa62ac860d0> =
assert_utils.true_within
19:34:33 E + and 1800 = constants.ADD_HOST_TIMEOUT
See e.g. https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/23661
Any idea what's wrong?
Thanks
Vojta
9:55 p.m.
Vojtech Juranek <vjuranek(a)redhat.com> writes:
Hi,
OST is constantly failing today in test_verify_add_hosts with
19:34:33 E assert find_up_host() -> False != True after 1800 seconds
19:34:33 E + where find_up_host() -> False != True after 1800
seconds = <function true_within at 0x7fa62ac860d0>(<function
test_verify_add_hosts.<locals>.find_up_host at
0x7fa628f97940>, timeout=1800)
19:34:33 E + where <function true_within at 0x7fa62ac860d0> =
assert_utils.true_within
19:34:33 E + and 1800 = constants.ADD_HOST_TIMEOUT
See e.g. https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/23661
Any idea what's wrong?
No. From another failed run:
The host installation succeeded:
2021-12-16 21:23:45,073+01 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID:
ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host ost-basic-suite-master-host-0.
restart libvirtd.
...
2021-12-16 21:23:48,105+01 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1) [5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID:
VDS_ANSIBLE_INSTALL_FINISHED(561), Ansible host-deploy playbook execution has successfully
finished on host ost-basic-suite-master-host-0.
And then start repeating tracebacks in engine.log, e.g.:
2021-12-16 21:52:48,301+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(SSL Stomp Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connecting to
ost-basic-suite-master-host-0/192.168.206.3
2021-12-16 21:52:48,301+01 INFO [org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient]
(SSL Stomp Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connected to
ost-basic-suite-master-host-0/192.168.206.3:54321
2021-12-16 21:52:48,301+01 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Unable to
RefreshCapabilities: ConnectException: Connection refused
2021-12-16 21:52:48,301+01 DEBUG
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80) [] Exception:
org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection issues during send
request
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:317)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand.executeVdsBrokerCommand(GetCapabilitiesAsyncVDSCommand.java:30)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
at
org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:409)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown
Source)
at jdk.internal.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
at
org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
at jdk.internal.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown
Source)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:782)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshCapabilities(HostMonitoring.java:187)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:146)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:103)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshImpl(VdsManager.java:302)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refresh(VdsManager.java:273)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227)
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:146)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:137)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:105)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:315)
... 38 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:120)
... 41 more
On the host side, Vdsm is restarted
2021-12-16 21:23:43,151+0100 INFO (MainThread) [vds] Received signal 15, shutting down
(vdsmd:66)
2021-12-16 21:23:54,048+0100 INFO (MainThread) [vds] (PID: 34647) I am the actual vdsm
4.50.0.3.32.git467747f7a ost-basic-suite-master-host-0 (4.18.0-348.2.1.el8_5.x86_64)
(vdsmd:153)
and then it looks like normally running.
1:37 p.m.
Milan Zamazal <mzamazal(a)redhat.com> writes:
Vojtech Juranek <vjuranek(a)redhat.com> writes:
> Hi,
> OST is constantly failing today in test_verify_add_hosts with
>
> 19:34:33 E assert find_up_host() -> False != True after 1800 seconds
> 19:34:33 E + where find_up_host() -> False != True after 1800
> seconds = <function true_within at 0x7fa62ac860d0>(<function
> test_verify_add_hosts.<locals>.find_up_host at
> 0x7fa628f97940>, timeout=1800)
> 19:34:33 E + where <function true_within at 0x7fa62ac860d0> =
assert_utils.true_within
> 19:34:33 E + and 1800 = constants.ADD_HOST_TIMEOUT
>
> See e.g. https://redir.apps.ovirt.org/dj/job/ds-ost-baremetal_manual/23661
>
> Any idea what's wrong?
No. From another failed run:
The host installation succeeded:
2021-12-16 21:23:45,073+01 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1)
[5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID:
ANSIBLE_RUNNER_EVENT_NOTIFICATION(559), Installing Host
ost-basic-suite-master-host-0. restart libvirtd.
...
2021-12-16 21:23:48,105+01 INFO
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(EE-ManagedThreadFactory-engine-Thread-1)
[5b20f413-4877-49c3-873a-ef0846b14cca] EVENT_ID:
VDS_ANSIBLE_INSTALL_FINISHED(561), Ansible host-deploy playbook
execution has successfully finished on host
ost-basic-suite-master-host-0.
And then start repeating tracebacks in engine.log, e.g.:
2021-12-16 21:52:48,301+01 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connecting to
ost-basic-suite-master-host-0/192.168.206.3
2021-12-16 21:52:48,301+01 INFO
[org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient] (SSL Stomp
Reactor) [5b20f413-4877-49c3-873a-ef0846b14cca] Connected to
ost-basic-suite-master-host-0/192.168.206.3:54321
2021-12-16 21:52:48,301+01 ERROR
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80)
[] Unable to RefreshCapabilities: ConnectException: Connection refused
2021-12-16 21:52:48,301+01 DEBUG
I also get IPv6 network errors when running locally. When I run it with
IPv4 only then it works.
[org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring]
(EE-ManagedScheduledExecutorService-engineScheduledThreadPool-Thread-80)
[] Exception:
org.ovirt.engine.core.vdsbroker.TransportRunTimeException: Connection
issues during send request
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:317)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesAsyncVDSCommand.executeVdsBrokerCommand(GetCapabilitiesAsyncVDSCommand.java:30)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVdsCommandWithNetworkEvent(VdsBrokerCommand.java:123)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:65)
at
org.ovirt.engine.core.dal//org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:409)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand$$super(Unknown
Source)
at jdk.internal.reflect.GeneratedMethodAccessor238.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78)
at
org.ovirt.engine.core.common//org.ovirt.engine.core.common.di.interceptor.LoggingInterceptor.apply(LoggingInterceptor.java:12)
at jdk.internal.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.reader.SimpleInterceptorInvocation$SimpleMethodInvocation.invoke(SimpleInterceptorInvocation.java:73)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeAroundInvoke(InterceptorMethodHandler.java:84)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.executeInterception(InterceptorMethodHandler.java:72)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.interceptor.proxy.InterceptorMethodHandler.invoke(InterceptorMethodHandler.java:56)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:79)
at
org.jboss.weld.core@3.1.7.SP1//org.jboss.weld.bean.proxy.CombinedInterceptorAndDecoratorStackMethodHandler.invoke(CombinedInterceptorAndDecoratorStackMethodHandler.java:68)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.ResourceManager$Proxy$_$$_WeldSubclass.runVdsCommand(Unknown
Source)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:782)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshCapabilities(HostMonitoring.java:187)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refreshVdsRunTimeInfo(HostMonitoring.java:146)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.monitoring.HostMonitoring.refresh(HostMonitoring.java:103)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refreshImpl(VdsManager.java:302)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.VdsManager.refresh(VdsManager.java:273)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.access$201(ManagedScheduledThreadPoolExecutor.java:360)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedScheduledThreadPoolExecutor$ManagedScheduledFutureTask.run(ManagedScheduledThreadPoolExecutor.java:511)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
at
org.glassfish.javax.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:227)
Caused by: org.ovirt.vdsm.jsonrpc.client.ClientConnectionException: Connection failed
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:146)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.getClient(JsonRpcClient.java:137)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.JsonRpcClient.call(JsonRpcClient.java:105)
at
deployment.engine.ear//org.ovirt.engine.core.vdsbroker.jsonrpc.JsonRpcVdsServer.getCapabilities(JsonRpcVdsServer.java:315)
... 38 more
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
at
org.ovirt.vdsm-jsonrpc-java//org.ovirt.vdsm.jsonrpc.client.reactors.ReactorClient.connect(ReactorClient.java:120)
... 41 more
On the host side, Vdsm is restarted
2021-12-16 21:23:43,151+0100 INFO (MainThread) [vds] Received signal 15, shutting down
(vdsmd:66)
2021-12-16 21:23:54,048+0100 INFO (MainThread) [vds] (PID: 34647) I am
the actual vdsm 4.50.0.3.32.git467747f7a ost-basic-suite-master-host-0
(4.18.0-348.2.1.el8_5.x86_64) (vdsmd:153)
and then it looks like normally running.
1067
days inactive
1071
days old
2 comments
2 participants
participants (2)
-
Milan Zamazal -
Vojtech Juranek