This is a multi-part message in MIME format... ------------=_1511682298-27447-558 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1781?page=com.atlassian.jir... ] Barak Korren commented on OVIRT-1781: ------------------------------------- [~danken(a)redhat.com] this error has nothing to do with disk space, you can see in the beginning on the log, in the cleanup script that there are 27G available. This is talking about the `workspace` object which is the object that Jenkins uses to communicate with the slave. If the workspace object is lost it typically means that Jenkins lost the connection to the slave, here is the actual relevant exception, where we can see that is indeed a `{{java.io.EOFException}}` (Meaning - an EOF on the slave connection): {code} storage/guarded_test.py::TestContext::test_release_failure PASSED storage/guarded_test.py::TestContext::test_two_vols_different_domains PASSED storage/guarded_test.py::TestContext::test_two_vols_same_image PASSED storage/hsm_test.py::TestVerifyUntrustedVolume::test_compat_not_checked_for_raw FATAL: command execution failed java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:898) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy80.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496) at hudson.model.Run.execute(Run.java:1737) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421) Build step 'Execute shell' marked build as failure FATAL: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73) Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:898) at hudson.FilePath.act(FilePath.java:987) Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org at hudson.FilePath.act(FilePath.java:994) at hudson.FilePath.act(FilePath.java:976) at hudson.FilePath.deleteRecursive(FilePath.java:1178) at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84) at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113) at hudson.model.Build$BuildExecution.doRun(Build.java:174) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496) at hudson.model.Run.execute(Run.java:1737) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421) {code} Could something with the vdsm tests that ran on the slave cause it to disconnect? -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100073) ------------=_1511682298-27447-558 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1781?page=com.atlassian.jir... ]</pre> <h3>Barak Korren commented on OVIRT-1781:</h3> <p&gt;[~danken(a)redhat.com] this error has nothing to do with disk space, you can see in the beginning on the log, in the cleanup script that there are 27G available.</p> <p>This is talking about the `workspace` object which is the object that Jenkins uses to communicate with the slave.</p> <p>If the workspace object is lost it typically means that Jenkins lost the connection to the slave, here is the actual relevant exception, where we can see that is indeed a `{{java.io.EOFException}}` (Meaning &ndash; an EOF on the slave connection):</p> <p>{code} storage/guarded_test.py::TestContext::test_release_failure PASSED storage/guarded_test.py::TestContext::test_two_vols_different_domains PASSED storage/guarded_test.py::TestContext::test_two_vols_same_image PASSED storage/hsm_test.py::TestVerifyUntrustedVolume::test_compat_not_checked_for_raw FATAL: command execution failed java.io.EOFException</p> <pre>at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864) at java.io.ObjectInputStream.&lt;init&gt;(ObjectInputStream.java:360) at hudson.remoting.ObjectInputStreamEx.&lt;init&gt;(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)</pre> <p>Caused: java.io.IOException: Unexpected termination of the channel</p> <pre>at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)</pre> <p>Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down</p> <pre>at hudson.remoting.Channel.call(Channel.java:898) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257) at com.sun.proxy.$Proxy80.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496) at hudson.model.Run.execute(Run.java:1737) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421)</pre> <p>Build step &lsquo;Execute shell&rsquo; marked build as failure FATAL: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down java.io.EOFException</p> <pre>at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864) at java.io.ObjectInputStream.&lt;init&gt;(ObjectInputStream.java:360) at hudson.remoting.ObjectInputStreamEx.&lt;init&gt;(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)</pre> <p>Caused: java.io.IOException: Unexpected termination of the channel</p> <pre>at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)</pre> <p>Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down</p> <pre>at hudson.remoting.Channel.call(Channel.java:898) at hudson.FilePath.act(FilePath.java:987)</pre> <p>Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org</p> <pre>at hudson.FilePath.act(FilePath.java:994) at hudson.FilePath.act(FilePath.java:976) at hudson.FilePath.deleteRecursive(FilePath.java:1178) at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84) at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113) at hudson.model.Build$BuildExecution.doRun(Build.java:174) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496) at hudson.model.Run.execute(Run.java:1737) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:421)</pre> <p>{code}</p> <p>Could something with the vdsm tests that ran on the slave cause it to disconnect?</p> <blockquote><h3>ci failure due to no space on slave?</h3> <pre> Key: OVIRT-1781 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1781 Project: oVirt - virtualization made easy Issue Type: Outage Components: oVirt CI Reporter: danken Assignee: infra Priority: High Labels: vdsm</pre> <p>Exception when executing the batch command : no workspace from node hudson.slaves.DumbSlave[vm0014.workers-phx.ovirt.org] which is computer hudson.slaves.SlaveComputer@695d3d3c and has channel null <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_6... <p>&mdash; This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100073)</p> <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpS... alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1511682298-27447-558--