[JIRA] (OVIRT-1781) ci failure due to no space on slave?

26 Nov 2017

      This is a multi-part message in MIME format...

------------=_1511682298-27447-558
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

    [ https://ovirt-jira.atlassian.net/browse/OVIRT-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=35391#comment-35391 ] 

Barak Korren commented on OVIRT-1781:
-------------------------------------

[~danken@redhat.com] this error has nothing to do with disk space, you can see in the beginning on the log, in the cleanup script that there are 27G available.

This is talking about the `workspace` object which is the object that Jenkins uses to communicate with the slave.

If the workspace object is lost it typically means that Jenkins lost the connection to the slave, here is the actual relevant exception, where we can see that is indeed a `{{java.io.EOFException}}` (Meaning - an EOF on the slave connection):

{code}
storage/guarded_test.py::TestContext::test_release_failure PASSED
storage/guarded_test.py::TestContext::test_two_vols_different_domains PASSED
storage/guarded_test.py::TestContext::test_two_vols_same_image PASSED
storage/hsm_test.py::TestVerifyUntrustedVolume::test_compat_not_checked_for_raw FATAL: command execution failed
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:898)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy80.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
	at hudson.model.Run.execute(Run.java:1737)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)
Build step 'Execute shell' marked build as failure
FATAL: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:898)
	at hudson.FilePath.act(FilePath.java:987)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org
	at hudson.FilePath.act(FilePath.java:994)
	at hudson.FilePath.act(FilePath.java:976)
	at hudson.FilePath.deleteRecursive(FilePath.java:1178)
	at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84)
	at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113)
	at hudson.model.Build$BuildExecution.doRun(Build.java:174)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
	at hudson.model.Run.execute(Run.java:1737)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)
{code}

Could something with the vdsm tests that ran on the slave cause it to disconnect?
...
ci failure due to no space on slave?
------------------------------------
Key: OVIRT-1781
                URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1781
            Project: oVirt - virtualization made easy
         Issue Type: Outage
         Components: oVirt CI
           Reporter: danken
           Assignee: infra
           Priority: High
             Labels: vdsm
Exception when executing the batch command : no workspace from node
hudson.slaves.DumbSlave[vm0014.workers-phx.ovirt.org] which is
computer hudson.slaves.SlaveComputer@695d3d3c and has channel null
http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/19901/consol...
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100073)

------------=_1511682298-27447-558
Content-Type: text/html; charset="UTF-8"
Content-Disposition: inline
Content-Transfer-Encoding: 7bit

<html><body>
<pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1781?page=com.atlassian.jira.p... ]</pre>
<h3>Barak Korren commented on OVIRT-1781:</h3>
<p>[~danken@redhat.com] this error has nothing to do with disk space, you can see in the beginning on the log, in the cleanup script that there are 27G available.</p>
<p>This is talking about the `workspace` object which is the object that Jenkins uses to communicate with the slave.</p>
<p>If the workspace object is lost it typically means that Jenkins lost the connection to the slave, here is the actual relevant exception, where we can see that is indeed a `{{java.io.EOFException}}` (Meaning – an EOF on the slave connection):</p>
<p>{code} storage/guarded_test.py::TestContext::test_release_failure PASSED storage/guarded_test.py::TestContext::test_two_vols_different_domains PASSED storage/guarded_test.py::TestContext::test_two_vols_same_image PASSED storage/hsm_test.py::TestVerifyUntrustedVolume::test_compat_not_checked_for_raw FATAL: command execution failed java.io.EOFException</p>
<pre>at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)</pre>
<p>Caused: java.io.IOException: Unexpected termination of the channel</p>
<pre>at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)</pre>
<p>Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down</p>
<pre>at hudson.remoting.Channel.call(Channel.java:898)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
at com.sun.proxy.$Proxy80.isAlive(Unknown Source)
at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138)
at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130)
at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
at hudson.model.Run.execute(Run.java:1737)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)</pre>
<p>Build step ‘Execute shell’ marked build as failure FATAL: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down java.io.EOFException</p>
<pre>at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)</pre>
<p>Caused: java.io.IOException: Unexpected termination of the channel</p>
<pre>at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)</pre>
<p>Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down</p>
<pre>at hudson.remoting.Channel.call(Channel.java:898)
at hudson.FilePath.act(FilePath.java:987)</pre>
<p>Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64@tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel@391e411:vm0014.workers-phx.ovirt.org</p>
<pre>at hudson.FilePath.act(FilePath.java:994)
at hudson.FilePath.act(FilePath.java:976)
at hudson.FilePath.deleteRecursive(FilePath.java:1178)
at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84)
at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113)
at hudson.model.Build$BuildExecution.doRun(Build.java:174)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
at hudson.model.Run.execute(Run.java:1737)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:421)</pre>
<p>{code}</p>
<p>Could something with the vdsm tests that ran on the slave cause it to disconnect?</p>
<blockquote><h3>ci failure due to no space on slave?</h3>
<pre>     Key: OVIRT-1781
     URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1781
 Project: oVirt - virtualization made easy
         Issue Type: Outage
         Components: oVirt CI
Reporter: danken
Assignee: infra
Priority: High
  Labels: vdsm</pre>
<p>Exception when executing the batch command : no workspace from node hudson.slaves.DumbSlave[vm0014.workers-phx.ovirt.org] which is computer hudson.slaves.SlaveComputer@695d3d3c and has channel null <a href="http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/19901/consoleText">http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/19901/consoleText</a></p></blockquote>
<p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100073)</p>

<img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BCmpYLyzY..." alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/>
</body></html>

------------=_1511682298-27447-558--

Barak Korren (oVirt JIRA)

tags

participants (1)