[JIRA] (OVIRT-1781) ci failure due to no space on slave?

Sun Nov 26 07:44:58 UTC 2017

    [ https://ovirt-jira.atlassian.net/browse/OVIRT-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=35391#comment-35391 ] 

Barak Korren commented on OVIRT-1781:
-------------------------------------

[~danken at redhat.com] this error has nothing to do with disk space, you can see in the beginning on the log, in the cleanup script that there are 27G available.

This is talking about the `workspace` object which is the object that Jenkins uses to communicate with the slave.

If the workspace object is lost it typically means that Jenkins lost the connection to the slave, here is the actual relevant exception, where we can see that is indeed a `{{java.io.EOFException}}` (Meaning - an EOF on the slave connection):

{code}
storage/guarded_test.py::TestContext::test_release_failure PASSED
storage/guarded_test.py::TestContext::test_two_vols_different_domains PASSED
storage/guarded_test.py::TestContext::test_two_vols_same_image PASSED
storage/hsm_test.py::TestVerifyUntrustedVolume::test_compat_not_checked_for_raw FATAL: command execution failed
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:898)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:257)
	at com.sun.proxy.$Proxy80.isAlive(Unknown Source)
	at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1138)
	at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1130)
	at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109)
	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:736)
	at hudson.model.Build$BuildExecution.build(Build.java:206)
	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
	at hudson.model.Run.execute(Run.java:1737)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)
Build step 'Execute shell' marked build as failure
FATAL: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 at tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel at 391e411:vm0014.workers-phx.ovirt.org: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
java.io.EOFException
	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2680)
	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3155)
	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:864)
	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:360)
	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused: hudson.remoting.ChannelClosedException: Remote call on vm0014.workers-phx.ovirt.org failed. The channel is closing down or has closed down
	at hudson.remoting.Channel.call(Channel.java:898)
	at hudson.FilePath.act(FilePath.java:987)
Caused: java.io.IOException: remote file operation failed: /home/jenkins/workspace/vdsm_master_check-patch-el7-x86_64 at tmp/secretFiles/126de56b-84a8-43d8-8f8d-51ce21780f9c at hudson.remoting.Channel at 391e411:vm0014.workers-phx.ovirt.org
	at hudson.FilePath.act(FilePath.java:994)
	at hudson.FilePath.act(FilePath.java:976)
	at hudson.FilePath.deleteRecursive(FilePath.java:1178)
	at org.jenkinsci.plugins.credentialsbinding.impl.UnbindableDir$UnbinderImpl.unbind(UnbindableDir.java:84)
	at org.jenkinsci.plugins.credentialsbinding.impl.SecretBuildWrapper$1.tearDown(SecretBuildWrapper.java:113)
	at hudson.model.Build$BuildExecution.doRun(Build.java:174)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:496)
	at hudson.model.Run.execute(Run.java:1737)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:421)
{code}

Could something with the vdsm tests that ran on the slave cause it to disconnect?

> ci failure due to no space on slave?
> ------------------------------------
>
>                 Key: OVIRT-1781
>                 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1781
>             Project: oVirt - virtualization made easy
>          Issue Type: Outage
>          Components: oVirt CI
>            Reporter: danken
>            Assignee: infra
>            Priority: High
>              Labels: vdsm
>
> Exception when executing the batch command : no workspace from node
> hudson.slaves.DumbSlave[vm0014.workers-phx.ovirt.org] which is
> computer hudson.slaves.SlaveComputer at 695d3d3c and has channel null
> http://jenkins.ovirt.org/job/vdsm_master_check-patch-el7-x86_64/19901/consoleText

--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100073)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20171126/1b2f1fc7/attachment-0001.html>