[JIRA] (OVIRT-1067) Java updates without restarting Jenkins agent are probably causing jobs to fail

[Barak Korren (oVirt JIRA)]

Barak Korren created OVIRT-1067:
-----------------------------------

             Summary: Java updates without restarting Jenkins agent are probably causing jobs to fail
                 Key: OVIRT-1067
                 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1067
             Project: oVirt - virtualization made easy
          Issue Type: Bug
            Reporter: Barak Korren
            Assignee: infra


A [recently invoked|http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_3.6/4523/console] system teasts job failed with the following error:
{code}
java.io.IOException: remote file operation failed: /home/jenkins/workspace/test-repo_ovirt_experimental_3.6 at hudson.remoting.Channel at 432401f8:el7-vm26.phx.ovirt.org: java.io.IOException: Remote call on el7-vm26.phx.ovirt.org failed
	at hudson.FilePath.act(FilePath.java:992)
...

Caused by: java.lang.InternalError: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/lib/resources.jar
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1003)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:212)
...
{code}

Looking at {{el7-vm26.phx.ovirt.org failed}} where this happened proved that indeed the file '{{/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/lib/resources.jar}}' does not exist on the node. The file '{{/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/resources.jar}}' does exist however.

Further examination in the logs revealed that the system's java version was recently updated:
{code}
Jan 21 20:19:36 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-headless-1.8.0.121-0.b13.el7_3.x86_64
Jan 21 20:19:38 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64
Jan 21 20:19:40 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-devel-1.8.0.121-0.b13.el7_3.x86_64
{code}

I suppose the issue is does to the system Jave and libraries being updated while the Jenkins agent was still running and trying to use the older libraries.

I suggest we endure that the Jenkins agent is shut down every time package updates are done on the node and restarted afterwards. This could be done by setting the node to "offline" in Jenkins.



--
This message was sent by Atlassian JIRA
(v1000.695.3#100025)