[ https://ovirt-jira.atlassian.net/browse/OVIRT-1067?page=com.atlassian.jira.p... ] Barak Korren updated OVIRT-1067: -------------------------------- Epic Link: OVIRT-403

Java updates without restarting Jenkins agent are probably causing jobs to fail -------------------------------------------------------------------------------

Key: OVIRT-1067 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1067 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Barak Korren Assignee: infra

A [recently invoked|http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_3.6/4523/console] system teasts job failed with the following error: {code} java.io.IOException: remote file operation failed: /home/jenkins/workspace/test-repo_ovirt_experimental_3.6 at hudson.remoting.Channel@432401f8:el7-vm26.phx.ovirt.org: java.io.IOException: Remote call on el7-vm26.phx.ovirt.org failed at hudson.FilePath.act(FilePath.java:992) ... Caused by: java.lang.InternalError: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/lib/resources.jar at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1003) at sun.misc.URLClassPath.getResource(URLClassPath.java:212) ... {code} Looking at {{el7-vm26.phx.ovirt.org failed}} where this happened proved that indeed the file '{{/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/jre/lib/resources.jar}}' does not exist on the node. The file '{{/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/lib/resources.jar}}' does exist however. Further examination in the logs revealed that the system's java version was recently updated: {code} Jan 21 20:19:36 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-headless-1.8.0.121-0.b13.el7_3.x86_64 Jan 21 20:19:38 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64 Jan 21 20:19:40 el7-vm26 yum[31962]: Updated: 1:java-1.8.0-openjdk-devel-1.8.0.121-0.b13.el7_3.x86_64 {code} I suppose the issue is does to the system Jave and libraries being updated while the Jenkins agent was still running and trying to use the older libraries. I suggest we endure that the Jenkins agent is shut down every time package updates are done on the node and restarted afterwards. This could be done by setting the node to "offline" in Jenkins.

-- This message was sent by Atlassian JIRA (v1000.695.3#100025)