August 2019 - Infra - oVirt List Archives

[JIRA] (OVIRT-2776) 51 jobs in the queue, all workers are idle
by Evgheni Dereveanchin (oVirt JIRA) 14 Aug '19

14 Aug '19

1 0

Build failed in Jenkins: system-sync_mirrors-fedora-updates-fc29-x86_64 #835
by jenkins＠jenkins.phx.ovirt.org 14 Aug '19

14 Aug '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-updates-fc29-x86_64…> ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace /home/jenkins/workspace/system-sync_mirrors-fedora-updates-fc29-x86_64 No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 03d6b67f33d57527ef7d0f83470408546ec1fcf8 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 03d6b67f33d57527ef7d0f83470408546ec1fcf8 Commit message: "ovirt-log-collector: drop 4.2 jobs" > git rev-list --no-walk 03d6b67f33d57527ef7d0f83470408546ec1fcf8 # timeout=10 [system-sync_mirrors-fedora-updates-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins5036711870660329167.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-updates-fc29 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror fedora-updates-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror fedora-updates-fc29 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=fedora-updates-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs fedora-updates-fc29 yum + local repo_name=fedora-updates-fc29 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-updates-fc29 /var/www/html/repos/yum/fedora-updates-fc29/base + check_yum_sync_needed fedora-updates-fc29 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=fedora-updates-fc29 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/fedora-updates-fc29 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync fedora-updates-fc29 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=fedora-updates-fc29 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-updates-fc29 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-updates-fc29/base --norepopath --newest-only --urls --quiet FATAL: command execution failed java.io.IOException at hudson.remoting.Channel.close(Channel.java:1449) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:844) at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:811) at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:89) at jenkins.model.Jenkins.access$2100(Jenkins.java:309) at jenkins.model.Jenkins$19.run(Jenkins.java:3425) at hudson.model.Queue._withLock(Queue.java:1379) at hudson.model.Queue.withLock(Queue.java:1256) at jenkins.model.Jenkins._cleanUpDisconnectComputers(Jenkins.java:3419) at jenkins.model.Jenkins.cleanUp(Jenkins.java:3297) at hudson.WebAppMain.contextDestroyed(WebAppMain.java:379) at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:965) at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:566) at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:376) at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1503) at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1467) at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1009) at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:569) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:124) at org.eclipse.jetty.server.Server.doStop(Server.java:461) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at winstone.Launcher.shutdown(Launcher.java:310) at winstone.ShutdownHook.run(ShutdownHook.java:25) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on mirrors.phx.ovirt.org failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:950) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) at com.sun.proxy.$Proxy113.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142) at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:155) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:109) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1818) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) FATAL: Unable to delete script file /tmp/jenkins5036711870660329167.sh java.io.IOException at hudson.remoting.Channel.close(Channel.java:1449) at hudson.remoting.Channel.close(Channel.java:1405) at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:844) at hudson.slaves.SlaveComputer.kill(SlaveComputer.java:811) at hudson.model.AbstractCIBase.killComputer(AbstractCIBase.java:89) at jenkins.model.Jenkins.access$2100(Jenkins.java:309) at jenkins.model.Jenkins$19.run(Jenkins.java:3425) at hudson.model.Queue._withLock(Queue.java:1379) at hudson.model.Queue.withLock(Queue.java:1256) at jenkins.model.Jenkins._cleanUpDisconnectComputers(Jenkins.java:3419) at jenkins.model.Jenkins.cleanUp(Jenkins.java:3297) at hudson.WebAppMain.contextDestroyed(WebAppMain.java:379) at org.eclipse.jetty.server.handler.ContextHandler.callContextDestroyed(ContextHandler.java:965) at org.eclipse.jetty.servlet.ServletContextHandler.callContextDestroyed(ServletContextHandler.java:566) at org.eclipse.jetty.server.handler.ContextHandler.stopContext(ContextHandler.java:942) at org.eclipse.jetty.servlet.ServletContextHandler.stopContext(ServletContextHandler.java:376) at org.eclipse.jetty.webapp.WebAppContext.stopWebapp(WebAppContext.java:1503) at org.eclipse.jetty.webapp.WebAppContext.stopContext(WebAppContext.java:1467) at org.eclipse.jetty.server.handler.ContextHandler.doStop(ContextHandler.java:1009) at org.eclipse.jetty.servlet.ServletContextHandler.doStop(ServletContextHandler.java:288) at org.eclipse.jetty.webapp.WebAppContext.doStop(WebAppContext.java:569) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:149) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:170) at org.eclipse.jetty.server.handler.AbstractHandler.doStop(AbstractHandler.java:124) at org.eclipse.jetty.server.Server.doStop(Server.java:461) at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:89) at winstone.Launcher.shutdown(Launcher.java:310) at winstone.ShutdownHook.run(ShutdownHook.java:25) Caused: hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on mirrors.phx.ovirt.org failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:950) at hudson.FilePath.act(FilePath.java:1070) at hudson.FilePath.act(FilePath.java:1059) at hudson.FilePath.delete(FilePath.java:1540) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1818) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Build step 'Execute shell' marked build as failure

1 0

[JIRA] (OVIRT-2776) 51 jobs in the queue, all workers are idle
by Nir Soffer (oVirt JIRA) 14 Aug '19

14 Aug '19

1 0

[JIRA] (OVIRT-2774) system_update_puppet job does not work properly
by Evgheni Dereveanchin (oVirt JIRA) 14 Aug '19

14 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2774?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2774: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > system_update_puppet job does not work properly > ----------------------------------------------- > > Key: OVIRT-2774 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2774 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > Priority: Highest > > The job performs incorrect updates to the puppet repo. This seems to just have caused a Jenkins restart and downgraded its version -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

[CQ]: 102517,3 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 14 Aug '19

14 Aug '19

Change 102517,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/102517/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15398/

1 0

[CQ]: 102506,5 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 14 Aug '19

14 Aug '19

Change 102506,5 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/102506/5 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15397/

1 0

[CQ]: 102570,4 (vdsm) failed "ovirt-master" system tests
by oVirt Jenkins 14 Aug '19

14 Aug '19

Change 102570,4 (vdsm) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/102570/4 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/15395/

1 0

[JIRA] (OVIRT-2774) system_update_puppet job does not work properly
by Evgheni Dereveanchin (oVirt JIRA) 14 Aug '19

14 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2774?page=com.atlassian.jira.… ] Evgheni Dereveanchin edited comment on OVIRT-2774 at 8/14/19 12:12 PM: ----------------------------------------------------------------------- Looking at the foreman: grep jenkins_ver /etc/puppet/environments/production/hieradata/common.yaml ovirt_jenkins::jenkins_ver: 2.164.2-1.1 ls -l /etc/puppet/environments/production/hieradata/common.yaml rw-rw-r-. 1 puppet-repos puppet-repos 10586 Aug 1 15:05 /etc/puppet/environments/production/hieradata/common.yaml At the same time from the logs of the last job all looks fine: [https://jenkins.ovirt.org/job/system_update_puppet/176/console|https://jenk…] Looking into {{/var/lib/data/jenkins/jobs/system_update_puppet/workspace}} on the master the hieradata file is correct and contains Jenkins version 2.176.1 as needed. [~accountid:557058:5fc78873-359e-47c9-aa0b-4845b0da8143] do you know how exactly system_update_puppet invokes r10k? The job config only lists “ssh“ as the build step and on the foreman the shell of the puppet-repos user is plain bash so it doesn’t seem to run any hardcoded commands upon login. This stops me from understanding where the changes are lost. was (Author: ederevea): Looking at the foreman: {{# grep jenkins_ver /etc/puppet/environments/production/hieradata/common.yaml }} {{ovirt_jenkins::jenkins_ver: 2.164.2-1.1}} {{# ls -l /etc/puppet/environments/production/hieradata/common.yaml }} {{-rw-rw-r--. 1 puppet-repos puppet-repos 10586 Aug 1 15:05 /etc/puppet/environments/production/hieradata/common.yaml}} At the same time from the logs of the last job all looks fine: [https://jenkins.ovirt.org/job/system_update_puppet/176/console|https://jenk…] Looking into {{/var/lib/data/jenkins/jobs/system_update_puppet/workspace}} on the master the hieradata file is correct and contains Jenkins version 2.176.1 as needed. [~accountid:557058:5fc78873-359e-47c9-aa0b-4845b0da8143] do you know how exactly system_update_puppet invokes r10k? The job config only lists “ssh“ as the build step and on the foreman the shell of the puppet-repos user is plain bash so it doesn’t seem to run any hardcoded commands upon login. This stops me from understanding where the changes are lost. > system_update_puppet job does not work properly > ----------------------------------------------- > > Key: OVIRT-2774 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2774 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Highest > > The job performs incorrect updates to the puppet repo. This seems to just have caused a Jenkins restart and downgraded its version -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

[JIRA] (OVIRT-2774) system_update_puppet job does not work properly
by Evgheni Dereveanchin (oVirt JIRA) 14 Aug '19

14 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2774?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2774: --------------------------------------------- Looking at the foreman: {{# grep jenkins_ver /etc/puppet/environments/production/hieradata/common.yaml }} {{ovirt_jenkins::jenkins_ver: 2.164.2-1.1}} {{# ls -l /etc/puppet/environments/production/hieradata/common.yaml }} {{-rw-rw-r--. 1 puppet-repos puppet-repos 10586 Aug 1 15:05 /etc/puppet/environments/production/hieradata/common.yaml}} At the same time from the logs of the last job all looks fine: [https://jenkins.ovirt.org/job/system_update_puppet/176/console|https://jenk…] Looking into {{/var/lib/data/jenkins/jobs/system_update_puppet/workspace}} on the master the hieradata file is correct and contains Jenkins version 2.176.1 as needed. [~accountid:557058:5fc78873-359e-47c9-aa0b-4845b0da8143] do you know how exactly system_update_puppet invokes r10k? The job config only lists “ssh“ as the build step and on the foreman the shell of the puppet-repos user is plain bash so it doesn’t seem to run any hardcoded commands upon login. This stops me from understanding where the changes are lost. > system_update_puppet job does not work properly > ----------------------------------------------- > > Key: OVIRT-2774 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2774 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Priority: Highest > > The job performs incorrect updates to the puppet repo. This seems to just have caused a Jenkins restart and downgraded its version -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0

[JIRA] (OVIRT-2775) Jenkins unexpected restart 2019.08.14
by Evgheni Dereveanchin (oVirt JIRA) 14 Aug '19

14 Aug '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2775?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-2775: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > Jenkins unexpected restart 2019.08.14 > ------------------------------------- > > Key: OVIRT-2775 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2775 > Project: oVirt - virtualization made easy > Issue Type: Outage > Reporter: Evgheni Dereveanchin > Assignee: Evgheni Dereveanchin > > Jenkins restarted, looks like it was caused by puppet that also downgraded versions: > /Stage[main]/Jenkins::Package/Package[jenkins]/ensure ensure changed '2.176.1-1.1' to '2.164.2-1.1' > Logged OVIRT-2774 to see why the puppet issue happened but for now we need to upgrade Jenkins back and put it into operation -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100107)

1 0