Build failed in Jenkins: ovirt_4.0_system-tests #464
Evgheni Dereveanchin
ederevea at redhat.com
Mon Sep 26 12:03:43 UTC 2016
Yep, works for me as well now. Looks like it has been
something with network connectivity on the gerrit AWS
instance as it was also visible from outside PHX.
The instance did not reboot, I will open a Jira task
to follow up on the root cause of this short outage.
Regards,
Evgheni Dereveanchin
----- Original Message -----
From: "Gil Shinar" <gshinar at redhat.com>
To: "Evgheni Dereveanchin" <ederevea at redhat.com>
Cc: "Eyal Edri" <eedri at redhat.com>, "infra" <infra at ovirt.org>, "Shlomo Ben David" <sbendavi at redhat.com>
Sent: Monday, 26 September, 2016 2:01:08 PM
Subject: Re: Build failed in Jenkins: ovirt_4.0_system-tests #464
Yes, saw and now it looks like it has recovered.
It was a network issue
Gil
On Mon, Sep 26, 2016 at 2:59 PM, Evgheni Dereveanchin <ederevea at redhat.com>
wrote:
> Looks like there's some issue with the gerrit VM.
> I see alerts on Nagios and can't connect to the WebUI or SSH at the moment.
>
>
> Regards,
> Evgheni Dereveanchin
>
> ----- Original Message -----
> From: "Eyal Edri" <eedri at redhat.com>
> To: "Evgheni Dereveanchin" <ederevea at redhat.com>, "Gil Shinar" <
> gshinar at redhat.com>
> Cc: "infra" <infra at ovirt.org>
> Sent: Monday, 26 September, 2016 1:53:52 PM
> Subject: Re: Build failed in Jenkins: ovirt_4.0_system-tests #464
>
> connection issues to gerrit?
>
> On Mon, Sep 26, 2016 at 2:51 PM, <jenkins at jenkins.phx.ovirt.org> wrote:
>
> > See <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/>
> >
> > ------------------------------------------
> > Started by timer
> > [EnvInject] - Loading node environment variables.
> > Building remotely on ovirt-srv19.phx.ovirt.org (phx physical integ-tests
> > fc24) in workspace <http://jenkins.ovirt.org/job/
> > ovirt_4.0_system-tests/ws/>
> > Cloning the remote Git repository
> > Cloning repository git://gerrit.ovirt.org/ovirt-system-tests.git
> > > git init <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/
> > ovirt-system-tests> # timeout=10
> > Fetching upstream changes from git://gerrit.ovirt.org/ovirt-
> > system-tests.git
> > > git --version # timeout=10
> > > git -c core.askpass=true fetch --tags --progress git://
> > gerrit.ovirt.org/ovirt-system-tests.git +refs/heads/*:refs/remotes/
> > origin/*
> > > git config remote.origin.url git://gerrit.ovirt.org/ovirt-
> > system-tests.git # timeout=10
> > > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/
> origin/*
> > # timeout=10
> > > git config remote.origin.url git://gerrit.ovirt.org/ovirt-
> > system-tests.git # timeout=10
> > Cleaning workspace
> > > git rev-parse --verify HEAD # timeout=10
> > No valid HEAD. Skipping the resetting
> > > git clean -fdx # timeout=10
> > Pruning obsolete local branches
> > Fetching upstream changes from git://gerrit.ovirt.org/ovirt-
> > system-tests.git
> > > git -c core.askpass=true fetch --tags --progress git://
> > gerrit.ovirt.org/ovirt-system-tests.git refs/heads/master --prune
> > ERROR: Error fetching remote repo 'origin'
> > hudson.plugins.git.GitException: Failed to fetch from git://
> > gerrit.ovirt.org/ovirt-system-tests.git
> > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
> > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
> > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
> > at org.jenkinsci.plugins.multiplescms.MultiSCM.
> > checkout(MultiSCM.java:129)
> > at hudson.scm.SCM.checkout(SCM.java:485)
> > at hudson.model.AbstractProject.checkout(AbstractProject.java:
> > 1269)
> > at hudson.model.AbstractBuild$AbstractBuildExecution.
> > defaultCheckout(AbstractBuild.java:607)
> > at jenkins.scm.SCMCheckoutStrategy.checkout(
> > SCMCheckoutStrategy.java:86)
> > at hudson.model.AbstractBuild$AbstractBuildExecution.run(
> > AbstractBuild.java:529)
> > at hudson.model.Run.execute(Run.java:1738)
> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> > at hudson.model.ResourceController.execute(
> > ResourceController.java:98)
> > at hudson.model.Executor.run(Executor.java:410)
> > Caused by: hudson.plugins.git.GitException: Command "git -c
> > core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/ovirt-
> > system-tests.git refs/heads/master --prune" returned status code 128:
> > stdout:
> > stderr: fatal: unable to connect to gerrit.ovirt.org:
> > gerrit.ovirt.org[0: 107.22.212.69]: errno=Connection timed out
> >
> >
> > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.
> > launchCommandIn(CliGitAPIImpl.java:1640)
> > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.
> > launchCommandWithCredentials(CliGitAPIImpl.java:1388)
> > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.
> > access$300(CliGitAPIImpl.java:62)
> > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.
> > execute(CliGitAPIImpl.java:313)
> > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$
> > CommandInvocationHandler$1.call(RemoteGitImpl.java:152)
> > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$
> > CommandInvocationHandler$1.call(RemoteGitImpl.java:145)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:152)
> > at hudson.remoting.UserRequest.perform(UserRequest.java:50)
> > at hudson.remoting.Request$2.run(Request.java:332)
> > at hudson.remoting.InterceptingExecutorService$1.call(
> > InterceptingExecutorService.java:68)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > at ......remote call to ovirt-srv19.phx.ovirt.org(Native Method)
> > at hudson.remoting.Channel.attachCallSiteStackTrace(
> > Channel.java:1416)
> > at hudson.remoting.UserResponse.retrieve(UserRequest.java:252)
> > at hudson.remoting.Channel.call(Channel.java:781)
> > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$
> > CommandInvocationHandler.execute(RemoteGitImpl.java:145)
> > at sun.reflect.GeneratedMethodAccessor512.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$
> > CommandInvocationHandler.invoke(RemoteGitImpl.java:131)
> > at com.sun.proxy.$Proxy60.execute(Unknown Source)
> > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764)
> > ... 12 more
> > ERROR: null
> > Performing Post build task...
> > Match found for :.* : True
> > Logical operation result is TRUE
> > Running script : #!/bin/bash -xe
> > echo 'shell_scripts/system_tests.collect_logs.sh'
> >
> > #
> > # Required jjb vars:
> > # version
> > #
> > VERSION=4.0
> > SUITE_TYPE=
> >
> > WORKSPACE="$PWD"
> > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
> > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
> >
> > rm -rf "$WORKSPACE/exported-artifacts"
> > mkdir -p "$WORKSPACE/exported-artifacts"
> >
> > if [[ -d "$TESTS_LOGS" ]]; then
> > mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
> > fi
> >
> > [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/
> hudson5354774381234585939.sh
> > + echo shell_scripts/system_tests.collect_logs.sh
> > shell_scripts/system_tests.collect_logs.sh
> > + VERSION=4.0
> > + SUITE_TYPE=
> > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/>
> > + OVIRT_SUITE=4.0
> > + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_
> > system-tests/ws/ovirt-system-tests/exported-artifacts>
> > + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/
> > artifact/exported-artifacts>
> > + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/
> > artifact/exported-artifacts>
> > + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/
> > ovirt-system-tests/exported-artifacts> ]]
> > POST BUILD TASK : SUCCESS
> > END OF POST BUILD TASK : 0
> > Match found for :.* : True
> > Logical operation result is TRUE
> > Running script : #!/bin/bash -xe
> > echo "shell-scripts/mock_cleanup.sh"
> >
> > shopt -s nullglob
> >
> >
> > WORKSPACE="$PWD"
> >
> > # Make clear this is the cleanup, helps reading the jenkins logs
> > cat <<EOC
> > _______________________________________________________________________
> > #######################################################################
> > # #
> > # CLEANUP #
> > # #
> > #######################################################################
> > EOC
> >
> >
> > # Archive the logs, we want them anyway
> > logs=(
> > ./*log
> > ./*/logs
> > )
> > if [[ "$logs" ]]; then
> > tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
> > rm -rf "${logs[@]}"
> > fi
> >
> > # stop any processes running inside the chroot
> > failed=false
> > mock_confs=("$WORKSPACE"/*/mocker*)
> > # Clean current jobs mockroot if any
> > for mock_conf_file in "${mock_confs[@]}"; do
> > [[ "$mock_conf_file" ]] || continue
> > echo "Cleaning up mock $mock_conf"
> > mock_root="${mock_conf_file##*/}"
> > mock_root="${mock_root%.*}"
> > my_mock="/usr/bin/mock"
> > my_mock+=" --configdir=${mock_conf_file%/*}"
> > my_mock+=" --root=${mock_root}"
> > my_mock+=" --resultdir=$WORKSPACE"
> >
> > #TODO: investigate why mock --clean fails to umount certain dirs
> > sometimes,
> > #so we can use it instead of manually doing all this.
> > echo "Killing all mock orphan processes, if any."
> > $my_mock \
> > --orphanskill \
> > || {
> > echo "ERROR: Failed to kill orphans on $chroot."
> > failed=true
> > }
> >
> > mock_root="$(\
> > grep \
> > -Po "(?<=config_opts\['root'\] = ')[^']*" \
> > "$mock_conf_file" \
> > )" || :
> > [[ "$mock_root" ]] || continue
> > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
> > if [[ "$mounts" ]]; then
> > echo "Found mounted dirs inside the chroot $chroot. Trying to
> > umount."
> > fi
> > for mount in "${mounts[@]}"; do
> > sudo umount --lazy "$mount" \
> > || {
> > echo "ERROR: Failed to umount $mount."
> > failed=true
> > }
> > done
> > done
> >
> > # Clean any leftover chroot from other jobs
> > for mock_root in /var/lib/mock/*; do
> > this_chroot_failed=false
> > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
> > if [[ "$mounts" ]]; then
> > echo "Found mounted dirs inside the chroot $mock_root." \
> > "Trying to umount."
> > fi
> > for mount in "${mounts[@]}"; do
> > sudo umount --lazy "$mount" \
> > || {
> > echo "ERROR: Failed to umount $mount."
> > failed=true
> > this_chroot_failed=true
> > }
> > done
> > if ! $this_chroot_failed; then
> > sudo rm -rf "$mock_root"
> > fi
> > done
> >
> > if $failed; then
> > echo "Aborting."
> > exit 1
> > fi
> >
> > # remove mock system cache, we will setup proxies to do the caching and
> > this
> > # takes lots of space between runs
> > shopt -u nullglob
> > sudo rm -Rf /var/cache/mock/*
> >
> > # restore the permissions in the working dir, as sometimes it leaves
> files
> > # owned by root and then the 'cleanup workspace' from jenkins job fails
> to
> > # clean and breaks the jobs
> > sudo chown -R "$USER" "$WORKSPACE"
> >
> > [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/
> hudson7676240485039544411.sh
> > + echo shell-scripts/mock_cleanup.sh
> > shell-scripts/mock_cleanup.sh
> > + shopt -s nullglob
> > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/>
> > + cat
> > _______________________________________________________________________
> > #######################################################################
> > # #
> > # CLEANUP #
> > # #
> > #######################################################################
> > + logs=(./*log ./*/logs)
> > + [[ -n '' ]]
> > + failed=false
> > + mock_confs=("$WORKSPACE"/*/mocker*)
> > + false
> > + shopt -u nullglob
> > + sudo rm -Rf '/var/cache/mock/*'
> > + sudo chown -R jenkins <http://jenkins.ovirt.org/job/
> > ovirt_4.0_system-tests/ws/>
> > POST BUILD TASK : SUCCESS
> > END OF POST BUILD TASK : 1
> > Recording test results
> > ERROR: Step ?Publish JUnit test result report? failed: No test report
> > files were found. Configuration error?
> > Archiving artifacts
> > _______________________________________________
> > Infra mailing list
> > Infra at ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/infra
> >
> >
> >
>
>
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
>
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
>
More information about the Infra
mailing list