Re: Build failed in Jenkins: ovirt_4.0_system-tests #464

Looks like there's some issue with the gerrit VM. I see alerts on Nagios and can't connect to the WebUI or SSH at the moment. Regards, Evgheni Dereveanchin ----- Original Message ----- From: "Eyal Edri" <eedri(a)redhat.com&gt; To: "Evgheni Dereveanchin" <ederevea(a)redhat.com&gt;, "Gil Shinar" < gshinar(a)redhat.com&gt; Cc: "infra" <infra(a)ovirt.org&gt; Sent: Monday, 26 September, 2016 1:53:52 PM Subject: Re: Build failed in Jenkins: ovirt_4.0_system-tests #464 connection issues to gerrit? On Mon, Sep 26, 2016 at 2:51 PM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > See <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/> > > ------------------------------------------ > Started by timer > [EnvInject] - Loading node environment variables. > Building remotely on ovirt-srv19.phx.ovirt.org (phx physical integ-tests > fc24) in workspace <http://jenkins.ovirt.org/job/ > ovirt_4.0_system-tests/ws/> > Cloning the remote Git repository > Cloning repository git://gerrit.ovirt.org/ovirt-system-tests.git > > git init <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ > ovirt-system-tests> # timeout=10 > Fetching upstream changes from git://gerrit.ovirt.org/ovirt- > system-tests.git > > git --version # timeout=10 > > git -c core.askpass=true fetch --tags --progress git:// > gerrit.ovirt.org/ovirt-system-tests.git +refs/heads/*:refs/remotes/ > origin/* > > git config remote.origin.url git://gerrit.ovirt.org/ovirt- > system-tests.git # timeout=10 > > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/ origin/* > # timeout=10 > > git config remote.origin.url git://gerrit.ovirt.org/ovirt- > system-tests.git # timeout=10 > Cleaning workspace > > git rev-parse --verify HEAD # timeout=10 > No valid HEAD. Skipping the resetting > > git clean -fdx # timeout=10 > Pruning obsolete local branches > Fetching upstream changes from git://gerrit.ovirt.org/ovirt- > system-tests.git > > git -c core.askpass=true fetch --tags --progress git:// > gerrit.ovirt.org/ovirt-system-tests.git refs/heads/master --prune > ERROR: Error fetching remote repo 'origin' > hudson.plugins.git.GitException: Failed to fetch from git:// > gerrit.ovirt.org/ovirt-system-tests.git > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766) > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022) > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053) > at org.jenkinsci.plugins.multiplescms.MultiSCM. > checkout(MultiSCM.java:129) > at hudson.scm.SCM.checkout(SCM.java:485) > at hudson.model.AbstractProject.checkout(AbstractProject.java: > 1269) > at hudson.model.AbstractBuild$AbstractBuildExecution. > defaultCheckout(AbstractBuild.java:607) > at jenkins.scm.SCMCheckoutStrategy.checkout( > SCMCheckoutStrategy.java:86) > at hudson.model.AbstractBuild$AbstractBuildExecution.run( > AbstractBuild.java:529) > at hudson.model.Run.execute(Run.java:1738) > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > at hudson.model.ResourceController.execute( > ResourceController.java:98) > at hudson.model.Executor.run(Executor.java:410) > Caused by: hudson.plugins.git.GitException: Command "git -c > core.askpass=true fetch --tags --progress git://gerrit.ovirt.org/ovirt- > system-tests.git refs/heads/master --prune" returned status code 128: > stdout: > stderr: fatal: unable to connect to gerrit.ovirt.org: > gerrit.ovirt.org[0: 107.22.212.69]: errno=Connection timed out > > > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl. > launchCommandIn(CliGitAPIImpl.java:1640) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl. > launchCommandWithCredentials(CliGitAPIImpl.java:1388) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl. > access$300(CliGitAPIImpl.java:62) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1. > execute(CliGitAPIImpl.java:313) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$ > CommandInvocationHandler$1.call(RemoteGitImpl.java:152) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$ > CommandInvocationHandler$1.call(RemoteGitImpl.java:145) > at hudson.remoting.UserRequest.perform(UserRequest.java:152) > at hudson.remoting.UserRequest.perform(UserRequest.java:50) > at hudson.remoting.Request$2.run(Request.java:332) > at hudson.remoting.InterceptingExecutorService$1.call( > InterceptingExecutorService.java:68) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > at ......remote call to ovirt-srv19.phx.ovirt.org(Native Method) > at hudson.remoting.Channel.attachCallSiteStackTrace( > Channel.java:1416) > at hudson.remoting.UserResponse.retrieve(UserRequest.java:252) > at hudson.remoting.Channel.call(Channel.java:781) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$ > CommandInvocationHandler.execute(RemoteGitImpl.java:145) > at sun.reflect.GeneratedMethodAccessor512.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$ > CommandInvocationHandler.invoke(RemoteGitImpl.java:131) > at com.sun.proxy.$Proxy60.execute(Unknown Source) > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764) > ... 12 more > ERROR: null > Performing Post build task... > Match found for :.* : True > Logical operation result is TRUE > Running script : #!/bin/bash -xe > echo 'shell_scripts/system_tests.collect_logs.sh' > > # > # Required jjb vars: > # version > # > VERSION=4.0 > SUITE_TYPE= > > WORKSPACE="$PWD" > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" > > rm -rf "$WORKSPACE/exported-artifacts" > mkdir -p "$WORKSPACE/exported-artifacts" > > if [[ -d "$TESTS_LOGS" ]]; then > mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" > fi > > [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/ hudson5354774381234585939.sh > + echo shell_scripts/system_tests.collect_logs.sh > shell_scripts/system_tests.collect_logs.sh > + VERSION=4.0 > + SUITE_TYPE= > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> > + OVIRT_SUITE=4.0 > + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_ > system-tests/ws/ovirt-system-tests/exported-artifacts> > + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/ > artifact/exported-artifacts> > + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/ > artifact/exported-artifacts> > + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ > ovirt-system-tests/exported-artifacts> ]] > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 0 > Match found for :.* : True > Logical operation result is TRUE > Running script : #!/bin/bash -xe > echo "shell-scripts/mock_cleanup.sh" > > shopt -s nullglob > > > WORKSPACE="$PWD" > > # Make clear this is the cleanup, helps reading the jenkins logs > cat <<EOC > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > EOC > > > # Archive the logs, we want them anyway > logs=( > ./*log > ./*/logs > ) > if [[ "$logs" ]]; then > tar cvzf exported-artifacts/logs.tgz "${logs[@]}" > rm -rf "${logs[@]}" > fi > > # stop any processes running inside the chroot > failed=false > mock_confs=("$WORKSPACE"/*/mocker*) > # Clean current jobs mockroot if any > for mock_conf_file in "${mock_confs[@]}"; do > [[ "$mock_conf_file" ]] || continue > echo "Cleaning up mock $mock_conf" > mock_root="${mock_conf_file##*/}" > mock_root="${mock_root%.*}" > my_mock="/usr/bin/mock" > my_mock+=" --configdir=${mock_conf_file%/*}" > my_mock+=" --root=${mock_root}" > my_mock+=" --resultdir=$WORKSPACE" > > #TODO: investigate why mock --clean fails to umount certain dirs > sometimes, > #so we can use it instead of manually doing all this. > echo "Killing all mock orphan processes, if any." > $my_mock \ > --orphanskill \ > || { > echo "ERROR: Failed to kill orphans on $chroot." > failed=true > } > > mock_root="$(\ > grep \ > -Po "(?<=config_opts\['root'\] = ')[^']*" \ > "$mock_conf_file" \ > )" || : > [[ "$mock_root" ]] || continue > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $chroot. Trying to > umount." > fi > for mount in "${mounts[@]}"; do > sudo umount --lazy "$mount" \ > || { > echo "ERROR: Failed to umount $mount." > failed=true > } > done > done > > # Clean any leftover chroot from other jobs > for mock_root in /var/lib/mock/*; do > this_chroot_failed=false > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $mock_root." \ > "Trying to umount." > fi > for mount in "${mounts[@]}"; do > sudo umount --lazy "$mount" \ > || { > echo "ERROR: Failed to umount $mount." > failed=true > this_chroot_failed=true > } > done > if ! $this_chroot_failed; then > sudo rm -rf "$mock_root" > fi > done > > if $failed; then > echo "Aborting." > exit 1 > fi > > # remove mock system cache, we will setup proxies to do the caching and > this > # takes lots of space between runs > shopt -u nullglob > sudo rm -Rf /var/cache/mock/* > > # restore the permissions in the working dir, as sometimes it leaves files > # owned by root and then the 'cleanup workspace' from jenkins job fails to > # clean and breaks the jobs > sudo chown -R "$USER" "$WORKSPACE" > > [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/ hudson7676240485039544411.sh > + echo shell-scripts/mock_cleanup.sh > shell-scripts/mock_cleanup.sh > + shopt -s nullglob > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> > + cat > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > + logs=(./*log ./*/logs) > + [[ -n '' ]] > + failed=false > + mock_confs=("$WORKSPACE"/*/mocker*) > + false > + shopt -u nullglob > + sudo rm -Rf '/var/cache/mock/*' > + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ > ovirt_4.0_system-tests/ws/> > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 1 > Recording test results > ERROR: Step ?Publish JUnit test result report? failed: No test report > files were found. Configuration error? > Archiving artifacts > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > > -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)