<div dir="ltr">Yes, saw and now it looks like it has recovered.<div>It was a network issue</div><div><br></div><div>Gil</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 26, 2016 at 2:59 PM, Evgheni Dereveanchin <span dir="ltr"><<a href="mailto:ederevea@redhat.com" target="_blank">ederevea@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Looks like there's some issue with the gerrit VM.<br>
I see alerts on Nagios and can't connect to the WebUI or SSH at the moment.<br>
<br>
<br>
Regards,<br>
Evgheni Dereveanchin<br>
<div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
From: "Eyal Edri" <<a href="mailto:eedri@redhat.com">eedri@redhat.com</a>><br>
To: "Evgheni Dereveanchin" <<a href="mailto:ederevea@redhat.com">ederevea@redhat.com</a>>, "Gil Shinar" <<a href="mailto:gshinar@redhat.com">gshinar@redhat.com</a>><br>
Cc: "infra" <<a href="mailto:infra@ovirt.org">infra@ovirt.org</a>><br>
Sent: Monday, 26 September, 2016 1:53:52 PM<br>
Subject: Re: Build failed in Jenkins: ovirt_4.0_system-tests #464<br>
<br>
connection issues to gerrit?<br>
<br>
On Mon, Sep 26, 2016 at 2:51 PM, <<a href="mailto:jenkins@jenkins.phx.ovirt.org">jenkins@jenkins.phx.ovirt.org</a><wbr>> wrote:<br>
<br>
> See <<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_4.0_system-tests/464/</a>><br>
><br>
> ------------------------------<wbr>------------<br>
> Started by timer<br>
> [EnvInject] - Loading node environment variables.<br>
> Building remotely on <a href="http://ovirt-srv19.phx.ovirt.org" rel="noreferrer" target="_blank">ovirt-srv19.phx.ovirt.org</a> (phx physical integ-tests<br>
> fc24) in workspace <<a href="http://jenkins.ovirt.org/job/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/</a><br>
> ovirt_4.0_system-tests/ws/><br>
> Cloning the remote Git repository<br>
> Cloning repository git://<a href="http://gerrit.ovirt.org/ovirt-system-tests.git" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-<wbr>system-tests.git</a><br>
> > git init <<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_4.0_system-tests/ws/</a><br>
> ovirt-system-tests> # timeout=10<br>
> Fetching upstream changes from git://<a href="http://gerrit.ovirt.org/ovirt-" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-</a><br>
> system-tests.git<br>
> > git --version # timeout=10<br>
> > git -c core.askpass=true fetch --tags --progress git://<br>
> <a href="http://gerrit.ovirt.org/ovirt-system-tests.git" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-system-<wbr>tests.git</a> +refs/heads/*:refs/remotes/<br>
> origin/*<br>
> > git config remote.origin.url git://<a href="http://gerrit.ovirt.org/ovirt-" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-</a><br>
> system-tests.git # timeout=10<br>
> > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/<wbr>origin/*<br>
> # timeout=10<br>
> > git config remote.origin.url git://<a href="http://gerrit.ovirt.org/ovirt-" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-</a><br>
> system-tests.git # timeout=10<br>
> Cleaning workspace<br>
> > git rev-parse --verify HEAD # timeout=10<br>
> No valid HEAD. Skipping the resetting<br>
> > git clean -fdx # timeout=10<br>
> Pruning obsolete local branches<br>
> Fetching upstream changes from git://<a href="http://gerrit.ovirt.org/ovirt-" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-</a><br>
> system-tests.git<br>
> > git -c core.askpass=true fetch --tags --progress git://<br>
</div></div><div class="HOEnZb"><div class="h5">> <a href="http://gerrit.ovirt.org/ovirt-system-tests.git" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-system-<wbr>tests.git</a> refs/heads/master --prune<br>
> ERROR: Error fetching remote repo 'origin'<br>
> hudson.plugins.git.<wbr>GitException: Failed to fetch from git://<br>
> <a href="http://gerrit.ovirt.org/ovirt-system-tests.git" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-system-<wbr>tests.git</a><br>
> at hudson.plugins.git.GitSCM.<wbr>fetchFrom(GitSCM.java:766)<br>
> at hudson.plugins.git.GitSCM.<wbr>retrieveChanges(GitSCM.java:<wbr>1022)<br>
> at hudson.plugins.git.GitSCM.<wbr>checkout(GitSCM.java:1053)<br>
> at org.jenkinsci.plugins.<wbr>multiplescms.MultiSCM.<br>
> checkout(MultiSCM.java:129)<br>
> at hudson.scm.SCM.checkout(SCM.<wbr>java:485)<br>
> at hudson.model.AbstractProject.<wbr>checkout(AbstractProject.java:<br>
> 1269)<br>
> at hudson.model.AbstractBuild$<wbr>AbstractBuildExecution.<br>
> defaultCheckout(AbstractBuild.<wbr>java:607)<br>
> at jenkins.scm.<wbr>SCMCheckoutStrategy.checkout(<br>
> SCMCheckoutStrategy.java:86)<br>
> at hudson.model.AbstractBuild$<wbr>AbstractBuildExecution.run(<br>
> AbstractBuild.java:529)<br>
> at hudson.model.Run.execute(Run.<wbr>java:1738)<br>
> at hudson.model.FreeStyleBuild.<wbr>run(FreeStyleBuild.java:43)<br>
> at hudson.model.<wbr>ResourceController.execute(<br>
> ResourceController.java:98)<br>
> at hudson.model.Executor.run(<wbr>Executor.java:410)<br>
> Caused by: hudson.plugins.git.<wbr>GitException: Command "git -c<br>
> core.askpass=true fetch --tags --progress git://<a href="http://gerrit.ovirt.org/ovirt-" rel="noreferrer" target="_blank">gerrit.ovirt.org/ovirt-</a><br>
> system-tests.git refs/heads/master --prune" returned status code 128:<br>
> stdout:<br>
> stderr: fatal: unable to connect to <a href="http://gerrit.ovirt.org" rel="noreferrer" target="_blank">gerrit.ovirt.org</a>:<br>
> <a href="http://gerrit.ovirt.org" rel="noreferrer" target="_blank">gerrit.ovirt.org</a>[0: 107.22.212.69]: errno=Connection timed out<br>
><br>
><br>
> at org.jenkinsci.plugins.<wbr>gitclient.CliGitAPIImpl.<br>
> launchCommandIn(CliGitAPIImpl.<wbr>java:1640)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.CliGitAPIImpl.<br>
> launchCommandWithCredentials(<wbr>CliGitAPIImpl.java:1388)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.CliGitAPIImpl.<br>
> access$300(CliGitAPIImpl.java:<wbr>62)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.CliGitAPIImpl$1.<br>
> execute(CliGitAPIImpl.java:<wbr>313)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.RemoteGitImpl$<br>
> CommandInvocationHandler$1.<wbr>call(RemoteGitImpl.java:152)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.RemoteGitImpl$<br>
> CommandInvocationHandler$1.<wbr>call(RemoteGitImpl.java:145)<br>
> at hudson.remoting.UserRequest.<wbr>perform(UserRequest.java:152)<br>
> at hudson.remoting.UserRequest.<wbr>perform(UserRequest.java:50)<br>
> at hudson.remoting.Request$2.run(<wbr>Request.java:332)<br>
> at hudson.remoting.<wbr>InterceptingExecutorService$1.<wbr>call(<br>
> InterceptingExecutorService.<wbr>java:68)<br>
> at java.util.concurrent.<wbr>FutureTask.run(FutureTask.<wbr>java:266)<br>
> at java.util.concurrent.<wbr>ThreadPoolExecutor.runWorker(<br>
> ThreadPoolExecutor.java:1142)<br>
> at java.util.concurrent.<wbr>ThreadPoolExecutor$Worker.run(<br>
> ThreadPoolExecutor.java:617)<br>
> at java.lang.Thread.run(Thread.<wbr>java:745)<br>
> at ......remote call to <a href="http://ovirt-srv19.phx.ovirt.org" rel="noreferrer" target="_blank">ovirt-srv19.phx.ovirt.org</a>(<wbr>Native Method)<br>
> at hudson.remoting.Channel.<wbr>attachCallSiteStackTrace(<br>
> Channel.java:1416)<br>
> at hudson.remoting.UserResponse.<wbr>retrieve(UserRequest.java:252)<br>
> at hudson.remoting.Channel.call(<wbr>Channel.java:781)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.RemoteGitImpl$<br>
> CommandInvocationHandler.<wbr>execute(RemoteGitImpl.java:<wbr>145)<br>
> at sun.reflect.<wbr>GeneratedMethodAccessor512.<wbr>invoke(Unknown Source)<br>
> at sun.reflect.<wbr>DelegatingMethodAccessorImpl.<wbr>invoke(<br>
> DelegatingMethodAccessorImpl.<wbr>java:43)<br>
> at java.lang.reflect.Method.<wbr>invoke(Method.java:606)<br>
> at org.jenkinsci.plugins.<wbr>gitclient.RemoteGitImpl$<br>
> CommandInvocationHandler.<wbr>invoke(RemoteGitImpl.java:131)<br>
> at com.sun.proxy.$Proxy60.<wbr>execute(Unknown Source)<br>
> at hudson.plugins.git.GitSCM.<wbr>fetchFrom(GitSCM.java:764)<br>
> ... 12 more<br>
> ERROR: null<br>
> Performing Post build task...<br>
> Match found for :.* : True<br>
> Logical operation result is TRUE<br>
> Running script : #!/bin/bash -xe<br>
> echo 'shell_scripts/<a href="http://system_tests.collect_logs.sh" rel="noreferrer" target="_blank">system_tests.<wbr>collect_logs.sh</a>'<br>
><br>
> #<br>
> # Required jjb vars:<br>
> # version<br>
> #<br>
> VERSION=4.0<br>
> SUITE_TYPE=<br>
><br>
> WORKSPACE="$PWD"<br>
> OVIRT_SUITE="$SUITE_TYPE_<wbr>suite_$VERSION"<br>
> TESTS_LOGS="$WORKSPACE/ovirt-<wbr>system-tests/exported-<wbr>artifacts"<br>
><br>
> rm -rf "$WORKSPACE/exported-<wbr>artifacts"<br>
> mkdir -p "$WORKSPACE/exported-<wbr>artifacts"<br>
><br>
> if [[ -d "$TESTS_LOGS" ]]; then<br>
> mv "$TESTS_LOGS/"* "$WORKSPACE/exported-<wbr>artifacts/"<br>
> fi<br>
><br>
> [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/<wbr>hudson5354774381234585939.sh<br>
> + echo shell_scripts/<a href="http://system_tests.collect_logs.sh" rel="noreferrer" target="_blank">system_tests.<wbr>collect_logs.sh</a><br>
> shell_scripts/<a href="http://system_tests.collect_logs.sh" rel="noreferrer" target="_blank">system_tests.<wbr>collect_logs.sh</a><br>
> + VERSION=4.0<br>
> + SUITE_TYPE=<br>
> + WORKSPACE=<<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/" rel="noreferrer" target="_blank">http://jenkins.<wbr>ovirt.org/job/ovirt_4.0_<wbr>system-tests/ws/</a>><br>
> + OVIRT_SUITE=4.0<br>
> + TESTS_LOGS=<<a href="http://jenkins.ovirt.org/job/ovirt_4.0_" rel="noreferrer" target="_blank">http://jenkins.<wbr>ovirt.org/job/ovirt_4.0_</a><br>
> system-tests/ws/ovirt-system-<wbr>tests/exported-artifacts><br>
> + rm -rf <<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_4.0_system-tests/464/</a><br>
> artifact/exported-artifacts><br>
> + mkdir -p <<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/464/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_4.0_system-tests/464/</a><br>
> artifact/exported-artifacts><br>
> + [[ -d <<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>ovirt_4.0_system-tests/ws/</a><br>
> ovirt-system-tests/exported-<wbr>artifacts> ]]<br>
> POST BUILD TASK : SUCCESS<br>
> END OF POST BUILD TASK : 0<br>
> Match found for :.* : True<br>
> Logical operation result is TRUE<br>
> Running script : #!/bin/bash -xe<br>
> echo "shell-scripts/mock_cleanup.<wbr>sh"<br>
><br>
> shopt -s nullglob<br>
><br>
><br>
> WORKSPACE="$PWD"<br>
><br>
> # Make clear this is the cleanup, helps reading the jenkins logs<br>
> cat <<EOC<br>
> ______________________________<wbr>______________________________<wbr>___________<br>
> ##############################<wbr>##############################<wbr>###########<br>
> # #<br>
> # CLEANUP #<br>
> # #<br>
> ##############################<wbr>##############################<wbr>###########<br>
> EOC<br>
><br>
><br>
> # Archive the logs, we want them anyway<br>
> logs=(<br>
> ./*log<br>
> ./*/logs<br>
> )<br>
> if [[ "$logs" ]]; then<br>
> tar cvzf exported-artifacts/logs.tgz "${logs[@]}"<br>
> rm -rf "${logs[@]}"<br>
> fi<br>
><br>
> # stop any processes running inside the chroot<br>
> failed=false<br>
> mock_confs=("$WORKSPACE"/*/<wbr>mocker*)<br>
> # Clean current jobs mockroot if any<br>
> for mock_conf_file in "${mock_confs[@]}"; do<br>
> [[ "$mock_conf_file" ]] || continue<br>
> echo "Cleaning up mock $mock_conf"<br>
> mock_root="${mock_conf_file##*<wbr>/}"<br>
> mock_root="${mock_root%.*}"<br>
> my_mock="/usr/bin/mock"<br>
> my_mock+=" --configdir=${mock_conf_file%/<wbr>*}"<br>
> my_mock+=" --root=${mock_root}"<br>
> my_mock+=" --resultdir=$WORKSPACE"<br>
><br>
> #TODO: investigate why mock --clean fails to umount certain dirs<br>
> sometimes,<br>
> #so we can use it instead of manually doing all this.<br>
> echo "Killing all mock orphan processes, if any."<br>
> $my_mock \<br>
> --orphanskill \<br>
> || {<br>
> echo "ERROR: Failed to kill orphans on $chroot."<br>
> failed=true<br>
> }<br>
><br>
> mock_root="$(\<br>
> grep \<br>
> -Po "(?<=config_opts\['root'\] = ')[^']*" \<br>
> "$mock_conf_file" \<br>
> )" || :<br>
> [[ "$mock_root" ]] || continue<br>
> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :<br>
> if [[ "$mounts" ]]; then<br>
> echo "Found mounted dirs inside the chroot $chroot. Trying to<br>
> umount."<br>
> fi<br>
> for mount in "${mounts[@]}"; do<br>
> sudo umount --lazy "$mount" \<br>
> || {<br>
> echo "ERROR: Failed to umount $mount."<br>
> failed=true<br>
> }<br>
> done<br>
> done<br>
><br>
> # Clean any leftover chroot from other jobs<br>
> for mock_root in /var/lib/mock/*; do<br>
> this_chroot_failed=false<br>
> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :<br>
> if [[ "$mounts" ]]; then<br>
> echo "Found mounted dirs inside the chroot $mock_root." \<br>
> "Trying to umount."<br>
> fi<br>
> for mount in "${mounts[@]}"; do<br>
> sudo umount --lazy "$mount" \<br>
> || {<br>
> echo "ERROR: Failed to umount $mount."<br>
> failed=true<br>
> this_chroot_failed=true<br>
> }<br>
> done<br>
> if ! $this_chroot_failed; then<br>
> sudo rm -rf "$mock_root"<br>
> fi<br>
> done<br>
><br>
> if $failed; then<br>
> echo "Aborting."<br>
> exit 1<br>
> fi<br>
><br>
> # remove mock system cache, we will setup proxies to do the caching and<br>
> this<br>
> # takes lots of space between runs<br>
> shopt -u nullglob<br>
> sudo rm -Rf /var/cache/mock/*<br>
><br>
> # restore the permissions in the working dir, as sometimes it leaves files<br>
> # owned by root and then the 'cleanup workspace' from jenkins job fails to<br>
> # clean and breaks the jobs<br>
> sudo chown -R "$USER" "$WORKSPACE"<br>
><br>
> [ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/<wbr>hudson7676240485039544411.sh<br>
> + echo shell-scripts/mock_cleanup.sh<br>
> shell-scripts/mock_cleanup.sh<br>
> + shopt -s nullglob<br>
> + WORKSPACE=<<a href="http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/" rel="noreferrer" target="_blank">http://jenkins.<wbr>ovirt.org/job/ovirt_4.0_<wbr>system-tests/ws/</a>><br>
> + cat<br>
> ______________________________<wbr>______________________________<wbr>___________<br>
> ##############################<wbr>##############################<wbr>###########<br>
> # #<br>
> # CLEANUP #<br>
> # #<br>
> ##############################<wbr>##############################<wbr>###########<br>
> + logs=(./*log ./*/logs)<br>
> + [[ -n '' ]]<br>
> + failed=false<br>
> + mock_confs=("$WORKSPACE"/*/<wbr>mocker*)<br>
> + false<br>
> + shopt -u nullglob<br>
> + sudo rm -Rf '/var/cache/mock/*'<br>
> + sudo chown -R jenkins <<a href="http://jenkins.ovirt.org/job/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/</a><br>
> ovirt_4.0_system-tests/ws/><br>
> POST BUILD TASK : SUCCESS<br>
> END OF POST BUILD TASK : 1<br>
> Recording test results<br>
> ERROR: Step ?Publish JUnit test result report? failed: No test report<br>
> files were found. Configuration error?<br>
> Archiving artifacts<br>
> ______________________________<wbr>_________________<br>
> Infra mailing list<br>
> <a href="mailto:Infra@ovirt.org">Infra@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/infra" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/infra</a><br>
><br>
><br>
><br>
<br>
<br>
--<br>
Eyal Edri<br>
Associate Manager<br>
RHV DevOps<br>
EMEA ENG Virtualization R&D<br>
Red Hat Israel<br>
<br>
phone: <a href="tel:%2B972-9-7692018" value="+97297692018">+972-9-7692018</a><br>
irc: eedri (on #tlv #rhev-dev #rhev-integ)<br>
</div></div></blockquote></div><br></div>