Fwd: ssh connection fails after restarting engine VM
by Yedidyah Bar David
Forwarding to infra, as the email to infra-support didn't work.
---------- Forwarded message ---------
From: Yedidyah Bar David <didi(a)redhat.com>
Date: Wed, Feb 19, 2020 at 10:37 AM
Subject: ssh connection fails after restarting engine VM
To: <infra-support(a)ovirt.org>
Hi all,
Apparently my jira account was stuck in some state and all my emails
to infra-support from last years were dropped to /dev/null. Now saw
another case of a ticket I tried to open a month ago, and considered
updating it, so spent the time to actually login and see that I can't
find it anywhere. So copy/pasting below my original email.
The new case is:
https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.3/341/
lago.log has:
2020-02-19 05:10:50,890::log_utils.py::__exit__::611::lago.prefix::INFO::
# [Thread-2] lago-he-node-ng-suite-4-3-host-1: [32mSuccess [0m (in
0:00:08)
2020-02-19 05:10:52,116::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH
error connecting to lago-he-node-ng-suite-4-3-engine: No existing
session
2020-02-19 05:10:52,116::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still
got 0 tries for lago-he-node-ng-suite-4-3-engine
2020-02-19 05:10:53,117::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:e0b52607-6e65-4583-bf43-b615aa901cc7:Get ssh client for
lago-he-node-ng-suite-4-3-engine:
2020-02-19 05:10:53,232::log_utils.py::end_log_task::670::root::ERROR::
# [Thread-3] lago-he-node-ng-suite-4-3-engine: [31mERROR [0m (in
0:00:11)
2020-02-19 05:10:53,245::log_utils.py::__exit__::607::lago.prefix::DEBUG::
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in
_collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
748, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
468, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
259, in extract_paths
format(self.vm.name())
2020-02-19 05:10:53,245::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error
while running thread Thread-3
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in
_ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526,
in _collect_artifacts
vm.collect_artifacts(path, ignore_nopath)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
748, in collect_artifacts
ignore_nopath=ignore_nopath
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
468, in extract_paths
return self.provider.extract_paths(paths, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
259, in extract_paths
format(self.vm.name())
ExtractPathError: Unable to extract paths from
lago-he-node-ng-suite-4-3-engine: unreachable with SSH
Original email I wrote (and was never received) follows:
Hi all,
See e.g. [1].
In lago.log [2]:
2020-01-19 05:38:06,052::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@
Run test: 008_restart_he_vm.py: [32mSuccess [0m (in 0:23:27)
...
2020-01-19 05:38:07,680::log_utils.py::__enter__::600::lago.prefix::INFO::
# [Thread-3] lago-he-basic-suite-4-3-engine: [0m [0m
...
2020-01-19 05:38:07,686::log_utils.py::__enter__::600::lago.ssh::DEBUG::start
task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for
lago-he-basic-suite-4-3-engine:
...
2020-01-19 05:38:12,415::log_utils.py::__exit__::611::lago.prefix::INFO::
# [Thread-1] lago-he-basic-suite-4-3-host-0: [32mSuccess [0m (in
0:00:04)
2020-01-19 05:38:17,729::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH
error connecting to lago-he-basic-suite-4-3-engine: No existing
session
2020-01-19 05:38:17,730::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still
got 0 tries for lago-he-basic-suite-4-3-engine
2020-01-19 05:38:18,731::log_utils.py::__exit__::611::lago.ssh::DEBUG::end
task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for
lago-he-basic-suite-4-3-engine:
2020-01-19 05:38:18,739::log_utils.py::end_log_task::670::root::ERROR::
# [Thread-3] lago-he-basic-suite-4-3-engine: [31mERROR [0m (in
0:00:11)
...
ExtractPathError: Unable to extract paths from
lago-he-basic-suite-4-3-engine: unreachable with SSH
I wonder what might "No existing session" mean. Perhaps paramiko
caches the connection, and after the engine VM is restarted it does
not try to connect again? Or something similar?
Searching the net, [3] looks similar, although I am definitely not
sure we do not already do this (or similar) in lago.
Anyway, instead of (perhaps) spending time on this, it might be
better/possible to explicitly expose some "closeconnection" function
in lago, to be used by OST after such an engine vm restart (or similar
cases).
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/
[2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/a...
[3] https://stackoverflow.com/questions/57508919/paramiko-ssh-exception-sshex...
Thanks,
--
Didi
--
Didi
4 years, 9 months
Re: [oVirt Jenkins] ovirt-system-tests_he-basic-role-remote-suite-4.3
- Build # 283 - Failure!
by Yedidyah Bar David
On Wed, Feb 19, 2020 at 4:51 AM <jenkins(a)jenkins.phx.ovirt.org> wrote:
>
> Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-sui...
> Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-sui...
> Build Number: 283
> Build Status: Failure
> Triggered By: Started by timer
>
> -------------------------------------
> Changes Since Last Success:
> -------------------------------------
> Changes for Build #283
> [Anton Marchukov] Added "sar" system resources collection on VMs.
>
> [Yedidyah Bar David] Move ovirt-engine-extension-aaa-ldap master to stdci v2
>
> [Gal Ben Haim] Fix the return value of update_upstream_sources
>
>
>
>
> -----------------
> Failed Tests:
> -----------------
> 1 tests failed.
> FAILED: 008_restart_he_vm.restart_he_vm
>
> Error Message:
> 1 != 0
> -------------------- >> begin captured logging << --------------------
> lago.ssh: DEBUG: start task:02e64977-e3bd-4e7c-9084-61ddeaebb791:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-1:
> lago.ssh: DEBUG: end task:02e64977-e3bd-4e7c-9084-61ddeaebb791:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-1:
> lago.ssh: DEBUG: Running 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1: hosted-engine --vm-status --json
> lago.ssh: DEBUG: Command 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1 returned with 0
> lago.ssh: DEBUG: Command 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1 output:
> {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4583 (Tue Feb 18 21:48:18 2020)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4584 (Tue Feb 18 21:48:19 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eda7a0ea", "local_conf_timestamp": 4584, "host-ts": 4583}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4604 (Tue Feb 18 21:48:40 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4605 (Tue Feb 18 21:48:40 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "23d07ed7", "local_conf_timestamp": 4605, "host-ts": 4604}, "global_maintenance": true}
>
> root: INFO: Engine VM is on host lago-he-basic-role-remote-suite-4-3-host-0, restarting the VM
> root: INFO: Shutting down HE VM on host: lago-he-basic-role-remote-suite-4-3-host-0
> lago.ssh: DEBUG: start task:90671f4b-c54e-4efe-b0c1-cfa47015a9db:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0:
> lago.ssh: DEBUG: end task:90671f4b-c54e-4efe-b0c1-cfa47015a9db:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0:
> lago.ssh: DEBUG: Running 57a55eee on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-shutdown
> lago.ssh: DEBUG: Command 57a55eee on lago-he-basic-role-remote-suite-4-3-host-0 returned with 0
> root: INFO: Command succeeded
> root: INFO: Waiting for VM to be down...
> lago.ssh: DEBUG: start task:4b1d8dce-3c5b-4d53-b81f-1de8bb71eed7:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0:
> lago.ssh: DEBUG: end task:4b1d8dce-3c5b-4d53-b81f-1de8bb71eed7:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0:
> lago.ssh: DEBUG: Running 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-status --json
> lago.ssh: DEBUG: Command 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0 returned with 1
> lago.ssh: DEBUG: Command 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0 output:
> The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable.
agent.log has:
StatusStorageThread::ERROR::2020-02-18
21:48:30,263::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run)
Failed to update state.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 82, in run
if (self._status_broker._inquire_whiteboard_lock() or
File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py",
line 195, in _inquire_whiteboard_lock
self.host_id, self._lease_file)
SanlockException: (104, 'Sanlock lockspace inquire failure',
'Connection reset by peer')
StatusStorageThread::ERROR::2020-02-18
21:48:30,300::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_restart)
Trying to restart the broker
Another "Connection reset by peer", looks to me similar to the one
reported a few days ago (with subject "[oVirt Jenkins]
ovirt-system-tests_he-basic-scsi-suite-4.3 - Build # 350 - Failure!").
Are we ok with this? Stable 4.3 jobs failing with no clear and
acceptable reason and no further handling?
It seems like a communication issue, to me. Is anyone looking at it?
Thanks,
--
Didi
4 years, 9 months
Build failed in Jenkins:
system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64 #356
by jenkins＠jenkins.phx.ovirt.org
See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-...>
Changes:
------------------------------------------
Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-...>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: error: RPC failed; result=22, HTTP code = 503
fatal: The remote end hung up unexpectedly
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy95.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
4 years, 9 months
Build failed in Jenkins:
system-sync_mirrors-centos-updates-7.6-el7-x86_64 #356
by jenkins＠jenkins.phx.ovirt.org
See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-...>
Changes:
------------------------------------------
Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-...>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: error: RPC failed; result=22, HTTP code = 503
fatal: The remote end hung up unexpectedly
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy95.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
4 years, 9 months
Build failed in Jenkins:
system-sync_mirrors-centos-opstools-testing-el7-x86_64 #3384
by jenkins＠jenkins.phx.ovirt.org
See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-opstools-testing...>
Changes:
------------------------------------------
Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-opstools-testing...>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: error: RPC failed; result=22, HTTP code = 503
fatal: The remote end hung up unexpectedly
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
at com.sun.proxy.$Proxy95.execute(Unknown Source)
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192)
at hudson.scm.SCM.checkout(SCM.java:504)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
at hudson.model.Run.execute(Run.java:1815)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
ERROR: Error fetching remote repo 'origin'
4 years, 9 months
