- Infra - oVirt List Archives

Fwd: ssh connection fails after restarting engine VM
by Yedidyah Bar David 19 Feb '20

19 Feb '20

Forwarding to infra, as the email to infra-support didn't work. ---------- Forwarded message --------- From: Yedidyah Bar David <didi(a)redhat.com> Date: Wed, Feb 19, 2020 at 10:37 AM Subject: ssh connection fails after restarting engine VM To: <infra-support(a)ovirt.org> Hi all, Apparently my jira account was stuck in some state and all my emails to infra-support from last years were dropped to /dev/null. Now saw another case of a ticket I tried to open a month ago, and considered updating it, so spent the time to actually login and see that I can't find it anywhere. So copy/pasting below my original email. The new case is: https://jenkins.ovirt.org/job/ovirt-system-tests_he-node-ng-suite-4.3/341/ lago.log has: 2020-02-19 05:10:50,890::log_utils.py::__exit__::611::lago.prefix::INFO:: # [Thread-2] lago-he-node-ng-suite-4-3-host-1: [32mSuccess [0m (in 0:00:08) 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH error connecting to lago-he-node-ng-suite-4-3-engine: No existing session 2020-02-19 05:10:52,116::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still got 0 tries for lago-he-node-ng-suite-4-3-engine 2020-02-19 05:10:53,117::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:e0b52607-6e65-4583-bf43-b615aa901cc7:Get ssh client for lago-he-node-ng-suite-4-3-engine: 2020-02-19 05:10:53,232::log_utils.py::end_log_task::670::root::ERROR:: # [Thread-3] lago-he-node-ng-suite-4-3-engine: [31mERROR [0m (in 0:00:11) 2020-02-19 05:10:53,245::log_utils.py::__exit__::607::lago.prefix::DEBUG:: File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 259, in extract_paths format(self.vm.name()) 2020-02-19 05:10:53,245::utils.py::_ret_via_queue::63::lago.utils::DEBUG::Error while running thread Thread-3 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1526, in _collect_artifacts vm.collect_artifacts(path, ignore_nopath) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 748, in collect_artifacts ignore_nopath=ignore_nopath File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 468, in extract_paths return self.provider.extract_paths(paths, *args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 259, in extract_paths format(self.vm.name()) ExtractPathError: Unable to extract paths from lago-he-node-ng-suite-4-3-engine: unreachable with SSH Original email I wrote (and was never received) follows: Hi all, See e.g. [1]. In lago.log [2]: 2020-01-19 05:38:06,052::log_utils.py::__exit__::611::ovirtlago.prefix::INFO::@ Run test: 008_restart_he_vm.py: [32mSuccess [0m (in 0:23:27) ... 2020-01-19 05:38:07,680::log_utils.py::__enter__::600::lago.prefix::INFO:: # [Thread-3] lago-he-basic-suite-4-3-engine: [0m [0m ... 2020-01-19 05:38:07,686::log_utils.py::__enter__::600::lago.ssh::DEBUG::start task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for lago-he-basic-suite-4-3-engine: ... 2020-01-19 05:38:12,415::log_utils.py::__exit__::611::lago.prefix::INFO:: # [Thread-1] lago-he-basic-suite-4-3-host-0: [32mSuccess [0m (in 0:00:04) 2020-01-19 05:38:17,729::ssh.py::get_ssh_client::373::lago.ssh::DEBUG::SSH error connecting to lago-he-basic-suite-4-3-engine: No existing session 2020-01-19 05:38:17,730::ssh.py::get_ssh_client::381::lago.ssh::DEBUG::Still got 0 tries for lago-he-basic-suite-4-3-engine 2020-01-19 05:38:18,731::log_utils.py::__exit__::611::lago.ssh::DEBUG::end task:170b4eaa-fbf5-48ca-b81a-4ddf0c9a3bd5:Get ssh client for lago-he-basic-suite-4-3-engine: 2020-01-19 05:38:18,739::log_utils.py::end_log_task::670::root::ERROR:: # [Thread-3] lago-he-basic-suite-4-3-engine: [31mERROR [0m (in 0:00:11) ... ExtractPathError: Unable to extract paths from lago-he-basic-suite-4-3-engine: unreachable with SSH I wonder what might "No existing session" mean. Perhaps paramiko caches the connection, and after the engine VM is restarted it does not try to connect again? Or something similar? Searching the net, [3] looks similar, although I am definitely not sure we do not already do this (or similar) in lago. Anyway, instead of (perhaps) spending time on this, it might be better/possible to explicitly expose some "closeconnection" function in lago, to be used by OST after such an engine vm restart (or similar cases). [1] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/ [2] https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.3/324/art… [3] https://stackoverflow.com/questions/57508919/paramiko-ssh-exception-sshexce… Thanks, -- Didi -- Didi

3 3

gerrit whitelist
by Eitan Raviv 19 Feb '20

19 Feb '20

hi, I received the below message in https://gerrit.ovirt.org/#/c/106936/ Can it be verified that I am in the whitelist? Thanks ---------------------- Build Failed https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/4073/ : To avoid overloading the infrastructure, a whitelist for running gerrit triggered jobs has been set in place, if you feel like you should be in it, please contact infra at ovirt dot org.

2 1

Re: [oVirt Jenkins] ovirt-system-tests_he-basic-role-remote-suite-4.3 - Build # 283 - Failure!
by Yedidyah Bar David 19 Feb '20

19 Feb '20

On Wed, Feb 19, 2020 at 4:51 AM <jenkins(a)jenkins.phx.ovirt.org> wrote: > > Project: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… > Build: https://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-role-remote-suite… > Build Number: 283 > Build Status: Failure > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #283 > [Anton Marchukov] Added "sar" system resources collection on VMs. > > [Yedidyah Bar David] Move ovirt-engine-extension-aaa-ldap master to stdci v2 > > [Gal Ben Haim] Fix the return value of update_upstream_sources > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: 008_restart_he_vm.restart_he_vm > > Error Message: > 1 != 0 > -------------------- >> begin captured logging << -------------------- > lago.ssh: DEBUG: start task:02e64977-e3bd-4e7c-9084-61ddeaebb791:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-1: > lago.ssh: DEBUG: end task:02e64977-e3bd-4e7c-9084-61ddeaebb791:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-1: > lago.ssh: DEBUG: Running 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1: hosted-engine --vm-status --json > lago.ssh: DEBUG: Command 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1 returned with 0 > lago.ssh: DEBUG: Command 56796ff6 on lago-he-basic-role-remote-suite-4-3-host-1 output: > {"1": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4583 (Tue Feb 18 21:48:18 2020)\nhost-id=1\nscore=3400\nvm_conf_refresh_time=4584 (Tue Feb 18 21:48:19 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=EngineUp\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-0.lago.local", "host-id": 1, "engine-status": {"health": "good", "vm": "up", "detail": "Up"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "eda7a0ea", "local_conf_timestamp": 4584, "host-ts": 4583}, "2": {"conf_on_shared_storage": true, "live-data": true, "extra": "metadata_parse_version=1\nmetadata_feature_version=1\ntimestamp=4604 (Tue Feb 18 21:48:40 2020)\nhost-id=2\nscore=3400\nvm_conf_refresh_time=4605 (Tue Feb 18 21:48:40 2020)\nconf_on_shared_storage=True\nmaintenance=False\nstate=GlobalMaintenance\nstopped=False\n", "hostname": "lago-he-basic-role-remote-suite-4-3-host-1", "host-id": 2, "engine-status": {"reason": "vm not running on this host", "health": "bad", "vm": "down", "detail": "unknown"}, "score": 3400, "stopped": false, "maintenance": false, "crc32": "23d07ed7", "local_conf_timestamp": 4605, "host-ts": 4604}, "global_maintenance": true} > > root: INFO: Engine VM is on host lago-he-basic-role-remote-suite-4-3-host-0, restarting the VM > root: INFO: Shutting down HE VM on host: lago-he-basic-role-remote-suite-4-3-host-0 > lago.ssh: DEBUG: start task:90671f4b-c54e-4efe-b0c1-cfa47015a9db:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0: > lago.ssh: DEBUG: end task:90671f4b-c54e-4efe-b0c1-cfa47015a9db:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0: > lago.ssh: DEBUG: Running 57a55eee on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-shutdown > lago.ssh: DEBUG: Command 57a55eee on lago-he-basic-role-remote-suite-4-3-host-0 returned with 0 > root: INFO: Command succeeded > root: INFO: Waiting for VM to be down... > lago.ssh: DEBUG: start task:4b1d8dce-3c5b-4d53-b81f-1de8bb71eed7:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0: > lago.ssh: DEBUG: end task:4b1d8dce-3c5b-4d53-b81f-1de8bb71eed7:Get ssh client for lago-he-basic-role-remote-suite-4-3-host-0: > lago.ssh: DEBUG: Running 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0: hosted-engine --vm-status --json > lago.ssh: DEBUG: Command 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0 returned with 1 > lago.ssh: DEBUG: Command 5901a70c on lago-he-basic-role-remote-suite-4-3-host-0 output: > The hosted engine configuration has not been retrieved from shared storage. Please ensure that ovirt-ha-agent is running and the storage server is reachable. agent.log has: StatusStorageThread::ERROR::2020-02-18 21:48:30,263::status_broker::90::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(run) Failed to update state. Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 82, in run if (self._status_broker._inquire_whiteboard_lock() or File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/status_broker.py", line 195, in _inquire_whiteboard_lock self.host_id, self._lease_file) SanlockException: (104, 'Sanlock lockspace inquire failure', 'Connection reset by peer') StatusStorageThread::ERROR::2020-02-18 21:48:30,300::status_broker::70::ovirt_hosted_engine_ha.broker.status_broker.StatusBroker.Update::(trigger_restart) Trying to restart the broker Another "Connection reset by peer", looks to me similar to the one reported a few days ago (with subject "[oVirt Jenkins] ovirt-system-tests_he-basic-scsi-suite-4.3 - Build # 350 - Failure!"). Are we ok with this? Stable 4.3 jobs failing with no clear and acceptable reason and no further handling? It seems like a communication issue, to me. Is anyone looking at it? Thanks, -- Didi

2 1

Build failed in Jenkins: system-sync_mirrors-centos-sclo-rh-release-7.6-el7-x86_64 #356
by jenkins＠jenkins.phx.ovirt.org 18 Feb '20

18 Feb '20

See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-sclo-rh-release-7.…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: error: RPC failed; result=22, HTTP code = 503 fatal: The remote end hung up unexpectedly at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy95.execute(Unknown Source) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) ERROR: Error fetching remote repo 'origin'

1 1

Build failed in Jenkins: system-sync_mirrors-centos-updates-7.6-el7-x86_64 #356
by jenkins＠jenkins.phx.ovirt.org 18 Feb '20

18 Feb '20

See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-x8…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-updates-7.6-el7-x8…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: error: RPC failed; result=22, HTTP code = 503 fatal: The remote end hung up unexpectedly at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy95.execute(Unknown Source) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) ERROR: Error fetching remote repo 'origin'

1 1

Build failed in Jenkins: system-sync_mirrors-centos-opstools-testing-el7-x86_64 #3384
by jenkins＠jenkins.phx.ovirt.org 18 Feb '20

18 Feb '20

See <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-opstools-testing-e…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-centos-opstools-testing-e…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to fetch from http://gerrit.ovirt.org/jenkins.git at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:894) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: error: RPC failed; result=22, HTTP code = 503 fatal: The remote end hung up unexpectedly at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2172) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1864) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:78) at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:545) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146) at hudson.remoting.UserRequest.perform(UserRequest.java:212) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to mirrors.phx.ovirt.org at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146) at sun.reflect.GeneratedMethodAccessor625.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132) at com.sun.proxy.$Proxy95.execute(Unknown Source) at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:892) at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1161) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1192) at hudson.scm.SCM.checkout(SCM.java:504) at hudson.model.AbstractProject.checkout(AbstractProject.java:1208) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) ERROR: Error fetching remote repo 'origin'

1 1

[oVirt Jenkins] cleanup-gated-ovirt-master - Build #66 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 18 Feb '20

18 Feb '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-master/66/ Build Name: #66 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

oVirt infra daily report - unstable production jobs - 1105
by jenkins＠jenkins.phx.ovirt.org 18 Feb '20

18 Feb '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1105//artifact/exported… Cheers, Jenkins

1 0

[CQ]: 106791, 8 (vdsm) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 18 Feb '20

18 Feb '20

A system test invoked by the "ovirt-master" change queue including change 106791,8 (vdsm) failed. However, this change seems not to be the root cause for this failure. Change 106280,6 (vdsm) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 106280,6 (vdsm) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/106791/8 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/106280/6 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/20267/

1 0

[CQ]: 106992, 2 (ovirt-engine) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 18 Feb '20

18 Feb '20

A system test invoked by the "ovirt-master" change queue including change 106992,2 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 104391,14 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 104391,14 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/106992/2 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/104391/14 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/20261/

1 0