[JIRA] (OVIRT-2542) several failures failures on host not being
available - possible networking issue
by Dafna Ron (oVirt JIRA)
Dafna Ron created OVIRT-2542:
--------------------------------
Summary: several failures failures on host not being available - possible networking issue
Key: OVIRT-2542
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2542
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Dafna Ron
Assignee: infra
I have noticed that there are several issues that had failed recently with host not available.
the changes are not directly related to the issue and I cannot see anything in engine to suggest that there is a problem.
it could either be a network issue, a network test suite issue or infra related issue.
I am opening this Jira so we can follow all the failures and see if we can find a common denominator
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100094)
6 years, 2 months
[JIRA] (OVIRT-2541) Git fetch timeouts on jenkins
*-standar-check-pr jobs
by Ondra Machacek (oVirt JIRA)
Ondra Machacek created OVIRT-2541:
-------------------------------------
Summary: Git fetch timeouts on jenkins *-standar-check-pr jobs
Key: OVIRT-2541
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2541
Project: oVirt - virtualization made easy
Issue Type: Task
Reporter: Ondra Machacek
Assignee: infra
Example: https://jenkins.ovirt.org/job/oVirt_ovirt-ansible-vm-infra_standard-check...
git fetch --tags --progress https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*
08:47:21 ERROR: Timeout after 10 minutes
08:47:21 ERROR: Error cloning remote repo 'origin'
08:47:21 hudson.plugins.git.GitException: Command "git fetch --tags --progress https://gerrit.ovirt.org/jenkins +refs/heads/*:refs/remotes/origin/*" returned status code 143:
08:47:21 stdout:
08:47:21 stderr:
08:47:21 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2016)
08:47:21 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1735)
08:47:21 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)
08:47:21 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:420)
08:47:21 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:629)
08:47:21 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:153)
08:47:21 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:146)
08:47:21 at hudson.remoting.UserRequest.perform(UserRequest.java:212)
08:47:21 at hudson.remoting.UserRequest.perform(UserRequest.java:54)
08:47:21 at hudson.remoting.Request$2.run(Request.java:369)
08:47:21 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
08:47:21 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
08:47:21 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
08:47:21 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
08:47:21 at java.lang.Thread.run(Thread.java:748)
08:47:21 Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to vm0091.workers-phx.ovirt.org
08:47:21 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
08:47:21 at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
08:47:21 at hudson.remoting.Channel.call(Channel.java:955)
08:47:21 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
08:47:21 at sun.reflect.GeneratedMethodAccessor140.invoke(Unknown Source)
08:47:21 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
08:47:21 at java.lang.reflect.Method.invoke(Method.java:498)
08:47:21 at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
08:47:21 at com.sun.proxy.$Proxy77.execute(Unknown Source)
08:47:21 at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1146)
08:47:21 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
08:47:21 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:120)
08:47:21 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:90)
08:47:21 at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:77)
08:47:21 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:50)
08:47:21 at hudson.security.ACL.impersonate(ACL.java:290)
08:47:21 at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:47)
08:47:21 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
08:47:21 ... 4 more
[Pipeline] }
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100094)
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 -
Build # 660 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4....
Build Number: 660
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #660
[Gal Ben Haim] Add run-ost-container
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 012_local_maintenance_sdk.local_maintenance
Error Message:
The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
-------------------- >> begin captured logging << --------------------
root: INFO: * Waiting For System Stability...
root: INFO: * Performing Deactivation...
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f55c052b9b0>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 65, in <lambda>
lambda: host_service.get().status == types.HostStatus.MAINTENANCE or
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
return self._code(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback
self._check_fault(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
body = self._internal_read_body(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
self._connection.check_xml_content_type(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
response.headers
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
raise Error(msg)
Error: The response content type 'text/html; charset=iso-8859-1' isn't the expected XML
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper
prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 65, in local_maintenance
lambda: host_service.get().status == types.HostStatus.MAINTENANCE or
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long
assert_equals_within_long(func, True, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long
func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 65, in <lambda>
lambda: host_service.get().status == types.HostStatus.MAINTENANCE or
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait
return self._code(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback
self._check_fault(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault
body = self._internal_read_body(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body
self._connection.check_xml_content_type(response)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type
response.headers
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type
raise Error(msg)
'The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n-------------------- >> begin captured logging << --------------------\nroot: INFO: * Waiting For System Stability...\nroot: INFO: * Performing Deactivation...\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f55c052b9b0>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/012_local_maintenance_sdk.py", line 65, in <lambda>\n lambda: host_service.get().status == types.HostStatus.MAINTENANCE or\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 37093, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130, in _check_fault\n body = self._internal_read_body(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316, in _internal_read_body\n self._connection.check_xml_content_type(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849, in check_xml_content_type\n response.headers\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_content_type\n raise Error(msg)\nError: The response content type \'text/html; charset=iso-8859-1\' isn\'t the expected XML\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build #
787 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/787/
Build Number: 787
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #787
[Ales Musil] ovirt-4.2, ovirt-master: Retry Host setup Networks if it fails
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.add_hosts
Error Message:
Failed to read response: [(<pycurl.Curl object at 0x7fbd16b20050>, 7, 'TCP connection reset by peer')]
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fbd16b42230>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait
raise Error("Failed to read response: {}".format(err_list))
Error: Failed to read response: [(<pycurl.Curl object at 0x7fbd16b20050>, 7, 'TCP connection reset by peer')]
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts
add_hosts_4(prefix)
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4
testlib.assert_true_within(_host_is_up_4, timeout=15*60)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4
host_obj = host_service.get()
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get
return self._internal_get(headers, query, wait)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get
return future.wait() if wait else future
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait
response = self._connection.wait(self._context)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait
return self.__wait(context, failed_auth)
File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait
raise Error("Failed to read response: {}".format(err_list))
'Failed to read response: [(<pycurl.Curl object at 0x7fbd16b20050>, 7, \'TCP connection reset by peer\')]\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7fbd16b42230>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 253, in _host_is_up_4\n host_obj = host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 38104, in get\n return self._internal_get(headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211, in _internal_get\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 54, in wait\n response = self._connection.wait(self._context)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 497, in wait\n return self.__wait(context, failed_auth)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 511, in __wait\n raise Error("Failed to read response: {}".format(err_list))\nError: Failed to read response: [(<pycurl.Curl object at 0x7fbd16b20050>, 7, \'TCP connection reset by peer\')]\n--------------------- >> end captured logging << ---------------------'
6 years, 2 months
[JENKINS] Failed to setup proejct
kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org
Failed to run project_setup.sh for:
#2236 kubevirt [check-patch].
It probably means that docker_cleanup.py failed.
This step doesn't fail the job, but we do collect
data about such failures to find the root cause.
Infra owner, ensure that we're not running out of
disk space on ovirt-srv05.phx.ovirt.org-container-6.
6 years, 2 months
