March 2018 - Infra - oVirt List Archives

add_secondary_storage_domains failed (was: [oVirt Jenkins] ovirt-system-tests_he-basic-suite-4.1 - Build # 223 - Still Failing!)
by Yedidyah Bar David 19 Mar '18

19 Mar '18

On Mon, Mar 19, 2018 at 9:01 AM, Nir Soffer <nsoffer(a)redhat.com> wrote: > > > On Sun, Mar 18, 2018 at 4:21 PM Yedidyah Bar David <didi(a)redhat.com> wrote: >> >> On Sun, Mar 18, 2018 at 2:48 PM, Yedidyah Bar David <didi(a)redhat.com> >> wrote: >> > On Sun, Mar 18, 2018 at 1:45 PM, Yedidyah Bar David <didi(a)redhat.com> >> > wrote: >> >> On Sun, Mar 18, 2018 at 11:20 AM, <jenkins(a)jenkins.phx.ovirt.org> >> >> wrote: >> >>> Project: >> >>> http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/ >> >>> Build: >> >>> http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-4.1/223/ >> >>> Build Number: 223 >> >>> Build Status: Still Failing >> >>> Triggered By: Started by timer >> >> >> >> It was broken by: >> >> >> >> [1] https://gerrit.ovirt.org/88483 >> >> >> >> It should be fixed by: >> >> >> >> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1554283 >> >> >> >> [2] is on modified, no idea about its status other than that. >> >> >> >> I didn't intend to merge [1] before [2] is fixed, not sure if >> >> Sandro didn't notice my comment there, or thought that [2] is fixed. >> >> >> >> If it's annoying we can revert [1] and re-merge when [2] is fixed. >> > >> > Talked with Gal, and pushed this, which should hopefully fix: >> > >> > https://gerrit.ovirt.org/89136 >> >> It indeed seems to fix [1][2]: >> >> 13:13:24 # he_get_shared_config: >> 13:13:26 # he_get_shared_config: Success (in 0:00:01) >> 13:13:26 # sleep: >> 13:15:26 # sleep: Success (in 0:02:00) >> 13:15:26 # add_he_hosts: >> 13:16:18 # add_he_hosts: Success (in 0:00:52) >> 13:16:18 # he_check_ha_agent: >> 13:16:19 # he_check_ha_agent: Success (in 0:00:00) >> >> But later fails: >> >> 13:16:20 # add_secondary_storage_domains: >> 13:19:30 Error while running thread >> 13:19:30 Traceback (most recent call last): >> 13:19:30 File "/usr/lib/python2.7/site-packages/lago/utils.py", line >> 58, in _ret_via_queue >> 13:19:30 queue.put({'return': func()}) >> 13:19:30 File >> >> "/home/jenkins/workspace/ovirt-system-tests_master_check-patch-el7-x86_64/ovirt-system-tests/he-basic-suite-4.1/test-scenarios/002_bootstrap.py", >> line 491, in add_nfs_storage_domain >> 13:19:30 add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, >> SD_NFS_HOST_NAME, SD_NFS_PATH) >> 13:19:30 File >> >> "/home/jenkins/workspace/ovirt-system-tests_master_check-patch-el7-x86_64/ovirt-system-tests/he-basic-suite-4.1/test-scenarios/002_bootstrap.py", >> line 496, in add_generic_nfs_storage_domain >> 13:19:30 add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, >> nfs_host_name, mount_path, sd_format, sd_type, nfs_version) >> 13:19:30 File >> >> "/home/jenkins/workspace/ovirt-system-tests_master_check-patch-el7-x86_64/ovirt-system-tests/he-basic-suite-4.1/test-scenarios/002_bootstrap.py", >> line 552, in add_generic_nfs_storage_domain_4 >> 13:19:30 _add_storage_domain_4(api, p) >> 13:19:30 File >> >> "/home/jenkins/workspace/ovirt-system-tests_master_check-patch-el7-x86_64/ovirt-system-tests/he-basic-suite-4.1/test-scenarios/002_bootstrap.py", >> line 466, in _add_storage_domain_4 >> 13:19:30 id=sd.id, >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 2219, >> in add >> 13:19:30 return self._internal_add(storage_domain, headers, query, >> wait) >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 223, >> in _internal_add >> 13:19:30 return future.wait() if wait else future >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 53, in >> wait >> 13:19:30 return self._code(response) >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 220, >> in callback >> 13:19:30 self._check_fault(response) >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 123, >> in _check_fault >> 13:19:30 self._raise_error(response, body) >> 13:19:30 File >> "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 109, >> in _raise_error >> 13:19:30 raise error >> 13:19:30 Error: Fault reason is "Operation Failed". Fault detail is >> "[Storage domain cannot be reached. Please ensure it is accessible >> from the host(s).]". HTTP response code is 400. >> 13:19:30 Error while running thread >> >> I'll retrigger now, but perhaps someone from storage wants to check. >> vdsm log has[3]: >> >> 2018-03-18 08:59:52,918-0400 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] >> RPC call Host.getHardwareInfo succeeded in 0.00 seconds (__init__:539) >> 2018-03-18 08:59:52,924-0400 INFO (jsonrpc/4) [vdsm.api] START >> prepareImage(sdUUID=u'424e809d-b7ad-4ed5-b6a1-5426d373a5d2', >> spUUID=u'8509c64f-cdd7-4713-9e09-a79d90ba26ed', >> imgUUID=u'718090ba-b36e-45cc-bcd6-597c16a766b9', >> leafUUID=u'23ee843b-20e0-4afe-98dc-e165334ac710', allowIllegal=False) >> from=::1,36020, task_id=92aeabf2-084a-4c98-a6e5-03efae38e7b3 (api:46) >> 2018-03-18 08:59:52,928-0400 INFO (jsonrpc/4) [vdsm.api] FINISH >> prepareImage error=Volume does not exist: >> (u'23ee843b-20e0-4afe-98dc-e165334ac710',) from=::1,36020, >> task_id=92aeabf2-084a-4c98-a6e5-03efae38e7b3 (api:50) >> 2018-03-18 08:59:52,928-0400 ERROR (jsonrpc/4) >> [storage.TaskManager.Task] >> (Task='92aeabf2-084a-4c98-a6e5-03efae38e7b3') Unexpected error >> (task:872) >> Traceback (most recent call last): >> File "/usr/share/vdsm/storage/task.py", line 879, in _run >> return fn(*args, **kargs) >> File "<string>", line 2, in prepareImage >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in >> method >> ret = func(*args, **kwargs) >> File "/usr/share/vdsm/storage/hsm.py", line 3137, in prepareImage >> raise se.VolumeDoesNotExist(leafUUID) >> VolumeDoesNotExist: Volume does not exist: >> (u'23ee843b-20e0-4afe-98dc-e165334ac710',) > > > Immediately after this failed prepare, we see: > > 2018-03-18 08:59:52,941-0400 INFO (jsonrpc/7) [vdsm.api] START > createVolume(sdUUID=u'424e809d-b7ad-4ed5-b6a1-5426d373a5d2', > spUUID=u'8509c64f-cdd7-4713-9e09-a79d90ba26ed', > imgUUID=u'718090ba-b36e-45cc-bcd6-597c16a766b9', size=u'1048576', > volFormat=5, preallocate=1, diskType=2, > volUUID=u'23ee843b-20e0-4afe-98dc-e165334ac710', > desc=u'hosted-engine.lockspace', > srcImgUUID=u'00000000-0000-0000-0000-000000000000', > srcVolUUID=u'00000000-0000-0000-0000-000000000000', initialSize=None) > from=::1,36020, task_id=c4c73314-ee10-419a-8265-b8b87763e99f (api:46) > > So that volume really did not exit when it was prepared, which means it is a > client > error, not vdsm error. > > The caller (hosted engine agent?) The caller is ovirt-system-tests. The code for add_secondary_storage_domains is duplicated among most of the suites, didn't check variations. See e.g.: https://github.com/oVirt/ovirt-system-tests/blob/master/basic-suite-master/… It's not calling vdsm directly, but uses the engine api (through the python sdk). Not sure if it's, as you say, "a bug in the client" (the engine), or we use the sdk improperly, or something else. Can someone please advice? Patches to ovirt-system-tests are welcome :-), especially patches to stress-test this, as it seems it's a sensitive timing issue - usually does not fail. The specific suite that failed above is he-basic-suite-4.1 . I think that the code there is identical with basic-suite-4.1. Both are different from basic-suite-master. No idea if the diff is due to diff between the versions, or it's simply a bug fixed in the master suite and not fixed in 4.1 suites. > should use Volume.getInfo to check if a > volume > exits. Non existing volume is considered expected error in this call and > will be > logged as with INFO log level. > > See > https://github.com/oVirt/vdsm/blob/f286cf7825578d8997ca6abc0801a823dee247ef… > >> >> [1] >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… >> [2] >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… >> [3] >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_… >> >> > >> >> >> >>> >> >>> ------------------------------------- >> >>> Changes Since Last Success: >> >>> ------------------------------------- >> >>> Changes for Build #218 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> Changes for Build #219 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> Changes for Build #220 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> Changes for Build #221 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> Changes for Build #222 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> Changes for Build #223 >> >>> [Martin Sivak] Cleanup of test code - next() replaced with any() >> >>> >> >>> >> >>> >> >>> >> >>> ----------------- >> >>> Failed Tests: >> >>> ----------------- >> >>> 1 tests failed. >> >>> FAILED: 002_bootstrap.he_check_ha_agent >> >>> >> >>> Error Message: >> >>> False is not true >> >>> -------------------- >> begin captured logging << -------------------- >> >>> lago.ssh: DEBUG: start task:cb75d7f3-e333-47f1-b3e9-76ff9c0b4b21:Get >> >>> ssh client for lago-he-basic-suite-4-1-host0: >> >>> lago.ssh: DEBUG: end task:cb75d7f3-e333-47f1-b3e9-76ff9c0b4b21:Get ssh >> >>> client for lago-he-basic-suite-4-1-host0: >> >>> lago.ssh: DEBUG: Running 7e67a956 on lago-he-basic-suite-4-1-host0: >> >>> grep Permission denied /var/log/ovirt-hosted-engine-ha/agent.log >> >>> lago.ssh: DEBUG: Command 7e67a956 on lago-he-basic-suite-4-1-host0 >> >>> returned with 2 >> >>> lago.ssh: DEBUG: Command 7e67a956 on lago-he-basic-suite-4-1-host0 >> >>> output: >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> >>> Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf' >> >>> >> >>> lago.ssh: DEBUG: Command 7e67a956 on lago-he-basic-suite-4-1-host0 >> >>> errors: >> >>> grep: denied: No such file or directory >> >>> >> >>> --------------------- >> end captured logging << --------------------- >> >>> >> >>> Stack Trace: >> >>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >> >>> testMethod() >> >>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in >> >>> runTest >> >>> self.test(*self.arg) >> >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >> >>> 129, in wrapped_test >> >>> test() >> >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >> >>> 59, in wrapper >> >>> return func(get_test_prefix(), *args, **kwargs) >> >>> File >> >>> "/home/jenkins/workspace/ovirt-system-tests_he-basic-suite-4.1/ovirt-system-tests/he-basic-suite-4.1/test-scenarios/002_bootstrap.py", >> >>> line 1036, in he_check_ha_agent >> >>> 'Permission denied' not in result.out >> >>> File "/usr/lib64/python2.7/unittest/case.py", line 462, in >> >>> assertTrue >> >>> raise self.failureException(msg) >> >>> "False is not true\n-------------------- >> begin captured logging << >> >>> --------------------\nlago.ssh: DEBUG: start >> >>> task:cb75d7f3-e333-47f1-b3e9-76ff9c0b4b21:Get ssh client for >> >>> lago-he-basic-suite-4-1-host0:\nlago.ssh: DEBUG: end >> >>> task:cb75d7f3-e333-47f1-b3e9-76ff9c0b4b21:Get ssh client for >> >>> lago-he-basic-suite-4-1-host0:\nlago.ssh: DEBUG: Running 7e67a956 on >> >>> lago-he-basic-suite-4-1-host0: grep Permission denied >> >>> /var/log/ovirt-hosted-engine-ha/agent.log\nlago.ssh: DEBUG: Command 7e67a956 >> >>> on lago-he-basic-suite-4-1-host0 returned with 2\nlago.ssh: DEBUG: Command >> >>> 7e67a956 on lago-he-basic-suite-4-1-host0 output:\n >> >>> /var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] Permission >> >>> denied: >> >>> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> >>> [Errno 13] Permission denied: >> >>> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> >>> [Errno 13] Permission denied: '/var/run/ovirt-hosted-engine-ha/vm.conf >> '\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: [Errno 13] >> Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> [Errno 13] Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> [Errno 13] Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> [Errno 13] Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> [Errno 13] Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n/var/log/ovirt-hosted-engine-ha/agent.log:IOError: >> [Errno 13] Permission denied: >> '/var/run/ovirt-hosted-engine-ha/vm.conf'\n\nlago.ssh: DEBUG: Command >> 7e67a956 on lago-he-basic-suite-4-1-host0 errors:\n grep: denied: No such >> file or directory\n\n--------------------- >> end captured logging << >> ---------------------" >> >> >> >> >> >> >> >> -- >> >> Didi >> > >> > >> > >> > -- >> > Didi >> >> >> >> -- >> Didi >> _______________________________________________ >> Infra mailing list >> Infra(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/infra -- Didi

2 1

[CQ]: 88777,3 (ovirt-engine) failed "ovirt-master" system tests
by oVirt Jenkins 19 Mar '18

19 Mar '18

Change 88777,3 (ovirt-engine) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/88777/3 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/6429/

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 16 - Failure!
by jenkins＠jenkins.phx.ovirt.org 18 Mar '18

18 Mar '18

------=_Part_346_190385630.1521367477675 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2= /=20 Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/1= 6/ Build Number: 16 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #16 [Martin Sivak] Cleanup of test code - next() replaced with any() ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=3Diso-8859-1' isn't the expec= ted XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_u= p_4 at 0x3564c08> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i= n assert_equals_within res =3D func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt= -system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254= , in _host_is_up_4 host_obj =3D host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 362= 19, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211,= in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, = in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208,= in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130,= in _check_fault body =3D self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316,= in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849= , in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889= , in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=3Diso-8859-1' isn't th= e expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTes= t self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, i= n wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in= wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt= -system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 229= , in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt= -system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 273= , in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=3D15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, i= n assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i= n assert_equals_within res =3D func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt= -system-tests/hc-basic-suite-4.2/test-scenarios/002_bootstrap.py", line 254= , in _host_is_up_4 host_obj =3D host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 362= 19, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211,= in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, = in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208,= in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130,= in _check_fault body =3D self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316,= in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849= , in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889= , in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=3Diso-8859-1\' isn\'t the e= xpected XML\n-------------------- >> begin captured logging << ------------= --------\novirtlago.testlib: ERROR: * Unhandled exception in <function = _host_is_up_4 at 0x3564c08>\nTraceback (most recent call last):\n File "/u= sr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_e= quals_within\n res =3D func()\n File "/home/jenkins/workspace/ovirt-sys= tem-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-sce= narios/002_bootstrap.py", line 254, in _host_is_up_4\n host_obj =3D host= _service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/servic= es.py", line 36219, in get\n return self._internal_get(headers, query, w= ait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", lin= e 211, in _internal_get\n return future.wait() if wait else future\n Fi= le "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in w= ait\n return self._code(response)\n File "/usr/lib64/python2.7/site-pac= kages/ovirtsdk4/service.py", line 208, in callback\n self._check_fault(r= esponse)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",= line 130, in _check_fault\n body =3D self._internal_read_body(response)= \n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 31= 6, in _internal_read_body\n self._connection.check_xml_content_type(resp= onse)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", l= ine 849, in check_xml_content_type\n response.headers\n File "/usr/lib6= 4/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _check_conte= nt_type\n raise Error(msg)\nError: The response content type \'text/html= ; charset=3Diso-8859-1\' isn\'t the expected XML\n--------------------- >> = end captured logging << ---------------------' ------=_Part_346_190385630.1521367477675--

1 1

[JIRA] (OVIRT-1936) (nested)&&fc27 label has no online slaves
by Evgheni Dereveanchin (oVirt JIRA) 18 Mar '18

18 Mar '18

This is a multi-part message in MIME format... ------------=_1521202838-11894-197 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit [ https://ovirt-jira.atlassian.net/browse/OVIRT-1936?page=com.atlassian.jira.… ] Evgheni Dereveanchin reassigned OVIRT-1936: ------------------------------------------- Assignee: Evgheni Dereveanchin (was: infra) > (nested)&&fc27 label has no online slaves > ----------------------------------------- > > Key: OVIRT-1936 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1936 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: sbonazzo > Assignee: Evgheni Dereveanchin > -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100082) ------------=_1521202838-11894-197 Content-Type: text/html; charset="UTF-8" Content-Disposition: inline Content-Transfer-Encoding: 7bit <html><body> <pre>[ https://ovirt-jira.atlassian.net/browse/OVIRT-1936?page=com.atlassian.jira.… ]</pre> <h3>Evgheni Dereveanchin reassigned OVIRT-1936:</h3> <pre>Assignee: Evgheni Dereveanchin (was: infra)</pre> <blockquote><h3>(nested)&&fc27 label has no online slaves</h3> <pre> Key: OVIRT-1936 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1936 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: sbonazzo Assignee: Evgheni Dereveanchin</pre></blockquote> <p>— This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100082)</p> <img src="https://u4043402.ct.sendgrid.net/wf/open?upn=i5TMWGV99amJbNxJpSp2-2BJ33BSM3…" alt="" width="1" height="1" border="0" style="height:1px !important;width:1px !important;border-width:0 !important;margin-top:0 !important;margin-bottom:0 !important;margin-right:0 !important;margin-left:0 !important;padding-top:0 !important;padding-bottom:0 !important;padding-right:0 !important;padding-left:0 !important;"/> </body></html> ------------=_1521202838-11894-197--

2 1

OST Network suite is failing on "OSError: [Errno 28] No space left on device"
by Edward Haas 18 Mar '18

18 Mar '18

Good morning, The network suite is running Ansible in one of its tests and from the weekend it started failing without a good explanation. It raises "OSError: [Errno 28] No space left on device" when trying to take lock on the python multiprocess module. It smells like a problem on the slave, but I am not sure. Any ideas? Thanks, Edy. =================================== FAILURES =================================== ______________________ test_ovn_provider_create_scenario _______________________ os_client_config = None def test_ovn_provider_create_scenario(os_client_config): > _test_ovn_provider('create_scenario.yml') network-suite-master/tests/test_ovn_provider.py:68: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ network-suite-master/tests/test_ovn_provider.py:78: in _test_ovn_provider playbook.run() network-suite-master/lib/ansiblelib.py:127: in run self._run_playbook_executor() network-suite-master/lib/ansiblelib.py:138: in _run_playbook_executor pbex = PlaybookExecutor(**self._pbex_args) /usr/lib/python2.7/site-packages/ansible/executor/playbook_executor.py:60: in __init__ self._tqm = TaskQueueManager(inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords=self.passwords) /usr/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py:104: in __init__ self._final_q = multiprocessing.Queue() /usr/lib64/python2.7/multiprocessing/__init__.py:218: in Queue return Queue(maxsize) /usr/lib64/python2.7/multiprocessing/queues.py:63: in __init__ self._rlock = Lock() /usr/lib64/python2.7/multiprocessing/synchronize.py:147: in __init__ SemLock.__init__(self, SEMAPHORE, 1, 1) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <Lock(owner=unknown)>, kind = 1, value = 1, maxvalue = 1 def __init__(self, kind, value, maxvalue): > sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue) E OSError: [Errno 28] No space left on device /usr/lib64/python2.7/multiprocessing/synchronize.py:75: OSError

1 0

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 13 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Mar '18

17 Mar '18

------=_Part_316_1234843677.1521279432434 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/13/ Build Number: 13 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #13 [Martin Sivak] Cleanup of test code - next() replaced with any() ----------------- Failed Tests: ----------------- All tests passed ------=_Part_316_1234843677.1521279432434--

1 1

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-master - Build # 78 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Mar '18

17 Mar '18

------=_Part_284_2084206695.1521238462287 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-mast… Build Number: 78 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #78 [Martin Sivak] Cleanup of test code - next() replaced with any() ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_284_2084206695.1521238462287--

1 4

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 225 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Mar '18

17 Mar '18

------=_Part_263_1135888245.1521212507833 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-mas= ter/=20 Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-maste= r/225/ Build Number: 225 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #225 [Yedidyah Bar David] HE: Test hosted-engine --get-shared-config ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: The response content type 'text/html; charset=3Diso-8859-1' isn't the expec= ted XML -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_u= p_4 at 0x3433230> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i= n assert_equals_within res =3D func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ov= irt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", li= ne 255, in _host_is_up_4 host_obj =3D host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 363= 46, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211,= in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, = in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208,= in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130,= in _check_fault body =3D self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316,= in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849= , in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889= , in _check_content_type raise Error(msg) Error: The response content type 'text/html; charset=3Diso-8859-1' isn't th= e expected XML --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTes= t self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, i= n wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in= wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ov= irt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", li= ne 230, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ov= irt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", li= ne 274, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=3D15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 263, i= n assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, i= n assert_equals_within res =3D func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ov= irt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", li= ne 255, in _host_is_up_4 host_obj =3D host_service.get() File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 363= 46, in get return self._internal_get(headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 211,= in _internal_get return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, = in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 208,= in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 130,= in _check_fault body =3D self._internal_read_body(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 316,= in _internal_read_body self._connection.check_xml_content_type(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 849= , in check_xml_content_type response.headers File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889= , in _check_content_type raise Error(msg) 'The response content type \'text/html; charset=3Diso-8859-1\' isn\'t the e= xpected XML\n-------------------- >> begin captured logging << ------------= --------\novirtlago.testlib: ERROR: * Unhandled exception in <function = _host_is_up_4 at 0x3433230>\nTraceback (most recent call last):\n File "/u= sr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 219, in assert_e= quals_within\n res =3D func()\n File "/home/jenkins/workspace/ovirt-sys= tem-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/te= st-scenarios/002_bootstrap.py", line 255, in _host_is_up_4\n host_obj = =3D host_service.get()\n File "/usr/lib64/python2.7/site-packages/ovirtsdk= 4/services.py", line 36346, in get\n return self._internal_get(headers, = query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.= py", line 211, in _internal_get\n return future.wait() if wait else futu= re\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line = 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/= site-packages/ovirtsdk4/service.py", line 208, in callback\n self._check= _fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/serv= ice.py", line 130, in _check_fault\n body =3D self._internal_read_body(r= esponse)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py",= line 316, in _internal_read_body\n self._connection.check_xml_content_t= ype(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/__init_= _.py", line 849, in check_xml_content_type\n response.headers\n File "/= usr/lib64/python2.7/site-packages/ovirtsdk4/__init__.py", line 889, in _che= ck_content_type\n raise Error(msg)\nError: The response content type \'t= ext/html; charset=3Diso-8859-1\' isn\'t the expected XML\n-----------------= ---- >> end captured logging << ---------------------' ------=_Part_263_1135888245.1521212507833--

1 2

[oVirt Jenkins] ovirt-system-tests_he-basic-suite-master - Build # 237 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Mar '18

17 Mar '18

------=_Part_278_393064355.1521234372792 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-suite-master/237/ Build Number: 237 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #237 [Martin Sivak] Cleanup of test code - next() replaced with any() ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_278_393064355.1521234372792--

1 4

[oVirt Jenkins] ovirt-system-tests_he-basic-iscsi-suite-master - Build # 1 - Failure!
by jenkins＠jenkins.phx.ovirt.org 17 Mar '18

17 Mar '18

------=_Part_265_946388335.1521215444463 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-iscsi-suite-master… Build Number: 1 Build Status: Failure Triggered By: Started by user Sandro Bonazzola ------------------------------------- Changes Since Last Success: ------------------------------------- ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_265_946388335.1521215444463--

1 3