- Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_he-basic-ansible-suite-4.2 - Build # 609 - Failure!
by jenkins＠jenkins.phx.ovirt.org 29 Sep '18

29 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_he-basic-ansible-suite-4.2/… Build Number: 609 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #609 [Miguel Duarte Barroso] Revert "Repos: add python2-ovsdbapp to the reposync config" ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_secondary_storage_domains Error Message: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. -------------------- >> begin captured logging << -------------------- lago.utils: ERROR: Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24225, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409. --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 577, in add_secondary_storage_domains vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4 _add_storage_domain_4(api, p) File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4 sd = sds_service.add(p) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24225, in add return self._internal_add(storage_domain, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault self._raise_error(response, body) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error 'Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n-------------------- >> begin captured logging << --------------------\nlago.utils: ERROR: Error while running thread\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 491, in add_nfs_storage_domain\n add_generic_nfs_storage_domain(prefix, SD_NFS_NAME, SD_NFS_HOST_NAME, SD_NFS_PATH)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 496, in add_generic_nfs_storage_domain\n add_generic_nfs_storage_domain_4(prefix, sd_nfs_name, nfs_host_name, mount_path, sd_format, sd_type, nfs_version)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 552, in add_generic_nfs_storage_domain_4\n _add_storage_domain_4(api, p)\n File "/home/jenkins/workspace/ovirt-system-tests_he-basic-ansible-suite-4.2/ovirt-system-tests/he-basic-ansible-suite-4.2/test-scenarios/002_bootstrap.py", line 447, in _add_storage_domain_4\n sd = sds_service.add(p)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 24225, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 232, in _internal_add\n return future.wait() if wait else future\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait\n return self._code(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 229, in callback\n self._check_fault(response)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 132, in _check_fault\n self._raise_error(response, body)\n File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error\n raise error\nError: Fault reason is "Operation Failed". Fault detail is "[Cannot add storage server connection when Host status is not up]". HTTP response code is 409.\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-master - Build # 737 - Failure!
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-master/737/ Build Number: 737 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #737 [Miguel Duarte Barroso] Revert "Repos: add python2-ovsdbapp to the reposync config" ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.add_hosts Error Message: Host lago-hc-basic-suite-master-host-1 is in non operational state -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f5598098ed8> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4 raise RuntimeError('Host %s is in non operational state' % api_host.name) RuntimeError: Host lago-hc-basic-suite-master-host-1 is in non operational state --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 228, in add_hosts add_hosts_4(prefix) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 275, in add_hosts_4 testlib.assert_true_within(_host_is_up_4, timeout=15*60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4 raise RuntimeError('Host %s is in non operational state' % api_host.name) 'Host lago-hc-basic-suite-master-host-1 is in non operational state\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function _host_is_up_4 at 0x7f5598098ed8>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-master/ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 258, in _host_is_up_4\n raise RuntimeError(\'Host %s is in non operational state\' % api_host.name)\nRuntimeError: Host lago-hc-basic-suite-master-host-1 is in non operational state\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_hc-basic-suite-4.2 - Build # 516 - Failure!
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_hc-basic-suite-4.2/516/ Build Number: 516 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #516 [Miguel Duarte Barroso] Revert "Repos: add python2-ovsdbapp to the reposync config" ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.snapshot_merge Error Message: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 69, in wrapper return func(prefix.virt_env.engine_vm().get_api(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_hc-basic-suite-4.2/ovirt-system-tests/hc-basic-suite-4.2/test-scenarios/004_basic_sanity.py", line 194, in snapshot_merge api.vms.get(VM0_NAME).snapshots.add(dead_snap2_params) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/brokers.py", line 34511, in add headers={"Correlation-Id":correlation_id, "Expect":expect} File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 79, in add return self.request('POST', url, body, headers, cls=cls) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 122, in request persistent_auth=self.__persistent_auth File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request persistent_auth) File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 162, in __do_request raise errors.RequestError(response_code, response_reason, response_body) RequestError: status: 409 reason: Conflict detail: Cannot create Snapshot. Snapshot is currently being created for VM vm0.

1 1

oVirt infra daily report - unstable production jobs - 597
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/597//artifact/exported-a… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Failed to run project_setup.sh for: #2040 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv06.infra-phx.ovirt.org-container-1.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Failed to run project_setup.sh for: #2040 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv06.infra-phx.ovirt.org-container-6.

1 0

[JENKINS] Failed to setup proejct kubevirt_kubevirt_standard-check-pr
by jenkins＠jenkins.phx.ovirt.org 28 Sep '18

28 Sep '18

Failed to run project_setup.sh for: #2040 kubevirt [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on ovirt-srv05.phx.ovirt.org-container-1.

1 0

[JIRA] (OVIRT-2514) ovirt-engine failing on 4.2 in initialize_engine test for upgrade suite.
by Dafna Ron (oVirt JIRA) 28 Sep '18

28 Sep '18

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2514?page=com.atlassian.jira.… ] Dafna Ron commented on OVIRT-2514: ---------------------------------- Dominik Holler 8:27 PM (42 minutes ago) to me, Ales, Piotr, devel, infra https://gerrit.ovirt.org/#/c/94582/ fixes the issue > ovirt-engine failing on 4.2 in initialize_engine test for upgrade suite. > ------------------------------------------------------------------------- > > Key: OVIRT-2514 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2514 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Dafna Ron > Assignee: infra > Priority: Highest > Labels: ost_code_regression, ost_failures > > mail sent to developers. > seems related to CQ reported change. > https://gerrit.ovirt.org/#/c/94551/1 > http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3234/ -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100092)

1 0

[ OST Failure Report ] [ oVirt 4.2 (ovirt-engine) ] [ 27-09-20118 ] [ initialize_engine ]
by Dafna Ron 28 Sep '18

28 Sep '18

Hi, we are failing on ovirt-engine 4.1 on the upgrade suite. The issue seems to be related to this change: https://gerrit.ovirt.org/#/c/94551/ - packaging: Generate random MAC pool instead of hardcoded one Can you please have a look and issue a fix? Build log: https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3234/testReport… error: [ INFO ] Yum Verify: 100/100: ovirt-engine-tools.noarch 0:4.1.9.1-1.el7.centos - ud [ INFO ] Stage: Misc configuration [ INFO ] Upgrading CA [ INFO ] Installing PostgreSQL uuid-ossp extension into database [ INFO ] Creating/refreshing DWH database schema [ INFO ] Configuring WebSocket Proxy [ INFO ] Creating/refreshing Engine database schema [ INFO ] Creating/refreshing Engine 'internal' domain database schema Unregistering existing client registration info. [ INFO ] Creating default mac pool [ ERROR ] Failed to execute stage 'Misc configuration': insert or update on table "mac_pool_ranges" violates foreign key constraint "mac_pool_ranges_mac_pool_id_fkey" DETAIL: Key (mac_pool_id)=(58ca604b-017d-0374-0220-00000000014e) is not present in table "mac_pools". CONTEXT: SQL statement "INSERT INTO mac_pool_ranges ( mac_pool_id, from_mac, to_mac ) VALUES ( v_mac_pool_id, v_from_mac, v_to_mac )" PL/pgSQL function insertmacpoolrange(uuid,character varying,character varying) line 3 at SQL statement [ INFO ] Rolling back to the previous PostgreSQL instance (postgresql). [ INFO ] Stage: Clean up Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20180927090017-97fd5u.log [ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20180927090149-setup.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Execution of setup failed ('FATAL Internal error (main): insert or update on table "mac_pool_ranges" violates foreign key constraint "mac_pool_ranges_mac_pool_id_fkey"\nDETAIL: Key (mac_pool_id)=(58ca604b-017d-0374-0220-00000000014e) is not present in table "mac_pools".\nCONTEXT: SQL statement "INSERT INTO mac_pool_ranges (\n mac_pool_id,\n from_mac,\n to_mac\n )\n VALUES (\n v_mac_pool_id,\n v_from_mac,\n v_to_mac\n )"\nPL/pgSQL function insertmacpoolrange(uuid,character varying,character varying) line 3 at SQL statement\n',) lago.ssh: DEBUG: Command 483aadd2 on lago-upgrade-from-prevrelease-suite-4-2-engine errors: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/__main__.py", line 88, in main installer.execute() File "/usr/lib/python2.7/site-packages/otopi/main.py", line 157, in execute self.context.runSequence() File "/usr/lib/python2.7/site-packages/otopi/context.py", line 771, in runSequence util.raiseExceptionInformation(infos[0]) File "/usr/lib/python2.7/site-packages/otopi/util.py", line 81, in raiseExceptionInformation exec('raise info[1], None, info[2]') File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/network/macpools.py", line 98, in _misc_db_entries self._create_new_mac_pool_range(range_prefix) File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/network/macpools.py", line 73, in _create_new_mac_pool_range to_mac=range_prefix + ':ff:ff', File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 266, in execute args, IntegrityError: insert or update on table "mac_pool_ranges" violates foreign key constraint "mac_pool_ranges_mac_pool_id_fkey" DETAIL: Key (mac_pool_id)=(58ca604b-017d-0374-0220-00000000014e) is not present in table "mac_pools". CONTEXT: SQL statement "INSERT INTO mac_pool_ranges ( mac_pool_id, from_mac, to_mac ) VALUES ( v_mac_pool_id, v_from_mac, v_to_mac )" PL/pgSQL function insertmacpoolrange(uuid,character varying,character varying) line 3 at SQL statement Thanks, Dafna

2 4

[CQ]: 94571, 3 (ovirt-engine) failed "ovirt-4.2" system tests, but isn't the failure root cause
by oVirt Jenkins 28 Sep '18

28 Sep '18

A system test invoked by the "ovirt-4.2" change queue including change 94571,3 (ovirt-engine) failed. However, this change seems not to be the root cause for this failure. Change 94551,1 (ovirt-engine) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 94551,1 (ovirt-engine) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/94571/3 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/94551/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3249/

1 0