November 2020 - Infra - oVirt List Archives

[JIRA] (OVIRT-3053) [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources)
by Shlomi Zidmi (oVirt JIRA) 09 Nov '20

09 Nov '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3053?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-3053: -------------------------------- Resolution: Fixed Status: Done (was: To Do) DIsk now has a sufficient amount of free space > [FIRING:1] LowDiskSpace_NodeExporter_v1_0 resources.ovirt.org:9100 ovirt-resources (/dev/mapper/resources_lvm-data ext4 /srv/resources) > --------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3053 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3053 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="LowDiskSpace_NodeExporter_v1_0",instance="resources.ovirt.org:9100",job="ovirt-resources"} > > Labels: > - alertname = LowDiskSpace_NodeExporter_v1_0 > - device = /dev/mapper/resources_lvm-data > - fstype = ext4 > - instance = resources.ovirt.org:9100 > - job = ovirt-resources > - mountpoint = /srv/resources > Annotations: > - description = resources.ovirt.org:9100 of job ovirt-resources has less than 10% free disk space > Source: http://prometheus-0:9090/graph?g0.expr=node_filesystem_free_bytes+%2F+node_… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100149)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 561 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 561 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #561 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity.test_run_vms Error Message: Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: assert_vm_is_alive = <function is_alive at 0x7fc7d597b488> engine_api = <ovirtsdk4.Connection object at 0x7fc7d5979a50> management_gw_ip = '192.168.201.1' @order_by(_TEST_LIST) def test_run_vms(assert_vm_is_alive, engine_api, management_gw_ip): engine = engine_api.system_service() vm_params = types.Vm( initialization=types.Initialization( user_name=VM_USER_NAME, root_password=VM_PASSWORD ) ) vm_params.initialization.host_name = BACKUP_VM_NAME backup_vm_service = test_utils.get_vm_service(engine, BACKUP_VM_NAME) backup_vm_service.start(use_cloud_init=True, vm=vm_params) vm_params.initialization.host_name = VM2_NAME vm2_service = test_utils.get_vm_service(engine, VM2_NAME) vm2_service.start(use_cloud_init=True, vm=vm_params) # CirrOS cloud-init is different, networking doesn't work since it doesn't support the format oVirt is using vm_params.initialization.host_name = VM0_NAME # hostname seems to work, the others not vm_params.initialization.dns_search = 'lago.local' vm_params.initialization.domain = 'lago.local' vm_params.initialization.dns_servers = management_gw_ip vm0_service = test_utils.get_vm_service(engine, VM0_NAME) > vm0_service.start(use_cloud_init=True, vm=vm_params) ../basic-suite-master/test-scenarios/004_basic_sanity.py:754: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:32520: in start return self._internal_action(action, 'start', None, headers, query, wait) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in _internal_action return future.wait() if wait else future /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback self._check_fault(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault self._raise_error(response, body.fault) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <ovirtsdk4.http.Response object at 0x7fc7d6aefad0> detail = <ovirtsdk4.types.Fault object at 0x7fc7d6aefb50> @staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault. This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.' class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError error = class_(msg) error.code = response.code if response else None error.fault = fault > raise error E Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118: Error

2 2

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 540 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/54… Build Number: 540 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #540 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 697, in run_vms _verify_vm_state(engine, vm_name, types.VmStatus.UP) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 102, in _verify_vm_state lambda: File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 496 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 496 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #496 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 537 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/53… Build Number: 537 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #537 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.run_vms Error Message: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400. Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 694, in run_vms vm0_service.start(use_cloud_init=True, vm=vm_params) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py", line 32520, in start return self._internal_action(action, 'start', None, headers, query, wait) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 299, in _internal_action return future.wait() if wait else future File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 55, in wait return self._code(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 296, in callback self._check_fault(response) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 134, in _check_fault self._raise_error(response, body.fault) File "/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py", line 118, in _raise_error raise error Error: Fault reason is "Operation Failed". Fault detail is "[Network error during communication with the Host.]". HTTP response code is 400.

1 2

[oVirt Jenkins] ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly - Build # 495 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build Number: 495 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #495 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly/ovirt-system-tests/upgrade-from-prevrelease-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

oVirt infra daily report - unstable production jobs - 1369
by jenkins＠jenkins.phx.ovirt.org 09 Nov '20

09 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1369//artifact/exported… Cheers, Jenkins

1 0

[JENKINS] Failed to setup proejct standard-manual-runner
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Failed to run project_setup.sh for: #1594 vdsm [check-patch]. It probably means that docker_cleanup.py failed. This step doesn't fail the job, but we do collect data about such failures to find the root cause. Infra owner, ensure that we're not running out of disk space on vm0141.workers-phx.ovirt.org.

1 0

[oVirt Jenkins] ovirt-system-tests_upgrade-from-release-suite-4.3_nightly - Build # 493 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-release-suite… Build Number: 493 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #493 [Marcin Sobczyk] basic: Report start times and runnings time of tests ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.vm_run Error Message: False != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_upgrade-from-release-suite-4.3_nightly/ovirt-system-tests/upgrade-from-release-suite-4.3/test-scenarios-after-upgrade/004_basic_sanity.py", line 107, in vm_run lambda: api.vms.get(VM0_NAME).status.state == 'up', File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds

1 1

oVirt infra daily report - unstable production jobs - 1368
by jenkins＠jenkins.phx.ovirt.org 08 Nov '20

08 Nov '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1368//artifact/exported… Cheers, Jenkins

1 0