- Infra - oVirt List Archives

[oVirt Jenkins] cleanup-gated-ovirt-master - Build #241 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-master/241/ Build Name: #241 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 271 - Failure!
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 271 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #271 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_suspend_resume_vm0 Error Message: LagoSSHTimeoutException: Timed out (in 114 s) trying to ssh to 192.168.201.218 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7fc930d9fc10> @order_by(_TEST_LIST) def test_suspend_resume_vm0(prefix): vm_host = _vm_host(prefix, VM0_NAME) ret = vm_host.ssh(['tail', '-1', VDSM_LOG]) assert ret.code == EX_OK log_items = ret.out.decode('utf-8').split() global _log_time_before_suspend _log_time_before_suspend = log_items[0] + ' ' + log_items[1] # date + time > assert_vm0_is_alive(prefix) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:191: in assert_vm0_is_alive assert_vm_is_alive(prefix, VM0_NAME) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:199: in assert_vm_is_alive assert _vm_ssh(prefix, vm_hostname, ['true']).code == EX_OK ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:186: in _vm_ssh tries=tries, /usr/lib/python2.7/site-packages/lago/ssh.py:46: in ssh password=password, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ip_addr = '192.168.201.218', ssh_key = None, host_name = '192.168.201.218' ssh_tries = 0, propagate_fail = True, username = 'cirros', password = 'gocubsgo' def get_ssh_client( ip_addr, ssh_key=None, host_name=None, ssh_tries=None, propagate_fail=True, username='root', password='123456', ): """ Get a connected SSH client Args: ip_addr(str): IP address of the endpoint ssh_key(str or list of str): Path to a file which contains the private key hotname(str): The hostname of the endpoint ssh_tries(int): The number of attempts to connect to the endpoint propagate_fail(bool): If set to true, this event will be in the log and fail the outer stage. Otherwise, it will be discarded. username(str): The username to authenticate with password(str): Used for password authentication or for private key decryption Raises: :exc:`~LagoSSHTimeoutException`: If the client failed to connect after "ssh_tries" """ host_name = host_name or ip_addr with LogTask( 'Get ssh client for %s' % host_name, level='debug', propagate_fail=propagate_fail, ): ssh_timeout = int(config.get('ssh_timeout')) if ssh_tries is None: ssh_tries = int(config.get('ssh_tries', 10)) start_time = time.time() client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy(), ) while ssh_tries > 0: try: client.connect( ip_addr, username=username, password=password, key_filename=ssh_key, timeout=ssh_timeout, ) break except (socket.error, socket.timeout) as err: LOGGER.debug( 'Socket error connecting to %s: %s', host_name, err, ) except paramiko.ssh_exception.SSHException as err: LOGGER.debug( 'SSH error connecting to %s: %s', host_name, err, ) except EOFError as err: LOGGER.debug('EOFError connecting to %s: %s', host_name, err) ssh_tries -= 1 LOGGER.debug( 'Still got %d tries for %s', ssh_tries, host_name, ) time.sleep(1) else: end_time = time.time() raise LagoSSHTimeoutException( 'Timed out (in %d s) trying to ssh to %s' % > (end_time - start_time, host_name) ) E LagoSSHTimeoutException: Timed out (in 114 s) trying to ssh to 192.168.201.218 /usr/lib/python2.7/site-packages/lago/ssh.py:388: LagoSSHTimeoutException

1 1

oVirt infra daily report - unstable production jobs - 1280
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1280//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2991) [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 11 Aug '20

11 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2991?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2991: -------------------------------- Status: To Do (was: In Progress) > [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv05.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv05.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv05.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv05.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[CQ]: 110715,2 (jenkins) failed "dummy-master" system tests
by oVirt Jenkins 11 Aug '20

11 Aug '20

Change 110715,2 (jenkins) is probably the reason behind recent system test failures in the "dummy-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/110715/2 For failed test results see: https://jenkins.ovirt.org/job/dummy-master_change-queue-tester/523/

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 269 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Aug '20

11 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 269 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #269 [Michal Skrivanek] smaller OST VMs [Barak Korren] Dropped FCRAW tests for STDCI [Barak Korren] usrc: expand test for 'files' provider ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_suspend_resume_vm0 Error Message: LagoSSHTimeoutException: Timed out (in 328 s) trying to ssh to 192.168.201.117 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f3115f60c10> @order_by(_TEST_LIST) def test_suspend_resume_vm0(prefix): vm_host = _vm_host(prefix, VM0_NAME) ret = vm_host.ssh(['tail', '-1', VDSM_LOG]) assert ret.code == EX_OK log_items = ret.out.decode('utf-8').split() global _log_time_before_suspend _log_time_before_suspend = log_items[0] + ' ' + log_items[1] # date + time > assert_vm0_is_alive(prefix) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:191: in assert_vm0_is_alive assert_vm_is_alive(prefix, VM0_NAME) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:199: in assert_vm_is_alive assert _vm_ssh(prefix, vm_hostname, ['true']).code == EX_OK ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:186: in _vm_ssh tries=tries, /usr/lib/python2.7/site-packages/lago/ssh.py:46: in ssh password=password, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ip_addr = '192.168.201.117', ssh_key = None, host_name = '192.168.201.117' ssh_tries = 0, propagate_fail = True, username = 'cirros', password = 'gocubsgo' def get_ssh_client( ip_addr, ssh_key=None, host_name=None, ssh_tries=None, propagate_fail=True, username='root', password='123456', ): """ Get a connected SSH client Args: ip_addr(str): IP address of the endpoint ssh_key(str or list of str): Path to a file which contains the private key hotname(str): The hostname of the endpoint ssh_tries(int): The number of attempts to connect to the endpoint propagate_fail(bool): If set to true, this event will be in the log and fail the outer stage. Otherwise, it will be discarded. username(str): The username to authenticate with password(str): Used for password authentication or for private key decryption Raises: :exc:`~LagoSSHTimeoutException`: If the client failed to connect after "ssh_tries" """ host_name = host_name or ip_addr with LogTask( 'Get ssh client for %s' % host_name, level='debug', propagate_fail=propagate_fail, ): ssh_timeout = int(config.get('ssh_timeout')) if ssh_tries is None: ssh_tries = int(config.get('ssh_tries', 10)) start_time = time.time() client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy(), ) while ssh_tries > 0: try: client.connect( ip_addr, username=username, password=password, key_filename=ssh_key, timeout=ssh_timeout, ) break except (socket.error, socket.timeout) as err: LOGGER.debug( 'Socket error connecting to %s: %s', host_name, err, ) except paramiko.ssh_exception.SSHException as err: LOGGER.debug( 'SSH error connecting to %s: %s', host_name, err, ) except EOFError as err: LOGGER.debug('EOFError connecting to %s: %s', host_name, err) ssh_tries -= 1 LOGGER.debug( 'Still got %d tries for %s', ssh_tries, host_name, ) time.sleep(1) else: end_time = time.time() raise LagoSSHTimeoutException( 'Timed out (in %d s) trying to ssh to %s' % > (end_time - start_time, host_name) ) E LagoSSHTimeoutException: Timed out (in 328 s) trying to ssh to 192.168.201.117 /usr/lib/python2.7/site-packages/lago/ssh.py:388: LagoSSHTimeoutException

1 1

oVirt infra daily report - unstable production jobs - 1279
by jenkins＠jenkins.phx.ovirt.org 11 Aug '20

11 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1279//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2995) [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci)
by Alertmanager_Bot (oVirt JIRA) 10 Aug '20

10 Aug '20

Alertmanager_Bot created OVIRT-2995: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci) Key: OVIRT-2995 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2995 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = ibm-srv01.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = ibm-srv01.ovirt.org - node_role_kubernetes_io_compute = true - region = external - type = bare-metal-external - zone = ci Annotations: - description = ibm-srv01.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2992) [FIRING:1] TestAlert
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2992?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2992: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] TestAlert > --------------------- > > Key: OVIRT-2992 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2992 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="TestAlert"} > > Labels: > - alertname = TestAlert > - label1 = value1 > Annotations: > Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2994) Occasionally we see alerts on dead node-exporters on OpenShift nodes
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

Shlomi Zidmi created OVIRT-2994: ----------------------------------- Summary: Occasionally we see alerts on dead node-exporters on OpenShift nodes Key: OVIRT-2994 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2994 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Shlomi Zidmi Assignee: infra Priority: Low We need to check what is happening behind the scenes when node-exporters are crashing and probably other containers too. Most times OpenShift is able to recover automatically, yet sometimes we have to fix it manually. Creating a ticket for investigating this issue -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0