- Infra - oVirt List Archives

[oVirt Jenkins] cleanup-gated-ovirt-master - Build #241 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-master/241/ Build Name: #241 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 271 - Failure!
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 271 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #271 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_suspend_resume_vm0 Error Message: LagoSSHTimeoutException: Timed out (in 114 s) trying to ssh to 192.168.201.218 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7fc930d9fc10> @order_by(_TEST_LIST) def test_suspend_resume_vm0(prefix): vm_host = _vm_host(prefix, VM0_NAME) ret = vm_host.ssh(['tail', '-1', VDSM_LOG]) assert ret.code == EX_OK log_items = ret.out.decode('utf-8').split() global _log_time_before_suspend _log_time_before_suspend = log_items[0] + ' ' + log_items[1] # date + time > assert_vm0_is_alive(prefix) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:191: in assert_vm0_is_alive assert_vm_is_alive(prefix, VM0_NAME) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:199: in assert_vm_is_alive assert _vm_ssh(prefix, vm_hostname, ['true']).code == EX_OK ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:186: in _vm_ssh tries=tries, /usr/lib/python2.7/site-packages/lago/ssh.py:46: in ssh password=password, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ip_addr = '192.168.201.218', ssh_key = None, host_name = '192.168.201.218' ssh_tries = 0, propagate_fail = True, username = 'cirros', password = 'gocubsgo' def get_ssh_client( ip_addr, ssh_key=None, host_name=None, ssh_tries=None, propagate_fail=True, username='root', password='123456', ): """ Get a connected SSH client Args: ip_addr(str): IP address of the endpoint ssh_key(str or list of str): Path to a file which contains the private key hotname(str): The hostname of the endpoint ssh_tries(int): The number of attempts to connect to the endpoint propagate_fail(bool): If set to true, this event will be in the log and fail the outer stage. Otherwise, it will be discarded. username(str): The username to authenticate with password(str): Used for password authentication or for private key decryption Raises: :exc:`~LagoSSHTimeoutException`: If the client failed to connect after "ssh_tries" """ host_name = host_name or ip_addr with LogTask( 'Get ssh client for %s' % host_name, level='debug', propagate_fail=propagate_fail, ): ssh_timeout = int(config.get('ssh_timeout')) if ssh_tries is None: ssh_tries = int(config.get('ssh_tries', 10)) start_time = time.time() client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy(), ) while ssh_tries > 0: try: client.connect( ip_addr, username=username, password=password, key_filename=ssh_key, timeout=ssh_timeout, ) break except (socket.error, socket.timeout) as err: LOGGER.debug( 'Socket error connecting to %s: %s', host_name, err, ) except paramiko.ssh_exception.SSHException as err: LOGGER.debug( 'SSH error connecting to %s: %s', host_name, err, ) except EOFError as err: LOGGER.debug('EOFError connecting to %s: %s', host_name, err) ssh_tries -= 1 LOGGER.debug( 'Still got %d tries for %s', ssh_tries, host_name, ) time.sleep(1) else: end_time = time.time() raise LagoSSHTimeoutException( 'Timed out (in %d s) trying to ssh to %s' % > (end_time - start_time, host_name) ) E LagoSSHTimeoutException: Timed out (in 114 s) trying to ssh to 192.168.201.218 /usr/lib/python2.7/site-packages/lago/ssh.py:388: LagoSSHTimeoutException

1 1

oVirt infra daily report - unstable production jobs - 1280
by jenkins＠jenkins.phx.ovirt.org 12 Aug '20

12 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1280//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2991) [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 11 Aug '20

11 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2991?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2991: -------------------------------- Status: To Do (was: In Progress) > [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv05.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv05.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv05.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv05.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[CQ]: 110715,2 (jenkins) failed "dummy-master" system tests
by oVirt Jenkins 11 Aug '20

11 Aug '20

Change 110715,2 (jenkins) is probably the reason behind recent system test failures in the "dummy-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/110715/2 For failed test results see: https://jenkins.ovirt.org/job/dummy-master_change-queue-tester/523/

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 269 - Failure!
by jenkins＠jenkins.phx.ovirt.org 11 Aug '20

11 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 269 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #269 [Michal Skrivanek] smaller OST VMs [Barak Korren] Dropped FCRAW tests for STDCI [Barak Korren] usrc: expand test for 'files' provider ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_suspend_resume_vm0 Error Message: LagoSSHTimeoutException: Timed out (in 328 s) trying to ssh to 192.168.201.117 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f3115f60c10> @order_by(_TEST_LIST) def test_suspend_resume_vm0(prefix): vm_host = _vm_host(prefix, VM0_NAME) ret = vm_host.ssh(['tail', '-1', VDSM_LOG]) assert ret.code == EX_OK log_items = ret.out.decode('utf-8').split() global _log_time_before_suspend _log_time_before_suspend = log_items[0] + ' ' + log_items[1] # date + time > assert_vm0_is_alive(prefix) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:191: in assert_vm0_is_alive assert_vm_is_alive(prefix, VM0_NAME) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:199: in assert_vm_is_alive assert _vm_ssh(prefix, vm_hostname, ['true']).code == EX_OK ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:186: in _vm_ssh tries=tries, /usr/lib/python2.7/site-packages/lago/ssh.py:46: in ssh password=password, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ip_addr = '192.168.201.117', ssh_key = None, host_name = '192.168.201.117' ssh_tries = 0, propagate_fail = True, username = 'cirros', password = 'gocubsgo' def get_ssh_client( ip_addr, ssh_key=None, host_name=None, ssh_tries=None, propagate_fail=True, username='root', password='123456', ): """ Get a connected SSH client Args: ip_addr(str): IP address of the endpoint ssh_key(str or list of str): Path to a file which contains the private key hotname(str): The hostname of the endpoint ssh_tries(int): The number of attempts to connect to the endpoint propagate_fail(bool): If set to true, this event will be in the log and fail the outer stage. Otherwise, it will be discarded. username(str): The username to authenticate with password(str): Used for password authentication or for private key decryption Raises: :exc:`~LagoSSHTimeoutException`: If the client failed to connect after "ssh_tries" """ host_name = host_name or ip_addr with LogTask( 'Get ssh client for %s' % host_name, level='debug', propagate_fail=propagate_fail, ): ssh_timeout = int(config.get('ssh_timeout')) if ssh_tries is None: ssh_tries = int(config.get('ssh_tries', 10)) start_time = time.time() client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy(), ) while ssh_tries > 0: try: client.connect( ip_addr, username=username, password=password, key_filename=ssh_key, timeout=ssh_timeout, ) break except (socket.error, socket.timeout) as err: LOGGER.debug( 'Socket error connecting to %s: %s', host_name, err, ) except paramiko.ssh_exception.SSHException as err: LOGGER.debug( 'SSH error connecting to %s: %s', host_name, err, ) except EOFError as err: LOGGER.debug('EOFError connecting to %s: %s', host_name, err) ssh_tries -= 1 LOGGER.debug( 'Still got %d tries for %s', ssh_tries, host_name, ) time.sleep(1) else: end_time = time.time() raise LagoSSHTimeoutException( 'Timed out (in %d s) trying to ssh to %s' % > (end_time - start_time, host_name) ) E LagoSSHTimeoutException: Timed out (in 328 s) trying to ssh to 192.168.201.117 /usr/lib/python2.7/site-packages/lago/ssh.py:388: LagoSSHTimeoutException

1 1

oVirt infra daily report - unstable production jobs - 1279
by jenkins＠jenkins.phx.ovirt.org 11 Aug '20

11 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1279//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2995) [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci)
by Alertmanager_Bot (oVirt JIRA) 10 Aug '20

10 Aug '20

Alertmanager_Bot created OVIRT-2995: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci) Key: OVIRT-2995 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2995 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = ibm-srv01.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = ibm-srv01.ovirt.org - node_role_kubernetes_io_compute = true - region = external - type = bare-metal-external - zone = ci Annotations: - description = ibm-srv01.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2992) [FIRING:1] TestAlert
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2992?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2992: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] TestAlert > --------------------- > > Key: OVIRT-2992 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2992 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="TestAlert"} > > Labels: > - alertname = TestAlert > - label1 = value1 > Annotations: > Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2994) Occasionally we see alerts on dead node-exporters on OpenShift nodes
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

Shlomi Zidmi created OVIRT-2994: ----------------------------------- Summary: Occasionally we see alerts on dead node-exporters on OpenShift nodes Key: OVIRT-2994 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2994 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Shlomi Zidmi Assignee: infra Priority: Low We need to check what is happening behind the scenes when node-exporters are crashing and probably other containers too. Most times OpenShift is able to recover automatically, yet sometimes we have to fix it manually. Creating a ticket for investigating this issue -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2993) [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci)
by Alertmanager_Bot (oVirt JIRA) 10 Aug '20

10 Aug '20

Alertmanager_Bot created OVIRT-2993: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci) Key: OVIRT-2993 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2993 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = shift-n11.phx.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = shift-n11.phx.ovirt.org - node_role_kubernetes_io_compute = true - region = primary - type = vm - zone = ci Annotations: - description = shift-n11.phx.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 267 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/26… Build Number: 267 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #267 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_basic_ui_sanity.start_grid Error Message: could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local creating docker network for grid executing shell: docker network create grid 2e5f4fc816f6d795c564a5dab8b93c3b94818167957e7e65d94b37563da512d5 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local 2e5f4fc816f6 grid bridge local b3331b0f16ef host host local e00fb324e999 none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer edcd5e9f2f91: Waiting 0eeaf787f757: Waiting c949dee5af7e: Pulling fs layer 06c6d2f59700: Waiting df88a49b4162: Pulling fs layer c949dee5af7e: Waiting ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer 6c8e191c3abf: Waiting c9aca8b50247: Pulling fs layer c9aca8b50247: Waiting 4be3072e5a37: Waiting ce3c6f42fd24: Waiting df88a49b4162: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.7:53: read udp 10.131.3.159:57412->172.19.11.7:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ---------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 189, in start_grid raise RuntimeError("could not get ip address of selenium hub. See previous messages for probable docker failure") could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local creating docker network for grid executing shell: docker network create grid 2e5f4fc816f6d795c564a5dab8b93c3b94818167957e7e65d94b37563da512d5 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local 2e5f4fc816f6 grid bridge local b3331b0f16ef host host local e00fb324e999 none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer edcd5e9f2f91: Waiting 0eeaf787f757: Waiting c949dee5af7e: Pulling fs layer 06c6d2f59700: Waiting df88a49b4162: Pulling fs layer c949dee5af7e: Waiting ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer 6c8e191c3abf: Waiting c9aca8b50247: Pulling fs layer c9aca8b50247: Waiting 4be3072e5a37: Waiting ce3c6f42fd24: Waiting df88a49b4162: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.7:53: read udp 10.131.3.159:57412->172.19.11.7:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ----------------------

1 1

[oVirt Jenkins] cleanup-gated-dummy-master - Build #233 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-dummy-master/233/ Build Name: #233 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 264 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 264 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #264 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_hotplug_memory Error Message: assert 323 == 352 -323 +352 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f229a1ffc10> hotplug_mem_amount = 268435456 @order_by(_TEST_LIST) def test_hotplug_memory(prefix, hotplug_mem_amount): api_v4 = prefix.virt_env.engine_vm().get_api_v4() engine = api_v4.system_service() vm_service = test_utils.get_vm_service(engine, VM0_NAME) new_memory = vm_service.get().memory + hotplug_mem_amount with test_utils.TestEvent(engine, 2039): # HOT_SET_MEMORY(2,039) vm_service.update( vm=types.Vm( memory=new_memory ) ) assert vm_service.get().memory == new_memory assert_vm0_is_alive(prefix) > assert _vm_libvirt_memory_amount(prefix, VM0_NAME) // KB == new_memory // MB E assert 323 == 352 E -323 E +352 ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1050: AssertionError

1 2

oVirt infra daily report - unstable production jobs - 1278
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1278//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 261 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Aug '20

09 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 261 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #261 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_live_storage_migration Error Message: AssertionError: False != True after 600 seconds Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7f6cfbc40810> @order_by(_TEST_LIST) def test_live_storage_migration(api_v4): engine = api_v4.system_service() disk_service = test_utils.get_disk_service(engine, DISK0_NAME) correlation_id = 'live_storage_migration' disk_service.move( async=False, filter=False, storage_domain=types.StorageDomain( name=SD_ISCSI_NAME ), query={'correlation_id': correlation_id} ) testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) # Assert that the disk is on the correct storage domain, # its status is OK and the snapshot created for the migration # has been merged testlib.assert_true_within_long( lambda: api_v4.follow_link(disk_service.get().storage_domains[0]).name == SD_ISCSI_NAME ) vm0_snapshots_service = test_utils.get_vm_snapshots_service(engine, VM0_NAME) testlib.assert_true_within_long( > lambda: len(vm0_snapshots_service.list()) == 1 ) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:585: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:286: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:273: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7f6cf85b2a28>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10 ): allowed_exceptions = allowed_exceptions or [] with utils.EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: raise AssertionError( > '%s != %s after %s seconds' % (res, value, timeout) ) E AssertionError: False != True after 600 seconds /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:252: AssertionError

1 2

oVirt infra daily report - unstable production jobs - 1277
by jenkins＠jenkins.phx.ovirt.org 09 Aug '20

09 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1277//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2991) [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2991?page=com.atlassian.jira.… ] Shlomi Zidmi reopened OVIRT-2991: --------------------------------- > [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv05.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv05.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv05.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv05.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2989) [FIRING:1] InstanceUnreachable ibm-srv03.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv03.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2989?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2989: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv03.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv03.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2989 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2989 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv03.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv03.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv03.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv03.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2991) [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2991?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2991: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv05.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv05.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2991 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2991 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv05.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv05.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv05.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv05.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2990) [FIRING:1] InstanceUnreachable ibm-srv04.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv04.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2990?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2990: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv04.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv04.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2990 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2990 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv04.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv04.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv04.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv04.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2988) [FIRING:1] InstanceUnreachable ibm-srv02.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv02.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2988?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2988: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv02.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv02.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2988 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2988 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv02.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv02.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv02.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv02.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2987) [FIRING:1] InstanceUnreachable ibm-srv06.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv06.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2987?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2987: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv06.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv06.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2987 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2987 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv06.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv06.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv06.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv06.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

[JIRA] (OVIRT-2986) [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Aug '20

08 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2986?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2986: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2986 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2986 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv01.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv01.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv01.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv01.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100141)

1 0

oVirt infra daily report - unstable production jobs - 1276
by jenkins＠jenkins.phx.ovirt.org 08 Aug '20

08 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1276//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 251 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Aug '20

07 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 251 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #251 [Martin Necas] ansible: disable iscsi ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.002_bootstrap_pytest.test_verify_add_hosts Error Message: RuntimeError: 1 hosts failed installation: lago-basic-suite-master-host-0: install_failed Stack Trace: engine_api = <ovirtsdk4.Connection object at 0x7f4db412f5d0> @order_by(_TEST_LIST) def test_verify_add_hosts(engine_api): hosts_service = engine_api.system_service().hosts_service() hosts_status = hosts_service.list(search='datacenter={}'.format(DC_NAME)) total_hosts = len(hosts_status) dump_hosts = _host_status_to_print(hosts_service, hosts_status) LOGGER.debug('Host status, verify_add_hosts:\n {}'.format(dump_hosts)) testlib.assert_true_within( lambda: _single_host_up(hosts_service, total_hosts), > timeout=constants.ADD_HOST_TIMEOUT ) ../basic-suite-master/test-scenarios/002_bootstrap_pytest.py:434: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:278: in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:234: in assert_equals_within res = func() ../basic-suite-master/test-scenarios/002_bootstrap_pytest.py:433: in <lambda> lambda: _single_host_up(hosts_service, total_hosts), ../basic-suite-master/test-scenarios/002_bootstrap_pytest.py:230: in _single_host_up _check_problematic_hosts(hosts_service) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ hosts_service = <ovirtsdk4.services.HostsService object at 0x7f4db42c8050> def _check_problematic_hosts(hosts_service): problematic_hosts = hosts_service.list(search='datacenter={} AND status != installing and status != initializing and status != up'.format(DC_NAME)) if len(problematic_hosts): dump_hosts = '%s hosts failed installation:\n' % len(problematic_hosts) for host in problematic_hosts: host_service = hosts_service.host_service(host.id) dump_hosts += '%s: %s\n' % (host.name, host_service.get().status) > raise RuntimeError(dump_hosts) E RuntimeError: 1 hosts failed installation: E lago-basic-suite-master-host-0: install_failed ../basic-suite-master/test-scenarios/002_bootstrap_pytest.py:239: RuntimeError

1 8

[oVirt Jenkins] cleanup-gated-ovirt-4.3 - Build #231 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 07 Aug '20

07 Aug '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-4.3/231/ Build Name: #231 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_performance-suite-master - Build # 1540 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Aug '20

07 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/1… Build Number: 1540 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1540 [Martin Necas] ansible: disable iscsi ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 040_add_hosts_vms.verify_add_hosts Error Message: 1 hosts failed installation: lago-performance-suite-master-host-1: install_failed -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f585db998c0> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 306, in <lambda> lambda: _single_host_up(hosts_service, total_hosts) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 197, in _single_host_up _check_problematic_hosts(hosts_service) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 206, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-performance-suite-master-host-1: install_failed --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 298, in verify_add_hosts verify_add_hosts_4(api) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 306, in verify_add_hosts_4 lambda: _single_host_up(hosts_service, total_hosts) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 306, in <lambda> lambda: _single_host_up(hosts_service, total_hosts) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 197, in _single_host_up _check_problematic_hosts(hosts_service) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 206, in _check_problematic_hosts raise RuntimeError(dump_hosts) '1 hosts failed installation:\nlago-performance-suite-master-host-1: install_failed\n\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f585db998c0>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 306, in <lambda>\n lambda: _single_host_up(hosts_service, total_hosts)\n File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 197, in _single_host_up\n _check_problematic_hosts(hosts_service)\n File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/040_add_hosts_vms.py", line 206, in _check_problematic_hosts\n raise RuntimeError(dump_hosts)\nRuntimeError: 1 hosts failed installation:\nlago-performance-suite-master-host-1: install_failed\n\n--------------------- >> end captured logging << ---------------------'

1 2

oVirt infra daily report - unstable production jobs - 1275
by jenkins＠jenkins.phx.ovirt.org 07 Aug '20

07 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1275//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1274
by jenkins＠jenkins.phx.ovirt.org 06 Aug '20

06 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1274//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1273
by jenkins＠jenkins.phx.ovirt.org 05 Aug '20

05 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1273//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1272
by jenkins＠jenkins.phx.ovirt.org 04 Aug '20

04 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1272//artifact/exported… Cheers, Jenkins

1 0

[VDSM] Trouble merging patch, related to jenkins whitelist or new RHEL CI?
by Nir Soffer 03 Aug '20

03 Aug '20

I could not merge: https://gerrit.ovirt.org/c/109402/ Although it was approved and verified, and got +1 from Continuous Integration. There is a new "RHEL-Continuous-Integration", which does not run with this change, even when I trigger the tests manually with "ci test". I tried to add +1 for "RHEL-Continuous-Integration" but this is not possible, I see: RHEL-Continuous-Integration You don't have permission to edit this label. So finally I downloaded the patch and pushed it manually. I think this patch will fix the problem: https://gerrit.ovirt.org/#/c/110576/ But I need CI experts to review this. Since we have 50 projects that need this, this probably should be fixed elsewhere, and inherited by all projects. I know that Germano was not able to trigger tests because he was missing in the jenkins whitelist, but this was fixed last week. Germano, maybe just to check that everything works for you, you can post some trivial patch? Nir

4 9

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 245 - Failure!
by jenkins＠jenkins.phx.ovirt.org 03 Aug '20

03 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 245 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #245 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_suspend_resume_vm0 Error Message: LagoSSHTimeoutException: Timed out (in 116 s) trying to ssh to 192.168.201.149 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7fdfcf775c10> @order_by(_TEST_LIST) def test_suspend_resume_vm0(prefix): vm_host = _vm_host(prefix, VM0_NAME) ret = vm_host.ssh(['tail', '-1', VDSM_LOG]) assert ret.code == EX_OK log_items = ret.out.decode('utf-8').split() global _log_time_before_suspend _log_time_before_suspend = log_items[0] + ' ' + log_items[1] # date + time > assert_vm0_is_alive(prefix) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1210: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:191: in assert_vm0_is_alive assert_vm_is_alive(prefix, VM0_NAME) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:199: in assert_vm_is_alive assert _vm_ssh(prefix, vm_hostname, ['true']).code == EX_OK ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:186: in _vm_ssh tries=tries, /usr/lib/python2.7/site-packages/lago/ssh.py:46: in ssh password=password, _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ip_addr = '192.168.201.149', ssh_key = None, host_name = '192.168.201.149' ssh_tries = 0, propagate_fail = True, username = 'cirros', password = 'gocubsgo' def get_ssh_client( ip_addr, ssh_key=None, host_name=None, ssh_tries=None, propagate_fail=True, username='root', password='123456', ): """ Get a connected SSH client Args: ip_addr(str): IP address of the endpoint ssh_key(str or list of str): Path to a file which contains the private key hotname(str): The hostname of the endpoint ssh_tries(int): The number of attempts to connect to the endpoint propagate_fail(bool): If set to true, this event will be in the log and fail the outer stage. Otherwise, it will be discarded. username(str): The username to authenticate with password(str): Used for password authentication or for private key decryption Raises: :exc:`~LagoSSHTimeoutException`: If the client failed to connect after "ssh_tries" """ host_name = host_name or ip_addr with LogTask( 'Get ssh client for %s' % host_name, level='debug', propagate_fail=propagate_fail, ): ssh_timeout = int(config.get('ssh_timeout')) if ssh_tries is None: ssh_tries = int(config.get('ssh_tries', 10)) start_time = time.time() client = paramiko.SSHClient() client.set_missing_host_key_policy(paramiko.AutoAddPolicy(), ) while ssh_tries > 0: try: client.connect( ip_addr, username=username, password=password, key_filename=ssh_key, timeout=ssh_timeout, ) break except (socket.error, socket.timeout) as err: LOGGER.debug( 'Socket error connecting to %s: %s', host_name, err, ) except paramiko.ssh_exception.SSHException as err: LOGGER.debug( 'SSH error connecting to %s: %s', host_name, err, ) except EOFError as err: LOGGER.debug('EOFError connecting to %s: %s', host_name, err) ssh_tries -= 1 LOGGER.debug( 'Still got %d tries for %s', ssh_tries, host_name, ) time.sleep(1) else: end_time = time.time() raise LagoSSHTimeoutException( 'Timed out (in %d s) trying to ssh to %s' % > (end_time - start_time, host_name) ) E LagoSSHTimeoutException: Timed out (in 116 s) trying to ssh to 192.168.201.149 /usr/lib/python2.7/site-packages/lago/ssh.py:388: LagoSSHTimeoutException

1 2

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 246 - Failure!
by jenkins＠jenkins.phx.ovirt.org 03 Aug '20

03 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/24… Build Number: 246 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #246 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_basic_ui_sanity.initialize_chrome Error Message: Message: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, platform: LINUX, version: 64.0.3282.140} Stacktrace: at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:117) at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:84) at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:68) at javax.servlet.http.HttpServlet.service (HttpServlet.java:707) at javax.servlet.http.HttpServlet.service (HttpServlet.java:790) at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:860) at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:535) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188) at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188) at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1253) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:168) at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473) at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:166) at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1155) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:141) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.Server.handle (Server.java:530) at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:347) at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:256) at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:279) at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:102) at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:124) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:247) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce (EatWhatYouKill.java:140) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:131) at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:382) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:708) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:626) at java.lang.Thread.run (Thread.java:748) -------------------- >> begin captured stdout << --------------------- _init_browser, connecting to hub at http://172.18.0.2:4444/wd/hub Message: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, platform: LINUX, version: 64.0.3282.140} Stacktrace: at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:117) at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:84) at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:68) at javax.servlet.http.HttpServlet.service (HttpServlet.java:707) at javax.servlet.http.HttpServlet.service (HttpServlet.java:790) at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:860) at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:535) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188) at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188) at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1253) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:168) at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473) at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564) at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:166) at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1155) at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:141) at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132) at org.seleniumhq.jetty9.server.Server.handle (Server.java:530) at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:347) at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:256) at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:279) at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:102) at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:124) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:247) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce (EatWhatYouKill.java:140) at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:131) at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:382) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:708) at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:626) at java.lang.Thread.run (Thread.java:748) WebDriverException in _init_browser connecting to hub --------------------- >> end captured stdout << ---------------------- -------------------- >> begin captured logging << -------------------- selenium.webdriver.remote.remote_connection: DEBUG: POST http://172.18.0.2:4444/wd/hub/session {"capabilities": {"alwaysMatch": {"browserName": "chrome", "platformName": "linux", "browserVersion": "64.0.3282.140"}, "firstMatch": [{}]}, "desiredCapabilities": {"platform": "LINUX", "browserName": "chrome", "version": "64.0.3282.140"}} urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 172.18.0.2:4444 urllib3.connectionpool: DEBUG: http://172.18.0.2:4444 "POST /wd/hub/session HTTP/1.1" 500 7806 selenium.webdriver.remote.remote_connection: DEBUG: Finished Request --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 223, in initialize_chrome _init_driver(_get_chrome_capabilities()) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 249, in _init_driver raise e 'Message: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, platform: LINUX, version: 64.0.3282.140}\nStacktrace:\n at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:117)\n at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:84)\n at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:68)\n at javax.servlet.http.HttpServlet.service (HttpServlet.java:707)\n at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)\n at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:860)\n at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:535)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188)\n at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188)\n at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1253)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:168)\n at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473)\n at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:166)\n at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1155)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:141)\n at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)\n at org.seleniumhq.jetty9.server.Server.handle (Server.java:530)\n at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:347)\n at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:256)\n at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:279)\n at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:102)\n at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:124)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:247)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce (EatWhatYouKill.java:140)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:131)\n at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:382)\n at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:708)\n at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:626)\n at java.lang.Thread.run (Thread.java:748)\n-------------------- >> begin captured stdout << ---------------------\n_init_browser, connecting to hub at http://172.18.0.2:4444/wd/hub\nMessage: Error forwarding the new session Empty pool of VM for setup Capabilities {browserName: chrome, platform: LINUX, version: 64.0.3282.140}\nStacktrace:\n at org.openqa.grid.web.servlet.handler.RequestHandler.process (RequestHandler.java:117)\n at org.openqa.grid.web.servlet.DriverServlet.process (DriverServlet.java:84)\n at org.openqa.grid.web.servlet.DriverServlet.doPost (DriverServlet.java:68)\n at javax.servlet.http.HttpServlet.service (HttpServlet.java:707)\n at javax.servlet.http.HttpServlet.service (HttpServlet.java:790)\n at org.seleniumhq.jetty9.servlet.ServletHolder.handle (ServletHolder.java:860)\n at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle (ServletHandler.java:535)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188)\n at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle (SessionHandler.java:1595)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle (ScopedHandler.java:188)\n at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle (ContextHandler.java:1253)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:168)\n at org.seleniumhq.jetty9.servlet.ServletHandler.doScope (ServletHandler.java:473)\n at org.seleniumhq.jetty9.server.session.SessionHandler.doScope (SessionHandler.java:1564)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope (ScopedHandler.java:166)\n at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope (ContextHandler.java:1155)\n at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle (ScopedHandler.java:141)\n at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle (HandlerWrapper.java:132)\n at org.seleniumhq.jetty9.server.Server.handle (Server.java:530)\n at org.seleniumhq.jetty9.server.HttpChannel.handle (HttpChannel.java:347)\n at org.seleniumhq.jetty9.server.HttpConnection.onFillable (HttpConnection.java:256)\n at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded (AbstractConnection.java:279)\n at org.seleniumhq.jetty9.io.FillInterest.fillable (FillInterest.java:102)\n at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run (ChannelEndPoint.java:124)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce (EatWhatYouKill.java:247)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce (EatWhatYouKill.java:140)\n at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run (EatWhatYouKill.java:131)\n at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run (ReservedThreadExecutor.java:382)\n at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob (QueuedThreadPool.java:708)\n at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run (QueuedThreadPool.java:626)\n at java.lang.Thread.run (Thread.java:748)\nWebDriverException in _init_browser connecting to hub\n\n--------------------- >> end captured stdout << ----------------------\n-------------------- >> begin captured logging << --------------------\nselenium.webdriver.remote.remote_connection: DEBUG: POST http://172.18.0.2:4444/wd/hub/session {"capabilities": {"alwaysMatch": {"browserName": "chrome", "platformName": "linux", "browserVersion": "64.0.3282.140"}, "firstMatch": [{}]}, "desiredCapabilities": {"platform": "LINUX", "browserName": "chrome", "version": "64.0.3282.140"}}\nurllib3.connectionpool: DEBUG: Starting new HTTP connection (1): 172.18.0.2:4444\nurllib3.connectionpool: DEBUG: http://172.18.0.2:4444 "POST /wd/hub/session HTTP/1.1" 500 7806\nselenium.webdriver.remote.remote_connection: DEBUG: Finished Request\n--------------------- >> end captured logging << ---------------------'

1 1

[oVirt Jenkins] ovirt-system-tests_performance-suite-master - Build # 1536 - Failure!
by jenkins＠jenkins.phx.ovirt.org 03 Aug '20

03 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/1… Build Number: 1536 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1536 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 1000_analyze_postgresql.analyze_postgres Error Message: Failed to perform the postgres analysis. Exit code is 1 -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[0m?[0m lago.ssh: DEBUG: start task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine: lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[32mSuccess?[0m (in 0:00:00) lago.ssh: DEBUG: start task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: Running 93da8eda on lago-performance-suite-master-engine: /root/analyze_postgresql.sh lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine returned with 1 lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine output: Last metadata expiration check: 0:11:09 ago on Fri 31 Jul 2020 10:59:32 PM EDT. pgdg-redhat-repo-42.0-11.noarch.rpm 12 kB/s | 11 kB 00:00 Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Installing: pgdg-redhat-repo noarch 42.0-11 @commandline 11 k Transaction Summary ================================================================================ Install 1 Package Total size: 11 k Installed size: 11 k Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 Installed: pgdg-redhat-repo-42.0-11.noarch Complete! PostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 190 kB/s | 376 kB 00:01 PostgreSQL 12 for RHEL/CentOS 8 - x86_64 112 kB/s | 247 kB 00:02 PostgreSQL 11 for RHEL/CentOS 8 - x86_64 217 kB/s | 424 kB 00:01 PostgreSQL 10 for RHEL/CentOS 8 - x86_64 161 kB/s | 308 kB 00:01 PostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 127 kB/s | 300 kB 00:02 PostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 111 kB/s | 259 kB 00:02 Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: pgbadger noarch 11.3-1.rhel8 pgdg-common 348 k pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k Installing dependencies: perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k Transaction Summary ================================================================================ Install 4 Packages Total download size: 874 k Installed size: 3.5 M Downloading Packages: (1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 (2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 523 kB/s | 24 kB 00:00 (3/4): pgbadger-11.3-1.rhel8.noarch.rpm 252 kB/s | 348 kB 00:01 (4/4): pgcluu-3.1-2.rhel8.noarch.rpm 235 kB/s | 356 kB 00:01 -------------------------------------------------------------------------------- Total 561 kB/s | 874 kB 00:01 PostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 Key imported successfully Extra Packages for Enterprise Linux 8 - x86_64 7.4 kB/s | 1.6 kB 00:00 Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 Installing : pgbadger-11.3-1.rhel8.noarch 3/4 Installing : pgcluu-3.1-2.rhel8.noarch 4/4 Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 Verifying : pgbadger-11.3-1.rhel8.noarch 1/4 Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 Installed: perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch pgbadger-11.3-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch Complete! lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine errors: + pg_datadir=/var/lib/pgsql/data + pg_service=postgresql.service + pgdg= + scl_enable= ++ get_engine_pg_scl ++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf) ++ local engine_pg_scl_conf ++ local res= ++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]] ++ echo '' + engine_pg_scl= + [[ -n '' ]] ++ rpm -q postgresql + pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64 + case "${pgver}" in + pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/… + [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… ]] + yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… + yum install -y pgbadger pgcluu warning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.3-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY Importing GPG key 0x442DF0F8: Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers(a)pgfoundry.org>" Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG warning: /var/cache/dnf/epel-el8-fe1ce5303067b1fb/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY Importing GPG key 0x2F86D6A1: Userid : "Fedora EPEL (8) <epel(a)fedoraproject.org>" Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1 From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8 ++ su - postgres -c 'psql postgres postgres -t -c '\''show log_directory'\''' ++ sed 2d + pglogdir=' log' + pglogdir=log + cd /var/lib/pgsql/data/log/ ++ su - postgres -c 'psql postgres postgres -t -c '\''show log_filename'\''' ++ sed 2d + pglog=' postgresql-%m.log' ++ date +postgresql-%m.log + pglog=postgresql-07.log + mkdir pgbadger + pgbadger -f stderr postgresql-07.log --outdir pgbadger FATAL: logfile "postgresql-07.log" must exist! - Error at line 2410 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/1000_analyze_postgresql.py", line 52, in analyze_postgres ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) 'Failed to perform the postgres analysis. Exit code is 1\n-------------------- >> begin captured logging << --------------------\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine:\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[32mSuccess\x1b[0m (in 0:00:00)\nlago.ssh: DEBUG: start task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: Running 93da8eda on lago-performance-suite-master-engine: /root/analyze_postgresql.sh\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine returned with 1\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine output:\n Last metadata expiration check: 0:11:09 ago on Fri 31 Jul 2020 10:59:32 PM EDT.\npgdg-redhat-repo-42.0-11.noarch.rpm 12 kB/s | 11 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling:\n pgdg-redhat-repo noarch 42.0-11 @commandline 11 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size: 11 k\nInstalled size: 11 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 \n Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 \n\nInstalled:\n pgdg-redhat-repo-42.0-11.noarch \n\nComplete!\nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 190 kB/s | 376 kB 00:01 \nPostgreSQL 12 for RHEL/CentOS 8 - x86_64 112 kB/s | 247 kB 00:02 \nPostgreSQL 11 for RHEL/CentOS 8 - x86_64 217 kB/s | 424 kB 00:01 \nPostgreSQL 10 for RHEL/CentOS 8 - x86_64 161 kB/s | 308 kB 00:01 \nPostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 127 kB/s | 300 kB 00:02 \nPostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 111 kB/s | 259 kB 00:02 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n pgbadger noarch 11.3-1.rhel8 pgdg-common 348 k\n pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k\nInstalling dependencies:\n perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k\n perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k\n\nTransaction Summary\n================================================================================\nInstall 4 Packages\n\nTotal download size: 874 k\nInstalled size: 3.5 M\nDownloading Packages:\n(1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 \n(2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 523 kB/s | 24 kB 00:00 \n(3/4): pgbadger-11.3-1.rhel8.noarch.rpm 252 kB/s | 348 kB 00:01 \n(4/4): pgcluu-3.1-2.rhel8.noarch.rpm 235 kB/s | 356 kB 00:01 \n--------------------------------------------------------------------------------\nTotal 561 kB/s | 874 kB 00:01 \nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 \nKey imported successfully\nExtra Packages for Enterprise Linux 8 - x86_64 7.4 kB/s | 1.6 kB 00:00 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 \n Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 \n Installing : pgbadger-11.3-1.rhel8.noarch 3/4 \n Installing : pgcluu-3.1-2.rhel8.noarch 4/4 \n Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 \n Verifying : pgbadger-11.3-1.rhel8.noarch 1/4 \n Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 \n Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 \n Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 \n\nInstalled:\n perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch\n pgbadger-11.3-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch \n\nComplete!\n\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine errors:\n + pg_datadir=/var/lib/pgsql/data\n+ pg_service=postgresql.service\n+ pgdg=\n+ scl_enable=\n++ get_engine_pg_scl\n++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf)\n++ local engine_pg_scl_conf\n++ local res=\n++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]]\n++ echo \'\'\n+ engine_pg_scl=\n+ [[ -n \'\' ]]\n++ rpm -q postgresql\n+ pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64\n+ case "${pgver}" in\n+ pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/… [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… ]]\n+ yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… yum install -y pgbadger pgcluu\nwarning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.3-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY\nImporting GPG key 0x442DF0F8:\n Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers(a)pgfoundry.org>"\n Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG\nwarning: /var/cache/dnf/epel-el8-fe1ce5303067b1fb/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY\nImporting GPG key 0x2F86D6A1:\n Userid : "Fedora EPEL (8) <epel(a)fedoraproject.org>"\n Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1\n From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_directory\'\\\'\'\'\n++ sed 2d\n+ pglogdir=\' log\'\n+ pglogdir=log\n+ cd /var/lib/pgsql/data/log/\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_filename\'\\\'\'\'\n++ sed 2d\n+ pglog=\' postgresql-%m.log\'\n++ date +postgresql-%m.log\n+ pglog=postgresql-07.log\n+ mkdir pgbadger\n+ pgbadger -f stderr postgresql-07.log --outdir pgbadger\nFATAL: logfile "postgresql-07.log" must exist!\n - Error at line 2410\n\n--------------------- >> end captured logging << ---------------------'

1 2

oVirt infra daily report - unstable production jobs - 1271
by jenkins＠jenkins.phx.ovirt.org 03 Aug '20

03 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1271//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 243 - Failure!
by jenkins＠jenkins.phx.ovirt.org 02 Aug '20

02 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 243 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #243 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_next_run_unplug_cpu Error Message: assert 2 == 1 -2 +1 Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7f0a67125e50> @order_by(_TEST_LIST) def test_next_run_unplug_cpu(api_v4): engine = api_v4.system_service() vm_service = test_utils.get_vm_service(engine, VM0_NAME) new_cpu = vm_service.get().cpu new_cpu.topology.sockets = 1 vm_service.update( vm=types.Vm( cpu=new_cpu, ), next_run=True ) assert vm_service.get().cpu.topology.sockets == 2 assert vm_service.get(next_run=True).cpu.topology.sockets == 1 with test_utils.TestEvent(engine, 157): # USER_REBOOT_VM(157) vm_service.reboot() testlib.assert_true_within_long( lambda: vm_service.get().status == types.VmStatus.UP ) > assert vm_service.get().cpu.topology.sockets == 1 E assert 2 == 1 E -2 E +1 ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1112: AssertionError

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 243 - Failure!
by jenkins＠jenkins.phx.ovirt.org 02 Aug '20

02 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/24… Build Number: 243 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #243 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_basic_ui_sanity.start_grid Error Message: could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local creating docker network for grid executing shell: docker network create grid b986163a1f7e37430802635a5db3f7b965e863a81d763bfdd024233ccbead9f3 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local b986163a1f7e grid bridge local bd7765e6670f host host local 245daf9e104c none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer c949dee5af7e: Pulling fs layer df88a49b4162: Pulling fs layer ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer c9aca8b50247: Pulling fs layer 4be3072e5a37: Waiting 06c6d2f59700: Waiting edcd5e9f2f91: Waiting 0eeaf787f757: Waiting ce3c6f42fd24: Waiting c9aca8b50247: Waiting 6c8e191c3abf: Waiting df88a49b4162: Waiting c949dee5af7e: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.6:53: read udp 10.129.4.11:39190->172.19.11.6:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ---------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 189, in start_grid raise RuntimeError("could not get ip address of selenium hub. See previous messages for probable docker failure") could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local bd7765e6670f host host local 245daf9e104c none null local creating docker network for grid executing shell: docker network create grid b986163a1f7e37430802635a5db3f7b965e863a81d763bfdd024233ccbead9f3 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 7de924623102 bridge bridge local b986163a1f7e grid bridge local bd7765e6670f host host local 245daf9e104c none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer c949dee5af7e: Pulling fs layer df88a49b4162: Pulling fs layer ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer c9aca8b50247: Pulling fs layer 4be3072e5a37: Waiting 06c6d2f59700: Waiting edcd5e9f2f91: Waiting 0eeaf787f757: Waiting ce3c6f42fd24: Waiting c9aca8b50247: Waiting 6c8e191c3abf: Waiting df88a49b4162: Waiting c949dee5af7e: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.6:53: read udp 10.129.4.11:39190->172.19.11.6:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ----------------------

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 240 - Failure!
by jenkins＠jenkins.phx.ovirt.org 02 Aug '20

02 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 240 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #240 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_preview_snapshot_with_memory Error Message: AssertionError: False != True after 600 seconds Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7fa5aec7f810> @order_by(_TEST_LIST) def test_preview_snapshot_with_memory(api_v4): engine = api_v4.system_service() events = engine.events_service() testlib.assert_true_within_long( # wait for event 68 == USER_CREATE_SNAPSHOT_FINISHED_SUCCESS > lambda: any(e.code == 68 for e in events.list(max=6)) ) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:511: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:286: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:273: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7fa5ac0858c0>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10 ): allowed_exceptions = allowed_exceptions or [] with utils.EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: raise AssertionError( > '%s != %s after %s seconds' % (res, value, timeout) ) E AssertionError: False != True after 600 seconds /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:252: AssertionError

1 2

oVirt infra daily report - unstable production jobs - 1270
by jenkins＠jenkins.phx.ovirt.org 02 Aug '20

02 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1270//artifact/exported… Cheers, Jenkins

1 0

oVirt infra daily report - unstable production jobs - 1269
by jenkins＠jenkins.phx.ovirt.org 01 Aug '20

01 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1269//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 233 - Failure!
by jenkins＠jenkins.phx.ovirt.org 31 Jul '20

31 Jul '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/23… Build Number: 233 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #233 [Michal Skrivanek] verify hotplug memory and test unplug [Tristan Cacqueray] zuul-jobs-gate: repparent to base-gate-job [Ehud Yonasi] stdci_runner: added the rhel labels selection. ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 003_00_metrics_bootstrap.configure_vdsm_sos_plugin Error Message: Failed to copy files/directory /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin [Errno 2] No such file or directory: '/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin' -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin to lago-basic-suite-4-3-host-1:/tmp: ?[0m?[0m lago.ssh: DEBUG: start task:df0ccdbf-77a7-49e8-a3e3-46a2a949b9de:Get ssh client for lago-basic-suite-4-3-host-1: lago.ssh: DEBUG: end task:df0ccdbf-77a7-49e8-a3e3-46a2a949b9de:Get ssh client for lago-basic-suite-4-3-host-1: root: ERROR: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin to lago-basic-suite-4-3-host-1:/tmp: ?[31mERROR?[0m (in 0:00:01) lago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 506, in copy_to raise LagoCopyFilesToVMError(local_path, str(err)) --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/003_00_metrics_bootstrap.py", line 49, in configure_vdsm_sos_plugin '/tmp' File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 506, in copy_to raise LagoCopyFilesToVMError(local_path, str(err)) 'Failed to copy files/directory /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin\n[Errno 2] No such file or directory: \'/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin\'\n-------------------- >> begin captured logging << --------------------\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin to lago-basic-suite-4-3-host-1:/tmp: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:df0ccdbf-77a7-49e8-a3e3-46a2a949b9de:Get ssh client for lago-basic-suite-4-3-host-1:\nlago.ssh: DEBUG: end task:df0ccdbf-77a7-49e8-a3e3-46a2a949b9de:Get ssh client for lago-basic-suite-4-3-host-1:\nroot: ERROR: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/../common/test-scenarios-files/vdsm-sos-plugin to lago-basic-suite-4-3-host-1:/tmp: \x1b[31mERROR\x1b[0m (in 0:00:01)\nlago.plugins.vm: DEBUG: File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 506, in copy_to\n raise LagoCopyFilesToVMError(local_path, str(err))\n\n--------------------- >> end captured logging << ---------------------'

1 3

oVirt infra daily report - unstable production jobs - 1268
by jenkins＠jenkins.phx.ovirt.org 31 Jul '20

31 Jul '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1268//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2992) [FIRING:1] TestAlert
by Alertmanager_Bot (oVirt JIRA) 30 Jul '20

30 Jul '20

Alertmanager_Bot created OVIRT-2992: --------------------------------------- Summary: [FIRING:1] TestAlert Key: OVIRT-2992 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2992 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = TestAlert - label1 = value1 Annotations: Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100140)

1 0

[CQ]: 110470,5 (ovirt-provider-ovn) failed "ovirt-4.3" system tests
by oVirt Jenkins 30 Jul '20

30 Jul '20

Change 110470,5 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/110470/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/3734/

1 0

[CQ]: 110470, 5 (ovirt-provider-ovn) failed "ovirt-master" system tests
by oVirt Jenkins 30 Jul '20

30 Jul '20

Change 110470,5 (ovirt-provider-ovn) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/110470/5 For failed test results see: https://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/25148/

1 0

[oVirt Jenkins] deploy-to-gated-ovirt-master - Build #432 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 30 Jul '20

30 Jul '20

Build: https://jenkins.ovirt.org/job/deploy-to-gated-ovirt-master/432/ Build Name: #432 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

gerrit build whitelist
by Germano Veit Michel 30 Jul '20

30 Jul '20

Hi, Would it be possible to add me to the whitelist? I've got a build denied here: https://gerrit.ovirt.org/#/c/109402/ The account is germano(a)redhat.com Thanks, Germano

2 1

[oVirt Jenkins] cleanup-gated-ovirt-4.3 - Build #223 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 30 Jul '20

30 Jul '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-ovirt-4.3/223/ Build Name: #223 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0