August 2020 - Infra - oVirt List Archives

[JIRA] (OVIRT-2995) [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci)
by Alertmanager_Bot (oVirt JIRA) 10 Aug '20

10 Aug '20

Alertmanager_Bot created OVIRT-2995: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable ibm-srv01.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv01.ovirt.org true external bare-metal-external ci) Key: OVIRT-2995 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2995 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = ibm-srv01.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = ibm-srv01.ovirt.org - node_role_kubernetes_io_compute = true - region = external - type = bare-metal-external - zone = ci Annotations: - description = ibm-srv01.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2992) [FIRING:1] TestAlert
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2992?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2992: -------------------------------- Resolution: Fixed Status: Done (was: To Do) > [FIRING:1] TestAlert > --------------------- > > Key: OVIRT-2992 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2992 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="TestAlert"} > > Labels: > - alertname = TestAlert > - label1 = value1 > Annotations: > Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2994) Occasionally we see alerts on dead node-exporters on OpenShift nodes
by Shlomi Zidmi (oVirt JIRA) 10 Aug '20

10 Aug '20

Shlomi Zidmi created OVIRT-2994: ----------------------------------- Summary: Occasionally we see alerts on dead node-exporters on OpenShift nodes Key: OVIRT-2994 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2994 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Shlomi Zidmi Assignee: infra Priority: Low We need to check what is happening behind the scenes when node-exporters are crashing and probably other containers too. Most times OpenShift is able to recover automatically, yet sometimes we have to fix it manually. Creating a ticket for investigating this issue -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[JIRA] (OVIRT-2993) [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci)
by Alertmanager_Bot (oVirt JIRA) 10 Aug '20

10 Aug '20

Alertmanager_Bot created OVIRT-2993: --------------------------------------- Summary: [FIRING:1] InstanceUnreachable shift-n11.phx.ovirt.org kubernetes-nodes-exporter (amd64 linux shift-n11.phx.ovirt.org true primary vm ci) Key: OVIRT-2993 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2993 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = InstanceUnreachable - beta_kubernetes_io_arch = amd64 - beta_kubernetes_io_os = linux - instance = shift-n11.phx.ovirt.org - job = kubernetes-nodes-exporter - kubernetes_io_hostname = shift-n11.phx.ovirt.org - node_role_kubernetes_io_compute = true - region = primary - type = vm - zone = ci Annotations: - description = shift-n11.phx.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100142)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 267 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/26… Build Number: 267 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #267 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 008_basic_ui_sanity.start_grid Error Message: could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local creating docker network for grid executing shell: docker network create grid 2e5f4fc816f6d795c564a5dab8b93c3b94818167957e7e65d94b37563da512d5 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local 2e5f4fc816f6 grid bridge local b3331b0f16ef host host local e00fb324e999 none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer edcd5e9f2f91: Waiting 0eeaf787f757: Waiting c949dee5af7e: Pulling fs layer 06c6d2f59700: Waiting df88a49b4162: Pulling fs layer c949dee5af7e: Waiting ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer 6c8e191c3abf: Waiting c9aca8b50247: Pulling fs layer c9aca8b50247: Waiting 4be3072e5a37: Waiting ce3c6f42fd24: Waiting df88a49b4162: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.7:53: read udp 10.131.3.159:57412->172.19.11.7:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ---------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/008_basic_ui_sanity.py", line 189, in start_grid raise RuntimeError("could not get ip address of selenium hub. See previous messages for probable docker failure") could not get ip address of selenium hub. See previous messages for probable docker failure -------------------- >> begin captured stdout << --------------------- executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker rm -f grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: No such container: grid_node_chrome Error response from daemon: No such container: grid_node_firefox Error response from daemon: No such container: selenium-hub executing shell: docker kill grid_node_chrome grid_node_firefox selenium-hub Error response from daemon: Cannot kill container: grid_node_chrome: No such container: grid_node_chrome Error response from daemon: Cannot kill container: grid_node_firefox: No such container: grid_node_firefox Error response from daemon: Cannot kill container: selenium-hub: No such container: selenium-hub executing shell: docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network rm grid Error response from daemon: network grid not found executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local b3331b0f16ef host host local e00fb324e999 none null local creating docker network for grid executing shell: docker network create grid 2e5f4fc816f6d795c564a5dab8b93c3b94818167957e7e65d94b37563da512d5 executing shell: docker network ls NETWORK ID NAME DRIVER SCOPE 0172f0f60572 bridge bridge local 2e5f4fc816f6 grid bridge local b3331b0f16ef host host local e00fb324e999 none null local starting hub executing shell: docker run -d -p 4444:4444 --net grid --name selenium-hub selenium/hub:3.9.1-actinium Unable to find image 'selenium/hub:3.9.1-actinium' locally 3.9.1-actinium: Pulling from selenium/hub 1be7f2b886e8: Pulling fs layer 6fbc4a21b806: Pulling fs layer c71a6f8e1378: Pulling fs layer 4be3072e5a37: Pulling fs layer 06c6d2f59700: Pulling fs layer edcd5e9f2f91: Pulling fs layer 0eeaf787f757: Pulling fs layer edcd5e9f2f91: Waiting 0eeaf787f757: Waiting c949dee5af7e: Pulling fs layer 06c6d2f59700: Waiting df88a49b4162: Pulling fs layer c949dee5af7e: Waiting ce3c6f42fd24: Pulling fs layer 6c8e191c3abf: Pulling fs layer 6c8e191c3abf: Waiting c9aca8b50247: Pulling fs layer c9aca8b50247: Waiting 4be3072e5a37: Waiting ce3c6f42fd24: Waiting df88a49b4162: Waiting /usr/bin/docker-current: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blo…: dial tcp: lookup production.cloudflare.docker.com on 172.19.11.7:53: read udp 10.131.3.159:57412->172.19.11.7:53: i/o timeout. See '/usr/bin/docker-current run --help'. getting ip of hub --------------------- >> end captured stdout << ----------------------

1 1

[oVirt Jenkins] cleanup-gated-dummy-master - Build #233 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Build: https://jenkins.ovirt.org/job/cleanup-gated-dummy-master/233/ Build Name: #233 Build Status: FAILURE Gerrit change: null - title: null - project: null - branch: null - author: null <null>

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 264 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 264 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #264 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_hotplug_memory Error Message: assert 323 == 352 -323 +352 Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f229a1ffc10> hotplug_mem_amount = 268435456 @order_by(_TEST_LIST) def test_hotplug_memory(prefix, hotplug_mem_amount): api_v4 = prefix.virt_env.engine_vm().get_api_v4() engine = api_v4.system_service() vm_service = test_utils.get_vm_service(engine, VM0_NAME) new_memory = vm_service.get().memory + hotplug_mem_amount with test_utils.TestEvent(engine, 2039): # HOT_SET_MEMORY(2,039) vm_service.update( vm=types.Vm( memory=new_memory ) ) assert vm_service.get().memory == new_memory assert_vm0_is_alive(prefix) > assert _vm_libvirt_memory_amount(prefix, VM0_NAME) // KB == new_memory // MB E assert 323 == 352 E -323 E +352 ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:1050: AssertionError

1 2

oVirt infra daily report - unstable production jobs - 1278
by jenkins＠jenkins.phx.ovirt.org 10 Aug '20

10 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1278//artifact/exported… Cheers, Jenkins

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 261 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Aug '20

09 Aug '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 261 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #261 [Michal Skrivanek] smaller OST VMs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_live_storage_migration Error Message: AssertionError: False != True after 600 seconds Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7f6cfbc40810> @order_by(_TEST_LIST) def test_live_storage_migration(api_v4): engine = api_v4.system_service() disk_service = test_utils.get_disk_service(engine, DISK0_NAME) correlation_id = 'live_storage_migration' disk_service.move( async=False, filter=False, storage_domain=types.StorageDomain( name=SD_ISCSI_NAME ), query={'correlation_id': correlation_id} ) testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) # Assert that the disk is on the correct storage domain, # its status is OK and the snapshot created for the migration # has been merged testlib.assert_true_within_long( lambda: api_v4.follow_link(disk_service.get().storage_domains[0]).name == SD_ISCSI_NAME ) vm0_snapshots_service = test_utils.get_vm_snapshots_service(engine, VM0_NAME) testlib.assert_true_within_long( > lambda: len(vm0_snapshots_service.list()) == 1 ) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:585: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:286: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:273: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7f6cf85b2a28>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10 ): allowed_exceptions = allowed_exceptions or [] with utils.EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: raise AssertionError( > '%s != %s after %s seconds' % (res, value, timeout) ) E AssertionError: False != True after 600 seconds /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:252: AssertionError

1 2

oVirt infra daily report - unstable production jobs - 1277
by jenkins＠jenkins.phx.ovirt.org 09 Aug '20

09 Aug '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1277//artifact/exported… Cheers, Jenkins

1 0