- Infra - oVirt List Archives

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer (oVirt JIRA) 10 Sep '20

10 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3010?page=com.atlassian.jira.… ] Nir Soffer commented on OVIRT-3010: ----------------------------------- On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. This is still broken today. In the past sending mail to infra-support created a bug, and someone was handling issue quickly. This seems to be broken now. We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm for development and CI. When the rpm is missing, CI jobs fail, and developers cannot update their setup. Please fix it as soon as possible. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > > All seems to fail in: > > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > > Checking manually show: > > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > > ovirt-elease-master.rpm does not exist. > > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" > [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" > ------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3010 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Nir Soffer > Assignee: infra > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > All seems to fail in: > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > Checking manually show: > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > ovirt-elease-master.rpm does not exist. > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3010) [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer (oVirt JIRA) 10 Sep '20

10 Sep '20

Nir Soffer created OVIRT-3010: --------------------------------- Summary: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm" Key: OVIRT-3010 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3010 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Nir Soffer Assignee: infra This is the second time, last time there was an issue during a nightly job that left the master snapshot repo broken. I have 3 failure builds: https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ All seems to fail in: [2020-09-07T10:00:56.163Z] + dnf install -y http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm [2020-09-07T10:01:08.524Z] Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: 66.187.230.40) Checking manually show: $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>403 Forbidden</title> </head><body> <h1>Forbidden</h1> <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm on this server.</p> </body></html> Looking at: https://resources.ovirt.org/pub/yum-repo/ ovirt-elease-master.rpm does not exist. So we have 2 issues: - The file does not exist - We return "403 Forbidden" instead of "404 Not Found" -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3009) errors on infra
by Yedidyah Bar David (oVirt JIRA) 10 Sep '20

10 Sep '20

Yedidyah Bar David created OVIRT-3009: ----------------------------------------- Summary: errors on infra Key: OVIRT-3009 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3009 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Yedidyah Bar David Assignee: infra Hi, 1. Please see thread "[ovirt-devel] gerrit pull/clone timeout". 2. Please see [1] - it failed check-merged, I tried again, and it failed again. In both cases, e.g. [2]: Error: Error downloading packages: Cannot download x86_64/ovirt-engine-wildfly-19.1.0-2.el8.x86_64.rpm: All mirrors were tried Of course, I can't be sure these are due to a single root cause. Please fix :-) Thanks, [1] https://gerrit.ovirt.org/111098 [2] https://jenkins.ovirt.org/job/ovirt-dwh_standard-on-merge/124/artifact/buil… -- Didi -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3008) IMAPS certificate expired on lists.ovirt.org
by Evgheni Dereveanchin (oVirt JIRA) 10 Sep '20

10 Sep '20

Evgheni Dereveanchin created OVIRT-3008: ------------------------------------------- Summary: IMAPS certificate expired on lists.ovirt.org Key: OVIRT-3008 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3008 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra We've got reports about ticket creation issues from email. Looking at dovecot logs the email handler tries to connect but disconnects immediately. Connecting to port 993 shows an expired cert: openssl s_client lists.ovirt.org:993 CONNECTED(00000003) depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 verify return:1 depth=0 CN = lists.ovirt.org verify error:num=10:certificate has expired notAfter=Jul 1 09:47:58 2020 GMT verify return:1 depth=0 CN = lists.ovirt.org notAfter=Jul 1 09:47:58 2020 GMT verify return:1 --- Certificate chain 0 s:CN = lists.ovirt.org i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 Looks like dovecot is not getting restarted after a new cert is issued. Leaving as-is for now for further review. A service restart should theoretically be enough to make use of the current cert. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 363 - Failure!
by jenkins＠jenkins.phx.ovirt.org 10 Sep '20

10 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 363 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #363 [Martin Necas] Ansible: remove nic_on_boot from cloud-init ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.100_basic_ui_sanity_pytest.test_login[firefox] Error Message: test setup failure Stack Trace: engine_fqdn = 'engine', engine_ip = '192.168.201.4' @pytest.fixture(scope="session") def hub_url(engine_fqdn, engine_ip): env_url = _env_hub_url() if env_url is not None: yield env_url else: backend = _grid_backend() if backend == "podman": if os_utils.on_centos(8) and os_utils.inside_mock(): pytest.skip("podman doesn't work in mock") hub_port = network_utils.find_free_port(4444, 4544) with podman.grid(engine_fqdn, engine_ip, hub_port=hub_port) as hub_url: yield hub_url elif backend == "docker": > with docker.grid(engine_fqdn, engine_ip) as hub_url: ../ost_utils/ost_utils/pytest/fixtures/selenium/__init__.py:76: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:118: in grid with _network(network_name): /usr/lib64/python2.7/contextlib.py:17: in __enter__ return self.gen.next() ../ost_utils/ost_utils/selenium/docker.py:61: in _network shell(["docker", "network", "create", network_name]) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ args = ['docker', 'network', 'create', 'grid'], bytes_output = False kwargs = {}, process = <subprocess.Popen object at 0x7f48c8c633d0>, out = '' err = 'Error response from daemon: network with name grid already exists ' def shell(args, bytes_output=False, **kwargs): process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, **kwargs) out, err = process.communicate() if not bytes_output: out = out.decode("utf-8") err = err.decode("utf-8") if process.returncode: > raise ShellError(process.returncode, out, err) E ShellError: Command failed with rc=1. Stdout: E E Stderr: E Error response from daemon: network with name grid already exists ../ost_utils/ost_utils/shell.py:51: ShellError

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 361 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Sep '20

09 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 361 Build Status: Failure Triggered By: Started by user Parth Dhanjal ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #361 No changes ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 1309
by jenkins＠jenkins.phx.ovirt.org 09 Sep '20

09 Sep '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1309//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3007) permission to run ovirt-system-tests_manual
by Pierre Lecomte (oVirt JIRA) 09 Sep '20

09 Sep '20

Pierre Lecomte created OVIRT-3007: ------------------------------------- Summary: permission to run ovirt-system-tests_manual Key: OVIRT-3007 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3007 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Components: OST Manual job Reporter: Pierre Lecomte Assignee: infra I emailed infra-support(a)ovirt.org but this does not appear to have created the ticket here. -------- Forwarded Message -------- Subject: Request for permission to run ovirt-system-tests_manual Date: Wed, 9 Sep 2020 18:04:08 +0200 From: Pierre Lecomte <pierre.lecomte(a)oracle.com> To: infra-support(a)ovirt.org Hello, I have been told by Sandro I should be emailing this email to get access rights to trigger manual OST jobs. My login is plecomte on jenkins. (Note : There appears to be a pierre.lecomte(a)oracle.com login on the Jenkins host but I did not create it- It's maybe tied with my login on the gerrit instance which is achieved using github credentials). Thanks! -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-2847) Certain substage fails continously with missing useradd in chroot
by Evgheni Dereveanchin (oVirt JIRA) 09 Sep '20

09 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2847?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2847: ---------------------------------------- Issue Type: Bug (was: By-EMAIL) > Certain substage fails continously with missing useradd in chroot > ----------------------------------------------------------------- > > Key: OVIRT-2847 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2847 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Bell Levin > Assignee: infra > > Hello! > I have had a contentious error in a certain job. > The error seems to be: > ERROR: Could not find useradd in chroot, maybe the install failed? > you can see it in the logs here: > https://jenkins.ovirt.org/job/standard-manual-runner/983/artifact/check-net… > In the check-network.functional-ovs-switch-lib job, in the following runs: > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > Please advice. > Thanks, > Bell -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-2847) Certain substage fails continously with missing useradd in chroot
by Evgheni Dereveanchin (oVirt JIRA) 09 Sep '20

09 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2847?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2847: ---------------------------------------- Resolution: Cannot Reproduce Status: Done (was: To Do) This seems like an old issue with Fedora mocks. Closing since I've not seen this issue in a while > Certain substage fails continously with missing useradd in chroot > ----------------------------------------------------------------- > > Key: OVIRT-2847 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2847 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Bell Levin > Assignee: infra > > Hello! > I have had a contentious error in a certain job. > The error seems to be: > ERROR: Could not find useradd in chroot, maybe the install failed? > you can see it in the logs here: > https://jenkins.ovirt.org/job/standard-manual-runner/983/artifact/check-net… > In the check-network.functional-ovs-switch-lib job, in the following runs: > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > https://jenkins.ovirt.org/blue/organizations/jenkins/standard-manual-runner… > Please advice. > Thanks, > Bell -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 359 - Failure!
by jenkins＠jenkins.phx.ovirt.org 09 Sep '20

09 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 359 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #359 [Marcin Sobczyk] basic: Drop outdated code ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.098_ovirt_provider_ovn_pytest.test_use_ovn_provider Error Message: Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Network. The name of the logical network 'network_1' is already used by an existing logical network in the same data-center. -Please choose a different name.]". HTTP response code is 409. Stack Trace: prefix = <ovirtlago.prefix.OvirtPrefix object at 0x7f8782aba190> api_v4 = <ovirtsdk4.Connection object at 0x7f8781fee590> @versioning.require_version(4, 2) def test_use_ovn_provider(prefix, api_v4): engine = api_v4.system_service() engine_ip = prefix.virt_env.engine_vm().ip() provider_id = network_utils_v4.get_default_ovn_provider_id(engine) token_id = _get_auth_token(engine_ip) _validate_db_empty(token_id, engine_ip) with _disable_auto_sync(api_v4, provider_id): network1_id = _add_network( token_id, engine_ip, NETWORK_1, ) subnet1_id = _add_subnet( token_id, engine_ip, SUBNET_1, network1_id, ) port1_id = _add_port( token_id, engine_ip, PORT_1, network1_id, ) _validate_network(token_id, engine_ip, NETWORK_1, network1_id) _validate_port(token_id, engine_ip, PORT_1, port1_id, network1_id) _validate_subnet(token_id, engine_ip, SUBNET_1, subnet1_id, network1_id) datacenter_id = _get_datacenter_id(api_v4) > _import_network_to_ovirt(api_v4, provider_id, network1_id, datacenter_id) ../basic-suite-master/test-scenarios/098_ovirt_provider_ovn_pytest.py:474: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test-scenarios/098_ovirt_provider_ovn_pytest.py:364: in _import_network_to_ovirt id=datacenter_id /usr/lib64/python2.7/site-packages/ovirtsdk4/services.py:18492: in import_ return self._internal_action(action, 'import', None, headers, query, wait) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:299: in _internal_action return future.wait() if wait else future /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:55: in wait return self._code(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:296: in callback self._check_fault(response) /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:134: in _check_fault self._raise_error(response, body.fault) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ response = <ovirtsdk4.http.Response object at 0x7f8781fee450> detail = <ovirtsdk4.types.Fault object at 0x7f8781fee690> @staticmethod def _raise_error(response, detail=None): """ Creates and raises an error containing the details of the given HTTP response and fault. This method is intended for internal use by other components of the SDK. Refrain from using it directly, as backwards compatibility isn't guaranteed. """ fault = detail if isinstance(detail, types.Fault) else None msg = '' if fault: if fault.reason: if msg: msg += ' ' msg = msg + 'Fault reason is "%s".' % fault.reason if fault.detail: if msg: msg += ' ' msg = msg + 'Fault detail is "%s".' % fault.detail if response: if response.code: if msg: msg += ' ' msg = msg + 'HTTP response code is %s.' % response.code if response.message: if msg: msg += ' ' msg = msg + 'HTTP response message is "%s".' % response.message if isinstance(detail, six.string_types): if msg: msg += ' ' msg = msg + detail + '.' class_ = Error if response is not None: if response.code in [401, 403]: class_ = AuthError elif response.code == 404: class_ = NotFoundError error = class_(msg) error.code = response.code if response else None error.fault = fault > raise error E Error: Fault reason is "Operation Failed". Fault detail is "[Cannot add Network. The name of the logical network 'network_1' is already used by an existing logical network in the same data-center. E -Please choose a different name.]". HTTP response code is 409. /usr/lib64/python2.7/site-packages/ovirtsdk4/service.py:118: Error

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 355 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/35… Build Number: 355 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #355 [Marcin Sobczyk] ost_utils: network_utils: Add 'get_ips[6]' functions ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 005_network_by_label.assign_labeled_network Error Message: False != True after 180 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/005_network_by_label.py", line 149, in assign_labeled_network host_service, LABELED_NET_NAME)) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 282, in assert_true_within_short assert_equals_within_short(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 266, in assert_equals_within_short func, value, SHORT_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 180 seconds

1 1

oVirt infra daily report - unstable production jobs - 1308
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1308//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-3000) [FIRING:1] InstanceUnreachable ibm-srv08.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv08.ovirt.org external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Sep '20

08 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3000?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-3000: -------------------------------- Resolution: Done Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv08.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv08.ovirt.org external bare-metal-external ci) > ---------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3000 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3000 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv08.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv08.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv08.ovirt.org > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv08.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-2998) [FIRING:1] InstanceUnreachable ibm-srv07.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv07.ovirt.org external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Sep '20

08 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2998?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2998: -------------------------------- Resolution: Done Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv07.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv07.ovirt.org external bare-metal-external ci) > ---------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-2998 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2998 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv07.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv07.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv07.ovirt.org > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv07.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3001) [FIRING:1] InstanceUnreachable ibm-srv09.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv09.ovirt.org true external bare-metal-external ci)
by Shlomi Zidmi (oVirt JIRA) 08 Sep '20

08 Sep '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-3001?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-3001: -------------------------------- Resolution: Done Status: Done (was: To Do) > [FIRING:1] InstanceUnreachable ibm-srv09.ovirt.org kubernetes-nodes-exporter (amd64 linux ibm-srv09.ovirt.org true external bare-metal-external ci) > --------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: OVIRT-3001 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3001 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="InstanceUnreachable",instance="ibm-srv09.ovirt.org",job="kubernetes-nodes-exporter"} > > Labels: > - alertname = InstanceUnreachable > - beta_kubernetes_io_arch = amd64 > - beta_kubernetes_io_os = linux > - instance = ibm-srv09.ovirt.org > - job = kubernetes-nodes-exporter > - kubernetes_io_hostname = ibm-srv09.ovirt.org > - node_role_kubernetes_io_compute = true > - region = external > - type = bare-metal-external > - zone = ci > Annotations: > - description = ibm-srv09.ovirt.org of job kubernetes-nodes-exporter has been possibly down for more than 10 minutes. > Source: http://prometheus-0:9090/graph?g0.expr=up%7Bjob%3D%22kubernetes-nodes-expor… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

[JIRA] (OVIRT-3006) HTTP/500 errors when accessing lists.ovirt.org
by Evgheni Dereveanchin (oVirt JIRA) 08 Sep '20

08 Sep '20

Evgheni Dereveanchin created OVIRT-3006: ------------------------------------------- Summary: HTTP/500 errors when accessing lists.ovirt.org Key: OVIRT-3006 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-3006 Project: oVirt - virtualization made easy Issue Type: Outage Reporter: Evgheni Dereveanchin Assignee: infra I can't access lists.ovirt.org via the browser, getting a 500 Internal Server Error after a considerable time that looks like a timeout. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100145)

1 0

Re: [CI] Builds broken by "Status code: 403 for http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm"
by Nir Soffer 08 Sep '20

08 Sep '20

On Mon, Sep 7, 2020 at 2:30 PM Nir Soffer <nsoffer(a)redhat.com> wrote: > > This is the second time, last time there was an issue during a nightly job that > left the master snapshot repo broken. This is still broken today. In the past sending mail to infra-support created a bug, and someone was handling issue quickly. This seems to be broken now. We depend on http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm for development and CI. When the rpm is missing, CI jobs fail, and developers cannot update their setup. Please fix it as soon as possible. > I have 3 failure builds: > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7744/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7751/ > https://jenkins.ovirt.org/job/ovirt-engine_standard-check-patch/7755/ > > All seems to fail in: > > [2020-09-07T10:00:56.163Z] + dnf install -y > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > [2020-09-07T10:01:08.524Z] Status code: 403 for > http://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm (IP: > 66.187.230.40) > > Checking manually show: > > $ curl https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm > <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> > <html><head> > <title>403 Forbidden</title> > </head><body> > <h1>Forbidden</h1> > <p>You don't have permission to access /pub/yum-repo/ovirt-release-master.rpm > on this server.</p> > </body></html> > > Looking at: > https://resources.ovirt.org/pub/yum-repo/ > > ovirt-elease-master.rpm does not exist. > > So we have 2 issues: > - The file does not exist > - We return "403 Forbidden" instead of "404 Not Found"

3 5

Jenkins build became unstable: ovirt_4.2_publish-rpms_nightly #1099
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

See <https://jenkins.ovirt.org/job/ovirt_4.2_publish-rpms_nightly/1099/display/r…>

1 3

Jenkins build became unstable: ovirt_4.3_publish-rpms_nightly #613
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

See <https://jenkins.ovirt.org/job/ovirt_4.3_publish-rpms_nightly/613/display/re…>

1 3

Jenkins build became unstable: ovirt_master_publish-rpms_nightly #1856
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

See <https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/1856/displa…>

1 5

Ansible checks
by Marc Dequènes (Duck) 08 Sep '20

08 Sep '20

Quack, ansible-lint added new checks in 4.3.0 and especially one to detect when you create a file and forget to set permissions. Of course most of the time that's not a big deal but sometimes it can deploy bad permissions or even create security problems. One way to make a deployment is to do it manually and when you found the proper things to do you write the rules, right? One problem is that you might forget a few steps but since it's already manually done it deploys nicely until things change for another reason or you need to redeploy elsewhere or another instance. Another problem are these permissions which mean it's deployed but the service cannot access its files because too restrictive permissions or on the contrary it's too open. So it's better to recycle the VM or test in a container first. Currently we use the packaged version of ansible-lint so you did not see any problem _yet_. I have fixed the rules in all roles and also in the infra repo: https://gerrit.ovirt.org/c/111158/ I'm not pointing at people to blame them, and I am myself responsible for many of these mistakes (look at the history in the community-cage-infra-ansible repo or the roles), but I wanted to make sure we think about these problems and try to do better in the future. And btw I also added a YAMLlint check. Most of the time it's not a big deal but sometimes bad indenting causes bugs, especially inside blocks, or make things hardly readable, so I fixed these problems too. Regards. \_o<

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 356 - Failure!
by jenkins＠jenkins.phx.ovirt.org 08 Sep '20

08 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 356 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #356 [Marcin Sobczyk] ost_utils: network_utils: Add 'get_ips[6]' functions ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.100_basic_ui_sanity_pytest.test_virtual_machines[firefox] Error Message: TimeoutException: Message: Wait until VM list view is displayed failed Stack Trace: ovirt_driver = <test_utils.navigation.driver.Driver object at 0x7f4944070410> setup_virtual_machines = None save_screenshot = <function save at 0x7f4944aa5d70> save_page_source = <function save at 0x7f49455e4d70> def test_virtual_machines(ovirt_driver, setup_virtual_machines, save_screenshot, save_page_source): try: webadmin_menu = WebAdminLeftMenu(ovirt_driver) > vm_list_view = webadmin_menu.open_vm_list_view() ../basic-suite-master/test-scenarios/100_basic_ui_sanity_pytest.py:266: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../basic-suite-master/test_utils/page_objects/WebAdminLeftMenu.py:22: in open_vm_list_view vm_list_view.wait_for_displayed() ../basic-suite-master/test_utils/page_objects/Displayable.py:14: in wait_for_displayed self.ovirt_driver.wait_until('Wait until ' + self.get_displayable_name() + ' is displayed failed', self.is_displayed) ../basic-suite-master/test_utils/navigation/driver.py:282: in wait_until self._wait_until(message, testlib.SHORT_TIMEOUT, condition_method, *args) ../basic-suite-master/test_utils/navigation/driver.py:288: in _wait_until WebDriverWait(self.driver, timeout).until(ConditionClass(condition_method, *args), message) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <selenium.webdriver.support.wait.WebDriverWait (session="94efcd6d-2eef-47fb-b98b-8f59d5a01d8e")> method = <test_utils.navigation.driver.ConditionClass object at 0x7f4944ad9410> message = 'Wait until VM list view is displayed failed' def until(self, method, message=''): """Calls the method provided with the driver as an argument until the \ return value is not False.""" screen = None stacktrace = None end_time = time.time() + self._timeout while True: try: value = method(self._driver) if value: return value except self._ignored_exceptions as exc: screen = getattr(exc, 'screen', None) stacktrace = getattr(exc, 'stacktrace', None) time.sleep(self._poll) if time.time() > end_time: break > raise TimeoutException(message, screen, stacktrace) E TimeoutException: Message: Wait until VM list view is displayed failed ../.local/lib/python2.7/site-packages/selenium/webdriver/support/wait.py:80: TimeoutException

1 1

[oVirt Jenkins] ovirt-system-tests_upgrade-from-prevrelease-suite-4.3_nightly - Build # 310 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Sep '20

07 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build: https://jenkins.ovirt.org/job/ovirt-system-tests_upgrade-from-prevrelease-s… Build Number: 310 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #310 [Marcin Sobczyk] ost_utils: network_utils: Add 'get_ips[6]' functions ----------------- Failed Tests: ----------------- No tests ran.

1 1

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 352 - Failure!
by jenkins＠jenkins.phx.ovirt.org 07 Sep '20

07 Sep '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 352 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #352 [Marcin Sobczyk] ost_utils: network_utils: Add 'get_ips[6]' functions [Gal Ben Haim] dockerfile-utils: Better handling for nested repos ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_live_storage_migration Error Message: AssertionError: False != True after 600 seconds Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7f8b454470d0> @order_by(_TEST_LIST) def test_live_storage_migration(api_v4): engine = api_v4.system_service() disk_service = test_utils.get_disk_service(engine, DISK0_NAME) correlation_id = 'live_storage_migration' disk_service.move( async=False, filter=False, storage_domain=types.StorageDomain( name=SD_ISCSI_NAME ), query={'correlation_id': correlation_id} ) testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) # Assert that the disk is on the correct storage domain, # its status is OK and the snapshot created for the migration # has been merged testlib.assert_true_within_long( > lambda: api_v4.follow_link(disk_service.get().storage_domains[0]).name == SD_ISCSI_NAME ) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:580: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:286: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:273: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7f8b45200050>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10 ): allowed_exceptions = allowed_exceptions or [] with utils.EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: raise AssertionError( > '%s != %s after %s seconds' % (res, value, timeout) ) E AssertionError: False != True after 600 seconds /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:252: AssertionError

1 3