July 2020 - Infra - oVirt List Archives

[oVirt Jenkins] ovirt-system-tests_basic-suite-4.3_nightly - Build # 211 - Failure!
by jenkins＠jenkins.phx.ovirt.org 22 Jul '20

22 Jul '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-4.3_nightly/21… Build Number: 211 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #211 [Marcin Sobczyk] lago: Skip sysprep when ssh key is available ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 004_basic_sanity.disk_operations Error Message: False != True after 600 seconds -------------------- >> begin captured logging << -------------------- lago.utils: DEBUG: Error while running thread Thread-70 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: False != True after 600 seconds --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 932, in disk_operations vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) 'False != True after 600 seconds\n-------------------- >> begin captured logging << --------------------\nlago.utils: DEBUG: Error while running thread Thread-70\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/lago/utils.py", line 58, in _ret_via_queue\n queue.put({\'return\': func()})\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper\n return func(get_test_prefix(), *args, **kwargs)\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 79, in wrapper\n prefix.virt_env.engine_vm().get_api(api_ver=4), *args, **kwargs\n File "/home/jenkins/agent/workspace/ovirt-system-tests_basic-suite-4.3_nightly/ovirt-system-tests/basic-suite-4.3/test-scenarios/004_basic_sanity.py", line 508, in live_storage_migration\n testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id))\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 286, in assert_true_within_long\n assert_equals_within_long(func, True, allowed_exceptions)\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 273, in assert_equals_within_long\n func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 252, in assert_equals_within\n \'%s != %s after %s seconds\' % (res, value, timeout)\nAssertionError: False != True after 600 seconds\n--------------------- >> end captured logging << ---------------------'

1 1

oVirt infra daily report - unstable production jobs - 1260
by jenkins＠jenkins.phx.ovirt.org 22 Jul '20

22 Jul '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1260//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2976) [FIRING:1] MirrorIsOutOfSync (web-ovirt-mirrorchecker.apps.ovirt.org:80 oVirt-mirrors-health http://www.gtlib.gatech.edu/pub/oVirt/pub)
by Alertmanager_Bot (oVirt JIRA) 22 Jul '20

22 Jul '20

Alertmanager_Bot created OVIRT-2976: --------------------------------------- Summary: [FIRING:1] MirrorIsOutOfSync (web-ovirt-mirrorchecker.apps.ovirt.org:80 oVirt-mirrors-health http://www.gtlib.gatech.edu/pub/oVirt/pub) Key: OVIRT-2976 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2976 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = MirrorIsOutOfSync - instance = web-ovirt-mirrorchecker.apps.ovirt.org:80 - job = oVirt-mirrors-health - url = http://www.gtlib.gatech.edu/pub/oVirt/pub Annotations: - description = http://www.gtlib.gatech.edu/pub/oVirt/pub (measured by oVirt-mirrors-health) has not been synced in the past two days. Source: http://prometheus-0:9090/graph?g0.expr=mirror_health_check+%3E+172800&g0.ta… -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2268) Enable IPv6 for resources.ovirt.org
by Anton Marchukov (oVirt JIRA) 22 Jul '20

22 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2268?page=com.atlassian.jira.… ] Anton Marchukov commented on OVIRT-2268: ---------------------------------------- Since we cannot provide IPv6 on [resources.ovirt.org|http://resources.ovirt.org] let’s redefine this ticket to make sure IPv6-enabled users will get mirrorlist wit h IPv6-enables mirrors only. The target is to make sure IPv6-only people can get oVirt. > Enable IPv6 for resources.ovirt.org > ----------------------------------- > > Key: OVIRT-2268 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2268 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Components: oVirt Infra > Reporter: Dominik Holler > Assignee: infra > > To allow access to resources.ovirt.org in IPv6 only setups, IPv6 should be enabled for resources.ovirt.org. > This would allow users in IPv6 only setups to install oVirt. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2975) upload CirrOS 0.5.1 to glance
by Evgheni Dereveanchin (oVirt JIRA) 22 Jul '20

22 Jul '20

Evgheni Dereveanchin created OVIRT-2975: ------------------------------------------- Summary: upload CirrOS 0.5.1 to glance Key: OVIRT-2975 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2975 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra http://download.cirros-cloud.net/0.5.1/cirros-0.5.1-x86_64-disk.img -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

Re: [ovirt-users] invalid spf record for ovirt.org
by Yedidyah Bar David 22 Jul '20

22 Jul '20

On Wed, Jul 22, 2020 at 3:57 PM Jorick Astrego <jorick(a)netbulae.eu> wrote: > Hi, > > During routine maintenance on our mailserver I noticed the following in > the log: > > [22/Jul/2020 14:33:33] Error when parsing SPF TXT record for domain: > ovirt.org, envelope-from=users-bounces(a)ovirt.org, message: Invalid > character found near "" in "66.187.233.88" > > A check on MXtoolbox also gives an invalid systax error: > > v=spf1 a:mail.ovirt.org a:gerrit.ovirt.org 66.187.233.88 ~all > > Prefix Type Value PrefixDesc Description Error > > v spf1 > The SPF record version > + a mail.ovirt.org Pass Match if IP has a DNS 'A' record in given domain. > + a gerrit.ovirt.org Pass Match if IP has a DNS 'A' record in given > domain. > + 66.187.233.88 > Pass Unknown Unknown mechanisms are not allowed > ~ all > SoftFail Always matches. It goes at the end of your record. > > > Test Result > > <https://mxtoolbox.com/problem/spf/spf-syntax-check?page=prob_spf&action=spf…> SPF > Syntax Check Invalid syntax found > > Thanks for the report. Adding some more relevant people. Best regards, -- Didi

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 208 - Failure!
by jenkins＠jenkins.phx.ovirt.org 21 Jul '20

21 Jul '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 208 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #208 [Marcin Sobczyk] performance: Fix broken links for pgdg repos ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: basic-suite-master.test-scenarios.004_basic_sanity_pytest.test_live_storage_migration Error Message: AssertionError: False != True after 600 seconds Stack Trace: api_v4 = <ovirtsdk4.Connection object at 0x7f86a012a050> @order_by(_TEST_LIST) def test_live_storage_migration(api_v4): engine = api_v4.system_service() disk_service = test_utils.get_disk_service(engine, DISK0_NAME) correlation_id = 'live_storage_migration' disk_service.move( async=False, filter=False, storage_domain=types.StorageDomain( name=SD_ISCSI_NAME ), query={'correlation_id': correlation_id} ) testlib.assert_true_within_long(lambda: test_utils.all_jobs_finished(engine, correlation_id)) # Assert that the disk is on the correct storage domain, # its status is OK and the snapshot created for the migration # has been merged testlib.assert_true_within_long( lambda: api_v4.follow_link(disk_service.get().storage_domains[0]).name == SD_ISCSI_NAME ) vm0_snapshots_service = test_utils.get_vm_snapshots_service(engine, VM0_NAME) testlib.assert_true_within_long( > lambda: len(vm0_snapshots_service.list()) == 1 ) ../basic-suite-master/test-scenarios/004_basic_sanity_pytest.py:576: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:286: in assert_true_within_long assert_equals_within_long(func, True, allowed_exceptions) /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:273: in assert_equals_within_long func, value, LONG_TIMEOUT, allowed_exceptions=allowed_exceptions _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ func = <function <lambda> at 0x7f869c96daa0>, value = True, timeout = 600 allowed_exceptions = [], initial_wait = 0 def assert_equals_within( func, value, timeout, allowed_exceptions=None, initial_wait=10 ): allowed_exceptions = allowed_exceptions or [] with utils.EggTimer(timeout) as timer: while not timer.elapsed(): try: res = func() if res == value: return except Exception as exc: if _instance_of_any(exc, allowed_exceptions): time.sleep(3) continue LOGGER.exception("Unhandled exception in %s", func) raise if initial_wait == 0: time.sleep(3) else: time.sleep(initial_wait) initial_wait = 0 try: raise AssertionError( > '%s != %s after %s seconds' % (res, value, timeout) ) E AssertionError: False != True after 600 seconds /usr/lib/python2.7/site-packages/ovirtlago/testlib.py:252: AssertionError

1 1

oVirt infra daily report - unstable production jobs - 1259
by jenkins＠jenkins.phx.ovirt.org 21 Jul '20

21 Jul '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1259//artifact/exported… Cheers, Jenkins

1 0

Build failed in Jenkins: system-sync_mirrors-fedora-updates-fc30-x86_64 #1319
by jenkins＠jenkins.phx.ovirt.org 21 Jul '20

21 Jul '20

See <https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-updates-fc30-x86_6…> Changes: ------------------------------------------ Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <https://jenkins.ovirt.org/job/system-sync_mirrors-fedora-updates-fc30-x86_6…> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress --prune http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* # timeout=10 > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision e74e3d234ca539ec7d6be740575b720d5577e5a9 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f e74e3d234ca539ec7d6be740575b720d5577e5a9 # timeout=10 Commit message: "Removing rgolan(a)redhat.com from email notifs" > git rev-list --no-walk e74e3d234ca539ec7d6be740575b720d5577e5a9 # timeout=10 [system-sync_mirrors-fedora-updates-fc30-x86_64] $ /bin/bash -xe /tmp/jenkins141163348633797447.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-updates-fc30 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + YUM_PATH=/yum/ + MIRRORS_BASE_PREFIX=/var/www/html/repos/yum/ + REPOQUERY_HTTP_BASE=http://mirrors.phx.ovirt.org/repos/yum/ + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + MAX_AGE=14 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror fedora-updates-fc30 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror fedora-updates-fc30 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=fedora-updates-fc30 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs fedora-updates-fc30 yum + local repo_name=fedora-updates-fc30 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-updates-fc30 /var/www/html/repos/yum/fedora-updates-fc30/base + modify_reposync_conf fedora-updates-fc30 jenkins/data/mirrors-reposync.conf + local repo_name=fedora-updates-fc30 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local tmp_file=reposync.tmp + awk -v RS= -v 'ORS=\n\n' /main/ jenkins/data/mirrors-reposync.conf + awk -v RS= -v 'ORS=\n\n' /fedora-updates-fc30/ jenkins/data/mirrors-reposync.conf + mv reposync.tmp jenkins/data/mirrors-reposync.conf + check_yum_sync_needed fedora-updates-fc30 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=fedora-updates-fc30 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/fedora-updates-fc30 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync fedora-updates-fc30 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=fedora-updates-fc30 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-updates-fc30 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-updates-fc30/base --norepopath --urls --quiet Error setting up repositories: Cannot retrieve metalink for repository: fedora-updates-fc30. Please verify its path and try again + reposync_out= Build step 'Execute shell' marked build as failure

1 1

[JIRA] (OVIRT-2974) Tickets aren't created for fired alerts in Prometheus
by Shlomi Zidmi (oVirt JIRA) 21 Jul '20

21 Jul '20

Shlomi Zidmi created OVIRT-2974: ----------------------------------- Summary: Tickets aren't created for fired alerts in Prometheus Key: OVIRT-2974 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2974 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Shlomi Zidmi Assignee: infra There is a broken node_exporter on one of openshift nodes (shift-n12), however we haven't been notified about it with a ticket. For some reason, even though all application are running (Prometheus, Alertmanager, Jiralert) no ticket has been created for the broken exporter. Opening a ticket for investigating the issue -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[oVirt Jenkins] ovirt-system-tests_performance-suite-master - Build # 1521 - Still Failing!
by jenkins＠jenkins.phx.ovirt.org 20 Jul '20

20 Jul '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/1… Build Number: 1521 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1516 [Marcin Sobczyk] network: Run network suite master on el8 Changes for Build #1517 [Andrej Cernek] network, hostlib: refactor setup_networks method [Barak Korren] usrc: Support *.yml files Changes for Build #1518 [Galit] Remove basic-suite-4.2 suite Changes for Build #1519 [Marcin Sobczyk] basic: Verify glance template creation before tearing down the engine Changes for Build #1520 [Marcin Sobczyk] ost-images: Use OST images when available Changes for Build #1521 [Ehud Yonasi] fix el7 container label. [Roy Golan] Removing rgolan(a)redhat.com from email notifs ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 1000_analyze_postgresql.analyze_postgres Error Message: Failed to perform the postgres analysis. Exit code is 1 -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[0m?[0m lago.ssh: DEBUG: start task:a9b6bf76-d6a4-4f6d-938d-59d14cc805d8:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:a9b6bf76-d6a4-4f6d-938d-59d14cc805d8:Get ssh client for lago-performance-suite-master-engine: lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[32mSuccess?[0m (in 0:00:00) lago.ssh: DEBUG: start task:3cbcc980-918f-4be9-ac5e-e10f86305521:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:3cbcc980-918f-4be9-ac5e-e10f86305521:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: Running aee507f2 on lago-performance-suite-master-engine: /root/analyze_postgresql.sh lago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine returned with 1 lago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine output: Last metadata expiration check: 0:11:06 ago on Thu 16 Jul 2020 11:03:48 PM EDT. [MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) [MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) [MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) [MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) [FAILED] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) lago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine errors: + pg_datadir=/var/lib/pgsql/data + pg_service=postgresql.service + pgdg= + scl_enable= ++ get_engine_pg_scl ++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf) ++ local engine_pg_scl_conf ++ local res= ++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]] ++ echo '' + engine_pg_scl= + [[ -n '' ]] ++ rpm -q postgresql + pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64 + case "${pgver}" in + pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/… + [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… ]] + yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246) --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/1000_analyze_postgresql.py", line 52, in analyze_postgres ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) 'Failed to perform the postgres analysis. Exit code is 1\n-------------------- >> begin captured logging << --------------------\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:a9b6bf76-d6a4-4f6d-938d-59d14cc805d8:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:a9b6bf76-d6a4-4f6d-938d-59d14cc805d8:Get ssh client for lago-performance-suite-master-engine:\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[32mSuccess\x1b[0m (in 0:00:00)\nlago.ssh: DEBUG: start task:3cbcc980-918f-4be9-ac5e-e10f86305521:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:3cbcc980-918f-4be9-ac5e-e10f86305521:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: Running aee507f2 on lago-performance-suite-master-engine: /root/analyze_postgresql.sh\nlago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine returned with 1\nlago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine output:\n Last metadata expiration check: 0:11:06 ago on Thu 16 Jul 2020 11:03:48 PM EDT.\n[MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n[MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n[MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n[MIRROR] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n[FAILED] pgdg-redhat-repo-latest.noarch.rpm: Status code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n\nlago.ssh: DEBUG: Command aee507f2 on lago-performance-suite-master-engine errors:\n + pg_datadir=/var/lib/pgsql/data\n+ pg_service=postgresql.service\n+ pgdg=\n+ scl_enable=\n++ get_engine_pg_scl\n++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf)\n++ local engine_pg_scl_conf\n++ local res=\n++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]]\n++ echo \'\'\n+ engine_pg_scl=\n+ [[ -n \'\' ]]\n++ rpm -q postgresql\n+ pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64\n+ case "${pgver}" in\n+ pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/… [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… ]]\n+ yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… code: 404 for https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-… (IP: 72.32.157.246)\n\n--------------------- >> end captured logging << ---------------------'

1 4

oVirt infra daily report - unstable production jobs - 1258
by jenkins＠jenkins.phx.ovirt.org 20 Jul '20

20 Jul '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1258//artifact/exported… Cheers, Jenkins

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Evgheni Dereveanchin (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2972: --------------------------------------------- Suspect jobs that were reporting issues this weekend: [https://jenkins.ovirt.org/job/ovirt_master_publish-rpms_nightly/|https://je…] [https://jenkins.ovirt.org/job/ovirt_4.3_publish-rpms_nightly/|https://jenki…] [https://jenkins.ovirt.org/job/ovirt_4.2_publish-rpms_nightly/|https://jenki…] The errors are as follows: {noformat}13:15:32 2020-07-19 11:15:32,180::INFO::root::Saving /home/jenkins/ovirt-master-snapshot.tmp/rpm/fc29/x86_64/ovirt-engine-wildfly-18.0.0-1.fc29.x86_64.rpm 13:15:32 Traceback (most recent call last): 13:15:32 File "/bin/repoman", line 10, in <module> 13:15:32 sys.exit(main()) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/cmd.py", line 461, in main 13:15:32 exit_code = do_add(args, config, repo) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/cmd.py", line 361, in do_add 13:15:32 repo.save() 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/repo.py", line 42, in _func 13:15:32 return func(self, *args, **kwargs) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/repo.py", line 201, in save 13:15:32 store.save() 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/stores/RPM/__init__.py", line 263, in save 13:15:32 self._save(**args) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/stores/RPM/__init__.py", line 302, in _save 13:15:32 save_file(pkg.path, dst_path) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/utils.py", line 436, in save_file 13:15:32 copy(src_path, dst_path) 13:15:32 File "/usr/lib/python2.7/site-packages/repoman/common/utils.py", line 296, in copy 13:15:32 shutil.copy2(what, where) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 130, in copy2 13:15:32 copyfile(src, dst) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 84, in copyfile 13:15:32 copyfileobj(fsrc, fdst) 13:15:32 File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj 13:15:32 fdst.write(buf) 13:15:32 IOError: [Errno 28] No space left on device 13:15:32 2020-07-19 11:15:32,917::INFO::repoman.common.repo::Cleaning up temporary dir /tmp/tmpRKo543 13:15:34 Waiting for scan_for_artifacts.sh cron job to finish (attempt 1 of 50) {noformat} looks like the job ran out of tmpfs space - did we publish too many files for some reason? Not entirely sure what tmp is used for in these jobs. > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png, prom.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Evgheni Dereveanchin (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2972: --------------------------------------------- I did get emails from Nagios so we were alerted, just want to confirm the cause here as that’s something that doesn’t happen often: looks like some publishing job failed and did not clean up after itself which caused the next one to bring in even more files and fill up the partition. > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png, prom.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2973) [FIRING:1] TestAlert
by Shlomi Zidmi (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2973?page=com.atlassian.jira.… ] Shlomi Zidmi updated OVIRT-2973: -------------------------------- Resolution: Fixed Status: Done (was: To Do) Making sure jiralert is functioning properly > [FIRING:1] TestAlert > --------------------- > > Key: OVIRT-2973 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2973 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Alertmanager_Bot > Assignee: infra > Labels: ALERT{alertname="TestAlert"} > > Labels: > - alertname = TestAlert > - key = value > Annotations: > Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2973) [FIRING:1] TestAlert
by Alertmanager_Bot (oVirt JIRA) 20 Jul '20

20 Jul '20

Alertmanager_Bot created OVIRT-2973: --------------------------------------- Summary: [FIRING:1] TestAlert Key: OVIRT-2973 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2973 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Alertmanager_Bot Assignee: infra Labels: - alertname = TestAlert - key = value Annotations: Source: -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Shlomi Zidmi (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Shlomi Zidmi commented on OVIRT-2972: ------------------------------------- Actually we are monitoring [resources.ovirt.org|http://resources.ovirt.org] in Prometheus, and should have caught this on time !prom.png|width=83.33333333333334%! Looking at the alert rules defined, i see the following: {noformat}node_filesystem_free_bytes / node_filesystem_size_bytes{device!="tmpfs"} < 0.1{noformat} The blue line (/dev/sdb - mounted on /home/jenkins) in the picture is slightly above 0.1, and that’s why no alerts were fired. I’ll readjust the threshold so that Prometheus would be able to catch similar issues sooner next time. > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png, prom.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Galit Rosenthal (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Galit Rosenthal edited comment on OVIRT-2972 at 7/20/20 9:13 AM: ----------------------------------------------------------------- When I fixed the issues, the nagios already send a message that the disk space came back to normal. I just removed the *.ready directory that prevented from the publish to work was (Author: grosenth(a)redhat.com) When I fixed the issues, the jenkins already send a message that the disk space came back to normal. I just removed the *.ready directory that prevented from the publish to work > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Galit Rosenthal (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Galit Rosenthal commented on OVIRT-2972: ---------------------------------------- When I fixed the issues, the jenkins already send a message that the disk space came back to normal. I just removed the *.ready directory that prevented from the publish to work > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

ovirt-engine-wildfly 19.1.0-2 publishing
by Yedidyah Bar David 20 Jul '20

20 Jul '20

Hi all, 1. Yesterday mperina pushed and merged a patch [1] to hopefully solve a rather-long-time breakage of, among others, he-basic-suite. I still do not see it in master-snapshot [2] - only 19.1.0-1. I do see it in tested [3]. Can someone please publish it to snapshot? 2. I tried to debug this myself a bit, but failed to find documentation for the publishing part. A few months ago it was said (not very clearly/officially, IIRC) that we do not anymore pass built-artifacts results through OST, because it's too fragile. Did we update docs? Where can I find a clear overview of the process from pressing Merge to having an rpm in master snapshot? Thanks and best regards, [1] https://gerrit.ovirt.org/110324 [2] https://resources.ovirt.org/pub/ovirt-master-snapshot/rpm/el8/ [3] https://resources.ovirt.org/repos/ovirt/tested/master/rpm/el8/ -- Didi

3 4

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Ehud Yonasi (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Ehud Yonasi commented on OVIRT-2972: ------------------------------------ I read that on Sunday the repo was not accessible. I think [~accountid:5b25ec3f8b08c009c48b25c9] did cleanup there and might add more info on what used a lot of storage. > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Evgheni Dereveanchin (oVirt JIRA) 20 Jul '20

20 Jul '20

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2972?page=com.atlassian.jira.… ] Evgheni Dereveanchin commented on OVIRT-2972: --------------------------------------------- !image-20200720-083159.png|width=597,height=215! Here’s the usage graph, looks like something happened on the night of 17th that caused files not to clean up, then repeated on the 18th and has been fixed during the 19th. [~accountid:5aa0f39f5a4d022884128a0f] [~accountid:5c977456c430371a3c67dbf6] did you get any reports on Sunday? May this be related to some nightly jobs? > resources.ovirt.org ran out of space for artifact publishing > ------------------------------------------------------------ > > Key: OVIRT-2972 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 > Project: oVirt - virtualization made easy > Issue Type: Bug > Reporter: Evgheni Dereveanchin > Assignee: infra > Attachments: image-20200720-083159.png > > > This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org > AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. > The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[JIRA] (OVIRT-2972) resources.ovirt.org ran out of space for artifact publishing
by Evgheni Dereveanchin (oVirt JIRA) 20 Jul '20

20 Jul '20

Evgheni Dereveanchin created OVIRT-2972: ------------------------------------------- Summary: resources.ovirt.org ran out of space for artifact publishing Key: OVIRT-2972 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2972 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Evgheni Dereveanchin Assignee: infra This weekend Nagios sent disk space alerts for the /home/jenkins on resources.ovirt.org AFAIR this is used as intermediate artifact storage for publishing and shouldn't fill up that much. Logging a ticket to investigate the root cause. The partition is free again now so this is not blocking anything right now. -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100133)

1 0

[oVirt Jenkins] ovirt-system-tests_basic-suite-master_nightly - Build # 204 - Failure!
by jenkins＠jenkins.phx.ovirt.org 20 Jul '20

20 Jul '20

Project: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_basic-suite-master_nightly… Build Number: 204 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #204 [Marcin Sobczyk] ost_utils: Add ansible fixtures ----------------- Failed Tests: ----------------- No tests ran.

1 1

oVirt infra daily report - unstable production jobs - 1257
by jenkins＠jenkins.phx.ovirt.org 19 Jul '20

19 Jul '20

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - https://jenkins.ovirt.org/job/system_jenkins-report/1257//artifact/exported… Cheers, Jenkins

1 0