March 2019 - Infra - oVirt List Archives

[Ovirt] [CQ weekly status] [29-03-2019]
by Dafna Ron 29 Mar '19

29 Mar '19

Hi, This mail is to provide the current status of CQ and allow people to review status before and after the weekend. Please refer to below colour map for further information on the meaning of the colours. *CQ-4.2*: GREEN (#1) Last CQ job failure on 4.2 was 25-03-2019 on project ovirt-ansible-hosted-engine-setup due missing polkit package. *CQ-4.3*: RED (#1) failures in 4.3 and master this week were caused by two issues: 1. we have had random failures on 4 different tests which we are working on debugging with Marcin, Martin and several more people. this has been disruptive this week but did not cause any delays as I was re-triggering failed projects to insure they have their packages buit in tested. Currently we suspect api issue or changes to poastgres are causing the failures 2. we had libcgroup-tools package missing which cause failures for all projects in initialize engine. I merged a patch to fix the issue and provide the package. *CQ-Master:* RED (#1) We have had the same issues as in 4.3 this week. Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found here: [1] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-qu… [2] https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change-q… [3] http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change… Happy week! Dafna ------------------------------------------------------------------------------------------------------------------- COLOUR MAP Green = job has been passing successfully ** green for more than 3 days may suggest we need a review of our test coverage 1. 1-3 days GREEN (#1) 2. 4-7 days GREEN (#2) 3. Over 7 days GREEN (#3) Yellow = intermittent failures for different projects but no lasting or current regressions ** intermittent would be a healthy project as we expect a number of failures during the week ** I will not report any of the solved failures or regressions. 1. Solved job failures YELLOW (#1) 2. Solved regressions YELLOW (#2) Red = job has been failing ** Active Failures. The colour will change based on the amount of time the project/s has been broken. Only active regressions would be reported. 1. 1-3 days RED (#1) 2. 4-7 days RED (#2) 3. Over 7 days RED (#3)

1 0

[JIRA] (OVIRT-2711) Kernel soft lockup on vm0085.workers-phx.ovirt.org while building ovirt-node
by sbonazzo (oVirt JIRA) 29 Mar '19

29 Mar '19

sbonazzo created OVIRT-2711: ------------------------------- Summary: Kernel soft lockup on vm0085.workers-phx.ovirt.org while building ovirt-node Key: OVIRT-2711 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2711 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: sbonazzo Assignee: infra *https://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-fc28-x86_64/151/console <https://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-fc28-…>* *07:51:38* 06:51:37,952 EMERG kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [scsi_eh_1:85]*07:51:38* 06:51:37,952 WARNING kernel:Modules linked in: xfs fcoe libfcoe libfc scsi_transport_fc zram scsi_dh_rdac scsi_dh_emc scsi_dh_alua parport_pc i2c_piix4 parport joydev loop nls_utf8 isofs 8021q garp mrp stp llc virtio_console serio_raw qemu_fw_cfg virtio_pci e1000 bochs_drm drm_kms_helper ttm drm ata_generic pata_acpi sunrpc mcryptd sha256_ssse3 dm_crypt dm_round_robin dm_multipath linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs zstd_decompress xxhash cramfs edd virtio_rng virtio_ring virtio*07:51:38* 06:51:37,953 WARNING kernel:CPU: 0 PID: 85 Comm: scsi_eh_1 Not tainted 4.16.3-301.fc28.x86_64 #1*07:51:38* 06:51:37,954 WARNING kernel:Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28 04/01/2014*07:51:38* 06:51:37,954 WARNING kernel:RIP: 0010:_raw_spin_unlock_irqrestore+0xd/0x20*07:51:38* 06:51:37,954 WARNING kernel:RSP: 0018:ffffa53b8048bdf0 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff12*07:51:38* 06:51:37,954 WARNING kernel:RAX: 0000000000000000 RBX: ffff9898e8c58000 RCX: 0000000000000000*07:51:38* 06:51:37,955 WARNING kernel:RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202*07:51:38* 06:51:37,955 WARNING kernel:RBP: ffffffff9860bd20 R08: 0000000000000038 R09: 000000000000029e*07:51:38* 06:51:37,956 WARNING kernel:R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff9860b050*07:51:38* 06:51:37,956 WARNING kernel:R13: ffff9898e8c58130 R14: 0000000000000202 R15: 0000000000000000*07:51:38* 06:51:37,957 WARNING kernel:FS: 0000000000000000(0000) GS:ffff9898fbc00000(0000) knlGS:0000000000000000*07:51:38* 06:51:37,957 WARNING kernel:CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033*07:51:38* 06:51:37,957 WARNING kernel:CR2: 00007f544a48d6d7 CR3: 0000000068dfc000 CR4: 00000000000006f0*07:51:38* 06:51:37,957 WARNING kernel:Call Trace:*07:51:38* 06:51:37,958 WARNING kernel: ata_sff_error_handler+0x83/0xe0*07:51:38* 06:51:37,958 WARNING kernel: ata_scsi_port_error_handler+0x354/0x770*07:51:38* 06:51:37,958 WARNING kernel: ? scsi_try_target_reset+0x90/0x90*07:51:38* 06:51:37,959 WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*07:51:38* 06:51:37,959 WARNING kernel: ata_scsi_error+0x91/0xc0*07:51:38* 06:51:37,959 WARNING kernel: scsi_error_handler+0xd0/0x5b0*07:51:38* 06:51:37,959 WARNING kernel: ? __schedule+0x23f/0x850*07:51:38* 06:51:37,960 WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*07:51:38* 06:51:37,960 WARNING kernel: kthread+0x112/0x130*07:51:38* 06:51:37,960 WARNING kernel: ? kthread_create_worker_on_cpu+0x70/0x70*07:51:38* 06:51:37,960 WARNING kernel: ? kthread_create_worker_on_cpu+0x70/0x70*07:51:38* 06:51:37,960 WARNING kernel: ret_from_fork+0x35/0x40*07:51:38* 06:51:37,961 WARNING kernel:Code: a8 08 74 0b 65 81 25 6f 2c 76 67 ff ff ff 7f 89 d0 c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 48 89 f7 57 9d <0f> 1f 44 00 00 c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f Running yum check-update on the slave shows several updates available including kernel. I would recommend to upgrade slaves. -- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://red.ht/sig> -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

Build failed in Jenkins: system-sync_mirrors-glusterfs-5-el7-x86_64 #205
by jenkins＠jenkins.phx.ovirt.org 29 Mar '19

29 Mar '19

See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-5-el7-x86_64/205…> ------------------------------------------ Started by timer [EnvInject] - Loading node environment variables. Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-5-el7-x86_64/ws/> No credentials specified > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 Pruning obsolete local branches Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git > git --version # timeout=10 > git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision e2723ce1cb5307c98934c8f527598905fdf087ce (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f e2723ce1cb5307c98934c8f527598905fdf087ce Commit message: "stdci_slaves: increase memory requirement" > git rev-list --no-walk e2723ce1cb5307c98934c8f527598905fdf087ce # timeout=10 [system-sync_mirrors-glusterfs-5-el7-x86_64] $ /bin/bash -xe /tmp/jenkins504825524289935617.sh + jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf + MIRRORS_MP_BASE=/var/www/html/repos + MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos + MIRRORS_CACHE=/home/jenkins/mirrors_cache + MAX_LOCK_ATTEMPTS=120 + LOCK_WAIT_INTERVAL=5 + LOCK_BASE=/home/jenkins + OLD_MD_TO_KEEP=100 + HTTP_SELINUX_TYPE=httpd_sys_content_t + HTTP_FILE_MODE=644 + main resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf + local command=resync_yum_mirror + command_args=("${@:2}") + local command_args + cmd_resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf + local repo_name=glusterfs-5-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local sync_needed + mkdir -p /home/jenkins/mirrors_cache + verify_repo_fs glusterfs-5-el7 yum + local repo_name=glusterfs-5-el7 + local repo_type=yum + sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/glusterfs-5-el7 /var/www/html/repos/yum/glusterfs-5-el7/base + check_yum_sync_needed glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed + local repo_name=glusterfs-5-el7 + local repo_archs=x86_64 + local reposync_conf=jenkins/data/mirrors-reposync.conf + local p_sync_needed=sync_needed + local reposync_out + echo 'Checking if mirror needs a resync' Checking if mirror needs a resync + rm -rf /home/jenkins/mirrors_cache/glusterfs-5-el7 ++ IFS=, ++ echo x86_64 + for arch in '$(IFS=,; echo $repo_archs)' ++ run_reposync glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet ++ local repo_name=glusterfs-5-el7 ++ local repo_arch=x86_64 ++ local reposync_conf=jenkins/data/mirrors-reposync.conf ++ extra_args=("${@:4}") ++ local extra_args ++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=glusterfs-5-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/glusterfs-5-el7/base --norepopath --newest-only --urls --quiet Error setting up repositories: glusterfs-5-el7: Check uncompressed DB failed + reposync_out= Build step 'Execute shell' marked build as failure

1 1

[oVirt Jenkins] ovirt-system-tests_compat-4.2-suite-master - Build # 526 - Failure!
by jenkins＠jenkins.phx.ovirt.org 29 Mar '19

29 Mar '19

Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/ Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/526/ Build Number: 526 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #526 No changes ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.verify_add_all_hosts_42 Error Message: 1 hosts failed installation: lago-compat-4-2-suite-master-host-1: non_operational -------------------- >> begin captured logging << -------------------- ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2597ab1488> Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts), File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts raise RuntimeError(dump_hosts) RuntimeError: 1 hosts failed installation: lago-compat-4-2-suite-master-host-1: non_operational --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 343, in verify_add_all_hosts timeout=constants.ADD_HOST_TIMEOUT File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within res = func() File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda> lambda: _all_hosts_up(hosts_service, total_hosts), File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up _check_problematic_hosts(hosts_service) File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts raise RuntimeError(dump_hosts) '1 hosts failed installation:\nlago-compat-4-2-suite-master-host-1: non_operational\n\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2597ab1488>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda>\n lambda: _all_hosts_up(hosts_service, total_hosts),\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up\n _check_problematic_hosts(hosts_service)\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts\n raise RuntimeError(dump_hosts)\nRuntimeError: 1 hosts failed installation:\nlago-compat-4-2-suite-master-host-1: non_operational\n\n--------------------- >> end captured logging << ---------------------'

1 1

oVirt infra daily report - unstable production jobs - 778
by jenkins＠jenkins.phx.ovirt.org 29 Mar '19

29 Mar '19

Good morning! Attached is the HTML page with the jenkins status report. You can see it also here: - http://jenkins.ovirt.org/job/system_jenkins-report/778//artifact/exported-a… Cheers, Jenkins

1 0

[CQ]: 99014, 1 (ovirt-engine-metrics) failed "ovirt-master" system tests
by oVirt Jenkins 28 Mar '19

28 Mar '19

Change 99014,1 (ovirt-engine-metrics) is probably the reason behind recent system test failures in the "ovirt-master" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/99014/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13616/

1 0

[JIRA] (OVIRT-2705) Re: way to link my Github account with Gerrit?
by Evgheni Dereveanchin (oVirt JIRA) 28 Mar '19

28 Mar '19

[ https://ovirt-jira.atlassian.net/browse/OVIRT-2705?page=com.atlassian.jira.… ] Evgheni Dereveanchin updated OVIRT-2705: ---------------------------------------- Resolution: Fixed Status: Done (was: To Do) > Re: way to link my Github account with Gerrit? > ---------------------------------------------- > > Key: OVIRT-2705 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2705 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Anton Marchukov > Assignee: Evgheni Dereveanchin > > Hello Sahina. > Yes, we can merge those accounts. I am adding infra-support to make sure the ticket is opened on this. > Anton. > > On 22 Mar 2019, at 11:36, Sahina Bose <sabose(a)redhat.com> wrote: > > > > Hi, > > It looks like I ended up creating 2 different gerrit accounts. Is > > there a way to link the two? or delete one ? > > > > thanks > > sahina > > _______________________________________________ > > Infra mailing list -- infra(a)ovirt.org > > To unsubscribe send an email to infra-leave(a)ovirt.org > > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > > List Archives: https://lists.ovirt.org/archives/list/infra@ovirt.org/message/F5QSCR42G5KQP… > -- > Anton Marchukov > Associate Manager - RHV DevOps - Red Hat -- This message was sent by Atlassian Jira (v1001.0.0-SNAPSHOT#100099)

1 0

[CQ]: 98965, 1 (ovirt-engine-metrics) failed "ovirt-master" system tests, but isn't the failure root cause
by oVirt Jenkins 28 Mar '19

28 Mar '19

A system test invoked by the "ovirt-master" change queue including change 98965,1 (ovirt-engine-metrics) failed. However, this change seems not to be the root cause for this failure. Change 98825,13 (ovirt-engine-metrics) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 98825,13 (ovirt-engine-metrics) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/98965/1 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/98825/13 For failed test results see: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/13613/

1 0

[CQ]: 99008,1 (cockpit-ovirt) failed "ovirt-4.3" system tests
by oVirt Jenkins 28 Mar '19

28 Mar '19

Change 99008,1 (cockpit-ovirt) is probably the reason behind recent system test failures in the "ovirt-4.3" change queue and needs to be fixed. This change had been removed from the testing queue. Artifacts build from this change will not be released until it is fixed. For further details about the change see: https://gerrit.ovirt.org/#/c/99008/1 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/471/

1 0

[CQ]: 98945, 4 (ovirt-engine-metrics) failed "ovirt-4.3" system tests, but isn't the failure root cause
by oVirt Jenkins 28 Mar '19

28 Mar '19

A system test invoked by the "ovirt-4.3" change queue including change 98945,4 (ovirt-engine-metrics) failed. However, this change seems not to be the root cause for this failure. Change 98825,13 (ovirt-engine-metrics) that this change depends on or is based on, was detected as the cause of the testing failures. This change had been removed from the testing queue. Artifacts built from this change will not be released until either change 98825,13 (ovirt-engine-metrics) is fixed and this change is updated to refer to or rebased on the fixed version, or this change is modified to no longer depend on it. For further details about the change see: https://gerrit.ovirt.org/#/c/98945/4 For further details about the change that seems to be the root cause behind the testing failures see: https://gerrit.ovirt.org/#/c/98825/13 For failed test results see: http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/469/

1 0