[Ovirt] [CQ weekly status] [29-03-2019]
by Dafna Ron
Hi,
This mail is to provide the current status of CQ and allow people to review
status before and after the weekend.
Please refer to below colour map for further information on the meaning of
the colours.
*CQ-4.2*: GREEN (#1)
Last CQ job failure on 4.2 was 25-03-2019 on project
ovirt-ansible-hosted-engine-setup due missing polkit package.
*CQ-4.3*: RED (#1)
failures in 4.3 and master this week were caused by two issues:
1. we have had random failures on 4 different tests which we are working on
debugging with Marcin, Martin and several more people. this has been
disruptive this week but did not cause any delays as I was re-triggering
failed projects to insure they have their packages buit in tested.
Currently we suspect api issue or changes to poastgres are causing the
failures
2. we had libcgroup-tools package missing which cause failures for all
projects in initialize engine. I merged a patch to fix the issue and
provide the package.
*CQ-Master:* RED (#1)
We have had the same issues as in 4.3 this week.
Current running jobs for 4.2 [1], 4.3 [2] and master [3] can be found
here:
[1]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change-...
[2]
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.3_change...
[3]
http://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_chan...
Happy week!
Dafna
-------------------------------------------------------------------------------------------------------------------
COLOUR MAP
Green = job has been passing successfully
** green for more than 3 days may suggest we need a review of our test
coverage
1.
1-3 days GREEN (#1)
2.
4-7 days GREEN (#2)
3.
Over 7 days GREEN (#3)
Yellow = intermittent failures for different projects but no lasting or
current regressions
** intermittent would be a healthy project as we expect a number of
failures during the week
** I will not report any of the solved failures or regressions.
1.
Solved job failures YELLOW (#1)
2.
Solved regressions YELLOW (#2)
Red = job has been failing
** Active Failures. The colour will change based on the amount of time the
project/s has been broken. Only active regressions would be reported.
1.
1-3 days RED (#1)
2.
4-7 days RED (#2)
3.
Over 7 days RED (#3)
5 years, 7 months
[JIRA] (OVIRT-2711) Kernel soft lockup on
vm0085.workers-phx.ovirt.org while building ovirt-node
by sbonazzo (oVirt JIRA)
sbonazzo created OVIRT-2711:
-------------------------------
Summary: Kernel soft lockup on vm0085.workers-phx.ovirt.org while building ovirt-node
Key: OVIRT-2711
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2711
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: sbonazzo
Assignee: infra
*https://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-fc28-x86_64/151/console
<https://jenkins.ovirt.org/job/ovirt-node-ng-image_4.3_build-artifacts-fc2...>*
*07:51:38* 06:51:37,952 EMERG kernel:watchdog: BUG: soft lockup -
CPU#0 stuck for 21s! [scsi_eh_1:85]*07:51:38* 06:51:37,952 WARNING
kernel:Modules linked in: xfs fcoe libfcoe libfc scsi_transport_fc
zram scsi_dh_rdac scsi_dh_emc scsi_dh_alua parport_pc i2c_piix4
parport joydev loop nls_utf8 isofs 8021q garp mrp stp llc
virtio_console serio_raw qemu_fw_cfg virtio_pci e1000 bochs_drm
drm_kms_helper ttm drm ata_generic pata_acpi sunrpc mcryptd
sha256_ssse3 dm_crypt dm_round_robin dm_multipath linear raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs
zstd_decompress xxhash cramfs edd virtio_rng virtio_ring
virtio*07:51:38* 06:51:37,953 WARNING kernel:CPU: 0 PID: 85 Comm:
scsi_eh_1 Not tainted 4.16.3-301.fc28.x86_64 #1*07:51:38* 06:51:37,954
WARNING kernel:Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28
04/01/2014*07:51:38* 06:51:37,954 WARNING kernel:RIP:
0010:_raw_spin_unlock_irqrestore+0xd/0x20*07:51:38* 06:51:37,954
WARNING kernel:RSP: 0018:ffffa53b8048bdf0 EFLAGS: 00000202 ORIG_RAX:
ffffffffffffff12*07:51:38* 06:51:37,954 WARNING kernel:RAX:
0000000000000000 RBX: ffff9898e8c58000 RCX: 0000000000000000*07:51:38*
06:51:37,955 WARNING kernel:RDX: 0000000000000000 RSI:
0000000000000202 RDI: 0000000000000202*07:51:38* 06:51:37,955 WARNING
kernel:RBP: ffffffff9860bd20 R08: 0000000000000038 R09:
000000000000029e*07:51:38* 06:51:37,956 WARNING kernel:R10:
0000000000000000 R11: 0000000000000001 R12: ffffffff9860b050*07:51:38*
06:51:37,956 WARNING kernel:R13: ffff9898e8c58130 R14:
0000000000000202 R15: 0000000000000000*07:51:38* 06:51:37,957 WARNING
kernel:FS: 0000000000000000(0000) GS:ffff9898fbc00000(0000)
knlGS:0000000000000000*07:51:38* 06:51:37,957 WARNING kernel:CS: 0010
DS: 0000 ES: 0000 CR0: 0000000080050033*07:51:38* 06:51:37,957 WARNING
kernel:CR2: 00007f544a48d6d7 CR3: 0000000068dfc000 CR4:
00000000000006f0*07:51:38* 06:51:37,957 WARNING kernel:Call
Trace:*07:51:38* 06:51:37,958 WARNING kernel:
ata_sff_error_handler+0x83/0xe0*07:51:38* 06:51:37,958 WARNING kernel:
ata_scsi_port_error_handler+0x354/0x770*07:51:38* 06:51:37,958 WARNING
kernel: ? scsi_try_target_reset+0x90/0x90*07:51:38* 06:51:37,959
WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*07:51:38* 06:51:37,959
WARNING kernel: ata_scsi_error+0x91/0xc0*07:51:38* 06:51:37,959
WARNING kernel: scsi_error_handler+0xd0/0x5b0*07:51:38* 06:51:37,959
WARNING kernel: ? __schedule+0x23f/0x850*07:51:38* 06:51:37,960
WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*07:51:38* 06:51:37,960
WARNING kernel: kthread+0x112/0x130*07:51:38* 06:51:37,960 WARNING
kernel: ? kthread_create_worker_on_cpu+0x70/0x70*07:51:38*
06:51:37,960 WARNING kernel: ?
kthread_create_worker_on_cpu+0x70/0x70*07:51:38* 06:51:37,960 WARNING
kernel: ret_from_fork+0x35/0x40*07:51:38* 06:51:37,961 WARNING
kernel:Code: a8 08 74 0b 65 81 25 6f 2c 76 67 ff ff ff 7f 89 d0 c3 90
90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 48 89 f7 57
9d <0f> 1f 44 00 00 c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
Running yum check-update on the slave shows several updates available
including kernel.
I would recommend to upgrade slaves.
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://red.ht/sig>
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100099)
5 years, 7 months
Build failed in Jenkins: system-sync_mirrors-glusterfs-5-el7-x86_64
#205
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-5-el7-x86_64/2...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-glusterfs-5-el7-x86_64/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision e2723ce1cb5307c98934c8f527598905fdf087ce (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f e2723ce1cb5307c98934c8f527598905fdf087ce
Commit message: "stdci_slaves: increase memory requirement"
> git rev-list --no-walk e2723ce1cb5307c98934c8f527598905fdf087ce # timeout=10
[system-sync_mirrors-glusterfs-5-el7-x86_64] $ /bin/bash -xe /tmp/jenkins504825524289935617.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf
+ MIRRORS_MP_BASE=/var/www/html/repos
+ MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos
+ MIRRORS_CACHE=/home/jenkins/mirrors_cache
+ MAX_LOCK_ATTEMPTS=120
+ LOCK_WAIT_INTERVAL=5
+ LOCK_BASE=/home/jenkins
+ OLD_MD_TO_KEEP=100
+ HTTP_SELINUX_TYPE=httpd_sys_content_t
+ HTTP_FILE_MODE=644
+ main resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf
+ local command=resync_yum_mirror
+ command_args=("${@:2}")
+ local command_args
+ cmd_resync_yum_mirror glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf
+ local repo_name=glusterfs-5-el7
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local sync_needed
+ mkdir -p /home/jenkins/mirrors_cache
+ verify_repo_fs glusterfs-5-el7 yum
+ local repo_name=glusterfs-5-el7
+ local repo_type=yum
+ sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/glusterfs-5-el7 /var/www/html/repos/yum/glusterfs-5-el7/base
+ check_yum_sync_needed glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf sync_needed
+ local repo_name=glusterfs-5-el7
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local p_sync_needed=sync_needed
+ local reposync_out
+ echo 'Checking if mirror needs a resync'
Checking if mirror needs a resync
+ rm -rf /home/jenkins/mirrors_cache/glusterfs-5-el7
++ IFS=,
++ echo x86_64
+ for arch in '$(IFS=,; echo $repo_archs)'
++ run_reposync glusterfs-5-el7 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet
++ local repo_name=glusterfs-5-el7
++ local repo_arch=x86_64
++ local reposync_conf=jenkins/data/mirrors-reposync.conf
++ extra_args=("${@:4}")
++ local extra_args
++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=glusterfs-5-el7 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/glusterfs-5-el7/base --norepopath --newest-only --urls --quiet
Error setting up repositories: glusterfs-5-el7: Check uncompressed DB failed
+ reposync_out=
Build step 'Execute shell' marked build as failure
5 years, 7 months
[oVirt Jenkins] ovirt-system-tests_compat-4.2-suite-master - Build
# 526 - Failure!
by jenkins＠jenkins.phx.ovirt.org
Project: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/
Build: http://jenkins.ovirt.org/job/ovirt-system-tests_compat-4.2-suite-master/526/
Build Number: 526
Build Status: Failure
Triggered By: Started by timer
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #526
No changes
-----------------
Failed Tests:
-----------------
1 tests failed.
FAILED: 002_bootstrap.verify_add_all_hosts_42
Error Message:
1 hosts failed installation:
lago-compat-4-2-suite-master-host-1: non_operational
-------------------- >> begin captured logging << --------------------
ovirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2597ab1488>
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda>
lambda: _all_hosts_up(hosts_service, total_hosts),
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up
_check_problematic_hosts(hosts_service)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts
raise RuntimeError(dump_hosts)
RuntimeError: 1 hosts failed installation:
lago-compat-4-2-suite-master-host-1: non_operational
--------------------- >> end captured logging << ---------------------
Stack Trace:
File "/usr/lib64/python2.7/unittest/case.py", line 369, in run
testMethod()
File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test
test()
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper
return func(get_test_prefix(), *args, **kwargs)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 343, in verify_add_all_hosts
timeout=constants.ADD_HOST_TIMEOUT
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 278, in assert_true_within
assert_equals_within(func, True, timeout, allowed_exceptions)
File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within
res = func()
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda>
lambda: _all_hosts_up(hosts_service, total_hosts),
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up
_check_problematic_hosts(hosts_service)
File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts
raise RuntimeError(dump_hosts)
'1 hosts failed installation:\nlago-compat-4-2-suite-master-host-1: non_operational\n\n-------------------- >> begin captured logging << --------------------\novirtlago.testlib: ERROR: * Unhandled exception in <function <lambda> at 0x7f2597ab1488>\nTraceback (most recent call last):\n File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 234, in assert_equals_within\n res = func()\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 342, in <lambda>\n lambda: _all_hosts_up(hosts_service, total_hosts),\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 137, in _all_hosts_up\n _check_problematic_hosts(hosts_service)\n File "/home/jenkins/workspace/ovirt-system-tests_compat-4.2-suite-master/ovirt-system-tests/compat-4.2-suite-master/test-scenarios/002_bootstrap.py", line 157, in _check_problematic_hosts\n raise RuntimeError(dump_hosts)\nRuntimeError: 1 hosts failed installation:\nlago-compat-4-2-suite-master-host-1: non_operational\n\n--------------------- >> end captured logging << ---------------------'
5 years, 8 months
