[JIRA] (OVIRT-2703) oVirt Node build fails due to CPU stuck
by sbonazzo (oVirt JIRA)
sbonazzo created OVIRT-2703:
-------------------------------
Summary: oVirt Node build fails due to CPU stuck
Key: OVIRT-2703
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-2703
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: sbonazzo
Assignee: infra
*CPU is getting stuck for the VM running on the slave.*
*Error is:*
*https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-fc28-x86_64/240/console
<https://jenkins.ovirt.org/job/ovirt-node-ng-image_master_build-artifacts-...>*
*10:44:14* 09:44:13,825 WARNING kernel:ata2: lost interrupt (Status
0x58)*10:44:14* 09:44:13,834 DEBUG kernel:ata2: drained 65536 bytes to
clear DRQ*10:44:14* 09:44:13,835 EMERG kernel:watchdog: BUG: soft
lockup - CPU#0 stuck for 32s! [scsi_eh_1:85]*10:44:14* 09:44:13,835
WARNING kernel:Modules linked in: xfs fcoe libfcoe libfc
scsi_transport_fc zram scsi_dh_rdac scsi_dh_emc scsi_dh_alua
parport_pc i2c_piix4 parport joydev loop nls_utf8 isofs 8021q garp mrp
stp llc virtio_console serio_raw qemu_fw_cfg virtio_pci e1000
bochs_drm drm_kms_helper ttm drm ata_generic pata_acpi sunrpc mcryptd
sha256_ssse3 dm_crypt dm_round_robin dm_multipath linear raid10
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor
raid6_pq libcrc32c raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs
zstd_decompress xxhash cramfs edd virtio_rng virtio_ring
virtio*10:44:14* 09:44:13,844 WARNING kernel:CPU: 0 PID: 85 Comm:
scsi_eh_1 Not tainted 4.16.3-301.fc28.x86_64 #1*10:44:14* 09:44:13,844
WARNING kernel:Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28
04/01/2014*10:44:21* 09:44:13,845 WARNING kernel:RIP:
0010:_raw_spin_unlock_irqrestore+0xd/0x20*10:44:21* 09:44:13,846
WARNING kernel:RSP: 0018:ffffa31e804cfdf0 EFLAGS: 00000202 ORIG_RAX:
ffffffffffffff12*10:44:21* 09:44:13,855 WARNING kernel:RAX:
0000000000000000 RBX: ffff9654e8c5c000 RCX: 0000000000000000*10:44:21*
09:44:13,856 WARNING kernel:RDX: 0000000000000000 RSI:
0000000000000202 RDI: 0000000000000202*10:44:21* 09:44:13,856 WARNING
kernel:RBP: ffffffffbd60bd20 R08: 0000000000000038 R09:
00000000000002a4*10:44:21* 09:44:13,857 WARNING kernel:R10:
0000000000000000 R11: 0000000000000001 R12: ffffffffbd60b050*10:44:21*
09:44:13,857 WARNING kernel:R13: ffff9654e8c5c130 R14:
0000000000000202 R15: 0000000000000000*10:44:21* 09:44:13,858 WARNING
kernel:FS: 0000000000000000(0000) GS:ffff9654fbc00000(0000)
knlGS:0000000000000000*10:44:21* 09:44:13,865 WARNING kernel:CS: 0010
DS: 0000 ES: 0000 CR0: 0000000080050033*10:44:21* 09:44:14,008 WARNING
kernel:CR2: 00007fece8177000 CR3: 0000000069c18000 CR4:
00000000000006f0*10:44:21* 09:44:14,008 WARNING kernel:Call
Trace:*10:44:21* 09:44:14,008 WARNING kernel:
ata_sff_error_handler+0x83/0xe0*10:44:21* 09:44:14,009 WARNING kernel:
ata_scsi_port_error_handler+0x354/0x770*10:44:21* 09:44:14,009 WARNING
kernel: ? scsi_try_target_reset+0x90/0x90*10:44:21* 09:44:14,009
WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*10:44:21* 09:44:14,010
WARNING kernel: ata_scsi_error+0x91/0xc0*10:44:21* 09:44:14,010
WARNING kernel: scsi_error_handler+0xd0/0x5b0*10:44:21* 09:44:14,010
WARNING kernel: ? scsi_eh_get_sense+0x220/0x220*10:44:21* 09:44:14,010
WARNING kernel: kthread+0x112/0x130*10:44:21* 09:44:14,011 WARNING
kernel: ? kthread_create_worker_on_cpu+0x70/0x70*10:44:21*
09:44:14,026 WARNING kernel: ?
kthread_create_worker_on_cpu+0x70/0x70*10:44:21* 09:44:14,026 WARNING
kernel: ret_from_fork+0x35/0x40*10:44:21* 09:44:14,026 WARNING
kernel:Code: a8 08 74 0b 65 81 25 6f 2c 76 42 ff ff ff 7f 89 d0 c3 90
90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 c6 07 00 48 89 f7 57
9d <0f> 1f 44 00 00 c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
*10:44:21* 09:44:14,027 ERR kernel:ata2.00: exception Emask 0x0 SAct
0x0 SErr 0x0 action 0x6 frozen*10:44:21* 09:44:14,027 ERR
kernel:ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio
16392 in#012 Get event status notification 4a 01 00 00 10 00
00 00 08 00res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4
(timeout)*10:44:21* 09:44:14,028 ERR kernel:ata2.00: status: { DRDY
}*10:44:21* 09:44:14,028 INFO kernel:ata2: soft resetting
link*10:44:21* 09:44:19,296 WARNING kernel:ata2.00: qc timeout (cmd
0xa1)*10:44:21* 09:44:19,305 WARNING kernel:ata2.00: failed to
IDENTIFY (I/O error, err_mask=0x4)*10:44:21* 09:44:19,305 ERR
kernel:ata2.00: revalidation failed (errno=-5)*10:44:21* 09:44:19,305
INFO kernel:ata2: soft resetting link*10:44:21* 09:44:21,510 INFO
kernel:ata2.00: configured for MWDMA2*10:44:21* 09:44:21,558 INFO
kernel:ata2: EH complete
The slave is vm0034.workers-phx.ovirt.org
<https://jenkins.ovirt.org/computer/vm0034.workers-phx.ovirt.org>
Looking at the slave, it looks like several updates are available including
a kernel update.
--
This message was sent by Atlassian Jira
(v1001.0.0-SNAPSHOT#100099)
5 years, 8 months
Build failed in Jenkins:
system-sync_mirrors-fedora-base-fc29-x86_64 #385
by jenkins@jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-fedora-base-fc29-x86_64/ws/>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/heads/*:refs/remotes/origin/* --prune # timeout=10
> git rev-parse origin/master^{commit} # timeout=10
Checking out Revision f8813ff56974265c0ff2ba3a13f8300df323066e (origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f f8813ff56974265c0ff2ba3a13f8300df323066e # timeout=10
Commit message: "pytest: refactor: Move fixtures to main conftest"
> git rev-list --no-walk f8813ff56974265c0ff2ba3a13f8300df323066e # timeout=10
[system-sync_mirrors-fedora-base-fc29-x86_64] $ /bin/bash -xe /tmp/jenkins8651585706643226731.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ MIRRORS_MP_BASE=/var/www/html/repos
+ MIRRORS_HTTP_BASE=http://mirrors.phx.ovirt.org/repos
+ MIRRORS_CACHE=/home/jenkins/mirrors_cache
+ MAX_LOCK_ATTEMPTS=120
+ LOCK_WAIT_INTERVAL=5
+ LOCK_BASE=/home/jenkins
+ OLD_MD_TO_KEEP=100
+ HTTP_SELINUX_TYPE=httpd_sys_content_t
+ HTTP_FILE_MODE=644
+ main resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ local command=resync_yum_mirror
+ command_args=("${@:2}")
+ local command_args
+ cmd_resync_yum_mirror fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf
+ local repo_name=fedora-base-fc29
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local sync_needed
+ mkdir -p /home/jenkins/mirrors_cache
+ verify_repo_fs fedora-base-fc29 yum
+ local repo_name=fedora-base-fc29
+ local repo_type=yum
+ sudo install -o jenkins -d /var/www/html/repos/yum /var/www/html/repos/yum/fedora-base-fc29 /var/www/html/repos/yum/fedora-base-fc29/base
+ check_yum_sync_needed fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf sync_needed
+ local repo_name=fedora-base-fc29
+ local repo_archs=x86_64
+ local reposync_conf=jenkins/data/mirrors-reposync.conf
+ local p_sync_needed=sync_needed
+ local reposync_out
+ echo 'Checking if mirror needs a resync'
Checking if mirror needs a resync
+ rm -rf /home/jenkins/mirrors_cache/fedora-base-fc29
++ IFS=,
++ echo x86_64
+ for arch in '$(IFS=,; echo $repo_archs)'
++ run_reposync fedora-base-fc29 x86_64 jenkins/data/mirrors-reposync.conf --urls --quiet
++ local repo_name=fedora-base-fc29
++ local repo_arch=x86_64
++ local reposync_conf=jenkins/data/mirrors-reposync.conf
++ extra_args=("${@:4}")
++ local extra_args
++ reposync --config=jenkins/data/mirrors-reposync.conf --repoid=fedora-base-fc29 --arch=x86_64 --cachedir=/home/jenkins/mirrors_cache --download_path=/var/www/html/repos/yum/fedora-base-fc29/base --norepopath --newest-only --urls --quiet
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML
result = self._getFileRepoXML(local, text)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML
size=102400) # setting max size as 100K
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile
result = self.grab.urlgrab(misc.to_utf8(relative), local,
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda>
grab = property(lambda self: self._getgrab())
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab
self._setupGrab()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab
urls = self.urls
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda>
urls = property(fget=lambda self: self._geturls(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls
self._baseurlSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup
mirrorurls.extend(list(self.metalink_data.urls()))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda>
metalink_data = property(fget=lambda self: self._getMetalink(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 914, in _getMetalink
self._metalink = metalink.MetaLinkRepoMD(self.metalink_filename)
File "/usr/lib/python2.7/site-packages/yum/metalink.py", line 185, in __init__
raise MetaLinkRepoErrorParseFail, "File %s does not exist" %filename
yum.metalink.MetaLinkRepoErrorParseFail: File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml does not exist
+ reposync_out='Could not parse metalink https://mirrors.fedoraproject.org/metalink?repo=fedora-29&arch=x86_64 error was
File /home/jenkins/mirrors_cache/fedora-base-fc29/metalink.xml.tmp is not XML'
Build step 'Execute shell' marked build as failure
5 years, 8 months
Build failed in Jenkins:
system-sync_mirrors-centos-kvm-common-el7-x86_64 #2284
by jenkins@jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x8...>
------------------------------------------
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on mirrors.phx.ovirt.org (mirrors) in workspace <http://jenkins.ovirt.org/job/system-sync_mirrors-centos-kvm-common-el7-x8...>
No credentials specified
> git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
> git config remote.origin.url http://gerrit.ovirt.org/jenkins.git # timeout=10
Cleaning workspace
> git rev-parse --verify HEAD # timeout=10
Resetting working tree
> git reset --hard # timeout=10
> git clean -fdx # timeout=10
Pruning obsolete local branches
Fetching upstream changes from http://gerrit.ovirt.org/jenkins.git
> git --version # timeout=10
> git fetch --tags --progress http://gerrit.ovirt.org/jenkins.git +refs/changes/59/95959/1:test --prune # timeout=10
> git rev-parse origin/test^{commit} # timeout=10
> git rev-parse test^{commit} # timeout=10
Checking out Revision 05b40dfb4ec43a82530e8c471395d1858e5c59e1 (test)
> git config core.sparsecheckout # timeout=10
> git checkout -f 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10
Commit message: "mirror-reposync: remove gluster-3.10 mirror"
> git rev-list --no-walk 05b40dfb4ec43a82530e8c471395d1858e5c59e1 # timeout=10
[system-sync_mirrors-centos-kvm-common-el7-x86_64] $ /bin/bash -xe /tmp/jenkins8102270137299837991.sh
+ jenkins/scripts/mirror_mgr.sh resync_yum_mirror centos-kvm-common-el7 x86_64 jenkins/data/mirrors-reposync.conf
Checking if mirror needs a resync
Traceback (most recent call last):
File "/usr/bin/reposync", line 343, in <module>
main()
File "/usr/bin/reposync", line 175, in main
my.doRepoSetup()
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 681, in doRepoSetup
return self._getRepos(thisrepo, True)
File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 721, in _getRepos
self._repos.doSetup(thisrepo)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1479, in _commonLoadRepoXML
result = self._getFileRepoXML(local, text)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1256, in _getFileRepoXML
size=102400) # setting max size as 100K
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1022, in _getFile
result = self.grab.urlgrab(misc.to_utf8(relative), local,
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 700, in <lambda>
grab = property(lambda self: self._getgrab())
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 695, in _getgrab
self._setupGrab()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 632, in _setupGrab
urls = self.urls
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 878, in <lambda>
urls = property(fget=lambda self: self._geturls(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 875, in _geturls
self._baseurlSetup()
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 821, in _baseurlSetup
mirrorurls.extend(list(self.metalink_data.urls()))
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 918, in <lambda>
metalink_data = property(fget=lambda self: self._getMetalink(),
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 897, in _getMetalink
raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve metalink for repository: fedora-base-fc29. Please verify its path and try again
Build step 'Execute shell' marked build as failure
5 years, 8 months