Build failed in Jenkins: ovirt_3.6_he-system-tests #790
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/changes>
Changes:
[Yaniv Kaul] Fixes and changes to storage tests
[Dan Kenigsberg] vdsm: exclude f23 from 4.1 branch
[Yaniv Bronhaim] Update Jenkins slaves for vdsm to f24 and f25 and remove excludes
[Eyal Edri] update vdsm jobs to use 4.1 jobs
[ngoldin] Add fc25 lago jobs
[Juan Hernandez] Build and check API model from branch 4.1
------------------------------------------
[...truncated 608 lines...]
+ local ID=3
+ lvcreate -L20G -n lun3_bdev vg1_storage
+ targetcli /backstores/block create name=lun3_bdev dev=/dev/vg1_storage/lun3_bdev
+ targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun3_bdev
+ for I in '$(seq $NUM_LUNS)'
+ create_lun 4
+ local ID=4
+ lvcreate -L20G -n lun4_bdev vg1_storage
+ targetcli /backstores/block create name=lun4_bdev dev=/dev/vg1_storage/lun4_bdev
+ targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1/luns/ create /backstores/block/lun4_bdev
+ targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set auth userid=username password=password
+ targetcli /iscsi/iqn.2014-07.org.ovirt:storage/tpg1 set attribute demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1 default_cmdsn_depth=64
+ targetcli saveconfig
+ systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.
+ systemctl start target
+ sed -i 's/#node.session.auth.authmethod = CHAP/node.session.auth.authmethod = CHAP/g' /etc/iscsi/iscsid.conf
+ sed -i 's/#node.session.auth.username = username/node.session.auth.username = username/g' /etc/iscsi/iscsid.conf
+ sed -i 's/#node.session.auth.password = password/node.session.auth.password = password/g' /etc/iscsi/iscsid.conf
+ iscsiadm -m discovery -t sendtargets -p 127.0.0.1
+ iscsiadm -m node -L all
+ rescan-scsi-bus.sh
which: no multipath in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
+ lsscsi -i
+ grep 36
+ awk '{print $NF}'
+ sort
+ iscsiadm -m node -U all
+ iscsiadm -m node -o delete
+ systemctl stop iscsi.service
+ systemctl disable iscsi.service
Removed symlink /etc/systemd/system/sysinit.target.wants/iscsi.service.
+ install_deps_389ds
+ yum install -y --downloaddir=/dev/shm 389-ds-base
+ setup_389ds
+ DOMAIN=lago.local
+ PASSWORD=12345678
++ sed s/_/-/g
++ hostname
+ HOSTNAME=lago-he-basic-suite-3-6-storage.lago.local
++ /sbin/ip -4 -o addr show dev eth0
++ awk '{split($4,a,"."); print a[1] "." a[2] "." a[3] "." a[4]}'
++ awk -F/ '{print $1}'
+ ADDR=192.168.200.2
+ cat
+ sed -i s/@HOSTNAME(a)/lago-he-basic-suite-3-6-storage.lago.local/g answer_file.inf
+ sed -i s/@PASSWORD@/12345678/g answer_file.inf
+ sed -i s/@DOMAIN(a)/lago.local/g answer_file.inf
+ cat
+ /usr/sbin/setup-ds.pl --silent --file=answer_file.inf
Warning: using root as the server user id. You are strongly encouraged to use a non-root user.
* [Thread-2] Deploy VM lago-he-basic-suite-3-6-storage: ERROR (in 0:02:27)
Error while running thread
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host
host.name(),
RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> failed with status 1 on lago-he-basic-suite-3-6-storage
* [Thread-4] Deploy VM lago-he-basic-suite-3-6-host0: ERROR (in 0:08:50)
* [Thread-5] Deploy VM lago-he-basic-suite-3-6-host1: ERROR (in 0:08:51)
# Deploy environment: ERROR (in 0:08:51)
@ Deploy oVirt environment: ERROR (in 0:08:51)
Error occured, aborting
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 281, in do_run
self.cli_plugins[args.ovirtverb].do_run(args)
File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run
self._do_run(**vars(args))
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 489, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 500, in wrapper
return func(*args, prefix=prefix, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 187, in do_deploy
prefix.deploy()
File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/reposetup.py", line 68, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/ovirtlago/__init__.py", line 198, in deploy
return super(OvirtPrefix, self).deploy()
File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 621, in wrapper
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1249, in deploy
self._deploy_host, self.virt_env.get_vms().values()
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 97, in invoke_in_parallel
vt.join_all()
File "/usr/lib/python2.7/site-packages/lago/utils.py", line 55, in _ret_via_queue
queue.put({'return': func()})
File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 1242, in _deploy_host
host.name(),
RuntimeError: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> failed with status 1 on lago-he-basic-suite-3-6-storage
+ res=1
+ exit 1
+ cleanup
+ rm -rf exported-artifacts
+ mkdir -p exported-artifacts
+ [[ -d deployment-he-basic-suite-3.6/current/logs ]]
+ mv deployment-he-basic-suite-3.6/current/logs exported-artifacts/lago_logs
+ find deployment-he-basic-suite-3.6 -iname 'nose*.xml' -exec mv '{}' exported-artifacts/ ';'
+ [[ -d test_logs ]]
+ [[ -e failure_msg.txt ]]
+ mv failure_msg.txt exported-artifacts/
+ ./run_suite.sh --cleanup he-basic-suite-3.6
+ CLI=lago
+ DO_CLEANUP=false
+ RECOMMENDED_RAM_IN_MB=8196
+ EXTRA_SOURCES=()
++ getopt -o ho:e:n:b:cs:r: --long help,output:,engine:,node:,boot-iso:,cleanup --long extra-rpm-source,reposync-config: -n run_suite.sh -- --cleanup he-basic-suite-3.6
+ options=' --cleanup -- '\''he-basic-suite-3.6'\'''
+ [[ 0 != \0 ]]
+ eval set -- ' --cleanup -- '\''he-basic-suite-3.6'\'''
++ set -- --cleanup -- he-basic-suite-3.6
+ true
+ case $1 in
+ DO_CLEANUP=true
+ shift
+ true
+ case $1 in
+ shift
+ break
+ [[ -z he-basic-suite-3.6 ]]
++ realpath he-basic-suite-3.6
+ export SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...>
+ SUITE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...>
+ '[' -z '' ']'
+ export PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...>
+ PREFIX=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...>
+ true
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 607 seconds
===================================
logout
Finish: shell
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@ Tue Dec 27 13:49:16 UTC 2016 automation/he_basic_suite_3.6.sh chroot finished
@@ took 617 seconds
@@ rc = 1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
========== Scrubbing chroot
mock \
--configdir="<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-tests"> \
--root="mocker-epel-7-x86_64.el7" \
--resultdir="logs/mocker-epel-7-x86_64.el7.scrub" \
--scrub=chroot
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te....> Using default...
INFO: mock.py version 1.2.21 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Start: scrub ['chroot']
INFO: scrubbing chroot for mocker-epel-7-x86_64.el7
Finish: scrub ['chroot']
Finish: run
Scrub chroot took 5 seconds
============================
##########################################################
## Tue Dec 27 13:49:21 UTC 2016 Finished env: el7:epel-7-x86_64
## took 622 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log entries: logs/mocker-epel-7-x86_64.el7.he_basic_suite_3.6.sh/he_basic_suite_3.6.sh.log
##!
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 607 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=3.6
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_3.6_he-system-tests] $ /bin/bash -xe /tmp/hudson6318118267408142511.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=3.6
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/>
+ OVIRT_SUITE=3.6
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/expor...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/expor...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/790/artifact/expor...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Archiving artifacts
7 years, 3 months
oVirt infra daily report - unstable production jobs - 190
by jenkins＠jenkins.phx.ovirt.org
------=_Part_100_493996725.1483570862350
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Good morning!
Attached is the HTML page with the jenkins status report. You can see it also here:
- http://jenkins.ovirt.org/job/system_jenkins-report/190//artifact/exported...
Cheers,
Jenkins
------=_Part_100_493996725.1483570862350
Content-Type: text/html; charset=us-ascii; name=upstream_report.html
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=upstream_report.html
Content-ID: <upstream_report.html>
<!DOCTYPE html><head><style type="text/css">
table.gridtable {
border-collapse: collapse;
table-layout:fixed;
width:1600px;
font-family: monospace;
font-size:13px;
}
.head {
font-size:20px;
font-family: arial;
}
.sub {
font-size:18px;
background-color:#e5e5e5;
font-family: arial;
}
pre {
font-family: monospace;
display: inline;
white-space: pre-wrap;
white-space: -moz-pre-wrap !important;
white-space: -pre-wrap;
white-space: -o-pre-wrap;
word-wrap: break-word;
}
</style>
</head>
<body>
<table class="gridtable" border=2>
<tr><th colspan=2 class=head>
RHEVM CI Jenkins Daily Report - 04/01/2017
</th></tr><tr><th colspan=2 class=sub>
<font color="blue"><a href="http://jenkins.ovirt.org/">00 Unstable Critical</a></font>
</th></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/deploy-to-ovirt_experimental_3.6/">deploy-to-ovirt_experimental_3.6</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt-node-ng_ovirt-master-experimental_buil...">ovirt-node-ng_ovirt-master-experimental_build-artifacts-el7-x86_64</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_3.6_he-system-tests/">ovirt_3.6_he-system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_3.6_image-ng-system-tests/">ovirt_3.6_image-ng-system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/">ovirt_4.1_system-tests</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_4.0_check-closure_fc23_merged/">repos_4.0_check-closure_fc23_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_4.1_check-closure_fc24_merged/">repos_4.1_check-closure_fc24_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_master_check-closure_el7_merged/">repos_master_check-closure_el7_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/repos_master_check-closure_fc24_merged/">repos_master_check-closure_fc24_merged</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
<tr><td>
<a href="http://jenkins.ovirt.org/job/system-backup_jenkins_old_ovirt_org/">system-backup_jenkins_old_ovirt_org</a>
</td><td>
This job is automatically updated by jenkins job builder, any manual
change will be lost in the next update. If you want to make permanent
changes, check out the <a href="http://gerrit.ovirt.org/gitweb?p=jenkins.git;a=tree;h=refs/heads/master;h...">
jenkins</a> repo.
<!-- Managed by Jenkins Job Builder -->
</td></tr>
------=_Part_100_493996725.1483570862350--
7 years, 3 months
[JIRA] (OVIRT-1005) Fix race condition caused by experimental cleanup script
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-1005?page=com.atlassian.jir... ]
Barak Korren reassigned OVIRT-1005:
-----------------------------------
Assignee: Barak Korren (was: infra)
> Fix race condition caused by experimental cleanup script
> --------------------------------------------------------
>
> Key: OVIRT-1005
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-1005
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: Barak Korren
> Assignee: Barak Korren
> Priority: Highest
>
> The experimental repo has a cleanup script that is ran every hour via cron. The script is meant to delete old and unused package builds.
> The content of the script is as follows:
> {code}
> find \
> /srv/resources/repos/ovirt/experimental/*/* \
> -maxdepth 0 \
> -type d \
> -not -name 'latest.*' \
> -and \
> -mtime +1 \
> -exec rm -rf {} \;
> {code}
> The problem here is that 'latest' is not filtered out to not be deleted by this script. It is OK to delete it in most cases because then it gets recreated by the 'deploy-to-experimental' job, but there might be a case where 'test-repo_ovirt-experimental_*' can run before this happens, this may cause it to get an empty or partial 'latest' repo.
> To remedy this, the script needs to be changed to be the following:
> {code}
> find \
> /srv/resources/repos/ovirt/experimental/*/* \
> -maxdepth 0 \
> -type d \
> -not -name 'latest.*' \
> -and \
> -not -name 'latest' \
> -and \
> -mtime +1 \
> -exec rm -rf {} \;
> {code}
> The script itself on the resources server was already changed in-palce, this ticket is so we this change is also "backed up" to the 'infra-puppet' repo.
--
This message was sent by Atlassian JIRA
(v1000.656.2#100024)
7 years, 4 months