Build failed in Jenkins: ovirt_master_system-tests #388
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_master_system-tests/388/changes>
Changes:
[Gal Ben Haim] Removed the hard coded path of the template repo. The template repo file
------------------------------------------
[...truncated 649 lines...]
## took 807 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log enties: logs/mocker-fedora-24-x86_64.fc24.basic_suite_master.sh/basic_suite_master.sh.log
##!
+ true
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
Took 629 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=master
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson5606048360884548341.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=master
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
+ OVIRT_SUITE=master
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_master_system-tests/388/artifact/expor...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_master_system-tests/388/artifact/expor...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> <http://jenkins.ovirt.org/job/ovirt_master_system-tests/388/artifact/expor...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
rm -rf "${logs[@]}"
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
[ovirt_master_system-tests] $ /bin/bash -xe /tmp/hudson5680342711917502147.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
./ovirt-system-tests/logs/
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.init/
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.init/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.init/state.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.init/build.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.init/root.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.install_packages/
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.install_packages/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.clean_rpmdb/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.basic_suite_master.sh/
./ovirt-system-tests/logs/mocker-fedora-24-x86_64.fc24.basic_suite_master.sh/basic_suite_master.sh.log
+ rm -rf ./ovirt-system-tests/logs
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-fedora-24-x86_64.fc24.cfg
+ mock_root=mocker-fedora-24-x86_64.fc24
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-fedora-24-x86_64.fc24'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-tests> --root=mocker-fedora-24-x86_64.fc24 --resultdir=<http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te....> Using default...
INFO: mock.py version 1.2.18 starting (python version = 3.5.1)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/ovirt-system-te...>
+ mock_root=fedora-24-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ [[ -n fedora-24-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep fedora-24-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/fedora-24-x86_64-4ddbe48c8f8b7d8c2c3635b52313f04a
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_master_system-tests/ws/>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
Archiving artifacts
8 years, 4 months
[JIRA] (OVIRT-684) dev permissions
by Liron Aravot (oVirt JIRA)
Liron Aravot created OVIRT-684:
----------------------------------
Summary: dev permissions
Key: OVIRT-684
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-684
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Liron Aravot
Assignee: infra
Hi,
I need dev permissions for "laravot" to be able to trigger manually.
"laravot is missing the Gerrit/ManualTrigger permission"
thanks.
--
This message was sent by Atlassian JIRA
(v1000.245.0#100009)
8 years, 4 months
Re: CI failure related to setting up PK12?
by Eyal Edri
Adding infra.
But it looks like a real error, maybe Didi can help today with
understanding where it comes from, we might need to do update to answer
file due to recent change or something.
Relevant patch which touched PKI just recently:
commit f87e5c1ba7c58d342ba520422b0ec36fb01a866e
Author: Yedidyah Bar David <didi(a)redhat.com>
Date: Wed Aug 10 14:02:19 2016 +0300
packaging: setup: Enroll missing pki on upgrade
Without this patch, imageio-proxy pki will not be generated on upgrade
from 3.6, because it did not exist then.
Change-Id: Ib5bbe9ddb45bfcbf6b7bf8aa7c6e2fb19fa206fe
Bug-Url: https://bugzilla.redhat.com/1365451
Signed-off-by: Yedidyah Bar David <didi(a)redhat.com>
On Sun, Aug 14, 2016 at 8:48 AM, Allon Mureinik <amureini(a)redhat.com> wrote:
> Hi guys,
>
> Any recent engine patch (e.g., [1]) I look at seems to fail the CI in a
> similar way:
>
> [ INFO ] Stage: Setup validation
> [WARNING] Cannot validate host name settings, reason: resolved host does not match any of the local addresses
> [WARNING] Less than 16384MB of memory is available
>
> --== CONFIGURATION PREVIEW ==--
>
> Application mode : both
> Default SAN wipe after delete : False
> Firewall manager : firewalld
> Update Firewall : True
> Host FQDN : el7-vm31
> Require packages rollback : False
> Upgrade packages : True
> Engine database secured connection : False
> Engine database host : localhost
> Engine database user name : engine
> Engine database name : engine
> Engine database port : 5432
> Engine database host name validation : False
> Configure local DWH database : False
> Engine installation : True
> NFS setup : True
> PKI organization : ovirt.org
> NFS export ACL : 0.0.0.0/0.0.0.0(rw)
> NFS mount point : /var/lib/exports/iso
> Configure local Engine database : True
> Set application as default page : True
> Configure Apache SSL : True
> DWH installation : False
> Engine Host FQDN : el7-vm31
> Configure Image I/O Proxy : False
> Configure VMConsole Proxy : True
> Configure WebSocket Proxy : True
> [ INFO ] Stage: Transaction setup
> [ INFO ] Stopping engine service
> [ INFO ] Stopping ovirt-fence-kdump-listener service
> [ INFO ] Stopping websocket-proxy service
> [ INFO ] Stage: Misc configuration
> [ INFO ] Stage: Package installation
> [ INFO ] Stage: Misc configuration
> [ INFO ] Upgrading CA
> [ ERROR ] Failed to execute stage 'Misc configuration': Command '/usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh' failed to execute
>
>
>
> Can you take a look please?
>
>
>
> Thanks,
>
> Allon
>
>
> [1] http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade-from-master_el7_...
>
>
--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
8 years, 4 months
[JIRA] (OVIRT-682) Improve CI logging - check if its possible to include mock errors in console output
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-682?page=com.atlassian.jira... ]
eyal edri [Administrator] updated OVIRT-682:
--------------------------------------------
Epic Link: OVIRT-400
> Improve CI logging - check if its possible to include mock errors in console output
> -----------------------------------------------------------------------------------
>
> Key: OVIRT-682
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-682
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: eyal edri [Administrator]
> Assignee: infra
>
> On Fri, Aug 12, 2016 at 8:08 PM, Nir Soffer <nsoffer(a)redhat.com> wrote:
> > Hi all,
> >
> > Lately vdsm builds are failing in the install packages stage with
> > this unhelpful error:
> >
> > 13:59:42 INFO: installing package(s): autoconf automake gdb ...
> > 13:59:53 ERROR: Command failed. See logs for output.
> >
> > I downloaded the logs.tgz file from
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/artifact/exported-artifacts/logs.tgz
> >
> > And indeed in /./vdsm/logs/mocker-fedora-24-x86_64.fc24.install_packages/
> > root.log
> > I found found this error:
> >
> > DEBUG util.py:421: Error: Package:
> > python-ioprocess-0.17.0-1.201608111609.gitbd272f2.fc24.noarch
> > (ovirt-snapshot)
> > DEBUG util.py:421: Requires: ioprocess =
> > 0.17.0-1.201608111609.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.15.1-1.fc24.x86_64 (fedora)
> > DEBUG util.py:421: ioprocess = 0.15.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.16.1-1.fc24.x86_64 (updates)
> > DEBUG util.py:421: ioprocess = 0.16.1-1.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201607121058.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201607121058.gitbd272f2.fc24
> > DEBUG util.py:421: Available:
> > ioprocess-0.17.0-1.201608111129.gitbd272f2.fc24.x86_64
> > (ovirt-snapshot)
> > DEBUG util.py:421: ioprocess =
> > 0.17.0-1.201608111129.gitbd272f2.fc24
> >
> > So we have 2 issues here:
> >
> > 1. We need *all* errors in the console
> > http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-
> > x86_64/1154/console
> Not sure its possible or easy to do, these errors usually come from mock
> and are not relevant to the code running on the console.
> I've cc'd infra-support and updated topic to track it and check our options.
> Going foward, we might want to install a log collector like logstash and
> then searching errors there should be much easier.
> >
> >
> > 2. Someone need to fix the ovirt-snapshot repository - it should have
> > the missing ioprocess
> > package.
> > Maybe the project xml is not correct?
> >
> > Nir
> >
> --
> Eyal Edri
> Associate Manager
> RHV DevOps
> EMEA ENG Virtualization R&D
> Red Hat Israel
> phone: +972-9-7692018
> irc: eedri (on #tlv #rhev-dev #rhev-integ)
--
This message was sent by Atlassian JIRA
(v1000.245.0#100009)
8 years, 4 months