Build failed in Jenkins: ovirt_4.0_system-tests #629
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/629/changes>
Changes:
[Gal Ben Haim] Switched the mock automation scripts to symlinks
[Gil Shinar] Create source mount point
------------------------------------------
[...truncated 820 lines...]
## took 1549 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log entries: logs/mocker-fedora-23-x86_64.fc23.basic_suite_4.0.sh/basic_suite_4.0.sh.log
##!
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 1388 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=4.0
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/hudson9156453899063302634.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=4.0
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/>
+ OVIRT_SUITE=4.0
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/629/artifact/exported...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/629/artifact/exported...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/629/artifact/exported...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
rm -rf "${logs[@]}"
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
[ovirt_4.0_system-tests] $ /bin/bash -xe /tmp/hudson1105985712999322223.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
./ovirt-system-tests/logs/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_4.0.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_4.0.sh/basic_suite_4.0.sh.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
+ rm -rf ./ovirt-system-tests/logs
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-fedora-23-x86_64.fc23.cfg
+ mock_root=mocker-fedora-23-x86_64.fc23
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests....> Using default...
INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/ovirt-system-tests...>
+ mock_root=fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ [[ -n fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/fedora-23-x86_64-0c362156a2fa4a935ea8b988eb73b2ad
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.0_system-tests/ws/>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
Archiving artifacts
8 years, 1 month
[JIRA] (OVIRT-416) [standard-ci] build-artifacts should reuse the artifacts built on check-merged if any
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-416?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-416:
-------------------------------------------------
Any patches sent on this?
Also, wanted to mention #OVIRT-751 is handling similar purpose with enabling better maven caching, so it might suffice as a starting point in optimizing ovirt-engine runtime for builds.
> [standard-ci] build-artifacts should reuse the artifacts built on check-merged if any
> -------------------------------------------------------------------------------------
>
> Key: OVIRT-416
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-416
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: David Caro Estevez
> Assignee: infra
>
> Right now most projects just rebuild twice the artifacts, what for some means a big waste of resources, if we could just reuse what was built previously there would be no rebuilding.
> Something to take into account is that the distributions that check-merged and build-artifacts run for might not match, so you can be running check-merged only on fc23 but building artifacts for fc23, fc22 an el7 (as an example), maybe we should not allow different distros on each stage?|
--
This message was sent by Atlassian JIRA
(v1000.499.4#100018)
8 years, 1 month
[JIRA] (OVIRT-763) [ost][rfe] ability to use custom repos
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-763?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-763:
-------------------------------------------------
Getting into specific digits of versions sounds like very complicated solution which might break often.
We will add a job for it as mentioned before, and it will allow giving any 'custom refspec' as all CI jobs are allowing today. (e.g refs/heads/master for merged refs/changes/12/65912/4 for an open patch)
> [ost][rfe] ability to use custom repos
> ---------------------------------------
>
> Key: OVIRT-763
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-763
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Piotr Kliczewski
> Assignee: infra
>
> I started to test using lago with master repos and my custom engine built.
> It was built on 7.10 one test failed and I noticed that newer engine was
> installed instead of my specific version. It seems that we need an ability
> to define which version of the rpms should be used during the tests.
> Thanks,
> Piotr
--
This message was sent by Atlassian JIRA
(v1000.499.4#100018)
8 years, 1 month
[JIRA] (OVIRT-810) Cache issues
by Piotr Kliczewski (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-810?page=com.atlassian.jira... ]
Piotr Kliczewski commented on OVIRT-810:
----------------------------------------
Sandro, can you give more details?
6 lis 2016 09:07 "eyal edri [Administrator] (oVirt JIRA)" <
> Cache issues
> ------------
>
> Key: OVIRT-810
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-810
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Piotr Kliczewski
> Assignee: infra
>
> We saw artififactory cache refresh issues which zapping cache was not able
> to solve. Once that was overcame we saw issues with engine upgrade jobs due
> to local cache which is used by those jobs.
--
This message was sent by Atlassian JIRA
(v1000.499.4#100018)
8 years, 1 month
[JIRA] (OVIRT-763) [ost][rfe] ability to use custom repos
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-763?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-763:
-------------------------------------------------
I renamed the task, since you can choose version today by choosing to run the relevant suite (3.6,4.0,master), the requirement should be to allow running with custom repos.
So one option will be to use the '-s' param and another one will be add a jenkins job to support it as mentioned in this ticket.
> [ost][rfe] ability to use custom repos
> ---------------------------------------
>
> Key: OVIRT-763
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-763
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Piotr Kliczewski
> Assignee: infra
>
> I started to test using lago with master repos and my custom engine built.
> It was built on 7.10 one test failed and I noticed that newer engine was
> installed instead of my specific version. It seems that we need an ability
> to define which version of the rpms should be used during the tests.
> Thanks,
> Piotr
--
This message was sent by Atlassian JIRA
(v1000.499.4#100018)
8 years, 1 month
[JIRA] (OVIRT-763) [ost][rfe] ability to choose version to be installed
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-763?page=com.atlassian.jira... ]
eyal edri [Administrator] updated OVIRT-763:
--------------------------------------------
Summary: [ost][rfe] ability to choose version to be installed (was: [Lago][rfe] ability to choose version to be installed)
> [ost][rfe] ability to choose version to be installed
> ----------------------------------------------------
>
> Key: OVIRT-763
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-763
> Project: oVirt - virtualization made easy
> Issue Type: By-EMAIL
> Reporter: Piotr Kliczewski
> Assignee: infra
>
> I started to test using lago with master repos and my custom engine built.
> It was built on 7.10 one test failed and I noticed that newer engine was
> installed instead of my specific version. It seems that we need an ability
> to define which version of the rpms should be used during the tests.
> Thanks,
> Piotr
--
This message was sent by Atlassian JIRA
(v1000.499.4#100018)
8 years, 1 month