[JIRA] (OVIRT-988) standard-CI artifacts are not collected when mock runner fails
by Barak Korren (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-988?page=com.atlassian.jira... ]
Barak Korren reassigned OVIRT-988:
----------------------------------
Assignee: Barak Korren (was: infra)
> standard-CI artifacts are not collected when mock runner fails
> --------------------------------------------------------------
>
> Key: OVIRT-988
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-988
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Reporter: Barak Korren
> Assignee: Barak Korren
>
> Currently, when a standard-CI script fails (which makes mock_runner.sh fail), the exported-artifacts are not taken from the project directory but only from $WORKSPACE.
> It desirable to take the project's atficats too, even if the CI script failed, to allow debugging.
> It seems that pipeline-based jobs (Lago's CI, experimental, OST) do not exhibit this behaviour.
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years
[JIRA] (OVIRT-988) standard-CI artifacts are not collected when mock runner fails
by Barak Korren (oVirt JIRA)
Barak Korren created OVIRT-988:
----------------------------------
Summary: standard-CI artifacts are not collected when mock runner fails
Key: OVIRT-988
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-988
Project: oVirt - virtualization made easy
Issue Type: Bug
Reporter: Barak Korren
Assignee: infra
Currently, when a standard-CI script fails (which makes mock_runner.sh fail), the exported-artifacts are not taken from the project directory but only from $WORKSPACE.
It desirable to take the project's atficats too, even if the CI script failed, to allow debugging.
It seems that pipeline-based jobs (Lago's CI, experimental, OST) do not exhibit this behaviour.
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years
Build failed in Jenkins: ovirt_4.1_system-tests #9
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/changes>
Changes:
[Yaniv Kaul] Fixes and changes to storage tests
[Sandro Bonazzola] ovirt-hosted-engine-ha: added 4.1 branch
[Sandro Bonazzola] ovirt-release: move to 4.1 branch
[Sandro Bonazzola] cockpit-ovirt: exclude fc23 from 4.1
[Gil Shinar] Fix bugs in sdk yamls
------------------------------------------
[...truncated 750 lines...]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
+ exit
Took 62 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=4.1
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_4.1_system-tests] $ /bin/bash -xe /tmp/hudson2115846140055103178.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=4.1
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/>
+ OVIRT_SUITE=4.1
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-a...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-a...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-a...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -x
echo "shell-scripts/mock_cleanup.sh"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
shopt -s nullglob
WORKSPACE="${WORKSPACE:-$PWD}"
UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}"
UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}"
safe_umount() {
local mount="${1:?}"
local attempt
for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do
# If this is not the 1st time through the loop, Sleep a while to let
# the problem "solve itself"
[[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY"
# Try to umount
sudo umount --lazy "$mount" && return 0
# See if the mount is already not there despite failing
findmnt --kernel --first "$mount" > /dev/null && return 0
done
echo "ERROR: Failed to umount $mount."
return 1
}
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
for log in "${logs[@]}"; do
[[ "$log" = ./exported-artifacts/* ]] && continue
echo "Copying ${log} to exported-artifacts"
mv $log exported-artifacts/
done
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
safe_umount "$mount" || failed=true
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
safe_umount "$mount" && continue
# If we got here, we failed $UMOUNT_RETRIES attempts so we should make
# noise
failed=true
this_chroot_failed=true
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
# remove mock caches that are older then 2 days:
find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \
xargs -0 -tr sudo rm -rf
# We make no effort to leave around caches that may still be in use because
# packages installed in them may go out of date, so may as well recreate them
# Drop all left over libvirt domains
for UUID in $(virsh list --all --uuid); do
virsh destroy $UUID || :
sleep 2
virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || :
done
if $failed; then
echo "Cleanup script failed, propegating failure to job"
exit 1
fi
[ovirt_4.1_system-tests] $ /bin/bash -x /tmp/hudson5782931877352122916.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/>
+ UMOUNT_RETRIES=3
+ UMOUNT_RETRY_DELAY=1s
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/>
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ for log in '"${logs[@]}"'
+ [[ ./ovirt-system-tests/logs = ./exported-artifacts/* ]]
+ echo 'Copying ./ovirt-system-tests/logs to exported-artifacts'
Copying ./ovirt-system-tests/logs to exported-artifacts
+ mv ./ovirt-system-tests/logs exported-artifacts/
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-epel-7-x86_64.el7.cfg
+ mock_root=mocker-epel-7-x86_64.el7
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-epel-7-x86_64.el7'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests....> Using default...
INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests...>
+ mock_root=epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4
+ [[ -n epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4
+ :
+ [[ -n '' ]]
+ find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0
+ xargs -0 -tr sudo rm -rf
++ virsh list --all --uuid
+ false
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error?
Archiving artifacts
8 years
[JIRA] (OVIRT-987) Infra issues in Jenkins aren't reported properly on most of Jenkins jobs
by Yevgeny Zaspitsky (oVirt JIRA)
Yevgeny Zaspitsky created OVIRT-987:
---------------------------------------
Summary: Infra issues in Jenkins aren't reported properly on most of Jenkins jobs
Key: OVIRT-987
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-987
Project: oVirt - virtualization made easy
Issue Type: By-EMAIL
Reporter: Yevgeny Zaspitsky
Assignee: infra
Findbugs jobs reports an infra problem to gerrit properly: "There was an
infra issue, please contact infra(a)ovirt.org"
It'd be good to have same behavior on the rest of the Jenkins jobs.
Currently there is no differentiation between failure upon an infra issue
and a real patch problem in reporting to gerrit.
Regards,
Yevgeny
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years
[JIRA] (OVIRT-449) Add junit report to ovirt-engine standard ci
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-449?page=com.atlassian.jira... ]
eyal edri [Administrator] reassigned OVIRT-449:
-----------------------------------------------
Assignee: eyal edri [Administrator] (was: infra)
> Add junit report to ovirt-engine standard ci
> --------------------------------------------
>
> Key: OVIRT-449
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-449
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: David Caro Estevez
> Assignee: eyal edri [Administrator]
> Priority: Low
>
> Currently there are to ways of doing it:
> * Using a special job template for ovirt-engine (easy to do now, hard to maintain/reuse)
> * Using jenkins pipeline plugin to dynamically add it to any job that generates a report (hard to do right now, easy to maintain/reuse)
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years
[JIRA] (OVIRT-986) remove gerrit hooks from infra repos
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-986?page=com.atlassian.jira... ]
eyal edri [Administrator] updated OVIRT-986:
--------------------------------------------
Epic Link: OVIRT-411
> remove gerrit hooks from infra repos
> ------------------------------------
>
> Key: OVIRT-986
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-986
> Project: oVirt - virtualization made easy
> Issue Type: Task
> Reporter: eyal edri [Administrator]
> Assignee: infra
>
> We don't need to run oVirt gerrit hooks on infra repos like gerrit-admin or jenkins since we don't use bugzilla for tracking issues.
> We might want to add support for JIRA ticket added to a commit msg for handling infra repos, but that should require another ticket, for now we just need to remove the hooks from projects which are not relevant like jenkins,infra-docs,infra-puppet,gerrit-admin and others
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years
[JIRA] (OVIRT-986) remove gerrit hooks from infra repos
by eyal edri [Administrator] (oVirt JIRA)
eyal edri [Administrator] created OVIRT-986:
-----------------------------------------------
Summary: remove gerrit hooks from infra repos
Key: OVIRT-986
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-986
Project: oVirt - virtualization made easy
Issue Type: Task
Reporter: eyal edri [Administrator]
Assignee: infra
We don't need to run oVirt gerrit hooks on infra repos like gerrit-admin or jenkins since we don't use bugzilla for tracking issues.
We might want to add support for JIRA ticket added to a commit msg for handling infra repos, but that should require another ticket, for now we just need to remove the hooks from projects which are not relevant like jenkins,infra-docs,infra-puppet,gerrit-admin and others
--
This message was sent by Atlassian JIRA
(v1000.621.5#100023)
8 years