Build failed in Jenkins: ovirt_3.6_system-tests #453
by jenkins＠jenkins.phx.ovirt.org
See <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/changes>
Changes:
[Eyal Edri] Fail early on he setup first or second host install
------------------------------------------
[...truncated 703 lines...]
## took 2369 seconds
## rc = 1
##########################################################
##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
##!
+ true
+ env_cleanup
+ echo '#########################'
#########################
+ local res=0
+ local uuid
+ echo '======== Cleaning up'
======== Cleaning up
+ [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> ]]
+ echo '----------- Cleaning with lago'
----------- Cleaning with lago
+ lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> destroy --yes --all-prefixes
+ echo '----------- Cleaning with lago done'
----------- Cleaning with lago done
+ [[ 0 != \0 ]]
+ echo '======== Cleanup done'
======== Cleanup done
+ exit 0
Took 2203 seconds
===================================
##!
##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
##!########################################################
##########################################################
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo 'shell_scripts/system_tests.collect_logs.sh'
#
# Required jjb vars:
# version
#
VERSION=3.6
SUITE_TYPE=
WORKSPACE="$PWD"
OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
rm -rf "$WORKSPACE/exported-artifacts"
mkdir -p "$WORKSPACE/exported-artifacts"
if [[ -d "$TESTS_LOGS" ]]; then
mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
fi
[ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson3737170357820277489.sh
+ echo shell_scripts/system_tests.collect_logs.sh
shell_scripts/system_tests.collect_logs.sh
+ VERSION=3.6
+ SUITE_TYPE=
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
+ OVIRT_SUITE=3.6
+ TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...>
+ rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported...>
+ mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported...>
+ [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> ]]
+ mv <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported...>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 0
Match found for :.* : True
Logical operation result is TRUE
Running script : #!/bin/bash -xe
echo "shell-scripts/mock_cleanup.sh"
shopt -s nullglob
WORKSPACE="$PWD"
# Make clear this is the cleanup, helps reading the jenkins logs
cat <<EOC
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
EOC
# Archive the logs, we want them anyway
logs=(
./*log
./*/logs
)
if [[ "$logs" ]]; then
tar cvzf exported-artifacts/logs.tgz "${logs[@]}"
rm -rf "${logs[@]}"
fi
# stop any processes running inside the chroot
failed=false
mock_confs=("$WORKSPACE"/*/mocker*)
# Clean current jobs mockroot if any
for mock_conf_file in "${mock_confs[@]}"; do
[[ "$mock_conf_file" ]] || continue
echo "Cleaning up mock $mock_conf"
mock_root="${mock_conf_file##*/}"
mock_root="${mock_root%.*}"
my_mock="/usr/bin/mock"
my_mock+=" --configdir=${mock_conf_file%/*}"
my_mock+=" --root=${mock_root}"
my_mock+=" --resultdir=$WORKSPACE"
#TODO: investigate why mock --clean fails to umount certain dirs sometimes,
#so we can use it instead of manually doing all this.
echo "Killing all mock orphan processes, if any."
$my_mock \
--orphanskill \
|| {
echo "ERROR: Failed to kill orphans on $chroot."
failed=true
}
mock_root="$(\
grep \
-Po "(?<=config_opts\['root'\] = ')[^']*" \
"$mock_conf_file" \
)" || :
[[ "$mock_root" ]] || continue
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $chroot. Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
}
done
done
# Clean any leftover chroot from other jobs
for mock_root in /var/lib/mock/*; do
this_chroot_failed=false
mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
if [[ "$mounts" ]]; then
echo "Found mounted dirs inside the chroot $mock_root." \
"Trying to umount."
fi
for mount in "${mounts[@]}"; do
sudo umount --lazy "$mount" \
|| {
echo "ERROR: Failed to umount $mount."
failed=true
this_chroot_failed=true
}
done
if ! $this_chroot_failed; then
sudo rm -rf "$mock_root"
fi
done
if $failed; then
echo "Aborting."
exit 1
fi
# remove mock system cache, we will setup proxies to do the caching and this
# takes lots of space between runs
shopt -u nullglob
sudo rm -Rf /var/cache/mock/*
# restore the permissions in the working dir, as sometimes it leaves files
# owned by root and then the 'cleanup workspace' from jenkins job fails to
# clean and breaks the jobs
sudo chown -R "$USER" "$WORKSPACE"
[ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson677792111944984977.sh
+ echo shell-scripts/mock_cleanup.sh
shell-scripts/mock_cleanup.sh
+ shopt -s nullglob
+ WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
+ cat
_______________________________________________________________________
#######################################################################
# #
# CLEANUP #
# #
#######################################################################
+ logs=(./*log ./*/logs)
+ [[ -n ./ovirt-system-tests/logs ]]
+ tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs
./ovirt-system-tests/logs/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/
./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log
+ rm -rf ./ovirt-system-tests/logs
+ failed=false
+ mock_confs=("$WORKSPACE"/*/mocker*)
+ for mock_conf_file in '"${mock_confs[@]}"'
+ [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...> ]]
+ echo 'Cleaning up mock '
Cleaning up mock
+ mock_root=mocker-fedora-23-x86_64.fc23.cfg
+ mock_root=mocker-fedora-23-x86_64.fc23
+ my_mock=/usr/bin/mock
+ my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests'>
+ my_mock+=' --root=mocker-fedora-23-x86_64.fc23'
+ my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/'>
+ echo 'Killing all mock orphan processes, if any.'
Killing all mock orphan processes, if any.
+ /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> --orphanskill
WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests....> Using default...
INFO: mock.py version 1.2.18 starting (python version = 3.4.3)...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Finish: run
++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests...>
+ mock_root=fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c
+ [[ -n fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c ]]
+ mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
++ mount
++ awk '{print $3}'
++ grep fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c
+ :
+ [[ -n '' ]]
+ false
+ shopt -u nullglob
+ sudo rm -Rf /var/cache/mock/fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c
+ sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/>
POST BUILD TASK : SUCCESS
END OF POST BUILD TASK : 1
Recording test results
Archiving artifacts
8 years, 2 months
[oVirt Jenkins] repos_4.0_check-closure_merged - Build # 26 - Still Failing!
by jenkins＠jenkins.phx.ovirt.org
------=_Part_185_1755631428.1473003986840
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Project: http://jenkins.ovirt.org/job/repos_4.0_check-closure_merged/
Build: http://jenkins.ovirt.org/job/repos_4.0_check-closure_merged/26/
Build Number: 26
Build Status: Still Failing
Triggered By: Started by user Sharon Naftaly
-------------------------------------
Changes Since Last Success:
-------------------------------------
Changes for Build #18
[Sharon Naftaly] Adding ovirt-hosted-engine-ha std-ci jobs
Changes for Build #19
[Eyal Edri] align all job types to keep longer history of 120 builds
[Eyal Edri] splitting mom created and merged projects in yaml
Changes for Build #20
No changes
Changes for Build #21
No changes
Changes for Build #22
[Sharon Naftaly] Changing repos_check_closure jobs email recipients
[Eyal Edri] restore check-patch for master on el7
Changes for Build #23
No changes
Changes for Build #24
No changes
Changes for Build #25
No changes
Changes for Build #26
[Sharon Naftaly] Add recipients list to failure email in check closure jobs
[Irit Goihman] vdsm: move check merged job on master to el7 and remove fc22 jobs
[Eyal Edri] missed increasing ovirt-engine build-artifacts history
-----------------
Failed Tests:
-----------------
No tests ran.
------=_Part_185_1755631428.1473003986840--
8 years, 2 months
[JIRA] (OVIRT-719) check-patch jobs should check if the patch is still open
by eyal edri [Administrator] (oVirt JIRA)
eyal edri [Administrator] created OVIRT-719:
-----------------------------------------------
Summary: check-patch jobs should check if the patch is still open
Key: OVIRT-719
URL: https://ovirt-jira.atlassian.net/browse/OVIRT-719
Project: oVirt - virtualization made easy
Issue Type: Improvement
Reporter: eyal edri [Administrator]
Assignee: infra
We sometimes see multiple check-patch jobs still running while actually the patch for them was already merged.
This happens when Jenkins queue is too big and the committer merge the patch before the job in CI finish or even starts.
In the case where the job didn't even start because of long queue, there is not reason to continue running the job if the patch was merged.
Any post merge logic will be run in check-merged stage and there is no value in adding info on the check-patch to the merged commit.
It should be easy to check in the beginning of the check-patch job and part of standard CI if the patch that is triggered the job is still open and the job should continue.
If the patch is merged, then we should abort the job and thus cleaning the queue from check-patch jobs that won't have any value .
--
This message was sent by Atlassian JIRA
(v1000.305.0#100011)
8 years, 2 months
[JIRA] (OVIRT-719) check-patch jobs should check if the patch is still open
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-719?page=com.atlassian.jira... ]
eyal edri [Administrator] updated OVIRT-719:
--------------------------------------------
Epic Link: OVIRT-400
> check-patch jobs should check if the patch is still open
> ---------------------------------------------------------
>
> Key: OVIRT-719
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-719
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Reporter: eyal edri [Administrator]
> Assignee: infra
>
> We sometimes see multiple check-patch jobs still running while actually the patch for them was already merged.
> This happens when Jenkins queue is too big and the committer merge the patch before the job in CI finish or even starts.
> In the case where the job didn't even start because of long queue, there is not reason to continue running the job if the patch was merged.
> Any post merge logic will be run in check-merged stage and there is no value in adding info on the check-patch to the merged commit.
> It should be easy to check in the beginning of the check-patch job and part of standard CI if the patch that is triggered the job is still open and the job should continue.
> If the patch is merged, then we should abort the job and thus cleaning the queue from check-patch jobs that won't have any value .
--
This message was sent by Atlassian JIRA
(v1000.305.0#100011)
8 years, 2 months
[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by snaftaly (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira... ]
snaftaly commented on OVIRT-717:
--------------------------------
[~eedri], yes, currently the package is installed at the beginning of the job.
> livecd-tools RPM from ci-tools repo is missing dependencies
> -----------------------------------------------------------
>
> Key: OVIRT-717
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: General
> Reporter: Evgheni Dereveanchin
> Assignee: infra
>
> It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise.
--
This message was sent by Atlassian JIRA
(v1000.305.0#100011)
8 years, 2 months
[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira... ]
eyal edri [Administrator] commented on OVIRT-717:
-------------------------------------------------
[~snaftaly(a)redhat.com] did you end up installing locally the RPM in the job? do we still need to refresh the rpm on ci-tools repo?
> livecd-tools RPM from ci-tools repo is missing dependencies
> -----------------------------------------------------------
>
> Key: OVIRT-717
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: General
> Reporter: Evgheni Dereveanchin
> Assignee: infra
>
> It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise.
--
This message was sent by Atlassian JIRA
(v1000.305.0#100011)
8 years, 2 months
[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira... ]
eyal edri [Administrator] updated OVIRT-717:
--------------------------------------------
Epic Link: OVIRT-403
> livecd-tools RPM from ci-tools repo is missing dependencies
> -----------------------------------------------------------
>
> Key: OVIRT-717
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717
> Project: oVirt - virtualization made easy
> Issue Type: Bug
> Components: General
> Reporter: Evgheni Dereveanchin
> Assignee: infra
>
> It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise.
--
This message was sent by Atlassian JIRA
(v1000.305.0#100011)
8 years, 2 months
