Build failed in Jenkins: ovirt_4.0_he-system-tests #627
Milan Zamazal
mzamazal at redhat.com
Tue Jan 10 10:45:30 UTC 2017
Yaniv Kaul <ykaul at redhat.com> writes:
> On Tue, Jan 10, 2017 at 12:08 PM, Lev Veyde <lveyde at redhat.com> wrote:
>
>> This patch is one that caused it probably:
>> https://github.com/lago-project/lago/commit/05ccf7240976f91b0c14d6a1f88016
>> 376d5e87f0
>
>
> +Milan.
+Martin
> I must confess that I did not like the patch to begin with...
> I did not understand what real problem it solved, but Michal assured me
> there was a real issue.
Yes, there was a real issue with nested virtualization. Some CPU flags
are missing with Haswell and Lago doesn't run properly.
> I know have Engine with a Java@ 100% CPU - I hope it's unrelated to this as
> well.
>
> I suggest we do survey to see who doesn't have SandyBridge and above and
> perhaps move higher than Westmere.
We've got Westmere servers in the Brno lab.
> What do we have in CI?
> Y.
>
>
>>
>> Thanks in advance,
>> Lev Veyde.
>>
>> ----- Original Message -----
>> From: "Lev Veyde" <lveyde at redhat.com>
>> To: "Eyal Edri" <eedri at redhat.com>, sbonazzo at redhat.com
>> Cc: infra at ovirt.org, "Gal Ben Haim" <gbenhaim at redhat.com>
>> Sent: Tuesday, January 10, 2017 11:50:05 AM
>> Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627
>>
>> Hi,
>>
>> Checked the logs and see the following:
>>
>> 02:42:05 [WARNING] OVF does not contain a valid image description, using
>> default.
>> 02:42:05 The following CPU types are supported by this host:
>> 02:42:05 - model_Westmere: Intel Westmere Family
>> 02:42:05 - model_Nehalem: Intel Nehalem Family
>> 02:42:05 - model_Penryn: Intel Penryn Family
>> 02:42:05 - model_Conroe: Intel Conroe Family
>> 02:42:05 [ ERROR ] Failed to execute stage 'Environment customization':
>> Invalid CPU type specified: model_SandyBridge
>>
>> Barak thinks that it may be related to the recent update in the Lago code.
>>
>> Gal, any idea ?
>>
>> Thanks in advance,
>> Lev Veyde.
>>
>> ----- Original Message -----
>> From: jenkins at jenkins.phx.ovirt.org
>> To: sbonazzo at redhat.com, infra at ovirt.org, lveyde at redhat.com
>> Sent: Tuesday, January 10, 2017 4:42:14 AM
>> Subject: Build failed in Jenkins: ovirt_4.0_he-system-tests #627
>>
>> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/changes>
>>
>> Changes:
>>
>> [Lev Veyde] Mask NetworkManager service
>>
>> [Eyal Edri] fix imgbased job names in jjb
>>
>> [Daniel Belenky] fixing jjb version for cockpit-ovirt
>>
>> [Gil Shinar] Add some more 4.1 to experimental
>>
>> [Juan Hernandez] Don't build RPMs for the JBoss modules Maven plugin
>>
>> [pkliczewski] jsonrpc 4.1 branch
>>
>> ------------------------------------------
>> [...truncated 749 lines...]
>> Finish: shell
>> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> @@ Tue Jan 10 02:42:07 UTC 2017 automation/he_basic_suite_4.0.sh chroot
>> finished
>> @@ took 360 seconds
>> @@ rc = 1
>> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> ========== Scrubbing chroot
>> mock \
>> --configdir="<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> system-tests/ws/ovirt-system-tests"> \
>> --root="mocker-epel-7-x86_64.el7" \
>> --resultdir="./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.scrub"
>> \
>> --scrub=chroot
>> WARNING: Could not find required logging config file: <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/logging.ini.> Using default...
>> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
>> Start: init plugins
>> INFO: selinux enabled
>> Finish: init plugins
>> Start: run
>> Start: scrub ['chroot']
>> INFO: scrubbing chroot for mocker-epel-7-x86_64.el7
>> Finish: scrub ['chroot']
>> Finish: run
>> Scrub chroot took 6 seconds
>> ============================
>> ##########################################################
>> ## Tue Jan 10 02:42:13 UTC 2017 Finished env: el7:epel-7-x86_64
>> ## took 366 seconds
>> ## rc = 1
>> ##########################################################
>> find: ‘logs’: No such file or directory
>> No log files found, check command output
>> ##!########################################################
>> Collecting mock logs
>> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.clean_rpmdb’ ->
>> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb’
>> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’ ->
>> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_
>> basic_suite_4.0.sh’
>> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.init’ ->
>> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init’
>> ##########################################################
>> Build step 'Execute shell' marked build as failure
>> Performing Post build task...
>> Match found for :.* : True
>> Logical operation result is TRUE
>> Running script : #!/bin/bash -xe
>> echo 'shell_scripts/system_tests.collect_logs.sh'
>>
>> #
>> # Required jjb vars:
>> # version
>> #
>> VERSION=4.0
>> SUITE_TYPE=
>>
>> WORKSPACE="$PWD"
>> OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
>> TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
>>
>> rm -rf "$WORKSPACE/exported-artifacts"
>> mkdir -p "$WORKSPACE/exported-artifacts"
>>
>> if [[ -d "$TESTS_LOGS" ]]; then
>> mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
>> fi
>>
>> [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson302101162661598371.
>> sh
>> + echo shell_scripts/system_tests.collect_logs.sh
>> shell_scripts/system_tests.collect_logs.sh
>> + VERSION=4.0
>> + SUITE_TYPE=
>> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>> + OVIRT_SUITE=4.0
>> + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> system-tests/ws/ovirt-system-tests/exported-artifacts>
>> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> artifact/exported-artifacts>
>> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> artifact/exported-artifacts>
>> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/exported-artifacts> ]]
>> + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/exported-artifacts/failure_msg.txt> <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/exported-artifacts/lago_logs> <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/exported-artifacts/mock_logs> <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> artifact/exported-artifacts/>
>> POST BUILD TASK : SUCCESS
>> END OF POST BUILD TASK : 0
>> Match found for :.* : True
>> Logical operation result is TRUE
>> Running script : #!/bin/bash -x
>> echo "shell-scripts/mock_cleanup.sh"
>> # Make clear this is the cleanup, helps reading the jenkins logs
>> cat <<EOC
>> _______________________________________________________________________
>> #######################################################################
>> # #
>> # CLEANUP #
>> # #
>> #######################################################################
>> EOC
>>
>> shopt -s nullglob
>>
>> WORKSPACE="${WORKSPACE:-$PWD}"
>> UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}"
>> UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}"
>>
>> safe_umount() {
>> local mount="${1:?}"
>> local attempt
>> for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do
>> # If this is not the 1st time through the loop, Sleep a while to
>> let
>> # the problem "solve itself"
>> [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY"
>> # Try to umount
>> sudo umount --lazy "$mount" && return 0
>> # See if the mount is already not there despite failing
>> findmnt --kernel --first "$mount" > /dev/null && return 0
>> done
>> echo "ERROR: Failed to umount $mount."
>> return 1
>> }
>>
>> # restore the permissions in the working dir, as sometimes it leaves files
>> # owned by root and then the 'cleanup workspace' from jenkins job fails to
>> # clean and breaks the jobs
>> sudo chown -R "$USER" "$WORKSPACE"
>>
>> # stop any processes running inside the chroot
>> failed=false
>> mock_confs=("$WORKSPACE"/*/mocker*)
>> # Clean current jobs mockroot if any
>> for mock_conf_file in "${mock_confs[@]}"; do
>> [[ "$mock_conf_file" ]] || continue
>> echo "Cleaning up mock $mock_conf"
>> mock_root="${mock_conf_file##*/}"
>> mock_root="${mock_root%.*}"
>> my_mock="/usr/bin/mock"
>> my_mock+=" --configdir=${mock_conf_file%/*}"
>> my_mock+=" --root=${mock_root}"
>> my_mock+=" --resultdir=$WORKSPACE"
>>
>> #TODO: investigate why mock --clean fails to umount certain dirs
>> sometimes,
>> #so we can use it instead of manually doing all this.
>> echo "Killing all mock orphan processes, if any."
>> $my_mock \
>> --orphanskill \
>> || {
>> echo "ERROR: Failed to kill orphans on $chroot."
>> failed=true
>> }
>>
>> mock_root="$(\
>> grep \
>> -Po "(?<=config_opts\['root'\] = ')[^']*" \
>> "$mock_conf_file" \
>> )" || :
>> [[ "$mock_root" ]] || continue
>> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>> if [[ "$mounts" ]]; then
>> echo "Found mounted dirs inside the chroot $chroot. Trying to
>> umount."
>> fi
>> for mount in "${mounts[@]}"; do
>> safe_umount "$mount" || failed=true
>> done
>> done
>>
>> # Clean any leftover chroot from other jobs
>> for mock_root in /var/lib/mock/*; do
>> this_chroot_failed=false
>> mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) ||
>> :
>> if [[ "$mounts" ]]; then
>> echo "Found mounted dirs inside the chroot $mock_root." \
>> "Trying to umount."
>> fi
>> for mount in "${mounts[@]}"; do
>> safe_umount "$mount" && continue
>> # If we got here, we failed $UMOUNT_RETRIES attempts so we should
>> make
>> # noise
>> failed=true
>> this_chroot_failed=true
>> done
>> if ! $this_chroot_failed; then
>> sudo rm -rf "$mock_root"
>> fi
>> done
>>
>> # remove mock caches that are older then 2 days:
>> find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \
>> xargs -0 -tr sudo rm -rf
>> # We make no effort to leave around caches that may still be in use because
>> # packages installed in them may go out of date, so may as well recreate
>> them
>>
>> # Drop all left over libvirt domains
>> for UUID in $(virsh list --all --uuid); do
>> virsh destroy $UUID || :
>> sleep 2
>> virsh undefine --remove-all-storage --storage vda --snapshots-metadata
>> $UUID || :
>> done
>>
>> if $failed; then
>> echo "Cleanup script failed, propegating failure to job"
>> exit 1
>> fi
>>
>> [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/
>> hudson1888216492513466503.sh
>> + echo shell-scripts/mock_cleanup.sh
>> shell-scripts/mock_cleanup.sh
>> + cat
>> _______________________________________________________________________
>> #######################################################################
>> # #
>> # CLEANUP #
>> # #
>> #######################################################################
>> + shopt -s nullglob
>> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>> + UMOUNT_RETRIES=3
>> + UMOUNT_RETRY_DELAY=1s
>> + sudo chown -R jenkins <http://jenkins.ovirt.org/job/
>> ovirt_4.0_he-system-tests/ws/>
>> + failed=false
>> + mock_confs=("$WORKSPACE"/*/mocker*)
>> + for mock_conf_file in '"${mock_confs[@]}"'
>> + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]]
>> + echo 'Cleaning up mock '
>> Cleaning up mock
>> + mock_root=mocker-epel-7-x86_64.el7.cfg
>> + mock_root=mocker-epel-7-x86_64.el7
>> + my_mock=/usr/bin/mock
>> + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> system-tests/ws/ovirt-system-tests'>
>> + my_mock+=' --root=mocker-epel-7-x86_64.el7'
>> + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> system-tests/ws/'>
>> + echo 'Killing all mock orphan processes, if any.'
>> Killing all mock orphan processes, if any.
>> + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7
>> --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/>
>> --orphanskill
>> WARNING: Could not find required logging config file: <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/logging.ini.> Using default...
>> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
>> Start: init plugins
>> INFO: selinux enabled
>> Finish: init plugins
>> Start: run
>> Finish: run
>> ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <
>> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg>
>> + mock_root=epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b
>> + [[ -n epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b ]]
>> + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
>> ++ mount
>> ++ awk '{print $3}'
>> ++ grep epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b
>> + :
>> + [[ -n '' ]]
>> + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0
>> + xargs -0 -tr sudo rm -rf
>> ++ virsh list --all --uuid
>> + false
>> POST BUILD TASK : SUCCESS
>> END OF POST BUILD TASK : 1
>> Recording test results
>> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
>> files were found. Configuration error?
>> Archiving artifacts
>>
More information about the Infra
mailing list