Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627

Yaniv Kaul <ykaul(a)redhat.com&gt; writes: > On Tue, Jan 10, 2017 at 12:08 PM, Lev Veyde <lveyde(a)redhat.com&gt; wrote: > >> This patch is one that caused it probably: >> https://github.com/lago-project/lago/commit/ 05ccf7240976f91b0c14d6a1f88016 >> 376d5e87f0 > > > +Milan. +Martin > I must confess that I did not like the patch to begin with... > I did not understand what real problem it solved, but Michal assured me > there was a real issue. Yes, there was a real issue with nested virtualization. Some CPU flags are missing with Haswell and Lago doesn't run properly.

> I know have Engine with a Java@ 100% CPU - I hope it's unrelated to this as > well. > > I suggest we do survey to see who doesn't have SandyBridge and above and > perhaps move higher than Westmere. We've got Westmere servers in the Brno lab.

> What do we have in CI? > Y. > > >> >> Thanks in advance, >> Lev Veyde. >> >> ----- Original Message ----- >> From: "Lev Veyde" <lveyde(a)redhat.com&gt; >> To: "Eyal Edri" <eedri(a)redhat.com&gt;, sbonazzo(a)redhat.com >> Cc: infra(a)ovirt.org, "Gal Ben Haim" <gbenhaim(a)redhat.com&gt; >> Sent: Tuesday, January 10, 2017 11:50:05 AM >> Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627 >> >> Hi, >> >> Checked the logs and see the following: >> >> 02:42:05 [WARNING] OVF does not contain a valid image description, using >> default. >> 02:42:05 The following CPU types are supported by this host: >> 02:42:05 - model_Westmere: Intel Westmere Family >> 02:42:05 - model_Nehalem: Intel Nehalem Family >> 02:42:05 - model_Penryn: Intel Penryn Family >> 02:42:05 - model_Conroe: Intel Conroe Family >> 02:42:05 [ ERROR ] Failed to execute stage 'Environment customization': >> Invalid CPU type specified: model_SandyBridge >> >> Barak thinks that it may be related to the recent update in the Lago code. >> >> Gal, any idea ? >> >> Thanks in advance, >> Lev Veyde. >> >> ----- Original Message ----- >> From: jenkins(a)jenkins.phx.ovirt.org >> To: sbonazzo(a)redhat.com, infra(a)ovirt.org, lveyde(a)redhat.com >> Sent: Tuesday, January 10, 2017 4:42:14 AM >> Subject: Build failed in Jenkins: ovirt_4.0_he-system-tests #627 >> >> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/changes > >> >> Changes: >> >> [Lev Veyde] Mask NetworkManager service >> >> [Eyal Edri] fix imgbased job names in jjb >> >> [Daniel Belenky] fixing jjb version for cockpit-ovirt >> >> [Gil Shinar] Add some more 4.1 to experimental >> >> [Juan Hernandez] Don't build RPMs for the JBoss modules Maven plugin >> >> [pkliczewski] jsonrpc 4.1 branch >> >> ------------------------------------------ >> [...truncated 749 lines...] >> Finish: shell >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> @@ Tue Jan 10 02:42:07 UTC 2017 automation/he_basic_suite_4.0.sh chroot >> finished >> @@ took 360 seconds >> @@ rc = 1 >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ >> ========== Scrubbing chroot >> mock \ >> --configdir="<http://jenkins.ovirt.org/job/ovirt_4.0_he- >> system-tests/ws/ovirt-system-tests"> \ >> --root="mocker-epel-7-x86_64.el7" \ >> --resultdir="./mock_logs.xGGwEk6V/mocker-epel-7-x86_64. el7.scrub" >> \ >> --scrub=chroot >> WARNING: Could not find required logging config file: < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/logging.ini.> Using default... >> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... >> Start: init plugins >> INFO: selinux enabled >> Finish: init plugins >> Start: run >> Start: scrub ['chroot'] >> INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 >> Finish: scrub ['chroot'] >> Finish: run >> Scrub chroot took 6 seconds >> ============================ >> ########################################################## >> ## Tue Jan 10 02:42:13 UTC 2017 Finished env: el7:epel-7-x86_64 >> ## took 366 seconds >> ## rc = 1 >> ########################################################## >> find: ‘logs’: No such file or directory >> No log files found, check command output >> ##!######################################################## >> Collecting mock logs >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.clean_rpmdb’ -> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb’ >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’ -> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_ >> basic_suite_4.0.sh’ >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.init’ -> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init’ >> ########################################################## >> Build step 'Execute shell' marked build as failure >> Performing Post build task... >> Match found for :.* : True >> Logical operation result is TRUE >> Running script : #!/bin/bash -xe >> echo 'shell_scripts/system_tests.collect_logs.sh' >> >> # >> # Required jjb vars: >> # version >> # >> VERSION=4.0 >> SUITE_TYPE= >> >> WORKSPACE="$PWD" >> OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" >> TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" >> >> rm -rf "$WORKSPACE/exported-artifacts" >> mkdir -p "$WORKSPACE/exported-artifacts" >> >> if [[ -d "$TESTS_LOGS" ]]; then >> mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" >> fi >> >> [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson302101162661598371. >> sh >> + echo shell_scripts/system_tests.collect_logs.sh >> shell_scripts/system_tests.collect_logs.sh >> + VERSION=4.0 >> + SUITE_TYPE= >> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > >> + OVIRT_SUITE=4.0 >> + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he- >> system-tests/ws/ovirt-system-tests/exported-artifacts> >> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ >> artifact/exported-artifacts> >> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ >> artifact/exported-artifacts> >> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/exported-artifacts> ]] >> + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/exported-artifacts/failure_msg.txt> < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/exported-artifacts/lago_logs> < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/exported-artifacts/mock_logs> < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ >> artifact/exported-artifacts/> >> POST BUILD TASK : SUCCESS >> END OF POST BUILD TASK : 0 >> Match found for :.* : True >> Logical operation result is TRUE >> Running script : #!/bin/bash -x >> echo "shell-scripts/mock_cleanup.sh" >> # Make clear this is the cleanup, helps reading the jenkins logs >> cat <<EOC >> _______________________________________________________________________ >> ####################################################################### >> # # >> # CLEANUP # >> # # >> ####################################################################### >> EOC >> >> shopt -s nullglob >> >> WORKSPACE="${WORKSPACE:-$PWD}" >> UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" >> UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" >> >> safe_umount() { >> local mount="${1:?}" >> local attempt >> for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do >> # If this is not the 1st time through the loop, Sleep a while to >> let >> # the problem "solve itself" >> [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" >> # Try to umount >> sudo umount --lazy "$mount" && return 0 >> # See if the mount is already not there despite failing >> findmnt --kernel --first "$mount" > /dev/null && return 0 >> done >> echo "ERROR: Failed to umount $mount." >> return 1 >> } >> >> # restore the permissions in the working dir, as sometimes it leaves files >> # owned by root and then the 'cleanup workspace' from jenkins job fails to >> # clean and breaks the jobs >> sudo chown -R "$USER" "$WORKSPACE" >> >> # stop any processes running inside the chroot >> failed=false >> mock_confs=("$WORKSPACE"/*/mocker*) >> # Clean current jobs mockroot if any >> for mock_conf_file in "${mock_confs[@]}"; do >> [[ "$mock_conf_file" ]] || continue >> echo "Cleaning up mock $mock_conf" >> mock_root="${mock_conf_file##*/}" >> mock_root="${mock_root%.*}" >> my_mock="/usr/bin/mock" >> my_mock+=" --configdir=${mock_conf_file%/*}" >> my_mock+=" --root=${mock_root}" >> my_mock+=" --resultdir=$WORKSPACE" >> >> #TODO: investigate why mock --clean fails to umount certain dirs >> sometimes, >> #so we can use it instead of manually doing all this. >> echo "Killing all mock orphan processes, if any." >> $my_mock \ >> --orphanskill \ >> || { >> echo "ERROR: Failed to kill orphans on $chroot." >> failed=true >> } >> >> mock_root="$(\ >> grep \ >> -Po "(?<=config_opts\['root'\] = ')[^']*" \ >> "$mock_conf_file" \ >> )" || : >> [[ "$mock_root" ]] || continue >> mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : >> if [[ "$mounts" ]]; then >> echo "Found mounted dirs inside the chroot $chroot. Trying to >> umount." >> fi >> for mount in "${mounts[@]}"; do >> safe_umount "$mount" || failed=true >> done >> done >> >> # Clean any leftover chroot from other jobs >> for mock_root in /var/lib/mock/*; do >> this_chroot_failed=false >> mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || >> : >> if [[ "$mounts" ]]; then >> echo "Found mounted dirs inside the chroot $mock_root." \ >> "Trying to umount." >> fi >> for mount in "${mounts[@]}"; do >> safe_umount "$mount" && continue >> # If we got here, we failed $UMOUNT_RETRIES attempts so we should >> make >> # noise >> failed=true >> this_chroot_failed=true >> done >> if ! $this_chroot_failed; then >> sudo rm -rf "$mock_root" >> fi >> done >> >> # remove mock caches that are older then 2 days: >> find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ >> xargs -0 -tr sudo rm -rf >> # We make no effort to leave around caches that may still be in use because >> # packages installed in them may go out of date, so may as well recreate >> them >> >> # Drop all left over libvirt domains >> for UUID in $(virsh list --all --uuid); do >> virsh destroy $UUID || : >> sleep 2 >> virsh undefine --remove-all-storage --storage vda --snapshots-metadata >> $UUID || : >> done >> >> if $failed; then >> echo "Cleanup script failed, propegating failure to job" >> exit 1 >> fi >> >> [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/ >> hudson1888216492513466503.sh >> + echo shell-scripts/mock_cleanup.sh >> shell-scripts/mock_cleanup.sh >> + cat >> _______________________________________________________________________ >> ####################################################################### >> # # >> # CLEANUP # >> # # >> ####################################################################### >> + shopt -s nullglob >> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > >> + UMOUNT_RETRIES=3 >> + UMOUNT_RETRY_DELAY=1s >> + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ >> ovirt_4.0_he-system-tests/ws/> >> + failed=false >> + mock_confs=("$WORKSPACE"/*/mocker*) >> + for mock_conf_file in '"${mock_confs[@]}"' >> + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] >> + echo 'Cleaning up mock ' >> Cleaning up mock >> + mock_root=mocker-epel-7-x86_64.el7.cfg >> + mock_root=mocker-epel-7-x86_64.el7 >> + my_mock=/usr/bin/mock >> + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- >> system-tests/ws/ovirt-system-tests'> >> + my_mock+=' --root=mocker-epel-7-x86_64.el7' >> + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- >> system-tests/ws/'> >> + echo 'Killing all mock orphan processes, if any.' >> Killing all mock orphan processes, if any. >> + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- >> system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 >> --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > >> --orphanskill >> WARNING: Could not find required logging config file: < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/logging.ini.> Using default... >> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... >> Start: init plugins >> INFO: selinux enabled >> Finish: init plugins >> Start: run >> Finish: run >> ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ >> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> >> + mock_root=epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b >> + [[ -n epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b ]] >> + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) >> ++ mount >> ++ awk '{print $3}' >> ++ grep epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b >> + : >> + [[ -n '' ]] >> + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 >> + xargs -0 -tr sudo rm -rf >> ++ virsh list --all --uuid >> + false >> POST BUILD TASK : SUCCESS >> END OF POST BUILD TASK : 1 >> Recording test results >> ERROR: Step ‘Publish JUnit test result report’ failed: No test report >> files were found. Configuration error? >> Archiving artifacts >>