Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627

On Tue, Jan 10, 2017 at 12:08 PM, Lev Veyde <lveyde(a)redhat.com&gt; wrote: > This patch is one that caused it probably: > https://github.com/lago-project/lago/commit/05ccf7240976f91b0c14d6a1f88016 > 376d5e87f0 +Milan.

I must confess that I did not like the patch to begin with... I did not understand what real problem it solved, but Michal assured me there was a real issue.

I know have Engine with a Java@ 100% CPU - I hope it's unrelated to this as well. I suggest we do survey to see who doesn't have SandyBridge and above and perhaps move higher than Westmere.

What do we have in CI? Y. > > Thanks in advance, > Lev Veyde. > > ----- Original Message ----- > From: "Lev Veyde" <lveyde(a)redhat.com&gt; > To: "Eyal Edri" <eedri(a)redhat.com&gt;, sbonazzo(a)redhat.com > Cc: infra(a)ovirt.org, "Gal Ben Haim" <gbenhaim(a)redhat.com&gt; > Sent: Tuesday, January 10, 2017 11:50:05 AM > Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627 > > Hi, > > Checked the logs and see the following: > > 02:42:05 [WARNING] OVF does not contain a valid image description, using > default. > 02:42:05 The following CPU types are supported by this host: > 02:42:05 - model_Westmere: Intel Westmere Family > 02:42:05 - model_Nehalem: Intel Nehalem Family > 02:42:05 - model_Penryn: Intel Penryn Family > 02:42:05 - model_Conroe: Intel Conroe Family > 02:42:05 [ ERROR ] Failed to execute stage 'Environment customization': > Invalid CPU type specified: model_SandyBridge > > Barak thinks that it may be related to the recent update in the Lago code. > > Gal, any idea ? > > Thanks in advance, > Lev Veyde. > > ----- Original Message ----- > From: jenkins(a)jenkins.phx.ovirt.org > To: sbonazzo(a)redhat.com, infra(a)ovirt.org, lveyde(a)redhat.com > Sent: Tuesday, January 10, 2017 4:42:14 AM > Subject: Build failed in Jenkins: ovirt_4.0_he-system-tests #627 > > See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/changes> > > Changes: > > [Lev Veyde] Mask NetworkManager service > > [Eyal Edri] fix imgbased job names in jjb > > [Daniel Belenky] fixing jjb version for cockpit-ovirt > > [Gil Shinar] Add some more 4.1 to experimental > > [Juan Hernandez] Don't build RPMs for the JBoss modules Maven plugin > > [pkliczewski] jsonrpc 4.1 branch > > ------------------------------------------ > [...truncated 749 lines...] > Finish: shell > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > @@ Tue Jan 10 02:42:07 UTC 2017 automation/he_basic_suite_4.0.sh chroot > finished > @@ took 360 seconds > @@ rc = 1 > @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ > ========== Scrubbing chroot > mock \ > --configdir="<http://jenkins.ovirt.org/job/ovirt_4.0_he- > system-tests/ws/ovirt-system-tests"> \ > --root="mocker-epel-7-x86_64.el7" \ > --resultdir="./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.scrub" > \ > --scrub=chroot > WARNING: Could not find required logging config file: < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/logging.ini.> Using default... > INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... > Start: init plugins > INFO: selinux enabled > Finish: init plugins > Start: run > Start: scrub ['chroot'] > INFO: scrubbing chroot for mocker-epel-7-x86_64.el7 > Finish: scrub ['chroot'] > Finish: run > Scrub chroot took 6 seconds > ============================ > ########################################################## > ## Tue Jan 10 02:42:13 UTC 2017 Finished env: el7:epel-7-x86_64 > ## took 366 seconds > ## rc = 1 > ########################################################## > find: ‘logs’: No such file or directory > No log files found, check command output > ##!######################################################## > Collecting mock logs > ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.clean_rpmdb’ -> > ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb’ > ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’ -> > ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_ > basic_suite_4.0.sh’ > ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.init’ -> > ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init’ > ########################################################## > Build step 'Execute shell' marked build as failure > Performing Post build task... > Match found for :.* : True > Logical operation result is TRUE > Running script : #!/bin/bash -xe > echo 'shell_scripts/system_tests.collect_logs.sh' > > # > # Required jjb vars: > # version > # > VERSION=4.0 > SUITE_TYPE= > > WORKSPACE="$PWD" > OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" > TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" > > rm -rf "$WORKSPACE/exported-artifacts" > mkdir -p "$WORKSPACE/exported-artifacts" > > if [[ -d "$TESTS_LOGS" ]]; then > mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" > fi > > [ovirt_4.0_he-system-tests] $ /bin/bash -xe /tmp/hudson302101162661598371. > sh > + echo shell_scripts/system_tests.collect_logs.sh > shell_scripts/system_tests.collect_logs.sh > + VERSION=4.0 > + SUITE_TYPE= > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> > + OVIRT_SUITE=4.0 > + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he- > system-tests/ws/ovirt-system-tests/exported-artifacts> > + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ > artifact/exported-artifacts> > + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ > artifact/exported-artifacts> > + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/exported-artifacts> ]] > + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/exported-artifacts/failure_msg.txt> < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/exported-artifacts/lago_logs> < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/exported-artifacts/mock_logs> < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/ > artifact/exported-artifacts/> > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 0 > Match found for :.* : True > Logical operation result is TRUE > Running script : #!/bin/bash -x > echo "shell-scripts/mock_cleanup.sh" > # Make clear this is the cleanup, helps reading the jenkins logs > cat <<EOC > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > EOC > > shopt -s nullglob > > WORKSPACE="${WORKSPACE:-$PWD}" > UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" > UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" > > safe_umount() { > local mount="${1:?}" > local attempt > for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do > # If this is not the 1st time through the loop, Sleep a while to > let > # the problem "solve itself" > [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" > # Try to umount > sudo umount --lazy "$mount" && return 0 > # See if the mount is already not there despite failing > findmnt --kernel --first "$mount" > /dev/null && return 0 > done > echo "ERROR: Failed to umount $mount." > return 1 > } > > # restore the permissions in the working dir, as sometimes it leaves files > # owned by root and then the 'cleanup workspace' from jenkins job fails to > # clean and breaks the jobs > sudo chown -R "$USER" "$WORKSPACE" > > # stop any processes running inside the chroot > failed=false > mock_confs=("$WORKSPACE"/*/mocker*) > # Clean current jobs mockroot if any > for mock_conf_file in "${mock_confs[@]}"; do > [[ "$mock_conf_file" ]] || continue > echo "Cleaning up mock $mock_conf" > mock_root="${mock_conf_file##*/}" > mock_root="${mock_root%.*}" > my_mock="/usr/bin/mock" > my_mock+=" --configdir=${mock_conf_file%/*}" > my_mock+=" --root=${mock_root}" > my_mock+=" --resultdir=$WORKSPACE" > > #TODO: investigate why mock --clean fails to umount certain dirs > sometimes, > #so we can use it instead of manually doing all this. > echo "Killing all mock orphan processes, if any." > $my_mock \ > --orphanskill \ > || { > echo "ERROR: Failed to kill orphans on $chroot." > failed=true > } > > mock_root="$(\ > grep \ > -Po "(?<=config_opts\['root'\] = ')[^']*" \ > "$mock_conf_file" \ > )" || : > [[ "$mock_root" ]] || continue > mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $chroot. Trying to > umount." > fi > for mount in "${mounts[@]}"; do > safe_umount "$mount" || failed=true > done > done > > # Clean any leftover chroot from other jobs > for mock_root in /var/lib/mock/*; do > this_chroot_failed=false > mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || > : > if [[ "$mounts" ]]; then > echo "Found mounted dirs inside the chroot $mock_root." \ > "Trying to umount." > fi > for mount in "${mounts[@]}"; do > safe_umount "$mount" && continue > # If we got here, we failed $UMOUNT_RETRIES attempts so we should > make > # noise > failed=true > this_chroot_failed=true > done > if ! $this_chroot_failed; then > sudo rm -rf "$mock_root" > fi > done > > # remove mock caches that are older then 2 days: > find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ > xargs -0 -tr sudo rm -rf > # We make no effort to leave around caches that may still be in use because > # packages installed in them may go out of date, so may as well recreate > them > > # Drop all left over libvirt domains > for UUID in $(virsh list --all --uuid); do > virsh destroy $UUID || : > sleep 2 > virsh undefine --remove-all-storage --storage vda --snapshots-metadata > $UUID || : > done > > if $failed; then > echo "Cleanup script failed, propegating failure to job" > exit 1 > fi > > [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/ > hudson1888216492513466503.sh > + echo shell-scripts/mock_cleanup.sh > shell-scripts/mock_cleanup.sh > + cat > _______________________________________________________________________ > ####################################################################### > # # > # CLEANUP # > # # > ####################################################################### > + shopt -s nullglob > + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> > + UMOUNT_RETRIES=3 > + UMOUNT_RETRY_DELAY=1s > + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ > ovirt_4.0_he-system-tests/ws/> > + failed=false > + mock_confs=("$WORKSPACE"/*/mocker*) > + for mock_conf_file in '"${mock_confs[@]}"' > + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]] > + echo 'Cleaning up mock ' > Cleaning up mock > + mock_root=mocker-epel-7-x86_64.el7.cfg > + mock_root=mocker-epel-7-x86_64.el7 > + my_mock=/usr/bin/mock > + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- > system-tests/ws/ovirt-system-tests'> > + my_mock+=' --root=mocker-epel-7-x86_64.el7' > + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- > system-tests/ws/'> > + echo 'Killing all mock orphan processes, if any.' > Killing all mock orphan processes, if any. > + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he- > system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 > --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/> > --orphanskill > WARNING: Could not find required logging config file: < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/logging.ini.> Using default... > INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... > Start: init plugins > INFO: selinux enabled > Finish: init plugins > Start: run > Finish: run > ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' < > http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/ > ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> > + mock_root=epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b > + [[ -n epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b ]] > + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) > ++ mount > ++ awk '{print $3}' > ++ grep epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b > + : > + [[ -n '' ]] > + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 > + xargs -0 -tr sudo rm -rf > ++ virsh list --all --uuid > + false > POST BUILD TASK : SUCCESS > END OF POST BUILD TASK : 1 > Recording test results > ERROR: Step ‘Publish JUnit test result report’ failed: No test report > files were found. Configuration error? > Archiving artifacts >