On Tue, Jan 10, 2017 at 3:14 PM, Evgheni Dereveanchin <ederevea@redhat.com> wrote:
Not sure what the initial problem was, but on my laptop (Haswell-MB)
I always use the lowest possible CPU family to ensure it's using
as few features as possible in nested VMs:

;-)

I'm doing the exact opposite, for two reasons:
1. I want the best possible performance. Specifically, I'd like the tests to run as fast as possible.
2. I'd like to expose as many of the latest features up to the hosts (and VMs).
 
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>core2duo</model>
    <feature policy='require' name='vmx'/>
  </cpu>

Respectively, I use model_Conroe on oVirt side and didn't have
problems with it. Do we really need to use newer CPU families
in our tests?

We probably don't - we used to have Conroe hard-coded in the tests (until I changed it to use something different).

It does mean it'll be a bit challenging to run on AMD if we decide to go back to hard-code Conroe.
Y.
 

Regards,
Evgheni Dereveanchin

----- Original Message -----
From: "Milan Zamazal" <mzamazal@redhat.com>
To: "Yaniv Kaul" <ykaul@redhat.com>
Cc: "Lev Veyde" <lveyde@redhat.com>, "Eyal Edri" <eedri@redhat.com>, "Sandro Bonazzola" <sbonazzo@redhat.com>, "infra" <infra@ovirt.org>, "Gal Ben Haim" <gbenhaim@redhat.com>, "Martin Polednik" <mpoledni@redhat.com>, "Evgheni Dereveanchin" <ederevea@redhat.com>
Sent: Tuesday, 10 January, 2017 1:16:09 PM
Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627

Yaniv Kaul <ykaul@redhat.com> writes:

> On Tue, Jan 10, 2017 at 12:45 PM, Milan Zamazal <mzamazal@redhat.com> wrote:
>
>> Yaniv Kaul <ykaul@redhat.com> writes:
>>
>> > On Tue, Jan 10, 2017 at 12:08 PM, Lev Veyde <lveyde@redhat.com> wrote:
>> >
>> >> This patch is one that caused it probably:
>> >> https://github.com/lago-project/lago/commit/
>> 05ccf7240976f91b0c14d6a1f88016
>> >> 376d5e87f0
>> >
>> >
>> > +Milan.
>>
>> +Martin
>>
>> > I must confess that I did not like the patch to begin with...
>> > I did not understand what real problem it solved, but Michal assured me
>> > there was a real issue.
>>
>> Yes, there was a real issue with nested virtualization.  Some CPU flags
>> are missing with Haswell and Lago doesn't run properly.
>>
>
> Is this a libvirt bug btw?

I'm not sure.  When the sets of CPU flags on the host and in the VM with
a copied host CPU are different, it's not clear what's the right thing
to do.

> Perhaps we need a switch to turn this feature on and off?

I think it would be useful to have a possibility to specify a particular
CPU type in the Lago configuration.

>> > I know have Engine with a Java@ 100% CPU - I hope it's unrelated to
>> this as
>> > well.
>> >
>> > I suggest we do survey to see who doesn't have SandyBridge and above and
>> > perhaps move higher than Westmere.
>>
>> We've got Westmere servers in the Brno lab.
>>
>
> Do we know the scope of the problem? Does it happen only on Westmere, for
> example?

The problem was with Haswell-noTSX (on my Lenovo, but I think Martin has
observed the same problem too).  We don't know the scope of the problem,
but if we want to be able to run Lago on Brno servers then we must be
Westmere compatible.

>  Y.
>
>
>> > What do we have in CI?
>> > Y.
>> >
>> >
>> >>
>> >> Thanks in advance,
>> >> Lev Veyde.
>> >>
>> >> ----- Original Message -----
>> >> From: "Lev Veyde" <lveyde@redhat.com>
>> >> To: "Eyal Edri" <eedri@redhat.com>, sbonazzo@redhat.com
>> >> Cc: infra@ovirt.org, "Gal Ben Haim" <gbenhaim@redhat.com>
>> >> Sent: Tuesday, January 10, 2017 11:50:05 AM
>> >> Subject: Re: Build failed in Jenkins: ovirt_4.0_he-system-tests #627
>> >>
>> >> Hi,
>> >>
>> >> Checked the logs and see the following:
>> >>
>> >> 02:42:05 [WARNING] OVF does not contain a valid image description, using
>> >> default.
>> >> 02:42:05           The following CPU types are supported by this host:
>> >> 02:42:05                 - model_Westmere: Intel Westmere Family
>> >> 02:42:05                 - model_Nehalem: Intel Nehalem Family
>> >> 02:42:05                 - model_Penryn: Intel Penryn Family
>> >> 02:42:05                 - model_Conroe: Intel Conroe Family
>> >> 02:42:05 [ ERROR ] Failed to execute stage 'Environment customization':
>> >> Invalid CPU type specified: model_SandyBridge
>> >>
>> >> Barak thinks that it may be related to the recent update in the Lago
>> code.
>> >>
>> >> Gal, any idea ?
>> >>
>> >> Thanks in advance,
>> >> Lev Veyde.
>> >>
>> >> ----- Original Message -----
>> >> From: jenkins@jenkins.phx.ovirt.org
>> >> To: sbonazzo@redhat.com, infra@ovirt.org, lveyde@redhat.com
>> >> Sent: Tuesday, January 10, 2017 4:42:14 AM
>> >> Subject: Build failed in Jenkins: ovirt_4.0_he-system-tests #627
>> >>
>> >> See <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/changes
>> >
>> >>
>> >> Changes:
>> >>
>> >> [Lev Veyde] Mask NetworkManager service
>> >>
>> >> [Eyal Edri] fix imgbased job names in jjb
>> >>
>> >> [Daniel Belenky] fixing jjb version for cockpit-ovirt
>> >>
>> >> [Gil Shinar] Add some more 4.1 to experimental
>> >>
>> >> [Juan Hernandez] Don't build RPMs for the JBoss modules Maven plugin
>> >>
>> >> [pkliczewski] jsonrpc 4.1 branch
>> >>
>> >> ------------------------------------------
>> >> [...truncated 749 lines...]
>> >> Finish: shell
>> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> >> @@ Tue Jan 10 02:42:07 UTC 2017 automation/he_basic_suite_4.0.sh chroot
>> >> finished
>> >> @@      took 360 seconds
>> >> @@      rc = 1
>> >> @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
>> >> ========== Scrubbing chroot
>> >>     mock \
>> >>         --configdir="<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> >> system-tests/ws/ovirt-system-tests"> \
>> >>         --root="mocker-epel-7-x86_64.el7" \
>> >>         --resultdir="./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.
>> el7.scrub"
>> >> \
>> >>         --scrub=chroot
>> >> WARNING: Could not find required logging config file: <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/logging.ini.> Using default...
>> >> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
>> >> Start: init plugins
>> >> INFO: selinux enabled
>> >> Finish: init plugins
>> >> Start: run
>> >> Start: scrub ['chroot']
>> >> INFO: scrubbing chroot for mocker-epel-7-x86_64.el7
>> >> Finish: scrub ['chroot']
>> >> Finish: run
>> >> Scrub chroot took 6 seconds
>> >> ============================
>> >> ##########################################################
>> >> ## Tue Jan 10 02:42:13 UTC 2017 Finished env: el7:epel-7-x86_64
>> >> ##      took 366 seconds
>> >> ##      rc = 1
>> >> ##########################################################
>> >> find: ‘logs’: No such file or directory
>> >> No log files found, check command output
>> >> ##!########################################################
>> >> Collecting mock logs
>> >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.clean_rpmdb’ ->
>> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.clean_rpmdb’
>> >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.he_basic_suite_4.0.sh’
>> ->
>> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.he_
>> >> basic_suite_4.0.sh’
>> >> ‘./mock_logs.xGGwEk6V/mocker-epel-7-x86_64.el7.init’ ->
>> >> ‘exported-artifacts/mock_logs/mocker-epel-7-x86_64.el7.init’
>> >> ##########################################################
>> >> Build step 'Execute shell' marked build as failure
>> >> Performing Post build task...
>> >> Match found for :.* : True
>> >> Logical operation result is TRUE
>> >> Running script  : #!/bin/bash -xe
>> >> echo 'shell_scripts/system_tests.collect_logs.sh'
>> >>
>> >> #
>> >> # Required jjb vars:
>> >> #    version
>> >> #
>> >> VERSION=4.0
>> >> SUITE_TYPE=
>> >>
>> >> WORKSPACE="$PWD"
>> >> OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION"
>> >> TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts"
>> >>
>> >> rm -rf "$WORKSPACE/exported-artifacts"
>> >> mkdir -p "$WORKSPACE/exported-artifacts"
>> >>
>> >> if [[ -d "$TESTS_LOGS" ]]; then
>> >>     mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/"
>> >> fi
>> >>
>> >> [ovirt_4.0_he-system-tests] $ /bin/bash -xe
>> /tmp/hudson302101162661598371.
>> >> sh
>> >> + echo shell_scripts/system_tests.collect_logs.sh
>> >> shell_scripts/system_tests.collect_logs.sh
>> >> + VERSION=4.0
>> >> + SUITE_TYPE=
>> >> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >
>> >> + OVIRT_SUITE=4.0
>> >> + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> >> system-tests/ws/ovirt-system-tests/exported-artifacts>
>> >> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> >> artifact/exported-artifacts>
>> >> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> >> artifact/exported-artifacts>
>> >> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/exported-artifacts> ]]
>> >> + mv <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/exported-artifacts/failure_msg.txt> <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/exported-artifacts/lago_logs> <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/exported-artifacts/mock_logs> <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/627/
>> >> artifact/exported-artifacts/>
>> >> POST BUILD TASK : SUCCESS
>> >> END OF POST BUILD TASK : 0
>> >> Match found for :.* : True
>> >> Logical operation result is TRUE
>> >> Running script  : #!/bin/bash -x
>> >> echo "shell-scripts/mock_cleanup.sh"
>> >> # Make clear this is the cleanup, helps reading the jenkins logs
>> >> cat <<EOC
>> >> _______________________________________________________________________
>> >> #######################################################################
>> >> #                                                                     #
>> >> #                               CLEANUP                               #
>> >> #                                                                     #
>> >> #######################################################################
>> >> EOC
>> >>
>> >> shopt -s nullglob
>> >>
>> >> WORKSPACE="${WORKSPACE:-$PWD}"
>> >> UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}"
>> >> UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}"
>> >>
>> >> safe_umount() {
>> >>     local mount="${1:?}"
>> >>     local attempt
>> >>     for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do
>> >>         # If this is not the 1st time through the loop, Sleep a while to
>> >> let
>> >>         # the problem "solve itself"
>> >>         [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY"
>> >>         # Try to umount
>> >>         sudo umount --lazy "$mount" && return 0
>> >>         # See if the mount is already not there despite failing
>> >>         findmnt --kernel --first "$mount" > /dev/null && return 0
>> >>     done
>> >>     echo "ERROR:  Failed to umount $mount."
>> >>     return 1
>> >> }
>> >>
>> >> # restore the permissions in the working dir, as sometimes it leaves
>> files
>> >> # owned by root and then the 'cleanup workspace' from jenkins job fails
>> to
>> >> # clean and breaks the jobs
>> >> sudo chown -R "$USER" "$WORKSPACE"
>> >>
>> >> # stop any processes running inside the chroot
>> >> failed=false
>> >> mock_confs=("$WORKSPACE"/*/mocker*)
>> >> # Clean current jobs mockroot if any
>> >> for mock_conf_file in "${mock_confs[@]}"; do
>> >>     [[ "$mock_conf_file" ]] || continue
>> >>     echo "Cleaning up mock $mock_conf"
>> >>     mock_root="${mock_conf_file##*/}"
>> >>     mock_root="${mock_root%.*}"
>> >>     my_mock="/usr/bin/mock"
>> >>     my_mock+=" --configdir=${mock_conf_file%/*}"
>> >>     my_mock+=" --root=${mock_root}"
>> >>     my_mock+=" --resultdir=$WORKSPACE"
>> >>
>> >>     #TODO: investigate why mock --clean fails to umount certain dirs
>> >> sometimes,
>> >>     #so we can use it instead of manually doing all this.
>> >>     echo "Killing all mock orphan processes, if any."
>> >>     $my_mock \
>> >>         --orphanskill \
>> >>     || {
>> >>         echo "ERROR:  Failed to kill orphans on $chroot."
>> >>         failed=true
>> >>     }
>> >>
>> >>     mock_root="$(\
>> >>         grep \
>> >>             -Po "(?<=config_opts\['root'\] = ')[^']*" \
>> >>             "$mock_conf_file" \
>> >>     )" || :
>> >>     [[ "$mock_root" ]] || continue
>> >>     mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || :
>> >>     if [[ "$mounts" ]]; then
>> >>         echo "Found mounted dirs inside the chroot $chroot. Trying to
>> >> umount."
>> >>     fi
>> >>     for mount in "${mounts[@]}"; do
>> >>         safe_umount "$mount" || failed=true
>> >>     done
>> >> done
>> >>
>> >> # Clean any leftover chroot from other jobs
>> >> for mock_root in /var/lib/mock/*; do
>> >>     this_chroot_failed=false
>> >>     mounts=($(cut -d\  -f2 /proc/mounts | grep "$mock_root" | sort -r))
>> ||
>> >> :
>> >>     if [[ "$mounts" ]]; then
>> >>         echo "Found mounted dirs inside the chroot $mock_root." \
>> >>              "Trying to umount."
>> >>     fi
>> >>     for mount in "${mounts[@]}"; do
>> >>         safe_umount "$mount" && continue
>> >>         # If we got here, we failed $UMOUNT_RETRIES attempts so we
>> should
>> >> make
>> >>         # noise
>> >>         failed=true
>> >>         this_chroot_failed=true
>> >>     done
>> >>     if ! $this_chroot_failed; then
>> >>         sudo rm -rf "$mock_root"
>> >>     fi
>> >> done
>> >>
>> >> # remove mock caches that are older then 2 days:
>> >> find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0
>> | \
>> >>     xargs -0 -tr sudo rm -rf
>> >> # We make no effort to leave around caches that may still be in use
>> because
>> >> # packages installed in them may go out of date, so may as well recreate
>> >> them
>> >>
>> >> # Drop all left over libvirt domains
>> >> for UUID in $(virsh list --all --uuid); do
>> >>   virsh destroy $UUID || :
>> >>   sleep 2
>> >>   virsh undefine --remove-all-storage --storage vda --snapshots-metadata
>> >> $UUID || :
>> >> done
>> >>
>> >> if $failed; then
>> >>     echo "Cleanup script failed, propegating failure to job"
>> >>     exit 1
>> >> fi
>> >>
>> >> [ovirt_4.0_he-system-tests] $ /bin/bash -x /tmp/
>> >> hudson1888216492513466503.sh
>> >> + echo shell-scripts/mock_cleanup.sh
>> >> shell-scripts/mock_cleanup.sh
>> >> + cat
>> >> _______________________________________________________________________
>> >> #######################################################################
>> >> #                                                                     #
>> >> #                               CLEANUP                               #
>> >> #                                                                     #
>> >> #######################################################################
>> >> + shopt -s nullglob
>> >> + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >
>> >> + UMOUNT_RETRIES=3
>> >> + UMOUNT_RETRY_DELAY=1s
>> >> + sudo chown -R jenkins <http://jenkins.ovirt.org/job/
>> >> ovirt_4.0_he-system-tests/ws/>
>> >> + failed=false
>> >> + mock_confs=("$WORKSPACE"/*/mocker*)
>> >> + for mock_conf_file in '"${mock_confs[@]}"'
>> >> + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg> ]]
>> >> + echo 'Cleaning up mock '
>> >> Cleaning up mock
>> >> + mock_root=mocker-epel-7-x86_64.el7.cfg
>> >> + mock_root=mocker-epel-7-x86_64.el7
>> >> + my_mock=/usr/bin/mock
>> >> + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> >> system-tests/ws/ovirt-system-tests'>
>> >> + my_mock+=' --root=mocker-epel-7-x86_64.el7'
>> >> + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> >> system-tests/ws/'>
>> >> + echo 'Killing all mock orphan processes, if any.'
>> >> Killing all mock orphan processes, if any.
>> >> + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-
>> >> system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7
>> >> --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >
>> >> --orphanskill
>> >> WARNING: Could not find required logging config file: <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/logging.ini.> Using default...
>> >> INFO: mock.py version 1.2.21 starting (python version = 3.4.3)...
>> >> Start: init plugins
>> >> INFO: selinux enabled
>> >> Finish: init plugins
>> >> Start: run
>> >> Finish: run
>> >> ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <
>> >> http://jenkins.ovirt.org/job/ovirt_4.0_he-system-tests/ws/
>> >> ovirt-system-tests/mocker-epel-7-x86_64.el7.cfg>
>> >> + mock_root=epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b
>> >> + [[ -n epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b ]]
>> >> + mounts=($(mount | awk '{print $3}' | grep "$mock_root"))
>> >> ++ mount
>> >> ++ awk '{print $3}'
>> >> ++ grep epel-7-x86_64-6f628e6dc1a827c86d5e1bd9d3b3d38b
>> >> + :
>> >> + [[ -n '' ]]
>> >> + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2
>> -print0
>> >> + xargs -0 -tr sudo rm -rf
>> >> ++ virsh list --all --uuid
>> >> + false
>> >> POST BUILD TASK : SUCCESS
>> >> END OF POST BUILD TASK : 1
>> >> Recording test results
>> >> ERROR: Step ‘Publish JUnit test result report’ failed: No test report
>> >> files were found. Configuration error?
>> >> Archiving artifacts
>> >>
>>