September 2016 - Infra - oVirt List Archives

Build failed in Jenkins: ovirt_3.6_system-tests #453
by jenkins＠jenkins.phx.ovirt.org 04 Sep '16

04 Sep '16

See <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/changes> Changes: [Eyal Edri] Fail early on he setup first or second host install ------------------------------------------ [...truncated 703 lines...] ## took 2369 seconds ## rc = 1 ########################################################## ##! ERROR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv ##! Last 20 log enties: logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log ##! + true + env_cleanup + echo '#########################' ######################### + local res=0 + local uuid + echo '======== Cleaning up' ======== Cleaning up + [[ -e <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/d…> ]] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/d…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 Took 2203 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=3.6 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson3737170357820277489.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=3.6 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> + OVIRT_SUITE=3.6 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported-a…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported-a…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/453/artifact/exported-a…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo "shell-scripts/mock_cleanup.sh" shopt -s nullglob WORKSPACE="$PWD" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then tar cvzf exported-artifacts/logs.tgz "${logs[@]}" rm -rf "${logs[@]}" fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true } done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do sudo umount --lazy "$mount" \ || { echo "ERROR: Failed to umount $mount." failed=true this_chroot_failed=true } done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done if $failed; then echo "Aborting." exit 1 fi # remove mock system cache, we will setup proxies to do the caching and this # takes lots of space between runs shopt -u nullglob sudo rm -Rf /var/cache/mock/* # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" [ovirt_3.6_system-tests] $ /bin/bash -xe /tmp/hudson677792111944984977.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + tar cvzf exported-artifacts/logs.tgz ./ovirt-system-tests/logs ./ovirt-system-tests/logs/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.init/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/stdout_stderr.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/build.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/root.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.install_packages/state.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.basic_suite_3.6.sh/basic_suite_3.6.sh.log ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/ ./ovirt-system-tests/logs/mocker-fedora-23-x86_64.fc23.clean_rpmdb/stdout_stderr.log + rm -rf ./ovirt-system-tests/logs + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/m…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-fedora-23-x86_64.fc23.cfg + mock_root=mocker-fedora-23-x86_64.fc23 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-fedora-23-x86_64.fc23' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests> --root=mocker-fedora-23-x86_64.fc23 --resultdir=<http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/l….> Using default... INFO: mock.py version 1.2.18 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/ovirt-system-tests/m…> + mock_root=fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c + [[ -n fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c + : + [[ -n '' ]] + false + shopt -u nullglob + sudo rm -Rf /var/cache/mock/fedora-23-x86_64-235bec7d0621e95d1cae73d7cf9dc27c + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_3.6_system-tests/ws/> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results Archiving artifacts

1 2

[oVirt Jenkins] repos_4.0_check-closure_merged - Build # 26 - Still Failing!
by jenkins＠jenkins.phx.ovirt.org 04 Sep '16

04 Sep '16

------=_Part_185_1755631428.1473003986840 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/repos_4.0_check-closure_merged/ Build: http://jenkins.ovirt.org/job/repos_4.0_check-closure_merged/26/ Build Number: 26 Build Status: Still Failing Triggered By: Started by user Sharon Naftaly ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #18 [Sharon Naftaly] Adding ovirt-hosted-engine-ha std-ci jobs Changes for Build #19 [Eyal Edri] align all job types to keep longer history of 120 builds [Eyal Edri] splitting mom created and merged projects in yaml Changes for Build #20 No changes Changes for Build #21 No changes Changes for Build #22 [Sharon Naftaly] Changing repos_check_closure jobs email recipients [Eyal Edri] restore check-patch for master on el7 Changes for Build #23 No changes Changes for Build #24 No changes Changes for Build #25 No changes Changes for Build #26 [Sharon Naftaly] Add recipients list to failure email in check closure jobs [Irit Goihman] vdsm: move check merged job on master to el7 and remove fc22 jobs [Eyal Edri] missed increasing ovirt-engine build-artifacts history ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_185_1755631428.1473003986840--

1 0

[JIRA] (OVIRT-719) check-patch jobs should check if the patch is still open
by eyal edri [Administrator] (oVirt JIRA) 04 Sep '16

04 Sep '16

eyal edri [Administrator] created OVIRT-719: ----------------------------------------------- Summary: check-patch jobs should check if the patch is still open Key: OVIRT-719 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-719 Project: oVirt - virtualization made easy Issue Type: Improvement Reporter: eyal edri [Administrator] Assignee: infra We sometimes see multiple check-patch jobs still running while actually the patch for them was already merged. This happens when Jenkins queue is too big and the committer merge the patch before the job in CI finish or even starts. In the case where the job didn't even start because of long queue, there is not reason to continue running the job if the patch was merged. Any post merge logic will be run in check-merged stage and there is no value in adding info on the check-patch to the merged commit. It should be easy to check in the beginning of the check-patch job and part of standard CI if the patch that is triggered the job is still open and the job should continue. If the patch is merged, then we should abort the job and thus cleaning the queue from check-patch jobs that won't have any value . -- This message was sent by Atlassian JIRA (v1000.305.0#100011)

1 0

[JIRA] (OVIRT-719) check-patch jobs should check if the patch is still open
by eyal edri [Administrator] (oVirt JIRA) 04 Sep '16

04 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-719?page=com.atlassian.jira.p… ] eyal edri [Administrator] updated OVIRT-719: -------------------------------------------- Epic Link: OVIRT-400 > check-patch jobs should check if the patch is still open > --------------------------------------------------------- > > Key: OVIRT-719 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-719 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: eyal edri [Administrator] > Assignee: infra > > We sometimes see multiple check-patch jobs still running while actually the patch for them was already merged. > This happens when Jenkins queue is too big and the committer merge the patch before the job in CI finish or even starts. > In the case where the job didn't even start because of long queue, there is not reason to continue running the job if the patch was merged. > Any post merge logic will be run in check-merged stage and there is no value in adding info on the check-patch to the merged commit. > It should be easy to check in the beginning of the check-patch job and part of standard CI if the patch that is triggered the job is still open and the job should continue. > If the patch is merged, then we should abort the job and thus cleaning the queue from check-patch jobs that won't have any value . -- This message was sent by Atlassian JIRA (v1000.305.0#100011)

1 0

Ansible best practices
by Eyal Edri 04 Sep '16

04 Sep '16

Since we started writing some Ansible in the team lately, might be relevant: https://www.ansible.com/blog/ansible-best-practices-essentials -- Eyal Edri Associate Manager RHV DevOps EMEA ENG Virtualization R&D Red Hat Israel phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

2 1

[oVirt Jenkins] repos_master_check-closure_merged - Build # 44 - Still Failing!
by jenkins＠jenkins.phx.ovirt.org 04 Sep '16

04 Sep '16

------=_Part_171_193014536.1472981287813 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Project: http://jenkins.ovirt.org/job/repos_master_check-closure_merged/ Build: http://jenkins.ovirt.org/job/repos_master_check-closure_merged/44/ Build Number: 44 Build Status: Still Failing Triggered By: Started by user Sharon Naftaly ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #40 [Sharon Naftaly] Adding ovirt-hosted-engine-ha std-ci jobs Changes for Build #41 [Eyal Edri] align all job types to keep longer history of 120 builds [Eyal Edri] splitting mom created and merged projects in yaml Changes for Build #42 No changes Changes for Build #43 No changes Changes for Build #44 No changes ----------------- Failed Tests: ----------------- No tests ran. ------=_Part_171_193014536.1472981287813--

1 0

Where is vdsm_master_check-patch-el7 job?
by Nir Soffer 04 Sep '16

04 Sep '16

After rebasing this patch (https://gerrit.ovirt.org/61200) only the fc24 job runs - previously we had both el7 and fc24 jobs. Testing only on fc24 means we will not catch packaging issues like missing package. Nir

3 4

[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by snaftaly (oVirt JIRA) 04 Sep '16

04 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira.p… ] snaftaly commented on OVIRT-717: -------------------------------- [~eedri], yes, currently the package is installed at the beginning of the job. > livecd-tools RPM from ci-tools repo is missing dependencies > ----------------------------------------------------------- > > Key: OVIRT-717 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: General > Reporter: Evgheni Dereveanchin > Assignee: infra > > It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise. -- This message was sent by Atlassian JIRA (v1000.305.0#100011)

1 0

[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by eyal edri [Administrator] (oVirt JIRA) 04 Sep '16

04 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-717: ------------------------------------------------- [~snaftaly(a)redhat.com] did you end up installing locally the RPM in the job? do we still need to refresh the rpm on ci-tools repo? > livecd-tools RPM from ci-tools repo is missing dependencies > ----------------------------------------------------------- > > Key: OVIRT-717 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: General > Reporter: Evgheni Dereveanchin > Assignee: infra > > It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise. -- This message was sent by Atlassian JIRA (v1000.305.0#100011)

1 0

[JIRA] (OVIRT-717) livecd-tools RPM from ci-tools repo is missing dependencies
by eyal edri [Administrator] (oVirt JIRA) 04 Sep '16

04 Sep '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-717?page=com.atlassian.jira.p… ] eyal edri [Administrator] updated OVIRT-717: -------------------------------------------- Epic Link: OVIRT-403 > livecd-tools RPM from ci-tools repo is missing dependencies > ----------------------------------------------------------- > > Key: OVIRT-717 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-717 > Project: oVirt - virtualization made easy > Issue Type: Bug > Components: General > Reporter: Evgheni Dereveanchin > Assignee: infra > > It was reported in OVIRT-573 that sssd-client is not installed on EL7 slaves, yet it is needed for livecd-creator. After investigating, the version of livecd-tools we have is installed from our ci-tools repo. It has a newer version so it overrides the version from CentOS, yet it's missing the sssd-client dependency. If we really need the newer version, we should rebuild the RPM to ensure a dependency for sssd-client is in place since livecd-creator may fail otherwise. -- This message was sent by Atlassian JIRA (v1000.305.0#100011)

1 0