- Infra - oVirt List Archives

[JIRA] (OVIRT-988) standard-CI artifacts are not collected when mock runner fails
by Barak Korren (oVirt JIRA) 27 Dec '16

27 Dec '16

Barak Korren created OVIRT-988: ---------------------------------- Summary: standard-CI artifacts are not collected when mock runner fails Key: OVIRT-988 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-988 Project: oVirt - virtualization made easy Issue Type: Bug Reporter: Barak Korren Assignee: infra Currently, when a standard-CI script fails (which makes mock_runner.sh fail), the exported-artifacts are not taken from the project directory but only from $WORKSPACE. It desirable to take the project's atficats too, even if the CI script failed, to allow debugging. It seems that pipeline-based jobs (Lago's CI, experimental, OST) do not exhibit this behaviour. -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

Build failed in Jenkins: ovirt_4.1_system-tests #9
by jenkins＠jenkins.phx.ovirt.org 27 Dec '16

27 Dec '16

See <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/changes> Changes: [Yaniv Kaul] Fixes and changes to storage tests [Sandro Bonazzola] ovirt-hosted-engine-ha: added 4.1 branch [Sandro Bonazzola] ovirt-release: move to 4.1 branch [Sandro Bonazzola] cockpit-ovirt: exclude fc23 from 4.1 [Gil Shinar] Fix bugs in sdk yamls ------------------------------------------ [...truncated 750 lines...] + echo '----------- Cleaning with lago' ----------- Cleaning with lago + lago --workdir <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/d…> destroy --yes --all-prefixes + echo '----------- Cleaning with lago done' ----------- Cleaning with lago done + [[ 0 != \0 ]] + echo '======== Cleanup done' ======== Cleanup done + exit 0 + exit Took 62 seconds =================================== ##! ##! ERROR ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ##!######################################################## ########################################################## Build step 'Execute shell' marked build as failure Performing Post build task... Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -xe echo 'shell_scripts/system_tests.collect_logs.sh' # # Required jjb vars: # version # VERSION=4.1 SUITE_TYPE= WORKSPACE="$PWD" OVIRT_SUITE="$SUITE_TYPE_suite_$VERSION" TESTS_LOGS="$WORKSPACE/ovirt-system-tests/exported-artifacts" rm -rf "$WORKSPACE/exported-artifacts" mkdir -p "$WORKSPACE/exported-artifacts" if [[ -d "$TESTS_LOGS" ]]; then mv "$TESTS_LOGS/"* "$WORKSPACE/exported-artifacts/" fi [ovirt_4.1_system-tests] $ /bin/bash -xe /tmp/hudson2115846140055103178.sh + echo shell_scripts/system_tests.collect_logs.sh shell_scripts/system_tests.collect_logs.sh + VERSION=4.1 + SUITE_TYPE= + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + OVIRT_SUITE=4.1 + TESTS_LOGS=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> + rm -rf <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-art…> + mkdir -p <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-art…> + [[ -d <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> ]] + mv <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/e…> <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/9/artifact/exported-art…> POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 0 Match found for :.* : True Logical operation result is TRUE Running script : #!/bin/bash -x echo "shell-scripts/mock_cleanup.sh" # Make clear this is the cleanup, helps reading the jenkins logs cat <<EOC _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### EOC shopt -s nullglob WORKSPACE="${WORKSPACE:-$PWD}" UMOUNT_RETRIES="${UMOUNT_RETRIES:-3}" UMOUNT_RETRY_DELAY="${UMOUNT_RETRY_DELAY:-1s}" safe_umount() { local mount="${1:?}" local attempt for ((attempt=0 ; attempt < $UMOUNT_RETRIES ; attempt++)); do # If this is not the 1st time through the loop, Sleep a while to let # the problem "solve itself" [[ attempt > 0 ]] && sleep "$UMOUNT_RETRY_DELAY" # Try to umount sudo umount --lazy "$mount" && return 0 # See if the mount is already not there despite failing findmnt --kernel --first "$mount" > /dev/null && return 0 done echo "ERROR: Failed to umount $mount." return 1 } # restore the permissions in the working dir, as sometimes it leaves files # owned by root and then the 'cleanup workspace' from jenkins job fails to # clean and breaks the jobs sudo chown -R "$USER" "$WORKSPACE" # Archive the logs, we want them anyway logs=( ./*log ./*/logs ) if [[ "$logs" ]]; then for log in "${logs[@]}"; do [[ "$log" = ./exported-artifacts/* ]] && continue echo "Copying ${log} to exported-artifacts" mv $log exported-artifacts/ done fi # stop any processes running inside the chroot failed=false mock_confs=("$WORKSPACE"/*/mocker*) # Clean current jobs mockroot if any for mock_conf_file in "${mock_confs[@]}"; do [[ "$mock_conf_file" ]] || continue echo "Cleaning up mock $mock_conf" mock_root="${mock_conf_file##*/}" mock_root="${mock_root%.*}" my_mock="/usr/bin/mock" my_mock+=" --configdir=${mock_conf_file%/*}" my_mock+=" --root=${mock_root}" my_mock+=" --resultdir=$WORKSPACE" #TODO: investigate why mock --clean fails to umount certain dirs sometimes, #so we can use it instead of manually doing all this. echo "Killing all mock orphan processes, if any." $my_mock \ --orphanskill \ || { echo "ERROR: Failed to kill orphans on $chroot." failed=true } mock_root="$(\ grep \ -Po "(?<=config_opts\['root'\] = ')[^']*" \ "$mock_conf_file" \ )" || : [[ "$mock_root" ]] || continue mounts=($(mount | awk '{print $3}' | grep "$mock_root")) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $chroot. Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" || failed=true done done # Clean any leftover chroot from other jobs for mock_root in /var/lib/mock/*; do this_chroot_failed=false mounts=($(cut -d\ -f2 /proc/mounts | grep "$mock_root" | sort -r)) || : if [[ "$mounts" ]]; then echo "Found mounted dirs inside the chroot $mock_root." \ "Trying to umount." fi for mount in "${mounts[@]}"; do safe_umount "$mount" && continue # If we got here, we failed $UMOUNT_RETRIES attempts so we should make # noise failed=true this_chroot_failed=true done if ! $this_chroot_failed; then sudo rm -rf "$mock_root" fi done # remove mock caches that are older then 2 days: find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 | \ xargs -0 -tr sudo rm -rf # We make no effort to leave around caches that may still be in use because # packages installed in them may go out of date, so may as well recreate them # Drop all left over libvirt domains for UUID in $(virsh list --all --uuid); do virsh destroy $UUID || : sleep 2 virsh undefine --remove-all-storage --storage vda --snapshots-metadata $UUID || : done if $failed; then echo "Cleanup script failed, propegating failure to job" exit 1 fi [ovirt_4.1_system-tests] $ /bin/bash -x /tmp/hudson5782931877352122916.sh + echo shell-scripts/mock_cleanup.sh shell-scripts/mock_cleanup.sh + cat _______________________________________________________________________ ####################################################################### # # # CLEANUP # # # ####################################################################### + shopt -s nullglob + WORKSPACE=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + UMOUNT_RETRIES=3 + UMOUNT_RETRY_DELAY=1s + sudo chown -R jenkins <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> + logs=(./*log ./*/logs) + [[ -n ./ovirt-system-tests/logs ]] + for log in '"${logs[@]}"' + [[ ./ovirt-system-tests/logs = ./exported-artifacts/* ]] + echo 'Copying ./ovirt-system-tests/logs to exported-artifacts' Copying ./ovirt-system-tests/logs to exported-artifacts + mv ./ovirt-system-tests/logs exported-artifacts/ + failed=false + mock_confs=("$WORKSPACE"/*/mocker*) + for mock_conf_file in '"${mock_confs[@]}"' + [[ -n <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/m…> ]] + echo 'Cleaning up mock ' Cleaning up mock + mock_root=mocker-epel-7-x86_64.el7.cfg + mock_root=mocker-epel-7-x86_64.el7 + my_mock=/usr/bin/mock + my_mock+=' --configdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests'> + my_mock+=' --root=mocker-epel-7-x86_64.el7' + my_mock+=' --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/'> + echo 'Killing all mock orphan processes, if any.' Killing all mock orphan processes, if any. + /usr/bin/mock --configdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests> --root=mocker-epel-7-x86_64.el7 --resultdir=<http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/> --orphanskill WARNING: Could not find required logging config file: <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/l….> Using default... INFO: mock.py version 1.2.21 starting (python version = 3.4.3)... Start: init plugins INFO: selinux enabled Finish: init plugins Start: run Finish: run ++ grep -Po '(?<=config_opts\['\''root'\''\] = '\'')[^'\'']*' <http://jenkins.ovirt.org/job/ovirt_4.1_system-tests/ws/ovirt-system-tests/m…> + mock_root=epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + [[ -n epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 ]] + mounts=($(mount | awk '{print $3}' | grep "$mock_root")) ++ mount ++ awk '{print $3}' ++ grep epel-7-x86_64-b8fbc97503d2f9478015768d3094ffe4 + : + [[ -n '' ]] + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d -mtime +2 -print0 + xargs -0 -tr sudo rm -rf ++ virsh list --all --uuid + false POST BUILD TASK : SUCCESS END OF POST BUILD TASK : 1 Recording test results ERROR: Step ?Publish JUnit test result report? failed: No test report files were found. Configuration error? Archiving artifacts

1 11

[JIRA] (OVIRT-982) Fwd: http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade-from-4.1_el7_created/431
by Gil Shinar (oVirt JIRA) 27 Dec '16

27 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-982?page=com.atlassian.jira.p… ] Gil Shinar commented on OVIRT-982: ---------------------------------- We are currently checking if moving from git protocol to http, solves this issue. > Fwd: http://jenkins.ovirt.org/job/ovirt-engine_master_upgrade-from-4.1_el7_creat… > -------------------------------------------------------------------------------------- > > Key: OVIRT-982 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-982 > Project: oVirt - virtualization made easy > Issue Type: By-EMAIL > Reporter: Yevgeny Zaspitsky > Assignee: infra > > Looks like a lot of jobs fail on similar problem. > From the job console log: > git -c core.askpass=true fetch --tags --progress > git://gerrit.ovirt.org/ovirt-engine.git refs/changes/93/67593/22 > --prune > ERROR: Error fetching remote repo > 'origin'hudson.plugins.git.GitException > <http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitExcepti…>: > Failed to fetch from git://gerrit.ovirt.org/ovirt-engine.git > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.fe…> > at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.re…> > at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.plugins.git.GitSCM.ch…> > at org.jenkinsci.plugins.multiplescms.MultiSCM.checkout(MultiSCM.java:129) > <http://stacktrace.jenkins-ci.org/search/?query=org.jenkinsci.plugins.multip…> > at hudson.scm.SCM.checkout(SCM.java:485) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.scm.SCM.checkout&enti…> > at hudson.model.AbstractProject.checkout(AbstractProject.java:1269) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractProject…> > at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$A…> > at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) > <http://stacktrace.jenkins-ci.org/search/?query=jenkins.scm.SCMCheckoutStrat…> > at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$A…> > at hudson.model.Run.execute(Run.java:1738) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Run.execute&ent…> > at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.FreeStyleBuild.…> > at hudson.model.ResourceController.execute(ResourceController.java:98) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.ResourceControl…> > at hudson.model.Executor.run(Executor.java:410) > <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Executor.run&en…> > Caused by: hudson.plugins.git.GitException > <http://stacktrace.jenkins-ci.org/search?query=hudson.plugins.git.GitExcepti…>: > Command "git -c core.askpass=true fetch --tags --progress > git://gerrit.ovirt.org/ovirt-engine.git refs/changes/93/67593/22 > --prune" returned status code 128: > stdout: > stderr: fatal: read error: Connection reset by peer > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1640) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1388) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:62) > at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$1.execute(CliGitAPIImpl.java:313) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:152) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$1.call(RemoteGitImpl.java:145) > at hudson.remoting.UserRequest.perform(UserRequest.java:152) > at hudson.remoting.UserRequest.perform(UserRequest.java:50) > at hudson.remoting.Request$2.run(Request.java:332) > at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > at ......remote call to vm0040.workers-phx.ovirt.org(Native Method) > at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416) > at hudson.remoting.UserResponse.retrieve(UserRequest.java:252) > at hudson.remoting.Channel.call(Channel.java:781) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:145) > at sun.reflect.GeneratedMethodAccessor452.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:131) > at com.sun.proxy.$Proxy60.execute(Unknown Source) > at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:764) > ... 12 more > ERROR: null -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-987) Infra issues in Jenkins aren't reported properly on most of Jenkins jobs
by Yevgeny Zaspitsky (oVirt JIRA) 27 Dec '16

27 Dec '16

Yevgeny Zaspitsky created OVIRT-987: --------------------------------------- Summary: Infra issues in Jenkins aren't reported properly on most of Jenkins jobs Key: OVIRT-987 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-987 Project: oVirt - virtualization made easy Issue Type: By-EMAIL Reporter: Yevgeny Zaspitsky Assignee: infra Findbugs jobs reports an infra problem to gerrit properly: "There was an infra issue, please contact infra(a)ovirt.org" It'd be good to have same behavior on the rest of the Jenkins jobs. Currently there is no differentiation between failure upon an infra issue and a real patch problem in reporting to gerrit. Regards, Yevgeny -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-449) Add junit report to ovirt-engine standard ci
by eyal edri [Administrator] (oVirt JIRA) 27 Dec '16

27 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-449?page=com.atlassian.jira.p… ] eyal edri [Administrator] reassigned OVIRT-449: ----------------------------------------------- Assignee: eyal edri [Administrator] (was: infra) > Add junit report to ovirt-engine standard ci > -------------------------------------------- > > Key: OVIRT-449 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-449 > Project: oVirt - virtualization made easy > Issue Type: Improvement > Reporter: David Caro Estevez > Assignee: eyal edri [Administrator] > Priority: Low > > Currently there are to ways of doing it: > * Using a special job template for ovirt-engine (easy to do now, hard to maintain/reuse) > * Using jenkins pipeline plugin to dynamically add it to any job that generates a report (hard to do right now, easy to maintain/reuse) -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-986) remove gerrit hooks from infra repos
by eyal edri [Administrator] (oVirt JIRA) 27 Dec '16

27 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-986?page=com.atlassian.jira.p… ] eyal edri [Administrator] updated OVIRT-986: -------------------------------------------- Epic Link: OVIRT-411 > remove gerrit hooks from infra repos > ------------------------------------ > > Key: OVIRT-986 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-986 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: eyal edri [Administrator] > Assignee: infra > > We don't need to run oVirt gerrit hooks on infra repos like gerrit-admin or jenkins since we don't use bugzilla for tracking issues. > We might want to add support for JIRA ticket added to a commit msg for handling infra repos, but that should require another ticket, for now we just need to remove the hooks from projects which are not relevant like jenkins,infra-docs,infra-puppet,gerrit-admin and others -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-986) remove gerrit hooks from infra repos
by eyal edri [Administrator] (oVirt JIRA) 27 Dec '16

27 Dec '16

eyal edri [Administrator] created OVIRT-986: ----------------------------------------------- Summary: remove gerrit hooks from infra repos Key: OVIRT-986 URL: https://ovirt-jira.atlassian.net/browse/OVIRT-986 Project: oVirt - virtualization made easy Issue Type: Task Reporter: eyal edri [Administrator] Assignee: infra We don't need to run oVirt gerrit hooks on infra repos like gerrit-admin or jenkins since we don't use bugzilla for tracking issues. We might want to add support for JIRA ticket added to a commit msg for handling infra repos, but that should require another ticket, for now we just need to remove the hooks from projects which are not relevant like jenkins,infra-docs,infra-puppet,gerrit-admin and others -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[oVirt Jenkins] test-repo_ovirt_experimental_master - Build #4371 - FAILURE!
by jenkins＠jenkins.phx.ovirt.org 27 Dec '16

27 Dec '16

------=_Part_130_741767172.1482410173640 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Build: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4371/, Build Number: 4371, Build Status: FAILURE ------=_Part_130_741767172.1482410173640--

1 77

[JIRA] (OVIRT-985) Fix the way repoman deploy to experimental
by eyal edri [Administrator] (oVirt JIRA) 27 Dec '16

27 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-985?page=com.atlassian.jira.p… ] eyal edri [Administrator] commented on OVIRT-985: ------------------------------------------------- [~gshinar] please add the relevant failing job ( mark it to keep forever ) and paste the exception here. > Fix the way repoman deploy to experimental > ------------------------------------------ > > Key: OVIRT-985 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-985 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Gil Shinar > Assignee: infra > > Please consult with Eyal on that. > No idea what should be done. -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0

[JIRA] (OVIRT-984) Add emails to deploy to experimental jobs
by eyal edri [Administrator] (oVirt JIRA) 27 Dec '16

27 Dec '16

[ https://ovirt-jira.atlassian.net/browse/OVIRT-984?page=com.atlassian.jira.p… ] eyal edri [Administrator] reassigned OVIRT-984: ----------------------------------------------- Assignee: Pavel Zhukov (was: infra) > Add emails to deploy to experimental jobs > ----------------------------------------- > > Key: OVIRT-984 > URL: https://ovirt-jira.atlassian.net/browse/OVIRT-984 > Project: oVirt - virtualization made easy > Issue Type: Task > Reporter: Gil Shinar > Assignee: Pavel Zhukov > > When deploy to experimental fails, we need to alert. -- This message was sent by Atlassian JIRA (v1000.621.5#100023)

1 0