[OVIRT CI] Tests succeeded, build failed

[Nir Soffer]

Hi all,

Looking in another unrelated vdsm test failure, I found this:

18:36:52 ##########################################################
18:36:52 ## Tue Dec 13 18:36:52 UTC 2016 Finished env: fc24:fedora-24-x86_64
18:36:52 ##      took 417 seconds
18:36:52 ##      rc = 0
18:36:52 ##########################################################
18:36:52 ## FINISHED SUCCESSFULLY
18:36:52 ##########################################################
18:36:52 ##########################################################

The test succeeded, but during cleanup:

18:36:55 + sudo umount --lazy
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems
18:36:55 umount:
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems:
mountpoint not found
18:36:55 + echo 'ERROR:  Failed to umount
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems.'
18:36:55 ERROR:  Failed to umount
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems.
18:36:55 + failed=true
18:36:55 + this_chroot_failed=true
18:36:55 + true
18:36:55 + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d
-mtime +2 -print0
18:36:55 + xargs -0 -tr sudo rm -rf
18:36:55 ++ virsh list --all --uuid
18:36:55 + true
18:36:55 + echo 'Cleanup script failed, propegating failure to job'
18:36:55 Cleanup script failed, propegating failure to job
18:36:55 + exit 1
18:36:55 POST BUILD TASK : FAILURE
18:36:55 END OF POST BUILD TASK : 0
18:36:55 ESCALATE FAILED POST BUILD TASK TO JOB STATUS
18:36:55 Build step 'Post build task' changed build result to FAILURE
18:36:55 Archiving artifacts
18:37:00 Build step 'Groovy Postbuild' marked build as failure
18:37:00 Started calculate disk usage of build
18:37:00 Finished Calculation of disk usage of build in 0 seconds
18:37:00 Started calculate disk usage of workspace
18:37:01 Finished Calculation of disk usage of workspace in 0 seconds
18:37:01 Finished: FAILURE

The build failed because cleanup script failed.

I discussed this lot of time with David Caro, trying to convince him
that there are
3 possible results for a build:

1. Tests run and passed
2. Tests run and failed
3. System could not run the tests or had another error

I know that this adds 33% more work to the CI team, having to handle 3 results
instead of two, but we really need this distinction.

Barak, do you think we can change the script so setup and cleanup failures are
not treated as build failures but build errors?

In travis such failure seem to start another build automatically,
making developers
life much nicer.

I kept this build forever so people can inspect it:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/6291/

Nir