[OVIRT CI] Tests succeeded, build failed
by Nir Soffer
Hi all,
Looking in another unrelated vdsm test failure, I found this:
18:36:52 ##########################################################
18:36:52 ## Tue Dec 13 18:36:52 UTC 2016 Finished env: fc24:fedora-24-x86_64
18:36:52 ## took 417 seconds
18:36:52 ## rc = 0
18:36:52 ##########################################################
18:36:52 ## FINISHED SUCCESSFULLY
18:36:52 ##########################################################
18:36:52 ##########################################################
The test succeeded, but during cleanup:
18:36:55 + sudo umount --lazy
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems
18:36:55 umount:
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems:
mountpoint not found
18:36:55 + echo 'ERROR: Failed to umount
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems.'
18:36:55 ERROR: Failed to umount
/var/lib/mock/epel-7-x86_64-95d9ead9d725499a15a9021ba2fe9831-54661/root/proc/filesystems.
18:36:55 + failed=true
18:36:55 + this_chroot_failed=true
18:36:55 + true
18:36:55 + find /var/cache/mock/ -mindepth 1 -maxdepth 1 -type d
-mtime +2 -print0
18:36:55 + xargs -0 -tr sudo rm -rf
18:36:55 ++ virsh list --all --uuid
18:36:55 + true
18:36:55 + echo 'Cleanup script failed, propegating failure to job'
18:36:55 Cleanup script failed, propegating failure to job
18:36:55 + exit 1
18:36:55 POST BUILD TASK : FAILURE
18:36:55 END OF POST BUILD TASK : 0
18:36:55 ESCALATE FAILED POST BUILD TASK TO JOB STATUS
18:36:55 Build step 'Post build task' changed build result to FAILURE
18:36:55 Archiving artifacts
18:37:00 Build step 'Groovy Postbuild' marked build as failure
18:37:00 Started calculate disk usage of build
18:37:00 Finished Calculation of disk usage of build in 0 seconds
18:37:00 Started calculate disk usage of workspace
18:37:01 Finished Calculation of disk usage of workspace in 0 seconds
18:37:01 Finished: FAILURE
The build failed because cleanup script failed.
I discussed this lot of time with David Caro, trying to convince him
that there are
3 possible results for a build:
1. Tests run and passed
2. Tests run and failed
3. System could not run the tests or had another error
I know that this adds 33% more work to the CI team, having to handle 3 results
instead of two, but we really need this distinction.
Barak, do you think we can change the script so setup and cleanup failures are
not treated as build failures but build errors?
In travis such failure seem to start another build automatically,
making developers
life much nicer.
I kept this build forever so people can inspect it:
http://jenkins.ovirt.org/job/vdsm_master_check-patch-fc24-x86_64/6291/
Nir
7 years, 11 months
[JIRA] (OVIRT-881) prepare for bugizlla 5 upgrade - api changes
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-881?page=com.atlassian.jira... ]
eyal edri [Administrator] reassigned OVIRT-881:
-----------------------------------------------
Assignee: Shlomo Ben David (was: infra)
> prepare for bugizlla 5 upgrade - api changes
> --------------------------------------------
>
> Key: OVIRT-881
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-881
> Project: oVirt - virtualization made easy
> Issue Type: New Feature
> Reporter: eyal edri [Administrator]
> Assignee: Shlomo Ben David
> Priority: High
>
> oVirt Bugzilla will be upgraded to Bugzilla 5.0 in the following months,
> We should make sure all our automation still works with the new API and verify mostly Gerrit Hooks still works.
> Also, the new version will include RESTAPI as oppose to XMLRPC which was used until now.
> Worth looking for existing tools / SDKs which exists that we might be able to use.
> cc [~amarchuk] [~sbendavi(a)redhat.com] [~ederevea]
--
This message was sent by Atlassian JIRA
(v1000.621.2#100023)
7 years, 11 months
new default jenkins monitoring view for unstable jobs
by Eyal Edri
FYI,
To improve our monitoring and response on critical jobs that affect the
whole infra ( i.e system jobs, publishers,etc...) I've changed the default
view to include only jobs from:
- experimental flows
- system jobs
- publishers
- system jobs ( cleaners, etc...)
This will help infra team track and fix wide infra issues much faster.
All other 'check-merged' jobs should be monitored by the relevant
maintainers, same as check-patch jobs.
On any issue found that fail due to infra, the flow should be opening a
ticket to infra-support(a)ovirt.org or on jira.ovirt.org.
This will help us as a team to be alert on important things and handle
tickets for failing jobs due to infra issues in a more structured and
orderly way.
--
Eyal Edri
Associate Manager
RHV DevOps
EMEA ENG Virtualization R&D
Red Hat Israel
phone: +972-9-7692018
irc: eedri (on #tlv #rhev-dev #rhev-integ)
7 years, 11 months
[JIRA] (OVIRT-910) mock_runner.sh seems to run slower on newer slaves
by eyal edri [Administrator] (oVirt JIRA)
[ https://ovirt-jira.atlassian.net/browse/OVIRT-910?page=com.atlassian.jira... ]
eyal edri [Administrator] reassigned OVIRT-910:
-----------------------------------------------
Assignee: Evgheni Dereveanchin (was: infra)
I think its duplicate of the 10-50 min task, feel free to close duplicate if its the case.
> mock_runner.sh seems to run slower on newer slaves
> --------------------------------------------------
>
> Key: OVIRT-910
> URL: https://ovirt-jira.atlassian.net/browse/OVIRT-910
> Project: oVirt - virtualization made easy
> Issue Type: Improvement
> Components: Jenkins
> Reporter: Barak Korren
> Assignee: Evgheni Dereveanchin
>
> After implementing speedup improvements for mock_runner.sh (Specifically enabling it to deploy the chroot by extracting a tarball as opposed to installing it with yum every time), I started monitoring run times for vdsm check_patch jobs.
> I've noticed that jobs that are running on the old slaves (fc24-vm*) seem to initialize mock faster then ones running on new slaves (vm*) - 17seconds vs 57 seconds respectively.
> I suspect this may be because the older slaves have a different FS mounted on /var/cache/mock
--
This message was sent by Atlassian JIRA
(v1000.621.2#100023)
7 years, 11 months