[ovirt-devel] [RFC] Release process

Mon Sep 8 09:34:19 UTC 2014

Il 20/08/2014 10:05, Sandro Bonazzola ha scritto:
> Hi,
> while we're near to release 3.5.0 I would like to start the discussion on the release process for 3.5.z and 3.6.0.
> The reference page for the release process has been last updated 2 years ago and is not reflecting the way we're managing the releases anymore.

Following up to this email with proposal for missing bits. Comments are welcome.

> In order to start the discussion here's what we are currently doing:
> 
> = oVirt maintenance releases =
> 
> * After a new 3.y.0 release, maintenance releases are scheduled every month until a new 3.y+1.0 release is published.
> * After first 3.y.0 release candidate is published a new release management wiki page is created (see http://www.ovirt.org/OVirt_3.4.z_release-management)
> * For each 3.y.z release, the release management page is updated
> ** Defining the schedule for RC and GA date
> ** Pointing to the new release note page to be created (see http://www.ovirt.org/OVirt_3.4.4_Release_Notes)
> ** Pointing to a testing page to be created (see http://www.ovirt.org/Testing/oVirt_3.4.4_Testing)
> ** Pointing to a blocker bug to be created (see http://bugzilla.redhat.com/1118689)
> * Weekly status email are sent to users and devel lists
> * QA page is updated with pointers to the new release (see http://www.ovirt.org/OVirt_Quality_Assurance)
> * Release manager update the release notes pages starting from RC
> 
> Usually RC date is scheduled one week before GA.
> RC releases are created by taking the latest nightly snapshot available after verifying that the build passes basic sanity test and repository closure.
> A new stabilization branch is created with the same git hash used for building the snapshot.
> Between RC and GA release criteria are tested on the RC and if the release met the criteria a GA release will be built just updating the versioning
> code to drop master, git hash and timestamp suffix from tarballs and rpms.
> A new RC will be created and GA postponed by one week if release criteria are not met.
> 
> 
> 
> = oVirt Release Process =
> 
> * After a new 3.y.0 release, a new 3.y+1.0 release is tentatively scheduled after 6 months.
> * A new release management page is created (see http://www.ovirt.org/OVirt_3.5_release-management)
> * A new tracker bug is created (see http://bugzilla.redhat.com/1073943)
> * A discussion is started on devel and users mailing list gathering idea for next release features
> * Teams prepare a list of accepted features collecting / creating bug tracker, devel owner, qa owner, feature page for each of them
> * Several presentations are scheduled by the teams presenting the accepted features
> * An alpha release is scheduled 4 months before GA
> * Feature freeze is scheduled 3 months before GA
> * A beta release is scheduled 2 months before GA, git tree is branched for stabilization
> * A release candidate is scheduled 1 month before GA
> * Weekly status email are sent starting 3 weeks before alpha release.
> * Test days are scheduled after every milestone release
> * Release manager update the release notes pages starting from Alpha
> 
> All milestones releases are created by taking the latest nightly snapshot available after verifying that the build passes basic sanity test and
> repository closure.
> 
> RC will be postponed until all known blockers are fixed
> 
> Between RC and GA release criteria are tested on the RC and if the release met the criteria a GA release will be built just updating the versioning
> code to drop master, git hash and timestamp suffix from tarballs and rpms.
> A new RC will be created and GA postponed by one week if release criteria are not met.
> 
> [[Category:Release management]]
> 
> 
> What I think we're missing / doing wrong:
> * A discussion about release criteria *after* accepted features list is ready

1) I think we should require that each feature must come with a set of test cases covering the functionality provided by the new feature.
The test cases becomes part of the release criteria as
" * All test cases defined for each accepted feature must be verified and pass the test for the build to be released"

This means that the whole test suite must be verified on each release candidate for deciding if it's ready to be released.
Test suite should cover whatever is in the documentation of the feature.
This means that the feature must contain enough documentation to be considered "a contract" between the feature owner and the feature tester / user.

2) I think that the current criteria: "MUST: No blockers on the lower level components - libvirt, lvm,device-mapper,qemu-kvm, Jboss, postgres,
iscsi-initiator" should be dropped.
We're not acting as QA for lower level components. Instead I think we should change above statement with:
"MUST: all lower level components must be available on supported distributions at required version"

This means we're not hosting any package we don't develop inside oVirt repository just because it's not yet released officially.

I think we can allow to require third party repositories like JPackage, Gluster, virt-preview, epel-testing, fedora-updates-testing.
We shouldn't allow to add to oVirt repositories packages which belongs to third party repositories.
Example of those are libguestfs, python-cpopen, slf4j, kexec-tools and so on.

3) MUST: No known regressions from previous releases.
this means that bugs with Regression keyword will become automatically blockers.

4) MUST: Have Release Notes with feature specific information
I think this should be a requirement for entering beta phase: all features page should contains enough information for allowing features maintainers
to add some information to the release notes page.

5) MUST: Have release announcement for front page of ovirt.org and for mailing lists
Well, this is something usually we do after we released the packages, so it can't be a MUST.

6) MUST: All sources must be available on ovirt.org

I think all remaining existing requirements should be on a per subproject base.

7) MUST each subproject must allow upgrade from a previous version either providing documented manual or automated upgrade procedures.

8) MUST each subproject must pass it's own regression test matrix.
This means that each subproject maintainer should create one and ensure it's tested.
This can be rephrased as "All features working on previous release must still work in the new release if not dropped as deprecated"
While verifying the features, regression bugs can be found. We just can't re-verify all fixed bugs, can we?

9) MUST each subproject should list packages expected to be in the release.
This just for ensuring we can verify the final repository for completeness.

Points 1 and 8 ensures also we have a test day page ready for the first test day, just listing test cases to be covered.

> * A clear schedule like http://fedoraproject.org/wiki/Releases/21/Schedule

Will follow up on the schedule once intermediate steps are more defined.

> * A clear feature freeze policy like https://fedoraproject.org/wiki/Changes_Freeze_Policy

Ignoring the Feature Freeze
Introducing significant changes after Feature Freeze can result in your patch to be reverted.

Allowed after Feature Freeze:
- Adding code meant for automated testing of the feature
- Adding bug fixes
- Adding localization
- Cleaning or optimizing the code without introducing new features

Not allowed after Feature Freeze:
- Continue to add new enhancements (note that supporting a new distribution is an enhancement)
- Try to cover new enhancements under bug fixes
- Make changes that require other subprojects to make changes as well (API, ABI and configuration files included) if not absolutely required for
fixing a bug
- Make changes that require packages not yet available on supported distributions.

Exceptions
About this, I think we shouldn't allow exceptions.
But if needed, I think that a public vote on the exception should be required.
Exceptions should be asked at least one week before feature freeze and vote on them should be closed on feature freeze.

This means that you're not allowed to open new features but you can complete those that can't be completed for feature freeze if the vote allow it.

Incomplete features
I think we should require for each feature to define a contingency plan, allowing either to disable the incomplete feature or ensuring that the
feature code is isolated in a directory or package that can be easily dropped.
Those that can't be done that way may live in their own branch until they're finished and merged in the main tree just when completed.

> * A clear policy about changes we must allow and not allow between RC and GA

I think that only patches fixing blockers bug should be allowed between RC and GA.
This means that ystream branches must fork on RC release git hash and release maintainers must ensure nothing will be merged there if not related to a
blocker bug.

> * Alpha release should come after feature freeze

Nothing more to say about it

> * A string freeze schedule, allowing translators to have enough time to translate new strings

I think that we should freeze string between beta and 1 week before RC.
Considering we have 1 month between beta and RC this means that string freeze should go no later than 2 weeks before RC and translators have 2 weeks
for finalizing last translation bits and verify them.
So, I think we should have a beta release requirement:
- Supported localizations must be at least at 70% of completeness at beta stage for being included in the release.
Without such requirement translators won't make it on time for GA.

> * Maintainers should pay more attention to release notes and test day pages

This is enforced by rules on features acceptance requirements.

> 
> 

-- 
Sandro Bonazzola
Better technology. Faster innovation. Powered by community collaboration.
See how it works at redhat.com