Release Processes

Release Processes¶

This document describes the process and week-per-week steps related to preparing the release. It should be adapted to take team holidays and other commitments into account, and turned into a clear action plan for the release cycle.

Week after previous release¶

Process any late or blocked release requests for deliverables for any branch (treating the new series branch as stable).
Prepare for the next release cycle by adding deliverable files under the next cycle’s directory. Remove any deliverable files from the current cycle that ended up not having any releases. Then run the following command to use the current cycle deliverables to generate placeholders for the next cycle:
```
tox -e venv -- init-series $SERIES $NEXT_SERIES
```
Submit your changes by using the gerrit topic $series-init-series.

Warning

At this point, the trailing deliverables of the previous series are not necessarily released yet. Adding them here is not an issue, however, we should be vigilant and ensure to create new releases for the previous series first.
Coordinate with the Infrastructure team to swap out the previous cycle signing key and establish the new one for the starting cycle.
Create the $series-relmgt-tracking etherpad using the make-tracking-etherpad command. For example:
```
tox -e venv -- make-tracking-pad ussuri
```
The output from this command can be pasted into a $SERIES-relmgt-tracking etherpad. Set title formatting for the top sections. Then highlight all listed weeks and set to Heading 3 style. Fill in the contents of one of the weeks with the typical items, then copy and past that into each subsequent week to prepare for the rest of the cycle.

Email PTLs directly one time to explain the use of the “[release][ptl]” email tag on the openstack-discuss list and tell them to pay attention to [release] countdown emails. Use the following email template and send it with $SERIES Cycle Information as email subject:

This email is going out to all PTLs and release liaisons for $SERIES.

Welcome to the $SERIES series. On behalf of the release management team, I
just wanted to say hello and make sure you had the information you need
to know what to do and what to expect during this coming release cycle.

Release Schedule
================

If you haven't seen it yet, please take a look at the published schedule for
this cycle:

    https://releases.openstack.org/$SERIES/schedule.html

That lists all of the major milestones along the way as we work towards the
coordinated release on $release-date.

Community-wide deadlines are listed under the "Cross-project events" column.
You may notice some projects have added project-specific deadlines under the
"Project-specific events" column. That can be done by submitting a patch to the
openstack/releases repo updating the doc/source/$SERIES/schedule.* files, if
you would like to make sure those deadlines are published somewhere.

Countdown Emails
================

Throughout the cycle, we will be sending regular "countdown" emails to the
openstack-discuss mailing list. If you have filtering set up for mailing list
emails, please make sure you are not filtering out subject lines that contain
"[release]" and "[ptl]" tags.

These countdown emails will contain important updates on upcoming deadlines and
information about necessary actions throughout the cycle. We try to keep them
to a minimum, so earlier on in the cycle they will not be sent weekly, but
around deadline times and especially towards the end of the cycle, these will
go out weekly to make sure everyone has the information they need to get work
done in time. Your attention to these emails would be appreciated. We will try
to keep them short.

Deliverables
============

The release process may be new for a few of you. All official OpenStack
deliverables are released by updating deliverable files in the
openstack/releases repo:

    https://opendev.org/openstack/releases/src/branch/master/deliverables/$SERIES

Please take a look at your deliverable files and make sure the release-model
matches what the team plans to follow for this cycle. This model is a way to
communicate to downstream consumers what to expect, so the release team uses
your declared model to determine when and how to enforce different steps during
the cycle.

Details on the release models and what they mean, as well as documentation on
how to use the releases repo, can be found in the references section here:

    https://releases.openstack.org/index.html#references

That has documentation for some of our tooling too. The most important one you
may want to know about is the new-release tooling so you don't need to edit
those yaml files directly. That command can be used like this:

    tools/new_release.sh $SERIES cinder feature

The last part can be major (major version bump), feature (minor version bump),
and bugfix (bugfix, or the Z in X.Y.Z).

Reviews
=======

Because of the strict timing of the cycle schedule, it is important that you
review release patches within some days or usually until Thursdays EOB if
possible. Please look at the incoming review request emails (sent by gerrit)
from openstack/releases repository regularly or at least check your inbox at
least once or twice a week:

    https://review.opendev.org/q/project:openstack/releases+reviewer:self+is:open

The above link shows a customised list of release patches that only contains
patches that you are added as reviewer. Release managers use a script to add
PTLs and release liaisons to their relevant release patches as reviewers.

Release Liaisons
================

If anyone should be added as a release liaison, or removed, just submit a patch
to update the data/release_liaisons.yaml file in the openstack/releases repo
with current information for your team.


Please contact us at any point with any questions. We can be reached on the
openstack-discuss mailing list with the [release] tag, or on IRC in the
#openstack-release channel.

Thanks for your attention. I hope the $SERIES cycle goes well for everyone.

At the end of the week, send the following weekly email content:

Welcome back to the release countdown emails! These will be sent at
major points in the $SERIES development cycle, which should conclude
with a final release on $release-date.

Development Focus
-----------------

At this stage in the release cycle, focus should be on planning the
$SERIES development cycle, assessing $SERIES community goals and approving
$SERIES specs.

General Information
-------------------

$remark-on-series-length. In case you haven't seen it yet, please take
a look over the schedule for this release:

https://releases.openstack.org/$SERIES/schedule.html

By default, the team PTL is responsible for handling the release cycle
and approving release requests. This task can (and probably should) be
delegated to release liaisons. Now is a good time to review release
liaison information for your team and make sure it is up to date:

https://opendev.org/openstack/releases/src/branch/master/data/release_liaisons.yaml

By default, all your team deliverables from the $PREVIOUS_SERIES release are
continued in $SERIES with a similar release model.

Upcoming Deadlines & Dates
--------------------------

$other-upcoming-event_
$SERIES-1 milestone: $milestone1

Week before milestone-1¶

Review cycle-trailing projects to check which haven’t released yet. Ask them to prepare their releases, if they haven’t already. The list of pending cycle-trailing deliverables can be determined by running the command:
```
tox -e venv -- list-deliverables --series $LASTSERIES \
    --type trailing --missing-final
```

At the end of the week, send the following weekly email content:

Development Focus
-----------------

The $SERIES-1 milestone is next week, on $milestone1! Project team plans
for the $SERIES cycle should now be solidified.

General Information
-------------------

Libraries need to be released at least once per milestone period. Next
week, the release team will propose releases for any library which had
changes but has not been otherwise released since the $LASTSERIES release.
PTL's or release liaisons, please watch for these and give a +1 to
acknowledge them. If there is some reason to hold off on a release, let
us know that as well, by posting a -1. If we do not hear anything at all
by the end of the week, we will assume things are OK to proceed.

NB: If one of your libraries is still releasing 0.x versions, start
thinking about when it will be appropriate to do a 1.0 version. The
version number does signal the state, real or perceived, of the library,
so we strongly encourage going to a full major version once things are
in a good and usable state.

Upcoming Deadlines & Dates
--------------------------

$SERIES-1 milestone: $milestone1

Milestone-1¶

Ensure that all trailing projects have been branched for the previous series.
- List unbranched projects using:
```
tools/list_unbranched_projects.sh
```
- Propose a patch to branch the missing ones.
Propose autoreleases for cycle-with-intermediary libraries which did not release since the previous release.
- List them using:
```
tox -e venv -- \
     list-deliverables \
         --unreleased \
         --model cycle-with-intermediary \
         --type client-library \
         --type library \
> /tmp/deliverables.log
```
- Edit the generated file (/tmp/deliverables.log) to remove tox’s logs.
- Generate release requests for all cycle-with-intermediary libraries which had changes, but did not release since the previous release.
  
  Warning
  
  process_auto_releases will ask you to enter a topic for the patches. Please use $series-milestone-1 as topic.
  
  For this, run (c.f tools/process_auto_releases.sh):
```
tools/process_auto_releases.sh $SERIES $(cat /tmp/deliverables.log)
```
  That patch will be used as a base to communicate with the team: if a team wants to wait for a specific patch to make it to the library, someone from the team can -1 the patch to have it held, or update that patch with a different commit SHA.
  
  Here is an example of milestone 1 generated for Wallaby:
  
  https://review.opendev.org/q/topic:%22w1-c-w-i%22
- Between Tuesday and Thursday, merge as soon as possible the patches that get +1 from the PTL or the release liaison.
- On the Friday, merge patches that did not get any feedback from PTL or release liaison. Discuss standing -1s to see if they should be granted an exception and wait until next week.
To catch if there are acl issues in newly created repositories, run tools/aclissues.py to detect potential leftovers in Gerrit ACLs allowing official deliverables to be directly tagged or branched without going through openstack/releases. You need to specify the location of up-to-date checkouts for the governance and the project-config repositories. For example:
```
tools/aclissues.py ../project-config ../governance
```
If the tool reports any violation, you can re-run it with --patch to generate needed changes in ../project-config to align ACLs with governance, and propose the changes for review.

At the end of the week, send the following weekly email content:

Development Focus
-----------------

We are now past the $SERIES-1 milestone. Teams should now be focused on
feature development and completion of goals [0].

[0] https://governance.openstack.org/tc/goals/selected/index.html

General Information
-------------------

Our next milestone in this development cycle will be $SERIES-2, on
$milestone2. This milestone is when we freeze the list of deliverables
that will be included in the $SERIES final release, so if you plan to
introduce new deliverables in this release, please propose a change to
add an empty deliverable file in the deliverables/$SERIES directory of
the openstack/releases repository.

Now is also generally a good time to look at bugfixes that were
introduced in the master branch that might make sense to be backported
and released in a stable release.

If you have any question around the OpenStack release process, feel free
to ask on this mailing-list or on the #openstack-release channel on IRC.

Upcoming Deadlines & Dates
--------------------------

$SERIES-2 Milestone: $milestone2