In some orgs, responsibility for mobile releases falls on either a single knowledgeable person or a very small number of people. These folks are mobile release wizards, locked away in their proverbial release high towers crafting ever more elaborate scripts and checklists to improve the release process. This is hard, important work, but it can also make it difficult for anyone else to run a release if the wizard ever leaves their tower to go on vacation (or even leaves for good).
Handing everything off to a singular release manager has its benefits: there’s no need for knowledge transfers, only one person ever needs to do any release-related context switching, and they’ll probably know exactly what to do if any problems arise. But these benefits are countered by a number of potential problems, many of which arise for two primary reasons: silos are bad, and having a single point of failure will at some point almost certainly guarantee actual failure.
Ultimately, releases shouldn’t be treated as rickety, taped together processes that require special knowledge of magical spells (or Ruby scripts) to run. In most situations, any mobile engineer (or even a PM or someone from QA) should be able to tackle a release with the right resources in hand.
Note that throughout this post we refer to release managers alternatively as "release pilots". We prefer this nomenclature because it better describes the work that’s being done. Release managers aren’t simply acting as administrators, but are instead driving the entire process of bringing new features in for a landing. Some orgs use the term release drivers, some use release captains; we like release pilots because our name is Runway, of course, and the complexity of flying a plane matches well with the complexity of running a release.
Why even set up a mobile release manager rotation?
As noted above, having your releases handled by a single person or small release team ensures every single release is run by people who have the know-how to tackle the many possible issues that can crop up during the process, and it can be easier to ensure a standardized, consistent process. But the problems with limiting release management to a small number of people can easily outweigh the benefits:
A single point of failure could guarantee failure
People go on vacation, become ill, even quit their jobs. Rebuilding their knowledge from scratch — or from half-finished docs they’ve cobbled together along the way — could knock you off course for weeks.
Such an important process should not exist in a black box
All engineering work should (arguably) be done out in the open. This makes it easier for everyone to learn from the past and for engineers to learn from their colleagues and grow. Siloing releases off to a small number of people can lead to the entire process becoming indecipherable and foreign to the rest of the org, and empathy for the release manager role suffers, with some cascading negative effects across the team.
Putting all release work on one person usually isn’t fair
Unless you’re an enormous org with a team that does nothing but manage releases — which can have its own problems — then whoever is managing them is almost certainly being put in a position where they’re being kept away from doing other work.
Dedicated release managers cost money
A person or team focused mostly on releases means salary dollars going to engineers who are spending little to no time working on the product. This can get expensive fast.
Everyone gains greater ownership over their work
It’s one thing to contribute to a feature, it’s another thing entirely to be part of shepherding that feature all the way into the app store and then to monitor how it performs once it’s there. If an engineer builds it, they should arguably run it, unless you just desperately want to make waterfall-style development popular again by vesting responsibilities for different parts of the process to siloed groups. You 1) probably do not want to do that and 2) likely do not take that approach anywhere else in your engineering org.
That’s the why, but what’s the how?
Define a release strategy, scope, and schedule for every release manager to follow
Expanding the number of release managers opens up the possibility of inconsistency from week to week. Different people have different approaches to doing anything, including running a mobile release. There may not be one right way to do anything, but you need to determine the one specific right way you’ve decided to run your mobile releases or else variance in approaches between different release pilots could potentially cause headaches, confusion, and quality issues as you rotate from pilot to pilot.
Before starting your rotation, sit down with everyone and define:
- Release cadence
- Cutoff days and times for adding features to a release
- Rules for getting fixes into an in-flight release
- Who needs to sign off on each release and when
- Any other variables specific to your team and app that could impact the pace or quality of releases and the workload on the entire team
Release pilots need to know what to do, when to do it, and how to do it. Determine all the steps involved in that and it should be fairly smooth sailing — so long as everyone remembers what to do when their turn as pilot comes around.
Put together a detailed release rotation checklist or runbook
One potential negative of switching away from a single (or small number) of release managers is that people will forget how to do something (or many somethings) in the gap between their turns as rotating release manager. Which is why all the steps in the process need to be put into easily digestible words (or simple actions) that release pilots can follow along with when their turn is up. Every part of the approach that was determined when setting your strategy, scope, and schedule should be documented in a runbook, as well as each and every step that needs to be taken to actually run a release.
This release runbook should be a living document, kept up to date and evolving over time. Run a retro after every release, even if it was totally successful and fine, to discuss where problems may come up in the future, hear issues or ideas for improvements from the pilot, and hear issues or ideas for improvements from the rest of the team. Adjust your process and revise the runbook accordingly. A runbook that is rarely updated is one that will have gaps, get out of date, and end up ignored by release managers who then have to rely on memory to wing it.
The last release pilot should own their own retro and manage the release documentation during that time.
Putting all this stuff in a static doc, even one you diligently update, is the bare minimum you can do. Leveraging tools that automate rote tasks away and more actively help the release pilot coordinate the entire process is even better, something we discuss later in this post. Humanity has had shareable docs since the invention of the printing press;, software engineering teams have even newer technology at their disposal and should use it.
Give mobile engineers a reason to become release managers and give them ample time to manage the process when they do
Engineers often hate having to do anything admin or process related. This isn’t because they’re simple code monkeys who can’t function outside of their IDE, but rather because they’re so often put in release or on-call rotations, handed an out-of-date doc, and provided no other prep.
You’re obviously not going to do that to your engineers — you are reading this post, after all — but due to negative experiences they’ve had in the past, they may not be so excited to jump at this opportunity. Consider how you can reward engineers who become release pilots:
- Bonuses for every week they run a release
- Time off the following week
- Special patches or pins
- A nice dinner out (restaurant gift certificate) or in (DoorDash gift certificate)
A new release pilot should shadow an experienced release manager before their own shift comes up and then be shadowed by that same experienced pilot during their first couple of shifts.
Bake time into a release manager’s work every week. Unless you’re using a tool like Runway (and if you’re not, you should consider it while you’re here) you should expect additional release work to add anywhere from four to eight hours to their week, which is four to eight hours they shouldn’t be expected to put into their ongoing projects.
Release pilots should also be responsible for preparing the next scheduled manager and handing things off to them so that any needed context from the previous release is passed on.
Get as close to a single source of truth for the entire team as you can
Successfully managing a release goes well beyond following a checklist. There are a lot of moving parts involved with a release: deadlines for cutting the release branch, sign-offs from different people and teams, getting last minute fixes in place, submitting to the App Store and Play Store, dealing with rejection (should it happen), and keeping an eye on phased rollouts.
Centralize all of this information and work as much as you can. If a release pilot is jumping between twenty different tabs in their browser to keep track of work, running three scripts and watching two of them fail, and messaging fifteen different people in twelve different Slack channels… then things are going to get missed and your release is going to be far more stressful for everyone involved.
Creating a single source of truth isn’t just valuable for your release pilot, it’s useful for everyone else on the team who otherwise would barely be able to follow along — outside of the limited viewpoint of their ongoing work — without also opening up 20 tabs and trying to keep up.
If you can centralize everything, then you also ought to be able to automate away any manual / repetitive tasks. Everything from setting the schedule, to attaching artifacts to the release, to halting unstable rollouts, to deleting release branches and transitioning issue tracking tickets when the release is completed should be automated away so that the release pilot doesn’t need to lift a finger to do any of them.
How can you centralize and automate all this work? In a perfect world you’d be able to find a tool that integrated with your VCS, issue trackers, communication tools, the app stores, and your post-launch crash reporting / analytics tooling. If only such a tool existed and one of its employees had written a blog post you were reading right now, all your problems would be solved…
But even if you don’t use a tool like Runway, your team could benefit from a release pilot rotation. It’s good for the mobile engineering team as it gives them greater ownership of their work, it’s good for whoever runs your releases now as it gives them a chance to get back to building features, and it’s good for the org since it lets your whole team put your release process through the same process of trial and improvement that your actual product goes through. There is no reason to work inefficiently just because only one person knows how to do something.
