In terms of relative effort, releasing app updates feels like it shouldn’t be that big a deal. The hard work of planning, designing, building, and testing your new version is all or mostly done at that point. You’re in the home stretch and oughta be able to kick back to watch the money (and good reviews) roll in.
Perhaps because it feels like it shouldn’t be a big deal, it’s often not treated like a big deal by the larger organization. This is a problem considering it often is a big deal. Mobile releases are too often a time and resource vacuum that sucks up the attention, coding and planning time, and peace of mind of everyone on the mobile team.
Chaotic mobile releases cost more than just your mobile team’s sanity; they also cost your company the features the team would be able to build and ship if they didn’t spend a quarter of their time distracted by the unnecessarily turbulent process of trying to get something they just built out the door.
An inefficient release process automatically makes you more inefficient at building your product. Some of these inefficiencies cost you in obvious ways (you’re not shipping as quickly as you could be) and in hidden ways (you’re quietly building up technical debt that will one day come back to haunt you).
What exactly does a bad (or just OK) release process cost you?
Not-so-hidden-cost: You’re shipping fewer features for your mobile app
“We estimated we were losing around 30% of an engineer’s time within a release period just with all they had to do to coordinate and execute a release. Our engineers were getting burned out from the cognitive and mental load of shipping a release.”
Alan Cooke, Squarespace
The amount of time it takes a release manager or an engineer within a release pilot rotation to run a release will vary. Depending on your tooling, the supporting release infrastructure your engineers have built for themselves, and the number of coordination problems they encounter during the release, the amount of time may go up or down a bit, but it’s usually at least on the order of several hours per release — and that’s if you have a release train in place. If you’re doing ad hoc releases, it’s probably more than that. How much is 200 hours of engineering time worth to your product each year?
And it’s not just your mobile engineers who are putting their energy into this release time sink. Product has to plan around the speed at which the mobile engineers can work, and they often have no visibility into the release process — so they have to continually ping engineers to find out what’s happening or find out if they may be needed in some way.
Same with QA. They can’t perform their jobs thoroughly or efficiently if they’re not plugged into the process, aren’t effectively notified when they should switch the builds they’re testing against, can’t predictably start testing a new release candidate at a set time, or are being blindsided by new builds or hotfixes. They then work more slowly, which in turn slows engineering down, which slows Product down, which slows the overall development of your app.
This work even has an impact outside of the PDE side of the house. Marketing needs to prepare copy and media and screenshots for distribution, support needs to prepare to answer questions and help customers with any issues that come up, and recruiting needs to be ready to repost your LinkedIn announcement to try and lure new engineers in with the opportunity to work on your cool app. Their lack of visibility into release progress can make their own work more stressful.
All of this is adding significantly more hours to the rough 200 hour figure I gave above. How many extra features could you ship each year if your teams weren’t having to wrestle with your mobile release process? And what’s the additional cost to the business of not only missing out on these additional features, but also getting the features you do ship into the hands of your customers more slowly? Are your competitors getting updates to their customers every week instead of once a month?
Hidden cost: You’re stressing your mobile engineers out
“There was lots of fragmentation and context switching: making sure marketing has given us the right media, making sure QA knows when it’s their turn to step up to the plate to run regression testing, and communicating with PMs.”
Sanjay Thakur, Classpass
Maybe this all sounds fine. Even if a release manager is devoting 30% of their time to a release, then they’re still devoting 70% of their time to their more valuable feature work. This doesn’t necessarily seem unreasonable; it’s not like it’s eating all their time. But leaning on your skilled mobile engineers to devote 30% of their time to building and maintaining their own infrastructure only seems alright until the built up harm to their work output and happiness begins bubbling to the surface.
The issue is that release work and feature work are very, very different, requiring fairly extreme context switching. They’re not just taking a 30% chunk of time for the release and then going back to their other work. Instead they’re jumping back-and-forth between tasks, maybe doing 2% of the release work, then 4% of the feature work, then 1% of the release work, and on from there.
Increasing cognitive load in this way makes getting into the flow of either task next to impossible, creating extra stress and leading to mistakes. What was just four hours of time for a release, becomes six hours from the context switching, and then twelve hours when you include fixing the bugs that were written caused by the lack of focus from all the context switching.
“But Richard!” you say. “It’s fine. We have a release pilot rotation in place, so any individual engineer only needs to face this stress six times a year.”
While it may be true that this specific stress only comes up every so often throughout the year, the stress — as well the amount of time wasted — are both increased by the huge gaps in time between an individual engineer’s turn in the rotation. No one is going to remember the steps they need to take each time their turn comes up if that turn only arrives every two months. Which means the whole process ends up taking even more of their time, causing more context switching and more problems that impact everyone.
Amping up the cognitive load for your engineers like this is a surefire way to make them frustrated and unhappy. What’s the cost to your organization when some of your engineers become so annoyed by your process that it contributes to them leaving to go find a different company that is more respectful of their time and talents?
Too much context switching and stress does more than just create unhappiness and turnover. It also adds up to engineers making unplanned concessions in their work, and unplanned concessions are one of the issues that cause technical debt.
Hidden cost: You’re building a big pile of technical debt
“People were hesitant to manage releases, even though we had a board that covered step-by-step what needed to be done. Most of the time I had to get on a call to help whoever was running a release.”
Eran Lunenfeld, monday.com
If you’re like most teams, then your mobile release process isn’t like most teams. While how you release your app certainly resembles the shape of what every other team is doing (you’ve got to get a release candidate tested and signed off on and then sent to the App Store and Play Store), your own approach is likely somewhat bespoke and sprinkled with its own specific idiosyncrasies that reflect the individual, often unconnected decisions made by all the people who helped create it.
The thing about a bespoke process is that if it’s not thoroughly documented, planned, and updated over time, it can very quickly (and also quite commonly) become an inconsistent process. And an inconsistent process will slowly fill up with technical debt since nothing is ever done completely by the book (because there’s no standard book for things to be done by) and your mobile team will likely attempt to make changes and improvements to the process on the fly without a larger strategy. While these individual changes (like writing a new script) will make sense in the moment, stringing together a lot of unconnected alterations and attempts at improvements will turn your release process into a Frankenstein’s monster.
It always takes concessions to deliver your product. If you’re building your own bespoke release process, then that process is also part of your product and concessions will likely be made to build it. Those small concessions that end up being made as part of every release can lead to the sort of inconsistencies that build up into larger problems — and before you know it, you’re up on a Sunday night figuring out to how patch a script written in an esoteric language by an engineer that has long been gone.
The more inconsistencies you have, the more work the release process is going to take to manage. The more work it takes to manage, the more stressed everyone is going to feel about trying to ship new features on the near impossible timeline being set by leadership that doesn’t take the slow and inconsistent release process into account. This leads to more concessions being made building those features. More bugs end up in the hands of customers, which means more hotfixes have to be shipped, more builds have to be tested, and more (mostly unnecessary) work is generated for everyone.
The more debt that builds up, the more difficult it is for engineers who didn’t personally build this stuff to easily contribute. This is just one example of how a weak, inconsistent release process can make collaboration more difficult.
Not-so-hidden cost: You’re hurting collaboration across the org
“Understanding what was happening across all our different tools was a big pain point for our immediate team, and especially for other stakeholders, and we had spent a lot of time trying to solve it.”
Juliana Lara, iFood
An opaque, inscrutable release process keeps everyone in the dark about each and every new update that goes out the door. Instead of sitting in the dark wondering what’s going on, they’ll probably ask about it a lot or simply expect to be kept continually updated on progress.
You can’t give everyone (probably not even everyone on the mobile team) access to all the tools involved in your process. That’d be expensive, they wouldn’t know their way around each tool and would need to be pointed to the information they need, and they might knock something over without meaning to. You’d just be exporting some of the context switching pain the release managers have been dealing with to everyone else and creating an entirely new kind of pain in having to educate everyone on how to use a disparate set of tools.
You could ask the release manager to act as a kind of release town crier who goes into a Slack channel and says, “hear ye, hear ye, QA just finished ye olde regression tests” every time something is finished, but considering the amount of noise that would generate, people would probably ignore it and just keep asking questions. It would barely serve a purpose except to give the appearance something was being done.
Ideally you can find or build a single source of truth where everyone can keep up with progress and add their contributions. Imagine if marketing was automatically notified when it was time for them to provide new screenshots and updated copy, and could potentially just add it themselves. Imagine if QA could immediately see when it was time to start testing a build and that engineers could immediately see when QA was done with those tests. Imagine if every member of the mobile team could check on progress and contribute regardless of whether it was their turn in the pilot rotation.
You don’t have to just imagine these things. All you need to do is focus on truly getting good at mobile releases. Establish a release train. Implement kick-offs and retros for every release so that any problems are fixed for the next release and any changes are documented so your releases are consistent. Get as close as you can to a single source of truth for the entire release process, from kick-off to monitoring your rollouts, so no one is ever in the dark about what’s going on and can take quick action when they’re needed.
Start working on it now, improve your process, and get ahead of the game. Or you can wait and focus on other priorities. But the longer you wait, the more it’s going to cost you to build a release process that doesn’t cost you.
