How successful mobile teams release apps at scale

Bruno Rocha is a household name in the mobile world. He writes and speaks often on topics such as Swift and reverse engineering, but with particular attention to matters concerning teams looking to grow and scale successfully. Originally from Brasil, his current role at Spotify has since taken him to Sweden. The Runway team is thrilled to have worked with Bruno on this guest post!

The problem of releasing mobile apps takes many forms, and what I find most interesting is how different the processes and solutions teams put in place can be — even across companies of similar size and maturity. Despite everyone facing the same foundational problem of having to optimize the process of delivering an app to the world, I’ve never encountered two companies that approach it in the exact same way. But although technical implementations differ, the release processes of large companies seem to always focus on the same core principles: quality control and data-driven feedback loops. To understand why this is the case, we’ll take a look at the types of problems that large companies face as they scale and see how they differ from those of smaller companies.  

In my experience working on apps of all sizes, releases start to become a serious issue and an acute pain point at a very specific moment in time. When the scale of the product (and team) gets to a point where it's impossible for a single developer to be aware of everything the app does, and familiar with all the code it contains, releases become an important area of concern. This is because it's perfectly viable for developers of a small product to build out and fully trust a suite of tests that cover all possible scenarios and user paths, but when you have a sufficiently large codebase, guaranteeing the quality of the app purely from a CI perspective becomes practically impossible. This isn’t a technical limitation – rather, not having a full understanding of the project means that the developers lose the ability to know, with 100% certainty, that a given new feature will work in all possible scenarios (and not break existing functionality).

When a product is scaling, the release process starts becoming a more central piece in guaranteeing the quality of the product. While smaller teams will often conflate their release process with a simple automated pipeline that runs tests, generates a build and uploads it somewhere, shipping releases at larger companies is a much longer and more intricate process.

Stabilization

The first defining factor of releasing at scale is that the act of releasing usually takes multiple days instead of a couple of minutes. Instead of fully trusting the app's testing suite, almost every large company will first subject the release candidate build to a stabilization process, under which the objective is to increase the team's trust in that particular build. How long this process takes varies depending on how often that particular team ships, but it’s usually on the order of one week.

Generally, the process of stabilization revolves around releasing the release candidate build as a pre-production “alpha” or “beta,” subjecting it to some active usage, and waiting to see if a critical problem shows up. This beta testing process can be done in several ways, including these common techniques:

• Dogfooding, or "eating your own food": In mobile, this often refers to the practice of generating internal beta builds for employees of the company to use in a real world scenario. Dogfooding is great not just for release stabilization reasons, but also to generate general product feedback and to get the company as a whole more involved in the development and evolution of the product. One great way of enabling this is to encourage employees to use a "nightly build", which is the output of a CI job that runs every night and compiles and deploys whatever is in the repo's main branch. Some companies even force their employees to use such builds (banning the use of production versions from App Store and Google Play)!
• Public alpha and beta builds: Public, pre-prod alphas and betas are incredibly good at exposing critical issues before they can be rolled out to an entire user base, and it's quite typical for very large companies like Google, Facebook, and Uber to have quite significant alpha/beta programs in place for their mobile apps. The reasoning here is that large, complicated apps with a variety of users and use cases will often experience issues that are so specific that it's unlikely a small QA team will be able to catch them in time (or ever!). On the other hand, if you get a lot of people using the app in lots of different ways, the chance for these issues to emerge during your stabilization process is much higher. As a reward for testing the app, the public beta users get early access to exciting new features, as well as the opportunity to contribute to a product that they enjoy.
• Manual QA: In addition to having the assistance of developers and external beta testers, large companies usually employ dedicated QA teams that will actively try to break the app based on their experience with the app and its bug history. As we've mentioned before, this is not a bulletproof solution, so it's often done in conjunction with the other measures.

When issues are found during the stabilization process, how they are dealt with will generally depend on how critical they’re deemed to be, and how early in the process they were found.

Approaches to evaluating the importance of newly-discovered issues vary, but teams generally try to make this part of the process very standardized, to avoid gray areas and politics. Impact to the business should be estimated and issues categorized into buckets of relative importance accordingly. Items falling into certain top, critical levels will need to be addressed prior to release. Other, less critical items will be ticketed for resolution in a future update.

Timing also plays an important role. Fixes for issues identified early in the stabilization process are more likely to be entertained for that same release, while issues found relatively late might need to be addressed in a subsequent release cycle – or, worst case, they can cause the release to be delayed or even canceled if they’re deemed critical and there's not enough time to both fix and test within the current schedule.

Release cadence

While an indie developer or small team might release new builds only when specific new features are ready for the outside world, larger teams often release on fixed schedules called release trains. When a train "leaves the station", often at the end of the week, the code for that release is effectively frozen (e.g. a release branch is created from the repo's main branch), and a release candidate build is compiled and run through the stabilization process. If everything goes well, the end of the following week will mark the beginning of that build's rollout to production, and the subsequent release will branch and leave the station. In practice, this decouples feature work from releases and means that new app features can only ship in concert with the schedule of the release train. Unless a developer is pushing a critical hotfix directly to a release's frozen code, their changes will only be available as part of the next train.

Release trains are popular for large-scale products because it's a much easier way of synchronizing multiple feature teams. Instead of having to figure out a release date that aligns with the product deadlines of every single team, it's the teams themselves who have to reconcile their needs with the fixed schedule of the release train. This can sometimes cause teams to miss releases if their work gets delayed, but the consistency of the process and benefits to security and overall product quality make it an optimal approach in most large companies’ eyes. Another less obvious win that release trains deliver is that they save teams from having to deal with haggling and diplomacy over exactly what work from which teams can make it into a given release.

Data-driven releases

Exactly what causes a stabilization process to succeed or fail, and how release success is defined more generally, depends heavily on a company's particular product and goals. At a high level, though, definition of release success usually revolves around an evaluation of app stability and a set of other metrics important to the product and business (e.g. user engagement and conversion). 

Accordingly, you'll see that successful companies at scale boast teams who are not only great at building and maintaining release process and tooling, but who also have a strong competency in product analytics and data more generally. In addition to solid overall release practices, such companies often also have robust analytics, monitoring, and A/B testing data coming from their apps, and while these other areas are not directly related to shipping builds, they very much go hand-in-hand. It's quite normal for large companies to have pages and pages of dashboards conveying critical info and trends about a specific app version's usage, and release teams are keeping a close eye on all of it as builds advance through stabilization and especially when approved builds are rolled out to production.

Rollouts, especially those to production, are always incremental and their continuation is dependent on very specific conditions and expectations being met on the data side. This is perhaps the most concrete way in which releases become data-driven. To that end, to make data collection easier and to make the resulting data more available and actionable, the most mature companies end up building out their own internal platforms that bring together release and monitoring functions.

Conclusion

Ultimately, despite following similar best practices, large companies will always tailor the specifics of their mobile release process to fit their unique needs. But, while the exact techniques may differ between organizations, the most impactful strategies that allow teams to successfully deliver a mobile product at scale generally fall into two major buckets: enabling meaningful data-driven feedback loops, and ensuring consistently high quality releases by leveraging a regular release cadence and a robust stabilization process. 

Focusing on these areas as your team grows — and as product complexity increases — is a surefire win. But there’s no out-of-the-box solution here, and achieving those goals in a repeatable way and with minimal overhead isn’t guaranteed. That’s a challenge for you and your team to solve!