The early (and also not-so-early) days of building a tech startup means hiring and working with people who are capable of wearing a lot of hats. There are a ton of things to do and nowhere near an equal amount of resources to do them. Even if you join as a specialist, chances are you'll find yourself getting deep into other areas.
This is true across the organization, including in mobile engineering. Early engineers don’t just write the code that builds the foundation for the app’s future success (while building up the tech debt that future engineers will pull their hair out over) but also the initial infrastructure both for how teams get things done and how they ship code out the door.
This doesn’t last forever though. As teams grow they make the transition out of generalist, jack-of-all-trades roles and begin focusing on specialization. But one part of the org where there is often a lag in making this transition — and sometimes a very long one — is on the mobile team. Very few mobile engineers will have time to support both their team's app features and the series of Ruby scripts that support their release process, yet the expectation that they do both often lingers far beyond when other teams in the organization have fully specialized.  Â
The challenge of mobile release engineering
Releasing code is supposed to be easy, right? You merge some code, do a little testing, run CI, ship it to production, and then just fix any bugs live as they come up. Rinse and repeat. To be honest, this is usually true in very small companies. But something that's also true is that the more a company grows, the more complicated this process gets. This is especially the case when you start having multiple product areas running in parallel; a point where coordination is key in order to avoid different parts of the company from running into each other.
Mobile teams in particular face unique challenges when it comes to releases:
Required app store reviews add inherent complexity.
Mobile developers can't make use of fancy automated deployments and continuous delivery like web teams do, they must submit them for review and be at mercy of Apple and Google's goodwill. This means that you can never really be sure when a new version is going to be rolled out, as review conditions and timelines are kept intentionally vague and sometimes even non-sensensical.
Mobile workflow automation is often bespoke and ad hoc.
The need for all this manual work usually means mobile engineers start building their own scripts and automations (or relying on some they found online) to make improvements. These are built as time allows and problems arise, which leads to a bespoke Frankenstein’s monster of a process that wasn’t implemented as part of a coherent strategy, can’t be recreated from team to team, and is likely to be largely unscalable and difficult to maintain.
Mobile workflow automation requires regular maintenance and upkeep.
Even once you have some basic process in place, it's likely that you'll need to tweak it as the project evolves, either to support new features or to make it more automatic. Without a dedicated release engineering team, you'll eventually run into trouble when said tweaks start requiring more and more effort in order to be achieved (e.g dedicated backends/pipelines)
Release engineering happens when a company gets so big that the systems and processes that it requires in order to be able to comfortably release the app become impossible to maintain as a side gig, requiring dedicated engineers. This is very common across large tech companies, and you'll find that many of the top tier companies have a mobile release engineering effort in one way or the other.
What are mobile release engineering teams responsible for?
Release engineers build and maintain tools and processes to support the company in getting features shipped and apps updates out the door: from generating release candidates, to providing and maintaining beta testing tracks, to providing data about a release's usage, to triaging and tracking bug fixes, to updating media and release notes, to uploading to the store and managing the review process, to managing a staged or phased rollout, to springing into action when problems arise during that rollout. The job of a release engineering team is to support this entire process, maintaining and improving it as needed.
A release engineering team should not be confused with a mobile platform team! There might be a time during a company's growth where a platform team could wear both hats, but the scope of release engineering eventually becomes big enough that it becomes its own separate domain.
The required skills and responsibilities of release engineers are distinct from those of other mobile engineers. They include:
A solid understanding of the underlying tooling and frameworks.
Release engineering tends to be primarily backend work, meaning release engineers need to have a decent understanding of general backend and data engineering, as a lot of time will be spent building and maintaining things such as servers, CI/data pipelines, cron jobs, as well as data endpoints for general ad-hoc data inspection needs. This is quite different from the skill set of those working on feature teams.
A deep understanding of any internal tools.
A strong grasp on the intricacies of any internal tools your team has built and relies on is important. Internal feature flagging systems, CI/CD platforms, analytics tools, are all a part of the development and release workflow and are likely tools that folks are regularly interfacing with as part of the process. Any efforts made to streamline this process will likely require a good understanding of any of the tools the team is relying on regularly at any point in the process, especially when it comes to triaging bugs. Â
Ownership and consolidation of disparate release processes.
If your team is releasing an app on multiple platforms, then you can save a lot of time and effort by standardizing and unifying those platforms under the same "release umbrella" of tools and processes. A release manager should be able to unite the processes for releasing those apps so that each team is following an interchangeable process and using the same tooling to do it.
What results should you see in a mobile release engineering team’s work?
A mobile release engineer shouldn’t just support the work of releases, they should actively work towards improving that process and make the overall release experience better as the org progressively grows.
What does this mean?
- Significantly less manual work for everyone. With a dedicated release engineering team, things like generating and downloading test builds for a particular version can be as easy as pressing a button.
- Far more productive feature teams. Without all the manual work, engineers can focus on what they like best: building and shipping features. The more efficient and automated your release process is, the more builds you can get into your users' hands.
- Regular and consistent releases. Streamlined, predictable, and consistent releases on a steady cadence (like an every one or two weeks release train)Â are a sign of a functioning and productive release engineering team.
- Constant improvements and learnings. There’s always room to improve the process of getting updates out the door. And in the cases where something goes wrong with a release, there should always be learnings that can be applied towards continuing to improve and increase confidence in mobile releases.
Conclusion
As we've covered, mobile release engineering is an important component of large scale teams. As the operation becomes larger and larger, having mobile developers double-time as pseudo release engineers becomes infeasible, leading to the necessity of either a dedicated team focused on making this process bigger, better, and easier for everyone or significantly increased infrastructure support that provides mobile engineers with the tools they need to greatly lessen the effort it takes to manage their releases.
The "necessity" part is very important to keep in mind here. Whenever I mention something that is common in top tier companies, I always like to stress that there's a reason they have such setups: they experience problems that demand such solutions, often because of their sheer size. If your company does not experience those same problems, then you will probably gain nothing by replicating their setup!
Thus, instead of simply copying what larger and more successful companies are doing (thus preemptively solving problems that you don't actually have), consider focusing on the problems that you truly have. As noted above you can often solve problems caused by ad hoc workflow automations, the added complexity brought about by additional stakeholders in a growing org, and a too widely distributed workflow toolchain by improving the infrastructure and workflows your team is currently using. Then once you reach the size where your release process starts becoming pressured, then I hope the knowledge in this article can help you get started towards implementing a better solution.
‍
