Bugs are a normal, unavoidable part of the development process. They’re usually not that big of a deal, until suddenly one random Wednesday at 3:37pm there’s a major issue breaking your app for 50% of customers and now it’s a very big deal and all other work is forced into a holding pattern until this one problem is tracked down and fixed. This is not only stressful, but expensive: bugs found after release are up to 100x more costly in both time and resources than those taken care of beforehand.
This is why pre-release build distribution is important.
Lofty KPIs are set for customer usage of a new feature but, despite how cool and useful it is, the way in which the feature is connected to other parts of the app leads to only 10% of users trying it out after launch. As time goes by, this number barely budges no matter how many tweets and emails are sent out to tell customers that you are “excited to announce!!!” the feature. There is a fundamental flaw with how the feature is presented, but you don’t have time to dig into what it is as leadership wants you working on a new project.
This is why pre-release build distribution is important.
Users who do take advantage of the feature are generally happy, but begin discovering every possible flaw and limitation with it: “I wish it did this!” and “This is getting there but it’s not quite the right fit for us yet.” and “It solved this problem for us, but created this new one.”
This is why pre-release build distribution is important.
There is no combination of planning, design, and test automation that can absolutely perfect your product. All of these things are very important, of course, but there are too many ways to tap and scroll and interact with the screen for unforeseen questions, issues, and ideas not to arise.
The Shift Left testing approach in Mobile DevOps
It’s for the above reasons that the Shift Left movement has been gaining ground in recent years as organizations seek to move all forms of testing as early in the development process as possible.
This allows teams to:
- Catch and fix bugs before they gain the potential to become a much larger source of pain and frustration.
- Put usability to the test and collaboratively explore design and UX.
- Root out problems with the delivery process to make the release of the next feature even smoother.
- Get early feedback on features to tighten things up and make improvements before a larger audience gets their hands on the app.
We’re not necessarily the kind of company to give much credence to buzzwords, but we’re making note of Shift Left because its underlying principle is meaningful. And so you confidently claim you’re now at the forefront of a thought leadership movement. That way when company leadership reaches out to ask why you’re making such a big to-do about improving your build distribution process, you can just say “we’re Shifting Left!” without having to explain anything. They’ll say “hmmm” while secretly being upset they hadn’t already heard of this, and will then read a Gartner whitepaper on the topic while nodding thoughtfully to themselves.
Challenges with mobile build distribution
Distributing your mobile app internally for earlier testing is complicated by the primary factor that we’ve already been discussing: humans. Humans need a way to install builds, require guidance on how to report bugs and give feedback, and need the right context on what you’re updating and changing with each build so they’re not spinning their wheels testing the wrong things.
You’ll want to ensure easy, uncomplicated distribution to:
- QA, so they can catch and report bugs as early as possible.
- Design/UX, so they can put usability to the test and ensure designs look and feel just as good in action as they do in Figma.
- Engineering, so they can test out code during code reviews and provide better feedback.
- Product, so they can give early feedback on new features.
Getting your pre-release mobile app versions into testers’ hands
This means generating installable builds that the testers we mentioned above can actually put on their own mobile devices without having to do a lot of work from their side. What is done fairly easily via the App Store and Play Store when you release your app to the general public is done less easily pre-release. You have several factors to consider:
- Build installation and tester experience - You’ll need a tool for this, one which generates links that authorized users can click to install the app and that ideally also notifies them of a new build via chat, email, or even a native app through which installation is handled.
- Code signing and provisioning restrictions - Apple restricts the distribution of builds outside of the App Store and Testflight, so you'll need to have a plan in place for managing different types of provisioning profiles and signing certificates for test builds. Android apps luckily don't have this problem as Android devices can install signed apks from any source.
- Tester limits and external review - App distribution platforms sometimes impose caps on how many testers you can distribute builds to, and the size of these limits can vary depending on flavor of build.
Ensuring the right builds are being tested, with the right context
You’re probably iterating just as quickly as testers are testing; you don’t want to waste time hearing feedback from people who’ve been playing around with out-of-date builds from three weeks ago and they likely don’t want to waste time sending you that feedback.
On top of this, you’ll likely be sharing many types of build flavors: feature builds coming from different product teams and variants of your own builds that are to be tested for a variety of reasons (pre-release regression testing, debugging, usability, etc).
If you’re putting all these builds in front of testers without useful context detailing what makes one build distinct from another, they’ll get confused and stop providing useful feedback or even stop paying attention to new builds entirely.
Gathering feedback from testers
Your users are likely not all software developers, or even all that knowledgeable about tech; the more steps they have to follow to get feedback to you, the less likely you’ll receive actionable feedback from them. You can’t just send out lists of builds and ask that they email you with their thoughts, then sit back and watch the actionable information roll in. Well, you can do that since you can do anything you want, but just don’t expect it to work out. The less detailed your instructions and the more work people have to do on their side to get feedback over to you, the less feedback you’re gonna get and the fewer people who are going to stick around as active testers.
The input is not all that matters either, as the output of this information needs to get in the hands of your team in a usable format. Collecting feedback in a form whose entries only occasionally get looked at isn’t going to lead to many fixed bugs or informed improvements to your app.
Integrating your mobile build distribution process with your current tooling
Testers aren’t the only people you have to think about. You also need to make build distribution as easy as possible for both you and your team. If you’re having to manually tinker with spinning up every other test build that goes out the door, then you’re not going to get anywhere near as much value as you would if you’re able to set something up to generate and distribute builds (mostly) automatically, so sharing different kinds of builds with the right groups of people is seamless and built right into the tools your team is already using.
Setting up the best possible mobile build distribution process
Considering the various points of friction that work against getting your app in the hands of the right people across your team, gathering useful feedback, and then taking action on any of that feedback your team deems relevant, how can you set yourself up for success with the distribution of your own mobile builds?
Continuously deliver builds
Ensure your CI/CD pipeline is at the very least set up to generate builds when code is merged to the development branch or, even better, get your environment set up in such a way that you can also build and distribute off any team member’s work-in-progress branch on demand. This way you don’t have to restrict yourself to testing a series of linear builds, but can have builds all throughout the development process — alpha builds, feature complete builds, beta builds, and release candidates (all across the various teams in your mobile org) going out for regular testing without complication.
Contextualize each and every mobile build
If you’re not thoughtful about how and when and where you distribute your different kinds of builds, you’ll end up presenting your testers with a big, incomprehensible list. Even if you’re following a fairly simple approach and just doing night builds, how is anyone supposed to know what to do when presented with a list of builds like this:
- 3.3.11.5
- 3.3.11.2
- 3.3.11.0
- 3.3.10.8
- 3.3.10.7
- 3.3.10.4
- 3.3.10.3
- 3.3.10.0
Most people would probably open the top build to test as it’s the most recent, but is that what you want them to do? If it is, what if they just started testing an earlier build an hour before the most recent one became available? How do they know what they should be looking for in the build and what’s new in it compared to the previous build?
Label builds clearly and succinctly, noting what makes each build different from all the other ones around it, and make it clear which ones are release ready, which ones are still in an alpha stage, which ones are in a beta stage, and which ones need to be explicitly debugged. Provide detailed tester notes that clearly describe what changes are included in the build — if you can, avoid just throwing together a list of commits, which less technical folks might have trouble understanding. Putting some effort into providing the right context around internal builds empowers your team to distribute lots of different build flavors without causing chaos for yourselves and testers.
Most build distribution platforms don’t support this level of bucketing and categorization. But we know of at least one that does: Build Distro.
Choose the best mobile distribution platform
There are many of them: Firebase, App Center, TestFlight, Build Distro (this is by far our favorite, both because it’s the best one and also because we built it) and several others. Take time to consider how easy each platform makes it for your team to get builds in front of the right users, provide those users with context around each build, and then help support you in collecting feedback.
We recently wrote a post comparing the major build distribution platforms. It’s worth a read before you choose one.
Provide very clear instructions for how internal testers can get started
Even though you’re asking colleagues to be testers, don’t assume they’ll all know what they’re doing. Every instance of friction in the process will cause some people to drop off.
Everyone will want to know:
- How to join and get access to their first build.
- How to handle any of the more technical parts of the process (installing Enterprise certificates on iOS, enabling app installation from source on Android).
- How often they should be on the lookout for updated builds.
- What they should look for when testing. If this is likely to change from build-to-build then that information needs to be included with each build.
- How to provide feedback. Not just where they provide it, but how they should structure it.
- How they should write feedback. Freeform feedback will be harder for most people to write, and also harder for you to parse.
- How to opt out if they want to stop.
- If they get anything special for testing, other than the satisfaction that they’re helping their own company (which may or may not be enough).
Write a short guide or FAQ, and provide that to everyone who signs up. Host a monthly live session where testers can watch someone from the team go through the onboarding process and then ask follow-up questions. Record a video (perhaps even a video of that live session) that people can watch on demand. Even do all three to cover your bases.
Create an actionable process for triaging feedback
None of this testing and feedback is valuable if your team doesn’t put it to use. If comments go into a Google Doc and no one ever looks at them, do they actually provide any value? The answer to that question is no, they don’t.
Tie builds back to the work that created them; you should always be able to tell which branch a build was triggered from and which commits contributed to that branch so you’re not constantly playing a game of Clue connecting feedback to work. Determine an ownership structure for which teams or individuals will take action on feedback for which types builds, and if you can automate a system that assigns feedback to a Jira project (or a Linear issue if you work somewhere that doesn’t force you to use technology from 2004) or to a specific product manager — even better.
Shift Left and invest in your mobile build distribution process
It takes real effort to set up, grow, and maintain an internal testing program, but the benefits are well worth it. There is no better way to:
- Find and fix bugs ahead of launch.
- Refine a new feature before most users see it.
- Exercise your continuous delivery muscles and put them to good use.
- Break down silos by engaging the entire company in features from across teams.
- Get the org excited about all the new features you’re continually shipping.
Establishing an internal testing program is also the first step to launching an external beta testing program. Fresh eyes and lack of encyclopedic product knowledge can come in handy for new ideas, bug discovery, and to drum up excitement with your users ahead of a big feature launch. What happens when numerous testers with less familiarity with your product and on a wider variety of devices take actions you were not necessarily expecting? Only one way to find out.
If you don’t set up a build distribution process, each of the above things will take far more work to individually achieve or simply won’t happen at all. If you do, well, the opposite will happen and that will have a positive impact on both your team and business.
