No matter what kind of app you’re building, you have customers of some kind. Whether you’re B2C or B2B or C2C or C2B (somehow) you want to build an app that those customers find useful and genuinely enjoyable that is also mostly free of bugs and crashes. Beta testing provides an opportunity to find usability problems, get ahead of possible performance issues, gather early external feedback on new features, and even build excitement for upcoming releases among your user base. Plus, there is no better way to ensure each new feature you add is its best, bug-free self than by giving some of these customers a chance to try things out before the new feature is officially released.
Building out such a program requires almost as much thought as building your app. How will you get people to sign up? How will they send you feedback? What will you do with that feedback once you have it?
How can you start running a great beta program of your own?
Decide what kind of beta program you want to run
Maybe you have a huge new feature coming out soon and want to make sure select customers can bang around on it ahead of release to give feedback on its usability and usefulness while ensuring as many bugs as possible are caught before the greater public gets the opportunity to experience them.
Or maybe you want to continually test each release from now until the sun explodes or you’re no long working on the app or there is a cataclysmic event unrelated to the sun exploding, whichever comes first — ensuring that several sets of customer eyes and hands test out everything you’re shipping, no matter how big or small your changes might be.
You might even want to do a little of both, keeping a regular program going with an expanded program for big features that you want to hear more feedback on, bringing in more or even just specific types of customers for different situations.
How do you actually decide which approach you’re going to take?
Figure out what you want to get from your beta testing program
If you don’t know what you want then you’re just gonna have a pile of feedback that doesn’t provide any value whatsoever. That’ll be frustrating for you, for your team, and especially for the customers who provided feedback and don’t see any results from it since it’s not the best experience to send bug reports or feature feedback into a dark void of nothingness.
Figuring out what you want doesn’t have to be a forever thing either. You can start with one sort of program and evolve it into something else, so long as you set expectations for everyone involved and have the right tools in place to collect the data and information you need.
Your goal could be to simply toss your beta builds of release candidates at users so you have an initial limited blast radius to ensure things don’t fall apart with normal everyday usage while collecting quantitative data from tooling and keeping a close eye on your app's performance to make sure crashes are within acceptable limits.
It could be to get very specific feedback — that requires more thoughtfulness on behalf of the beta testers — on functionality and design. A more involved process that means ensuring testers are empowered to easily provide qualitative feedback and that engineering teams are enabled to take action on it.
Or, as with the section above, you might end up doing a little of both. What matters here is that you determine if qualitative feedback directly from testers or if quantitative feedback (some of which comes from testers, but most of which likely comes from the same analytics tools you use with the public app) is what you need most.
Make sure expectations are set with beta testers too. If you’re essentially running an early access program, make it clear to them what it means and what you’re asking for from them. If you need more feedback, tell them up front with clear CTAs on how they should provide that feedback. Provide deadlines for when their feedback is required so they don’t save up all their comments until it’s too late. Ideally you’re also communicating expectations alongside every build you send them so that they know what to do. Even if expectations are the same, it can’t hurt to repeat them.
Choose a distribution platform for sending builds out to testers
If you were a web developer, running your beta program wouldn’t be particularly complicated. You could just put a beta version of your app up at a beta subdomain then seamlessly ship updates whenever you feel like it. But, as you know, mobile includes the added complexity of needing to distribute builds to your customers (who may not be particularly technical) ensure they know how to install them, and make sure they’re looking at the right build candidate for a beta release at any given time. Some things to consider when choosing a beta distribution platform:
- Build installation and tester experience – You’ll need to establish a process for onboarding your new testers and set things up to notify them of a new beta build via chat, email, or even a native app through which installation is handled.
- Code signing and provisioning restrictions – Apple restricts the distribution of builds outside of the App Store and TestFlight, so if you choose to leave the Apple ecosystem for distribution, you'll need to have a plan in place for managing beta build code signing and provisioning. Android devices can install signed APKs from any source.
- Tester limits and external review – App distribution platforms sometimes impose caps on how many testers you can distribute builds to, and the size of these limits can vary depending on flavor of build. And distributing beta builds to larger groups of testers requires that builds go through a review process for both Apple and Google’s beta distribution platforms.
Getting into the difference between all the various build distribution tools could take up an entire blog post, which is exactly why we recently wrote a detailed post digging into the differences between tools like TestFlight, Google Play Console’s testing tracks, and our Build Distro platform. Give that previous post a read when you’re ready to choose.
Gather feedback
Even if you take the “limited blast radius” approach to beta testing, you’ll still want and need to hear from testers. The more you can prompt them to provide feedback, the more feedback you’re going to get, while the more steps that take them away from your app and into a freeform braindump (email us here with your thoughts!) the less you’re going to get.
- Provide them a way to give feedback in the app itself. This can be a little feedback icon that appears in the menu in the beta version that they can access at will, a link to a feedback form that magically pops up in the app once a day, or a form that appears after they encounter a crash.
- Ask them via email. Same concept, but taking them one step away from the product and so putting them in a position where they’re a little less likely to take action.
When possible, it’s a good idea to pair written feedback with session replay (available via tools like LogRocket and Sentry) that will show you exactly what happened during their described time in the app or when they ran into an error. That way you’re not just taking their word for it.
While you might provide them a place for freeform feedback, you should also give them writing prompts so they have a starting place. This can be as simple as asking them questions about what they encountered: did this work the way you expected? If not, what were you expecting? If yes, do you like that it works this way? Is there anything else you wish this feature did?
Beyond written feedback, in some cases it can be valuable to interview users about their experience and have them walk you through what they’re doing in the app. You obviously can't interview every single person (unless you've limited the scopt of your beta to a small number of people) but if you really want to hear what, say, power users think of a new feature, having a conversation about it is probably going to get you a lot more information than a written response.
Act on this feedback
No matter what you do, gather at least some of your feedback in public. Let users see what other people are saying and potentially give them the opportunity to add their own upvotes and additional comments. Tools like Productboard and Canny provide features allowing you to do exactly this.
Ensure feedback is logged not just where customers see it, but also where your team is likely to see it, read it, and prioritize it. If most of your work is in Jira or Linear, but the feedback isn’t consistently logged there, do you think it’ll actually get worked on or will other priorities push it aside? Not everyone in the engineering or product org is going to have access or even be clued in to what’s going on with the beta program, so they may not even see this feedback otherwise. Having a project or product manager keeping an eye on feedback and ensuring it’s always prioritized is helpful.
Triage feedback on a regular basis and update users when you’ve made a fix or change. Seeing that their feedback has an impact is probably the best way to encourage testers to continue testing. This could even be tied into the public feedback board described above, by simply having a shipped feedback tab of some kind.
Build beta testing into your release schedule. If you’re releasing every two weeks, then time for testing, gathering feedback, and taking action must be baked into this release cycle. At Runway, we refer to this as the beta “soaking” period, a set (or even open-ended) time frame during which the release candidate must be tested before the next steps in the release can continue.
Reward testers for their efforts
For many folks, helping you make a better app may be the only reward they’ll ever need — which is why it’s so important to take their feedback seriously and show them you’re taking action on it — but this won’t be true for everyone and it never hurts to provide incentives for both participation and feedback.
What could you give away?
- Cool swag – Why reserve your best swag only for conference sponsorships and employees? Make some sort of small thank you package with swag or even create beta program-specific swag that celebrates the length of time they’ve been in the program or their taking part in testing different releases (imagine a patch or enamel pin made to commemorate a specific release or the amount of builds they’ve tested).
- Product discounts – Anything you sell through your app is undoubtedly already priced perfectly, but giving testers a regular 20% discount or a credit could be very compelling.
- Gift cards – Luring people in with an Amazon or Apple gift card is a classic approach for a reason. Could feel a little spammy at this point, though, since so many companies do the same thing and it lacks the personal touch of items related to your own org. And it can get expensive fast. Doesn’t mean it won’t work though.
- Early access program – Just call your beta program an “early access program” and make the entire thing sound like a reward from the moment of sign-up.
Virtually any items humans have been known to like can be used here; just make sure you can scale the giveaway to whatever the size of your beta program is going to be and that rewards are delivered logically and consistently. Anything else will leave some testers feeling upset.
Taking the right steps when starting your beta program can have a huge impact on its success (and the success of your app)
A successful, impactful beta program will find the right balance of engaging more than enough users for the quantitative feedback you need, while also empowering them to provide written qualitative feedback that provides juicy details that can help you make improvements to the current release candidate as well as future variations to come. It will allow your app to see fewer bugs at launch, give you a clearer understanding of your customer’s needs, and increase loyalty from users who will feel like they’re getting a peek behind the scenes and a chance to help shape the future of your org.
Next step? Write a plan for what you’d like to accomplish with your program and then select the best beta app distribution platform for that plan.
