How Cleo uses Runway to modernize and democratize their release process.

Everyone has that friend — the one that can make you laugh, or that knows when you need some tough love but always has your back. Meet Cleo, the AI chatbot (and the team behind “her”) that’s taking a fresh approach to the often alien and opaque realm of money management. Instead of the usual deep breaths and cringing that comes with opening other financial services’ apps, Cleo’s 2.6 million users across the world stay engaged because she manages to make saving and monitoring a budget painless and — dare we say it — fun.

Cleo first arrived on Facebook Messenger in 2016, with her trademark humor and sass. To start with, Messenger proved itself to be a competent medium for the core chatbot experience. Still, as Cleo’s user base grew, the product’s larger potential became increasingly clear and the team understood that they’d need to extricate themselves from the limitations of the Messenger platform sooner rather than later. They didn’t want to continue subjecting their business to the risks of needing to conform to Facebook’s rules, and in any event they knew they had much bigger ambitions for Cleo and the service around her than a Messenger chat thread and some webviews would accommodate in time.

Enter Cleo’s React Native mobile app, which was a natural next step for the three frontend developers who were already working in React. With the mobile app, they had the freedom to build a much more engaging interface and experience beyond just chat — the full Cleo, as it were. The service has been app-only for two years now, and today Cleo’s scaled-up (and growing!) mobile team is laser-focused on continually improving retention and engagement, and running effective multivariate testing — all while releasing weekly and learning at a breakneck pace.

Challenge

Putting up with more of a release grind than the team even realized

“We weren't quite confident about best practices when we first set up our mobile release process. On top of that, even though our original team knew all of its ins and outs and quirks, onboarding new people to the process was a pain. Steps and nuances that experienced folks knew how to navigate would trip others up, wasting hours. So, we really started feeling the pain of releasing when we started scaling the team.”

Oli Bates
Tech Lead
‍

When Cleo’s core group of React frontend developers originally decided to expand their duties to React Native to spin up the mobile app, they had less experience than they would have liked on exactly how to package and ship an app to the stores as a distributed team. So, they threw together a release process with the help of a couple of temporary contract engineers. The team was pretty sure that this cobbled-together process needed some major cleanup in terms of best practices, but it was “good enough” for a while. Still, team growth was gradually moving things towards a breaking point.

As Cleo started hiring and onboarding multiple new engineers to work on the app, they soon realized that their particular learned release routine was hard to teach and hand over to others. It was easier to continue having the tenured engineers wrangle releases than to democratize the process, but this undue burden wasn’t a good use of resources.

“Releases used to be a super manual process, and you’d have to bounce around with half a dozen tabs open — App Center open, App Store open, Play Store open... We actually didn’t realize how irritating it was and how many steps of work there were (and which we could potentially get wrong!), because we were mindlessly just putting up with it all.”

Mick Horler
Senior Frontend Developer
‍

New hires aside, release management would sometimes eat up three or four hours of the team’s time, and they really felt that inefficiency and drain every time they dealt with an unanticipated bug or sensed an impending release schedule milestone coming up. Soon the team even started to skip releases outright, stalling until a particular product squad was desperate to finally push a given feature out the door.

Throughout the years, Cleo’s engineering team had managed to stay pretty flat — everyone wanted to work on improving the product, so there was no dedicated platform or release team, by design. Even as the pain of scaling releases grew close to a breaking point, the team wanted to solve the problem and get back to work on the app itself instead of dedicating time and energy to managing better releases.

Solution

Runway as an encapsulation of best practice, with guardrails and automations to support rapid releases

Just as the Cleo app team was starting to take real notice of their releases problem, Oli, a tech lead on the team, came across Runway. At first, the thought was that their issues weren’t big enough to merit adding a new tool, and the team continued with the status quo. But a few months and several hires later, Oli sought out Runway again once it became clear that Cleo needed to get more out of their expanding team — both in onboarding new engineers to spread release ownership and responsibility around, and in adopting a more efficient release process in the first place.

The team quickly saw that incorporating Runway into their toolchain brought a number of short-term wins. Cleo finally had a platform with guardrails that encouraged and led them towards best practices for all aspects of their mobile release process, and the team immediately set to work making related process improvements to modernize their approach.

“Runway encapsulates best practices for releasing apps, and helped us to adapt our own process to the industry standard.”

Oli Bates
Tech Lead
‍

Ultimately, Runway finally gave Cleo one place to document, monitor and collaborate on all aspects of the release process, as a team. It provided an intuitive interface for understanding progress at each step, brought to light any blockers along the way, and kept all stakeholders continuously apprised on Slack with timely notifications. Engineers who had been with Cleo for years loved the newfound transparency, and naturally found that onboarding new developers to the release process was now a breeze.

“Runway makes a huge difference. Anyone on the team can run a release now, with very little onboarding — it’s that easy!”

Kasia Jastrzębska
Senior Frontend Developer
‍

Soon after adopting Runway, Cleo found that releases themselves became much less of an event, with Runway providing both key automations and a single source of truth to instill confidence and de-risk the process. Cleo’s engineers now felt empowered to release early and often, knowing that Runway would make the affair pain-free, predictable, and repeatable – a far cry from earlier days of avoiding releases unless absolutely necessary.

“Before Runway, we were always like, ‘Oh f***, we’ve got to release and do all this stuff’... We dreaded releases and everything was a bit flaky. Now, we’re way more confident in releasing. We release weekly at a minimum, and it doesn’t have to be this big production anymore. If someone ships a bug, they can quickly release a fix themselves, even mid-cycle.”

Mick Horler
Senior Frontend Developer
‍

Results

No more “release dread”: less cognitive load, less time wasted, more confidence

Oli had calculated that, prior to Runway, the Cleo team was spending 3-4 hours per release purely on managing and shepherding the process. Today, Runway allows the team to take those hours saved and invest them in the product and in more impactful areas of the release process – for example, in more rigorous regression testing, which has consistently resulted in more stable and bug-free releases.

As a bonus, Runway is increasingly becoming Cleo’s hub for more and more post-release monitoring needs, as key information pulled from their Bugsnag integration in Runway is presented within the same interface where they’re already running and tracking each release. 

“Our exploration of Runway started on the release management side of things, but we’ve found the monitoring features to be just as valuable.”

Oli Bates
Tech Lead
‍

Today, with the help of Runway, Cleo’s team feels newly unleashed to build an app experience that serves as the beating heart of their business. All told, the efficiency gains that Runway has delivered have been a game-changer for Cleo. Onboarding new engineers and empowering them to run releases is now as easy as signing them into Runway and having them follow along in the interface. Releases have become a non-event, making Cleo’s speedy weekly release cadence less daunting, and freeing the team up to iterate on the product and learn continuously. And the team knows that as they continue to scale to meet tomorrow’s objectives, Runway will scale right along with them.

“Runway has reduced the cognitive load for new team members getting onboarded, and it’s even made things easier for those of us who have been at Cleo since the start — we hadn’t realized what a pain in the ass releases had become.”

Mick Horler
Senior Frontend Developer
‍