In our last post, we took a close look at a handful of key metrics — change lead time, cycle time, release frequency, failure rate, and time to recovery — that can help mobile teams get a better understanding of how well their team’s dev and release process is functioning. But the real value of measuring any metric comes from the insights that can be drawn from them — and how you apply these insights to actually improve the thing being measured. In this post, we’ll explore how to leverage Mobile DevOps metrics to make meaningful improvements to your team’s DevOps practice.
First, invest in measuring metrics the right way
Pulling together the data needed to compute Mobile DevOps metrics can be extremely challenging — and absolutely crucial to get right. The quality of your metrics will directly affect the quality of the insights you can draw from them and your confidence therein. Even slight differences between teams’ processes can influence how DevOps metrics are calculated: for example, consider the definition of release failure used for the failure rate metric, which can take into account app stability, critical-path conversion rates, or even user reviews. How your team chooses to define release failure impacts how the failure rate metric is measured, so there’s no one-size-fits-all approach. DevOps metrics are inherently difficult to measure across any stack, but as we touched on in our last piece, mobile platforms face some unique challenges that make measuring meaningful DevOps metrics even more complicated.
Offloading as much of the data collection and computation as possible to existing tools is a good way to produce reliable metrics without significant extra effort on the part of your team. And while there are some solid options out there (both dedicated SaaS solutions, and even open source options), they’re typically not designed to accommodate the unique aspects of the mobile use case. As mobile practitioners ourselves, the difficulty we’ve encountered in finding a reliable way to correctly measure DevOps metrics for mobile was one of the motivations for offering core Mobile DevOps metrics like release failure rate, time to recovery, and cycle times as part of Runway out of the box.
Whether you choose to invest resources into measuring metrics internally, or choose to leverage existing tools, feeling confident in the accuracy of your metrics is the first step towards being able to draw actionable insights from them — because there’s nothing worse than building on bad data!
Understand the context
DevOps metrics offer a measure of the outcome of your team’s DevOps practice, and they can be a powerful tool to surface potential problem areas in your process. But in order to gain actionable insights from your team’s DevOps metrics, it’s important to understand the factors that contribute to specific metrics, and their surrounding context. For example, a team with a “long” lead time can’t simply resolve to reduce lead times without understanding that factors like a lack of automated testing, a long stakeholder approvals process for releases, or intentionally long beta soak periods will all affect average lead times — and not all need to necessarily be considered potential areas to improve. What one team would consider to be a “long” lead time might be an aspirational goal for another team whose process and priorities differ.
For example, consider a team where product quality is of utmost importance – to ensure releases are free of regressions, their process involves a nightly suite of automated tests followed by a week-long soak period, resulting in lead times in the range of two weeks. Now consider an early stage startup whose priority is shipping new features and updates quickly in order to iterate and reach product market fit. They have no automated testing in place, and no formal beta testing program. The stability of their updates is much lower than that of the team focused on quality, but their lead times are in the range of only a few days. One might look at the two-week lead time of the first team and, without the extra context, think that this team should make reducing lead times a priority. But by understanding the extra context around the specific needs and priorities of the team, it becomes clear that their long lead times are a product of intentional trade-offs this team has made in order to accomplish their organizational goals.
While DevOps metrics can certainly surface potential areas of concern, it’s up to each individual team to do the work required to understand the underlying context and draw insights and improvement plans based on their goals and those of the organization.
Create a feedback loop — use DevOps metrics to validate the impact of any process improvements
Measuring the impact of any process improvements your team chooses to invest in is hugely valuable, because it enables teams to take a data-driven approach to their own processes — something that’s often fluid and difficult to quantify. The best way to approach process improvements is to do so incrementally, and to continuously validate the impact of any changes by tracking how they affect specific metrics. For example, a team looking to tighten up their incident management process might look closely at the time to recovery metric as they adopt an automated incident notification tool like PagerDuty. As your team continues to invest in process improvements, you may even consider using DevOps metrics to set short- or long-term goals for your org as a way to keep your team accountable to continuously improving how you build and ship products. Quantifying how improvements have moved the needle in a concrete way can also help to win leadership buy-in for additional improvements if necessary.
Consider how metrics affect each other
If your team is considering using DevOps metrics to set goals for your org, it’s important to consider dependencies between metrics in a holistic way. For example, targeting release frequency without considering the potential impact on failure rate (and even cycle time) could have some undesired consequences; maybe you succeed in increasing release frequency, but at the cost of increased failure rate (is code that’s shipped faster also less robustly tested?). Perhaps even cycle time is affected (does development slow when the culture shifts to more frequent releases that have less time to be beta tested prior to release?). Certain tradeoffs may be worth it for the goals of your team and organization, but it’s important to be aware of how focusing on any specific metric might have downstream effects on other parts of your team’s process.
Create visibility around DevOps metrics and track them over time
Establishing a good process isn’t a one-and-done task — it requires discipline and continuous improvement. Similarly, monitoring and evaluating DevOps metrics is an exercise that should happen continuously over time — and should be the collective responsibility of the entire mobile org. Tracking DevOps metrics as they change over time, and making metrics and insights easily accessible to the entire team can help create a sense of ownership and accountability, while also helping to keep DevOps top of mind.
Evolving your process strategically with Mobile DevOps metrics
Investing in properly defining, collecting, and tracking Mobile DevOps metrics over time can be a powerful way to measure how efficiently your mobile org is functioning. And while defining and collecting the necessary data to compute key metrics can be one of the most difficult parts of getting started with Mobile DevOps metrics, leaning on existing tools can help overcome this hurdle. It’s only by putting in place reliable metrics that accurately measure important facets of your team’s process that your team can have the necessary confidence in the insights that are drawn from them.
Once DevOps metrics are spun up, making thoughtful incremental changes to parts of your process — and continuing to use metrics as a system for measuring the impact on these changes — can be a tactical and effective approach to making improvements. Finally, it’s important to remember to always keep metrics accessible — helping keep your team’s DevOps practice top of mind, while driving transparency and accountability for your entire mobile org.
