How to Measure DevOps Success to Keep Winning Support from Top Management

Why Measure DevOps Success?

You cannot have what you cannot measure. It all starts with the balanced scorecard of a company. Perhaps business measurement of success at organizational layer is a faster time to market, better customer experience, and reduced operating expenses. These are some typical key performance indicators (KPIs) in business but how about in DevOps at project level?

The sole reason why the DevOps business case has been approved in the first place is to support the organization-level KPIs with project-level technology KPIs / DevOps KPI, e.g., faster deployment time, release frequency, and deployment downtime. Using project-level KPIs, during business reviews, the DevOps executive would be able to quantify success and map it to the balanced scorecard at the business level KPIs.

Once DevOps KPI metrics are identified and measured, they can then be converted to a dollar value. Make no mistake, DevOps involves technology, time, resources, and risks tolerance; therefore, the dollar value of the KPI should outweigh the investments. Though we will discuss these in detail later in the blog. Let me share with you what I mean.

As you can see in the image below, one KPI of the organization is Time to Market, typical for an online transactional product company. Some DevOps KPIs that directly correlate to that said scorecard are deployment speed, deployment downtime, and release frequency. The DevOps Team can then measure these KPIs, like a 15-minute deployment time, 5 minutes deployment downtime, 90 releases per month. DevOps executives can then convert these measurements to a dollar value by calculating the difference before vs after implementing DevOps. The delta value can then be multiplied by the revenue per hour of the company e.g. 10K USD.

Assuming it resulted to 3.4M USD, that is the dollar value of the DevOps KPI. It is classified as cost savings since it could have been an opportunity loss due to delays in the absence of DevOps. See sample report below.

What are the different categories to measure DevOps success?

To identify the categories of measurement of DevOps success one has to look at the IT supply value chain. IT supply value chain is defined as the pathway and stages that a product feature has to go through to reach a customer. See typical IT supply value chain in image below.

Let us describe each category of DevOps success measures derived from IT supply value chain. The deep dive of each category is called DevOps KPI, which we will discuss in the later part of this article.

1. Plan - mainly delivery, project and product related success indicator.

2. Develop - Measurement of success related to coding and how well developers are doing.

3. Build - This is a measurement of packaging and preparing the product for delivery.

4. Testing - A category use to measure the quality of product.

5. Release - The category to measure for the quality of each product release.

6. Deploy - A measurement of the speed of delivery once the product is ready to production.

7. Operate - A measurement of DevOps success once the product is running in production.

8. Monitor - The measurement of product or service reliability or availability.

What are the detailed DevOps KPI for each categories?

Do note that these are just some of the key KPIs for DevOps. There are more beyond these. I just wanted to share the things I think are most relevant.

1. Plan DevOps KPI's

2. Develop DevOps KPI

3. Build

4. Testing

5. Release

6. Deploy

Before proceeding, I'd like to differentiate between release and deployment. A release is a major set of changes or features being sent to production. Theoretically, one release should equate to one deployment. However, most of the time there is a need to send a release in multiple deployments to production due to mistakes, hot fixes, or urgent configuration needed during the release. Bug fixes or small changes after the release are classified as deployments.

7. Operate

8. Monitor

How to translate the DevOps metrics to dollar value?

To further explain how to calculate the dollar equivalent of each metrics. Let us use Time to Deploy and Deployment Frequency as an example. These KPI can be equated to cost savings because for every delays would cost an enterprise certain amount of opportunity loss. The formula will be as follows.

Cost Savings(Dollar Value) = [Deployment Time(before DevOps) - Deployment Time(After DevOps)] X Revenue Per unit of time * Monthly Frequency

This would then result in a report similar to the image below. Please note that the values here are hypothetical. Replace them with your actual business figures.

Therefore, applying the formula mentioned above would lead to cost savings. The concept works to the rest of the DevOps KPIs. Having the formula for each KPI dollar equivalent is beyond the scope of this article. The point here is that each DevOps KPI can be measured in dollar value to prove the ROI of investments to stakeholders. It can then be mapped to the corporate balanced scorecard as discussed earlier in this article.

Lastly; I'd like to share a tool that may not be related to DevOps KPI but is very useful to identify cost savings in the infrastructure layer of Kubernetes Cluster running in cloud. The tool is called Cloud Health and SysDig. both has cost optimization that yields to cost savings if recommendations from the tool are followed properly. DevOps champions can use these tools as part of DevOps cost savings strategy to improve numbers during top management presentations.