How to measure engineering productivity

Successful DevOps-driven companies looking to expand the scope of their business need to find efficient ways to scale their engineering organization’s productivity over time. Indeed, accelerated business growth often leads to increased communication challenges and, consequently, creates bottlenecks and inefficiencies in the software development and delivery processes.

Measuring engineering productivity helps organizations have a better understanding of how people work, what motivates and makes them productive, and take data-driven actions to improve engineering productivity and velocity as the business grows.

Why do we need to measure engineering productivity?

As companies enter a hyper-growth trajectory, measuring and improving software engineering productivity becomes vital not only to improve business outcomes but also to ensure the well-being and satisfaction of the engineering team. But despite its recognized importance, knowing how to measure engineering productivity efficiently remains inaccessible for many teams. From focusing on one metric to capture all aspects of engineering activity and performance to tracking a plethora of metrics, many organizations fail to get the full value of measuring engineering productivity. 

Effectively measuring and improving engineering productivity involves continuously gathering insights into what makes software engineers more productive, how they communicate and collaborate, finding performance bottlenecks in engineering processes, and taking data-informed steps to resolve the identified issues.

In order to successfully scale their business and continue to deliver higher-quality software at DevOps speed, executives and IT leaders need to get a holistic picture of how to define, measure, and improve engineering efficiency within their organizations. Despite broad studies on measuring engineering productivity, the industry has yet to establish a fixed set of metrics every organization should monitor; instead, teams should start by observing their specific processes and focus on using frameworks and adapting them to their organization.

How to measure and improve engineering productivity

Before jumping into measuring the productivity of engineers, it is worthwhile first to evaluate the potential value of monitoring a given metric. This preliminary step is essential to reduce the costs inherent to the measurement process (from the cost of people in charge of measuring, analyzing, and communicating the data to any slow down in production or change in the engineers’ behavior due to the observation). 

Common mistakes to avoid

A few common mistakes when approaching engineering productivity still prevail within the industry. First, organizations may try to find a single metric to assess the productivity of the whole engineering team. As a rule, it doesn’t work since various dimensions of work affect productivity, and a single metric won’t reflect the full picture. For example, checking baseline metrics such as the number of pull requests, commits, or how many code lines were written can be inconclusive and susceptible to errors when measured alone. Similarly, focusing on too many metrics may confuse teams and lower their motivation.

Tying metrics to rewards or outcomes could determine team members to look for ways to game it, which can eventually undermine the value of those metrics. Moreover, organizations may sometimes use a metric to measure the wrong performance indicator. For example, tracking agile velocity is a good metric to estimate a team’s capacity for sprints but not to measure performance or to compare teams as it may generate perverse incentives.

Choosing the right metrics

As previously mentioned, there is no industry standard for how to measure engineering productivity. And while several measurement techniques exist, organizations tend to define their own frameworks.

For example, GitLab defined its own KPIs around its goal to provide more value to customers and ultimately increase revenues by shipping monthly releases. The main metric used at GitLab is the Merge Request (MR). he purpose of this metric is to encourage team members to divide their work into smaller MRs, which is likely to result in faster reviews, quicker releases of new features, and improved quality of the codebase. However, GitLab acknowledges that not only does “one metric never tell the full story,” but it can also lead to biased results.

DevOps orchestration platform Cloudbees recommends the following three main performance metrics to start measuring engineering productivity:

• Average cycle time — the amount of time spent working on a task until it gets delivered to production;
• Time spent by focus area — such as bug fixing, refactoring, documenting;
• The ratio between time spent on planned work vs. unplanned work.

To solve the issue of measuring engineering productivity, teams at Google use the Goals/Signals/Metrics (or GSM) framework. Within this framework, “Goals” stands for the desired result, without any reference to a specific metric.

“Signals” refers to evidence that the desired outcome has been achieved and each goal should have one or more signals. Although it would be great to measure signals, not all can be. 

“Metrics” represents the measurable proxy of a signal. However, as not all signals are measurable, metrics may fail to capture all the desired signals. It is also worth using qualitative data to ensure that the selected metrics are telling the whole picture. Qualitative studies can help IT leaders determine what developers are focused on, why specific tools were used, and what events affected a team’s productivity. This allows organizations to see the bigger picture before making further decisions.

While working on the problem of measuring engineering productivity, researchers from GitHub, The University of Victoria, and Microsoft Research have introduced the SPACE framework. This method evaluates productivity as a complex interplay of many factors (satisfaction and well-being; performance; activity; communication and collaboration; and efficiency and flow) rather than focusing on one specific metric.

Tracking results and improving productivity

While tech companies are measuring engineering productivity differently, they all start with defining what the organization needs to track and tying those metrics to end goals (i.e., there should always be a decision that can be made based on data). Keeping track of these metrics should help ensure that teams work on the right tasks and perform quickly and efficiently. Ultimately, organizations want to take active steps to improve productivity based on measurements and track the results over time.