Do we stop trying to measure software development productivity?

Quite a few decades worth of thinking and efforts have gone into understanding how to measure productivity of the software development process. We have moved past the age of evaluating software development through lines of code, to counting stories completed to figuring out if output can be measured and attributed to each individual developer or development team.

More importantly, we now understand that the most important reason to measure development activity productivity or even individual developer/developer team productivity is to

• Accurately estimate project timelines and budgets
• Reward or review developer performance
• Figure out which processes and/or tools help or hinder faster time-to-market

There might be other reasons for measuring development or developer productivity, but even they can be attributed to one of these three reasons.

The oft-repeated adage that writing software is closer to art than it is to construct a building, is probably true. But the devil lies in the details. Not all software development is completely like painting a Mona Lisa.

It still needs to be said that on the spectrum of between painting the Sistine Chapel and construction of a 4 wall pre-fab home, writing software is still on the left of the midway point.

Most regular software development projects are to the left of the spectrum

I only say this in the context of the ability to estimate and predict.

Part of the reason why accurate project estimation, faster time-to-market, and developer reward mechanisms cannot be accurate is because of the creative aspects of writing code. There are in most cases, for most software development projects, there are a lot of variables which are hard to track, calculate and generate insights out of.

However, that is not true of ALL software projects. Let’s take developing a simple or mildly customized WordPress website as an example. I think it falls on the right of the spectrum example shown earlier. In the vast majority of WordPress website development projects, it is easier to predict and estimate budgets, development team productivity and get the project done by deadlines. I am sure there are more complex WordPress projects as well.

Most WordPress website projects falls towards the right of the spectrum

There are many software development projects which lie on the left most side of the spectrum.

Writing code which writes other code is a good example. Making the machines which make cars is a far more complex process than making the cars themselves. Building cars is fairly predictable, Ford will be able to predict within a few percentage points the number of cars they will be able to manufacture in any given month. Ignoring externalities of course.

Software has many closely related examples. Writing code which writes other code is a previously mentioned example, building platforms which helps you write better software or manage it is another great example.

Writing an IDE is far more complex than *most* code written using the IDE. I strongly believe this is true for more than 90% of the software written out there. There will be projects with a far more larger code base in most cases, but the complexity of the code used to build the IDE takes far more brainpower and creativity.

This is true with building platforms such as the Kubernetes project for example. Kubernetes will probably be close to the left most extreme of our *special* spectrum that we’ve developed.

This is just a representational image. The Kubernetes project would be even more towards the left than the image suggests.

In the software world, building Kubernetes is far more complex than a vast majority of what is built on/using Kubernetes. Exceptions in the hundreds would exist though, however with Kubernetes being used in thousands of projects my estimate of “vast majority” would still hold true for now.

We therefore know that a *vast majority* of code written by the *vast majority* of developers fall on the left of building things like Kubernetes.

The people who built Kubernetes are artists. It would be tough to put metrics and numbers on how productive the team is or was.

However, a vast majority of the software development industry strongly needs better insight into the development process than in the past, simply to improve project prediction capabilities, reduce the risk in not being able to accurately project timelines.

Even better, we would need better insight to understand which part of the development process hinders faster, accurate or efficient development spend, and which processes and tools help improve the process.

The amount of money being spent across the world in digitizing processes and transforming a vast majority of industries to work efficiently and effectively is mind-blowingly large.

I do not need to regurgitate analyst numbers for you to know how large an effort is being undertaken to build better processes and products using software, even before the current COVID crisis which only excarberated the need for more insight.

Just since writing software is more of an art and timelines for art cannot be predicted/estimated is a step in no direction. If no effort can be put in understanding how development team productivity can be measured we will be no closer to improving processes.

So start measuring and understanding your development processes better, understand why your organization can deploy once every three weeks while a competitor seems to be deploying new features once per week. Even better, figure out if you even need to deploy faster or would it be better for you to focus on quality of output.

Improving processes and better project cost and time estimates is another need of the hour IMHO.

Caveats exists, of course. Of the three reasons why developer/developer team productivity should be measured, the weakest reason is “ reward or review developer performance”. Individual developer performance is a tricky thing to understand and most of the metrics I’ve seen out there only provide weak indication or weak direction.

The fact that it is difficult to measure individual developer performance should not hinder organizations from considering measures and projects to understand what make the development process tick, how developer teams work and what is the idea-to-implementation timeline organization wide. Ignore individual productivity and focus on team and dev. org productivity.

So how do you do it?

The book “Accelerate” (by Nicole Forsgren, Jez Humble & Gene Kim) has shown us through rigorous statistical methods that connecting software delivery to business outcome is a good way of measuring your organization’s productivity.

Four metrics — Lead Time, Deployment Frequency, MTTR and Change Fail Percentage — helps you understand more about your development productivity than other metrics.

Those metrics help you understand how well you are doing relative to yourself in a similar period in the past, i.e. did you improve or did quality go down? They will not be able to help you completely understand why that is so. They are also just the start point of your improvement process.

It therefore still behooves you to understand through collection of data across the entire process and through the data generated by the various tools in the tool chain (test suite, Kanban boards, repo metrics and the many other tools in the tool chain) and match it with the outcomes to figure out which changes led to what outcomes.

At Syren we are in the process of figuring out what would help IT leaders make better judgments and predict budgets and timelines better. Syren’s Lucid platform is designed to help analyze development process productivity and provide actionable insight to engineering leads, CTO’s and Product Managers.

If you have an opinion on the matter I could be reached at sid@syrencloud.com.