Alternative metrics for software development?

Question

I am on a team where metric generation has been decreed by those higher above us. Specifically we are going to track team productivity by number of commits per week, PR review times, error log entries generated, and lines of code changed per week.

First of all, it now seems that I am going to be evaluated on very hackable shit and I wonder if my manager thinks I am an idiot that I cannot figure that out.

Second, these metrics are going to cause me to actively harm the codebase in the hunt for a better score. #NotMyProblem long term and I will do it if required, but I would rather not.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

Answer 1

No, there are no easy metrics that are also useful. There is no easy way to judge the value or productivity of someone who does creative/innovative/research-type work.

If there were, they'd be used everywhere.

There are only companies who acknowledge that it is hard and companies who deny reality. You find yourself in the latter.

Answer 2

The so-far best option we have in the Software Development world when it comes to productivity (?) metrics is to look into what the Accelerate book indicates as real indicators of effectiveness in a development department.

• Lead time: the average amount of time it takes from the time code is checked in to the version control system to the point in time where it is deployed to production (lower is better);
• Deployment frequency: the number of times deploys to production occur in a time period (higher is better);
• Mean time to restore (MTTR): how long it takes to resolve or rollback an error in production (lower is better);
• Change fail percentage: what percentage of changes to production (software releases and configuration changes) fail (lower is better).

Summary sourced from this blog, emphasis mine

The authors of the book claim with data that working on improving these metrics is going to drive overall improvement on your organization, and they're somewhat harder to game compared to the more typical performance indicators such as lines, commits frequencies, and the like.

If you want a rebuttal to the proposed metrics, here it is:

• number of commits per week: Commits don't equate to business value. I can make a commit that brings us in 1 more million dollars, or I can make you 100 commits that make us lose money;
• PR review times: I'll stop diving deep into a PR because it pays more to just approve/reject it superficially. It's actually better to just reject anything that comes my way;
• error log entries generated: I'll stop logging errors;
• lines of code changed per week: I'll keep gilding the latest function back and forth to look super productive.

Answer 3

Here's what an old and wise previous manager of mine said:

"Any intelligent software developer can double his output according to any performance metric, without any effort, and without any increase in productivity".

What you should do: Figure out how you can improve your metrics easily and do that. That's to get raises, bonuses, recognition etc. Doesn't matter how stupid it is. In addition, remember that your company pays you money to do your job, so do your job. That's first because you don't want to be someone who gets paid for doing nothing of real value, and second because eventually the company will figure it out.

Answer 4

I have been working in the software industry for over 20 years now and have not found any good numerical metric that can really reflect the work that people are doing.

Every metric can (and will) be abused.

• Counting commits? Developers will break big commits to smaller and smaller unnecessary commits beyond the reasonable (who said a commit for a comment? Or a commit for adding a space to the log message).
• Counting Jira tickets resolved? The same. Breaking them to smaller and smaller unnecessary ticket beyond reasonable (a ticket for adding a comment to the method X!)
• PR review times? The same. Break into smaller ones and reduce cycle time (break a reasonable sized PR into five different not-really-needed PRs.
• And so on.

Many of the metrics are theoretically showing something, but if you know the way a developer works, they are meaningless and can easily be manipulated.

Answer 5

You will get whatever you measure. Make sure that you're measuring something that will actually improve your product and further your goals.

A few metrics I've seen that were actually useful were:

• Number of defects discovered by customers but missed by QA
• Number of defects found by QA that should reasonably have been caught during a code review
• Number of meaningful¹ defects that correspond to a missing unit/automated test
• Number of work items that were planned for this iteration but not completed
• In a continuous deployment scenario, the number of critical issues that resulted in an emergency roll-back on the production system

These are product-level metrics, however, not individual metrics. They're also the sort of metrics that you gather over a longer period of time. Be careful that you aren't using metrics that create an adversarial relationship between your developers and testers. Everybody should win when the product improves, and everyone should lose when quality declines. And like any other metric, they should be re-evaluated periodically to ensure that they're producing the intended result.

One important bit is that these were measured as counts, not ratios. It's easy to game a ratio by watering down the denominator.

^{1Above a certain probability rating, in the FMEA sense.}

Answer 6

From experience and hearsay:

Any metrics that track the worker and not the work are bound to fail. Any metric that focuses on individuals and not teams is harmful. Any metric that considers raw code and manipulations to said code to be the thing of value to track are harmful.

The only reasonable metrics in software which I know of are (in no specific order):

• User satisfaction over time
• North Star style metrics (leading indicators)
• Time from commit to finished deployment without skipping critical automated tests and validations
• Time to recovery after production failures

In essence management should stop trying to make programming "efficient", because writing code is already efficient because collaboration is the true bottleneck and that is a whole separate beast from "typing lines of code in a timeframe".

They should track the outcomes of shipped changes and they should gather feedback from users. They should provide the space for people to collaborate cross-functionally without delays.

A single "this is nice" feedback from a customer tells you so much more than tracking things like lines of code changed or branches merged or so on.

Programming is not manufacturing work and tracking items like products on a conveyor belt is of no value. Ask them how they would track work done by an architect or a writer for instance.

Answer 7

Collecting metrics on a team of software developers doesn't work in my experience.

Anything like monitoring commits or logging time only introduces unnecessary stress. Not only are these things completely inaccurate, the developers will end up finding ways to manipulate the figures, and possibly waste their time focusing on that instead of doing their job.

For example, my last company logged time. Half the time... well more like 80% of the time, developers would be logging on the wrong thing. Either they (including myself) genuinely forgot to change task, or they (including myself) log time on the wrong task so they can work on another task and finish both tasks on time. - Making business decisions based on manipulated metrics is not the way to go.

Have clearly defined processes in place for estimating and handling change requests. Once that's in place, the only metric that matters is, was the task complete in the estimated time. (You don't need to log time for that, you can just set a date for when it's due)

The following questions can then be recorded at the end of the task:

• What was the estimated time
• Was it completed on time
• Why did it take less time than thought
• Why did it take us longer than expected and what can be done next time to avoid this
• If we did the same job again how long would you estimate
• Was the client (internal or external) easy to work with. If not, why

These questions can then help the business see where things are working and where things are failing. It will also provide some insight into which developers always deliver, and which ones need some extra support and reasons why.

This is basically the approach my current company takes, and it has been working extremely well.

Answer 8

Some of the most valuable software engineers in any given company, are not the most productive in terms of any code or process related metrics:

• They are the people who everybody else turns to for advice. They spend a huge fraction of their time sitting in on meetings, conference calls, hall/doorway conferences, and code reviews, and occasionally have something to say. When they speak, everybody in the room listens, and knows what they just said is important. Their utterances impact the company's profit/loss statements in increments that exceed an average engineering team's annual salaries.
• They are the facilitators, who tag and categorize bugs, find the right libraries for the tasks at hand, keep the wiki or other documentation up to date, listen to the customers, and connect the right people to each other, so that the whole company or team functions smoothly. They do the grunt work that most software engineers find tedious. They're like a swiss army knife; lots of tools, but maybe not the most expert at any of them.
• They are the noisy ones who poke holes in everybody's plans.
• They are the mentors who go out of their way to help others recognize their strengths/weaknesses, and how best to harness/mitigate them, for everyone's benefit.
• They are the coders who can bang out multiple KLOC's of functioning prototype code in a few days, rarely write a comment, unit test, or check for any but the most likely of failure codes/exceptions. They explore the problem space quickly, expose some of its more interesting features, and provide the reference artifact(s) required to make real-world decisions, and efficiently design high quality products. Except for KLOC, they often appear to be non-productive, because their prototypes are often rejected, and their code is rarely found unchanged in the product, unless management decided to ship a buggy prototype with a product label on it.
• They are the coders who never write a line of code that isn't absolutely required to complete the task at hand. They write sparse, concise comments, and tend to let the code describe itself. They are often the writers/maintainers of our reusable libraries.
• They are the ~50 other types who are required to get it done, but who I am too tired to label at this time.

Companies that fail to notice and reward these people, tend to fail. Most of us have some or all of the above qualities, to varying degrees. It is difficult, if not impossible to detect these people, using any reliable metric.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

None I am aware of. As pointed out by others, there are some, somewhat useful metrics for measuring team and company performance, but; while they seem reasonable on the surface, none of them have large bodies of empirical evidence to support them. Anything you measure, will be managed, so choose carefully.

You need to understand the reason(s) for applying any metric, to judge its suitability for purpose. I think current social trends toward eliminating bias in the workplace from promotions, is driving the trend towards heavier reliance on, hopefully objective, metrics. What is being ignored however, is the inherent subjective bias going into choosing which metrics to apply, and what their weights should be.

There are some very big experiments being run in some very large organizations, across the globe, that may or may not bear useful fruit. It would take decades of tracking individual performance, across multiple organizations, to achieve any reliable results. Given the tendency of most companies in the tech industry, to bend and sway with the buzzword tides, I would say it is unlikely any such pool of data will ever be available to us.

Answer 9

It is an easy mistake for management to make to base metrics for productivity on 'doing' stuff and that is in a sense what they have done. As the job being done involves all of the metrics mentioned.

The issue you rightfully point out is that you could do a lot of that activity without actually achieving anything meaningful/valuable to the company. As you state, if you game the system it will cause more harm than good and as a developer a lot of what you do is solving problems. So it is quite natural that you will solve the problem of getting good metrics by gaming the system, which no doubt you and your colleagues will do very well and appear on paper to be doing lots.

This will eventually fall over though, because of the lack of value being generated by the company. The challenge of metric measurement for development teams is a tricky one. As mentioned a large part of a developers job is problem solving, and how do you measure that using simple metrics? An approach is to set clear requirements from the business based on real deliverable value.

This is usually done as a high level user journey in the form of:

'As a [role in the business] I can [do something of tangible value for the business] so that [clear justification of the value to the business]'

This sets out the problem for development team to solve together that reflects actual value to the business and can be tested after delivery to determine that the value has been achieved, but without proposing the concrete solution, so that the development team gets to do the problem solving using the skills and tools available to them.

The fundamental issue is development is a very complex task, so creating simple metrics to measure its success will not work, and in the worse case scenario cause more problems. It takes cooperation from the business and the development team to describe, understand and the solve the problems the business faces.

Answer 10

My 2 cents:

As other answers suggest, anything can be manipulated with some shorter or longer-term consequences. Finding a bulletproof metric is futile.

The key is how you and your management is going to use the metric. If the management will use the number of commits to decide who gets a raise and who does not, the joke is on them. There should be no incentive to influence the metrics.

A better way is to treat metrics as indicators. For example, the team starts working from home due to a lockdown. You can compare before and after indicators to see if and how ways of working have changed. You cannot tell what is better, but you can see if there is a difference. From there, you can ask people what has happened, correlate this with other indicators to get a complete picture and use that to make any further decisions.

Answer 11

If you need a metric, best to start from first principles. What do you want?

• Meeting requirements is good.
• Writing lots of code to do simple tasks is bad.
• … but cutting out edge-case handling is also bad.
• Easy-to-understand is good.
• Buggy is bad.
• Efficiency (execution time) is good.
• Efficiency (usability) is good.
• Efficiency (development time) is good.

And so on. No one metric encapsulates any of these, so you need to use more than one metric. My advice is to have enough metrics that they will all gradually go up (or down, as required) over time, and each (logical, not necessarily atomic) change improves at least one of the metrics in an explainable way.

As some metrics will go up and some will go down short-term, you should be able to demonstrate to the higher-ups that focusing on making metrics go up will interfere with development.

But what metrics to use?

• Requirements coverage – to what extent is the software “finished”?
• Verification coverage – how much of the existing code is tested / proven? (Logical; not automated test coverage, if possible.)
• Various bug tracker stats:
  • User bug reports – a count of bug reports coming in (perhaps per day / week instead of cumulative)
  • User reported bugs – bugs reported by at least one user
  • Known correctness bugs
  • Known serious bugs
  • Known usability issues
• Mini-features supported
  This is a fairly arbitrary metric. Each part of an application supports little mini use-cases, like “underline all titles” or “turn off spell-check on code snippets”. An ideal re-design simplifies the whole system without breaking any of these mini-features, and enables new ones (e.g. replace all the special-cased functions with a styles system, where “underline” or “spellcheck” are properties of those styles); such a change would make the number go up. However, if the simplification removed spell-checking entirely, the number might go down. Each bit of code has an associated list consisting of a few representative examples.
• Mini-features broken – how many mini-use-cases have been broken that were possible in a previous (public) version or release.
• Code size (minified, compressed, in bytes)
• Documentation size (including comments, compressed, in bytes)
• Code obviousness – get a random non-programmer (who understands the problem domain) to look through the code for ten minutes or so, then rank how much they understand it.

Add or remove as necessary, depending on what management truly cares about. Aim for something easy-to-measure (a few seconds of thinking), non-arbitrary (can be justified, different people would give basically the same numbers) and related to things that management cares about.

Yes, most of these can't be calculated by a computer. They can, however, be calculated and kept track of manually – in fact, you're probably already doing something like this. And it's hard to make the metrics diverge from reality (without somebody noticing fairly quickly), so gaming is less likely.

Management doesn't care about the metrics. Management cares about the story those metrics tell – a (hopefully) objective one. If they have obscure, meaningless metrics like “lines of code changed”, they'll still see a story (such is human nature), but it'll be an almost completely meaningless one, even if the developers don't know it's being used. But if they have meaningful metrics… if they can't interpret them, they can always ask someone what the team was up to, and then they'll understand better. (Here, the metrics serve to improve communication, rather than supplant it entirely.)

• If “requirements coverage” goes up, but “verification coverage” goes down: more code has been written, but it hasn't been checked yet.
• If “requirements” and “verification” go down at once and “known correctness bugs” goes up, then “verification” creeps back up over time, then “known serious bugs” goes down by three: the root cause of some serious problem just got fixed.
• If “requirements coverage” goes down a lot, at once: the requirements changed.
• If “requirements coverage” goes up a lot but “mini-features broken” goes down a lot: perhaps the requirements weren't complete.

If you can easily provide management with metrics that are obviously more useful than what they're using, perhaps they'll use them instead.

Answer 12

Yes, there are metrics, that can be gamed in the way that won't be a pure waste of time.

For example, test code coverage. Achieving full code coverage is usually not worth fulfilling, or even counterproductive - example Java, where you have exceptions that you must catch but can't be thrown. For example, IOException using ByteArrayOutputStream. In order to get 100% coverage, you'll have to instrument your test to replace the implementation in the JRE, which is a big effort for nothing.

But at least, high code coverage will make your code easier to maintain and catch most accidental side effect before they go into production. So if you very need some metrics, choose those, that will provide some value added to the product.

But there's a saying, you should not give a razor to a monkey. The metrics in the hand of totally incompetent management can do more harm than profit, even the best metrics.

Answer 13

My company often used a metric they called "velocity". How this works is that developers assign points to each story that delivers user value before the story is worked on. The points are an estimate of how much time or complexity is involved in the story and are usually done on a Fibonacci scale, 1, 2, 3, 5, 8.

We use a tool called Pivotal Tracker that tracks how many points worth of work are completed in a sprint and that ends up being your velocity. You can technically game it by pointing things higher than they are, but we would point stories together as a group, which mostly kept people honest.

Here's a blog post from Pivotal Tracker about how you can use velocity https://www.pivotaltracker.com/help/articles/planning_with_velocity/.

Answer 14

This to me smells of a product owner, executive BA or non-tech-savvy director who really wants to see more productivity, while ignoring the true meaning of productivity in the software industry (some devs like to make a few, significant commits while others make loads of tiny commits of little impact).

Let me just address the many issues with each "demand" one at a time:

Number of commits per week

Lines of code changed per week

This tells you nothing, anyone can make 100 commits in half a day while having no impact on the overall system. If anything it encourages the less-than-engaged developers to just make loads of tiny commits like reformatting, quality control, removing white spaces, changing the target framework and then changing it back... a bored developer gets really creative if they only need to "make more commits".

I wonder if my manager thinks I am an idiot that I cannot figure that out.

Like you said, this is a VERY hackable way of boosting your review points, if you think the new policy is stupid (you're not the only one) then by all means hack it. If your manager is code-savvy then you just need a reasonable explanation/description of each commit. Otherwise, you can commit whatever you want, however you want so long as it doesn't break the build or pipeline.

PR review times

This tells you nothing on it's own. Yes, it's generally bad to need loads of PR reviews and revisions for one piece of work, however it can be very circumstantial. What if your trying to PR a major breaking change with integration tests but the system changes very frequently? You can either ask the rest of the devs to stop working for a bit or you just have to tweak, tweak and tweak that PR until it's ready for release. Sure, silly mistakes are a red flag but I've made dozens of PRs where they would have passed I raised the PR but then, hours later, someone else change the master branch and I have to tweak my PR in order to let it build. Happens all the time. This is why it's imperative to have a team lead who can orchestrate PRs in a way that will never break the build.

Error log entries generated

On face value, this seems appropriate. I.e. more error logs means your making more mistakes, but where do these errors come from? Compile-time errors are 100% unacceptable (even if they're possible to commit) but run-time errors and exceptions usually get bubbled up through the many layers in a large scale system. E.g. there may be a low-level programmer who pushes a core fundamental change to the system, which still works fine. But then there may be code in a much higher layer like the front-end which could malfunction for a multitude of reasons (sometimes it's down to the user's device, sometimes it's down to a bloat of 3rd party frameworks, sometimes it's down to other developers using different IDEs, the list goes on). In all honesty, this shouldn't even be part of a performance review process, it should be an ongoing discussion between Devops and developers. In today's standard of software development, it can be a real pain to even find where an error is logged (server, client application, browser window, cloud host instance) for this reason I would always have a central log repository like Splunk with a dedicated log watcher who knows exactly where each message originated.

Second, these metrics are going to cause me to actively harm the codebase in the hunt for a better score. #NotMyProblem long term and I will do it if required, but I would rather not.

I couldn't agree more! Your company has basically imposed a competition between the developers, except this competition has no real reward, no guarantee of improving company performance and honestly it's more likely to do the opposite.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

Yes, but you need a very experienced software engineer to keep track of these metrics (and the developers who do the work, which areas are they focusing on and how much control they have over the underlying system/legacy code).

Some useful examples of productivity metrics:

• (PR review time-to-number of changes) ratio, if a small PR needs loads of revisions then it's a bad sign, but it's normal for larger PRs to have
• Customer Feedback/ Bug Reports If you give customers the opportunity to rate your software then it's very beneficial to know what works and what needs work. If a certain dev gets repeatedly poor feedback on most of their work it's a bad sign.
• Gantt Chart Accuracy Dunno if your team uses Gantt charts, they are very helpful for long term goal tracking, If your developers can plan work 8 weeks ahead of time and finish said work on time you can rest assured they are not only reliable but also realistic in their goals. Those are the two most important factors in my opinion when it comes to having a solid, productive team. You can't ask developers to achieve any goal in any time frame, but if they can consistently achieve goals in the time frame they estimate, then THAT'S AS GOOD AS IT GETS (without hiring some Rockstar contractors)
• Story Board Points This is like a more agile way to keep track of Gantt charts, so long as your team is realistic when setting the points of each story board item, you can use the points per week metric as an indicator of progress.