94

I am on a team where metric generation has been decreed by those higher above us. Specifically we are going to track team productivity by number of commits per week, PR review times, error log entries generated, and lines of code changed per week.

First of all, it now seems that I am going to be evaluated on very hackable shit and I wonder if my manager thinks I am an idiot that I cannot figure that out.

Second, these metrics are going to cause me to actively harm the codebase in the hunt for a better score. #NotMyProblem long term and I will do it if required, but I would rather not.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

1

14 Answers 14

151

No, there are no easy metrics that are also useful. There is no easy way to judge the value or productivity of someone who does creative/innovative/research-type work.

If there were, they'd be used everywhere.

There are only companies who acknowledge that it is hard and companies who deny reality. You find yourself in the latter.

9
  • 1
    The easy way to judge your work would be how many products you have brought to market and how profitable they have been. Tech R&D is speculative by nature. Who are the higher ups the OP talks about? I would be willing to explain my methodology to someone similarly technically versed, a person with 4 coloured pens in there pocket less so – Neil Meyer Jun 3 at 19:45
  • 6
    Rather than tracking many metrics, how about finding a metric which is unlikely to negatively impact the actual productivity? Meta-gaming the system, basically. I'm not sure what such a metric would be, though. Anything which looks plausibly related to performance, when aggressively gamed, would probably seriously harm results. – l0b0 Jun 3 at 19:53
  • 11
    @Clumsycat are you being sarcastic? I'd consider a metric like that even more harmful then measuring lines of code. – Erik Jun 4 at 9:51
  • 6
    @Eric Not really, but then I don't work in industry. A big issue in my field is people working too long hours, so rewarding people for leaving work on time would probably be an improvement. Of course it wouldn't prevent them from going home and keeping on working, but it might reduce the peer pressure to work long hours. – Clumsy cat Jun 4 at 10:00
  • 10
    "If there were, they'd be used everywhere" is the answer to so many questions, such as "Does this magic weight loss pill work?" or "Is there a way I should write my resume to guarantee I get a job". – Andy Lester Jun 4 at 15:08
66

The so-far best option we have in the Software Development world when it comes to productivity (?) metrics is to look into what the Accelerate book indicates as real indicators of effectiveness in a development department.

  • Lead time: the average amount of time it takes from the time code is checked in to the version control system to the point in time where it is deployed to production (lower is better);
  • Deployment frequency: the number of times deploys to production occur in a time period (higher is better);
  • Mean time to restore (MTTR): how long it takes to resolve or rollback an error in production (lower is better);
  • Change fail percentage: what percentage of changes to production (software releases and configuration changes) fail (lower is better).

Summary sourced from this blog, emphasis mine

The authors of the book claim with data that working on improving these metrics is going to drive overall improvement on your organization, and they're somewhat harder to game compared to the more typical performance indicators such as lines, commits frequencies, and the like.

If you want a rebuttal to the proposed metrics, here it is:

  • number of commits per week: Commits don't equate to business value. I can make a commit that brings us in 1 more million dollars, or I can make you 100 commits that make us lose money;
  • PR review times: I'll stop diving deep into a PR because it pays more to just approve/reject it superficially. It's actually better to just reject anything that comes my way;
  • error log entries generated: I'll stop logging errors;
  • lines of code changed per week: I'll keep gilding the latest function back and forth to look super productive.
19
  • 26
    The indicators you gave are great for measuring how the organization is operating. But might be less relevant for measuring how a specific developer is doing – riorio Jun 3 at 10:20
  • 7
    @riorio yes, that's true. But it's a minefield to go there anyway. I would rather know from the team if everybody is pulling their own weight in the group, as long as the org indicators are healthy. – STT LCU Jun 3 at 12:53
  • 15
    Wow. Change fail percentage (or something like it) actually measures something useful and encourages developers to test their code – slebetman Jun 3 at 16:06
  • 13
    I'd disagree higher deployment frequency is better. It is up to a point, but at some point to push it higher you need to sacrifice quality and testing. The others are okish, but most aren't a degree of developer productivity, they're team or organization measurements. Totally disjoint from what the OP is being asked about – Gabe Sechan Jun 3 at 18:26
  • 5
    @GabeSechan that is exactly the point: you don't sacrifice quality and testing - to reach the higher levels of deployment frequency all your testing needs to be automated, and the metric is actually forcing you into that direction. – fgysin Jun 4 at 5:35
50

Here's what an old and wise previous manager of mine said:

"Any intelligent software developer can double his output according to any performance metric, without any effort, and without any increase in productivity".

What you should do: Figure out how you can improve your metrics easily and do that. That's to get raises, bonuses, recognition etc. Doesn't matter how stupid it is. In addition, remember that your company pays you money to do your job, so do your job. That's first because you don't want to be someone who gets paid for doing nothing of real value, and second because eventually the company will figure it out.

7
  • 5
    IOW: what can be measured will be gamed for maximum personal benefit. – BryanH Jun 3 at 18:36
  • 19
    any intelligent software developer can go much farther with a significant decrease in his productivity. – Danubian Sailor Jun 3 at 19:50
  • 11
    I found that gaming the system to an absurd degree is a good way to teach management that such metrics aren't useful. Evaluating me based on lines of code changed? I'll write a script that changes the line endings on every file from Windows-style to Linux-style, commits, changes back, commits, etc. and let it run in a loop all weekend. When my stats have five extra zeroes on the end compared to everyone else, even the pointiest-haired boss can see their metrics aren't actually meaningful. – bta Jun 4 at 2:11
  • 2
    +1: I am all in for company natural selection. Milk this one as much as you can by fulfilling whatever pointless metrics they invent, and be prepared to jump the boat. – lvella Jun 4 at 18:50
  • 2
    @bta: It probably needs to be more subtle (for plausible deniability reasons) – Peter Mortensen Jun 5 at 20:05
17

I have been working in the software industry for over 20 years now and have not found any good numerical metric that can really reflect the work that people are doing.

Every metric can (and will) be abused.

  • Counting commits? Developers will break big commits to smaller and smaller unnecessary commits beyond the reasonable (who said a commit for a comment? Or a commit for adding a space to the log message).
  • Counting Jira tickets resolved? The same. Breaking them to smaller and smaller unnecessary ticket beyond reasonable (a ticket for adding a comment to the method X!)
  • PR review times? The same. Break into smaller ones and reduce cycle time (break a reasonable sized PR into five different not-really-needed PRs.
  • And so on.

Many of the metrics are theoretically showing something, but if you know the way a developer works, they are meaningless and can easily be manipulated.

14
  • 5
    I agree with your sentiment, but breaking commits up and lowering PR review times is a good thing. It's something I actively teach developers to do – tddmonkey Jun 3 at 8:35
  • 16
    @tddmonkey not for the sake of doing it, though. You can lower PR review time to zero by just blindly accepting everything, but then you quickly realizing lowering PR review times is not the point. – Erik Jun 3 at 9:05
  • 3
    To be honest, if your company is setting these things as goals, then you do exactly the stupid things they ask for. Break big commits into small ones etc. etc. etc. Eventually they will learn. – gnasher729 Jun 3 at 11:31
  • 2
    @NeilMeyer lets say assignment 1 has a 95% chance of making the company 1 million dollars and #2 has a 1% chance of making 200 million. #2 has twice the EV of #1 yet people tend to be risk adverse and would much prefer to work #1. Obviously I've significantly simplified things, but your approach creates all sorts of perverse incentives. Just look at academics and the pressure to publish. Or consider salespeople that are compensated through commissions. High pressure sales techniques and/or lying to the customer might be great for the salesperson and terrible for the business in the long run. – eps Jun 3 at 21:52
  • 4
    ... not to mention the history is littered with examples of terrible things happening when people are pressured / incentivized to maximize profit regardless of what it does to safety, the environment, society etc. – eps Jun 3 at 21:57
11

You will get whatever you measure. Make sure that you're measuring something that will actually improve your product and further your goals.

A few metrics I've seen that were actually useful were:

  • Number of defects discovered by customers but missed by QA
  • Number of defects found by QA that should reasonably have been caught during a code review
  • Number of meaningful1 defects that correspond to a missing unit/automated test
  • Number of work items that were planned for this iteration but not completed
  • In a continuous deployment scenario, the number of critical issues that resulted in an emergency roll-back on the production system

These are product-level metrics, however, not individual metrics. They're also the sort of metrics that you gather over a longer period of time. Be careful that you aren't using metrics that create an adversarial relationship between your developers and testers. Everybody should win when the product improves, and everyone should lose when quality declines. And like any other metric, they should be re-evaluated periodically to ensure that they're producing the intended result.

One important bit is that these were measured as counts, not ratios. It's easy to game a ratio by watering down the denominator.

1Above a certain probability rating, in the FMEA sense.

1
  • 1
    @JoeStrazzere You'd obviously have to tailor it to your specific needs. For my group, it generally meant that customers had configurations or hardware that weren't part of QA's test plans. The point of that stat was to ensure we moved beyond "how do I fix this?" to "how did we miss this?", a question that frequently revealed the bad development practices, communication breakdowns, and organizational issues that allowed the bugs to exist in the first place. – bta Jun 4 at 18:43
5

From experience and hearsay:

Any metrics that track the worker and not the work are bound to fail. Any metric that focuses on individuals and not teams is harmful. Any metric that considers raw code and manipulations to said code to be the thing of value to track are harmful.

The only reasonable metrics in software which I know of are (in no specific order):

  • User satisfaction over time
  • North Star style metrics (leading indicators)
  • Time from commit to finished deployment without skipping critical automated tests and validations
  • Time to recovery after production failures

In essence management should stop trying to make programming "efficient", because writing code is already efficient because collaboration is the true bottleneck and that is a whole separate beast from "typing lines of code in a timeframe".

They should track the outcomes of shipped changes and they should gather feedback from users. They should provide the space for people to collaborate cross-functionally without delays.

A single "this is nice" feedback from a customer tells you so much more than tracking things like lines of code changed or branches merged or so on.

Programming is not manufacturing work and tracking items like products on a conveyor belt is of no value. Ask them how they would track work done by an architect or a writer for instance.

5
  • 3
    Focusing on "time from commit to finished deployment" encourages you to do the bare minimum in review and testing, and discourages trying to do critical-priority work as soon as possible. You should of course pay attention to that metric and try to minimise it through reasonable measures (like not having a ticket sit in review for 1-2 weeks), but as a hard metric to measure performance it's somewhere between useless and harmful. – Bernhard Barker Jun 3 at 16:33
  • Can you expand a bit on "North Star style metrics"? The link basically just seems to say "you should read this book". – Bernhard Barker Jun 3 at 16:36
  • "Time from commit to finished deployment" is not a reliable metric, as not all completed code is released immediately. New features may be dependent on other systems that have a different release date, or the product owner may want to release a suite of features at the same time. – BryanH Jun 3 at 18:41
  • 5
    I don't know that writers make a good example. It's rather common actually to pay writers by the word. It may be wrong to evaluate writers by words per day, but it certainly isn't unknown or surprising. Measuring by the word is the industry standard for freelance work among writers. Doing the same with a programmer would be madness: consider five lines of code with a million random words in a comment. It somewhat works for writers because publishers can reject junk and edit overly-wordy submissions. – mdfst13 Jun 3 at 23:53
  • 2
    I'd expand the first part to "Any metrics that track the worker and not the outcome of the work are bound to fail.". Even measuring the work is bound to fail, because work that is useless is always easier than work that is valuable and also precisely what you want to have less of. – Erik Jun 4 at 6:52
4

Collecting metrics on a team of software developers doesn't work in my experience.

Anything like monitoring commits or logging time only introduces unnecessary stress. Not only are these things completely inaccurate, the developers will end up finding ways to manipulate the figures, and possibly waste their time focusing on that instead of doing their job.

For example, my last company logged time. Half the time... well more like 80% of the time, developers would be logging on the wrong thing. Either they (including myself) genuinely forgot to change task, or they (including myself) log time on the wrong task so they can work on another task and finish both tasks on time. - Making business decisions based on manipulated metrics is not the way to go.

Have clearly defined processes in place for estimating and handling change requests. Once that's in place, the only metric that matters is, was the task complete in the estimated time. (You don't need to log time for that, you can just set a date for when it's due)

The following questions can then be recorded at the end of the task:

  • What was the estimated time
  • Was it completed on time
  • Why did it take less time than thought
  • Why did it take us longer than expected and what can be done next time to avoid this
  • If we did the same job again how long would you estimate
  • Was the client (internal or external) easy to work with. If not, why

These questions can then help the business see where things are working and where things are failing. It will also provide some insight into which developers always deliver, and which ones need some extra support and reasons why.

This is basically the approach my current company takes, and it has been working extremely well.

13
  • 5
    "the only metric that matters is, was the task complete in the estimated time" You got it, boss! I will use the clearly defined processes to pad my estimates for everything, and will always come in early! Win, win! – BryanH Jun 3 at 18:39
  • 1
    The main idea is to review and track the job, not the developer, and not get too granular. Dev's work better and make less mistakes when they're not under the pressure of being continually monitored. - Sure you'll get some that try and abuse any system, but they should be obvious, and the first to go. – flexi Jun 3 at 20:35
  • 4
    I find "was it done on time" to still be a bad target. It encourages padding, as well as cutting corners to finish on time. It's fine to use as another data-point in your process, though. Having an honest conversation about why things took too long is super useful, tying someone's bonus to it is just going to have it be gamed like everything else. – Erik Jun 4 at 6:49
  • 1
    Re "was the task complete in the estimated time": It sounds like a recipe for building up technical debt (e.g., by encouraging copy-paste reuse - the worst kind of reuse). Two years from now the tasks will still complete on time, but development has slowed by a factor of 5 due to the technical debt. – Peter Mortensen Jun 5 at 20:00
  • 1
    What happens when the dev and business side can't agree on the estimate to begin with? Also, what's "If we did the same job again how long would you estimate?" ...why would you do the same job again? The point of computers is that you only have to do the same thing once...then the computer can repeat as many times as necessary from there on out. – user3067860 Jun 7 at 13:20
4

Some of the most valuable software engineers in any given company, are not the most productive in terms of any code or process related metrics:

  • They are the people who everybody else turns to for advice. They spend a huge fraction of their time sitting in on meetings, conference calls, hall/doorway conferences, and code reviews, and occasionally have something to say. When they speak, everybody in the room listens, and knows what they just said is important. Their utterances impact the company's profit/loss statements in increments that exceed an average engineering team's annual salaries.
  • They are the facilitators, who tag and categorize bugs, find the right libraries for the tasks at hand, keep the wiki or other documentation up to date, listen to the customers, and connect the right people to each other, so that the whole company or team functions smoothly. They do the grunt work that most software engineers find tedious. They're like a swiss army knife; lots of tools, but maybe not the most expert at any of them.
  • They are the noisy ones who poke holes in everybody's plans.
  • They are the mentors who go out of their way to help others recognize their strengths/weaknesses, and how best to harness/mitigate them, for everyone's benefit.
  • They are the coders who can bang out multiple KLOC's of functioning prototype code in a few days, rarely write a comment, unit test, or check for any but the most likely of failure codes/exceptions. They explore the problem space quickly, expose some of its more interesting features, and provide the reference artifact(s) required to make real-world decisions, and efficiently design high quality products. Except for KLOC, they often appear to be non-productive, because their prototypes are often rejected, and their code is rarely found unchanged in the product, unless management decided to ship a buggy prototype with a product label on it.
  • They are the coders who never write a line of code that isn't absolutely required to complete the task at hand. They write sparse, concise comments, and tend to let the code describe itself. They are often the writers/maintainers of our reusable libraries.
  • They are the ~50 other types who are required to get it done, but who I am too tired to label at this time.

Companies that fail to notice and reward these people, tend to fail. Most of us have some or all of the above qualities, to varying degrees. It is difficult, if not impossible to detect these people, using any reliable metric.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

None I am aware of. As pointed out by others, there are some, somewhat useful metrics for measuring team and company performance, but; while they seem reasonable on the surface, none of them have large bodies of empirical evidence to support them. Anything you measure, will be managed, so choose carefully.

You need to understand the reason(s) for applying any metric, to judge its suitability for purpose. I think current social trends toward eliminating bias in the workplace from promotions, is driving the trend towards heavier reliance on, hopefully objective, metrics. What is being ignored however, is the inherent subjective bias going into choosing which metrics to apply, and what their weights should be.

There are some very big experiments being run in some very large organizations, across the globe, that may or may not bear useful fruit. It would take decades of tracking individual performance, across multiple organizations, to achieve any reliable results. Given the tendency of most companies in the tech industry, to bend and sway with the buzzword tides, I would say it is unlikely any such pool of data will ever be available to us.

4

It is an easy mistake for management to make to base metrics for productivity on 'doing' stuff and that is in a sense what they have done. As the job being done involves all of the metrics mentioned.

The issue you rightfully point out is that you could do a lot of that activity without actually achieving anything meaningful/valuable to the company. As you state, if you game the system it will cause more harm than good and as a developer a lot of what you do is solving problems. So it is quite natural that you will solve the problem of getting good metrics by gaming the system, which no doubt you and your colleagues will do very well and appear on paper to be doing lots.

This will eventually fall over though, because of the lack of value being generated by the company. The challenge of metric measurement for development teams is a tricky one. As mentioned a large part of a developers job is problem solving, and how do you measure that using simple metrics? An approach is to set clear requirements from the business based on real deliverable value.

This is usually done as a high level user journey in the form of:

'As a [role in the business] I can [do something of tangible value for the business] so that [clear justification of the value to the business]'

This sets out the problem for development team to solve together that reflects actual value to the business and can be tested after delivery to determine that the value has been achieved, but without proposing the concrete solution, so that the development team gets to do the problem solving using the skills and tools available to them.

The fundamental issue is development is a very complex task, so creating simple metrics to measure its success will not work, and in the worse case scenario cause more problems. It takes cooperation from the business and the development team to describe, understand and the solve the problems the business faces.

1
  • I'm not quite sure how you can program in a business setting without adding value or solving problems. Granted world class r&d often solves problem people don't realise they have, but at the very least there should be some sort of economic goal to achieve. – Neil Meyer Jun 3 at 19:53
2

My 2 cents:

As other answers suggest, anything can be manipulated with some shorter or longer-term consequences. Finding a bulletproof metric is futile.

The key is how you and your management is going to use the metric. If the management will use the number of commits to decide who gets a raise and who does not, the joke is on them. There should be no incentive to influence the metrics.

A better way is to treat metrics as indicators. For example, the team starts working from home due to a lockdown. You can compare before and after indicators to see if and how ways of working have changed. You cannot tell what is better, but you can see if there is a difference. From there, you can ask people what has happened, correlate this with other indicators to get a complete picture and use that to make any further decisions.

2
  • Even then; my colleagues working from home commit much less frequently because it is not as quick or easy to do - because of buggy firewalls and slow broadband - so if someone is counting commits then obviously they are less productive at home. – RedSonja Jun 5 at 11:45
  • True, but knowing that commit/push operations are slow, developers may start making bigger commits, e.g. one commit at the end of the day. Thus, productivity in terms of LOC remains the same, however the metric of commit count shifs. The job of the management is to figure out why and does it matter.. – Eriks Klotins Jun 7 at 20:37
1

If you need a metric, best to start from first principles. What do you want?

  • Meeting requirements is good.
  • Writing lots of code to do simple tasks is bad.
  • … but cutting out edge-case handling is also bad.
  • Easy-to-understand is good.
  • Buggy is bad.
  • Efficiency (execution time) is good.
  • Efficiency (usability) is good.
  • Efficiency (development time) is good.

And so on. No one metric encapsulates any of these, so you need to use more than one metric. My advice is to have enough metrics that they will all gradually go up (or down, as required) over time, and each (logical, not necessarily atomic) change improves at least one of the metrics in an explainable way.

As some metrics will go up and some will go down short-term, you should be able to demonstrate to the higher-ups that focusing on making metrics go up will interfere with development.

But what metrics to use?

  • Requirements coverage – to what extent is the software “finished”?
  • Verification coverage – how much of the existing code is tested / proven? (Logical; not automated test coverage, if possible.)
  • Various bug tracker stats:
    • User bug reports – a count of bug reports coming in (perhaps per day / week instead of cumulative)
    • User reported bugs – bugs reported by at least one user
    • Known correctness bugs
    • Known serious bugs
    • Known usability issues
  • Mini-features supported
    This is a fairly arbitrary metric. Each part of an application supports little mini use-cases, like “underline all titles” or “turn off spell-check on code snippets”. An ideal re-design simplifies the whole system without breaking any of these mini-features, and enables new ones (e.g. replace all the special-cased functions with a styles system, where “underline” or “spellcheck” are properties of those styles); such a change would make the number go up. However, if the simplification removed spell-checking entirely, the number might go down. Each bit of code has an associated list consisting of a few representative examples.
  • Mini-features broken – how many mini-use-cases have been broken that were possible in a previous (public) version or release.
  • Code size (minified, compressed, in bytes)
  • Documentation size (including comments, compressed, in bytes)
  • Code obviousness – get a random non-programmer (who understands the problem domain) to look through the code for ten minutes or so, then rank how much they understand it.

Add or remove as necessary, depending on what management truly cares about. Aim for something easy-to-measure (a few seconds of thinking), non-arbitrary (can be justified, different people would give basically the same numbers) and related to things that management cares about.

Yes, most of these can't be calculated by a computer. They can, however, be calculated and kept track of manually – in fact, you're probably already doing something like this. And it's hard to make the metrics diverge from reality (without somebody noticing fairly quickly), so gaming is less likely.

Management doesn't care about the metrics. Management cares about the story those metrics tell – a (hopefully) objective one. If they have obscure, meaningless metrics like “lines of code changed”, they'll still see a story (such is human nature), but it'll be an almost completely meaningless one, even if the developers don't know it's being used. But if they have meaningful metrics… if they can't interpret them, they can always ask someone what the team was up to, and then they'll understand better. (Here, the metrics serve to improve communication, rather than supplant it entirely.)

  • If “requirements coverage” goes up, but “verification coverage” goes down: more code has been written, but it hasn't been checked yet.
  • If “requirements” and “verification” go down at once and “known correctness bugs” goes up, then “verification” creeps back up over time, then “known serious bugs” goes down by three: the root cause of some serious problem just got fixed.
  • If “requirements coverage” goes down a lot, at once: the requirements changed.
  • If “requirements coverage” goes up a lot but “mini-features broken” goes down a lot: perhaps the requirements weren't complete.

If you can easily provide management with metrics that are obviously more useful than what they're using, perhaps they'll use them instead.

2
  • The problems with bug counts, is it's usually not up to the individual contributor, or even the team, whether or when their code will get a product label slapped on it and shipped to the customer. It can also be difficult to connect any one contribution, to the quality of the code. Integrated software components often pass extensive test suites, and then exhibit unpredictable behavior in the hands of customers. – jwdonahue Jun 5 at 20:41
  • 1
    @jwdonahue Oh, the bosses want to measure individuals? In that case, no metrics are any good; teams do work, and an individual's “team contribution” is a proxy, at best, for how much worse the team would be without them. – wizzwizz4 Jun 5 at 22:58
0

Yes, there are metrics, that can be gamed in the way that won't be a pure waste of time.

For example, test code coverage. Achieving full code coverage is usually not worth fulfilling, or even counterproductive - example Java, where you have exceptions that you must catch but can't be thrown. For example, IOException using ByteArrayOutputStream. In order to get 100% coverage, you'll have to instrument your test to replace the implementation in the JRE, which is a big effort for nothing.

But at least, high code coverage will make your code easier to maintain and catch most accidental side effect before they go into production. So if you very need some metrics, choose those, that will provide some value added to the product.

But there's a saying, you should not give a razor to a monkey. The metrics in the hand of totally incompetent management can do more harm than profit, even the best metrics.

3
  • I'm not sure that code coverage is different to any of the OP's examples. E.g. commits per week: I can (a) do extra useful commits per week (productive) or (b) do extra useless commits each week (unproductive). Code coverage: I can (a) increase coverage of the existing codebase (potentially productive) or (b) refactor or add new code just so that I can write new test code to increase coverage (unproductive). In both cases the system can be easily abused in a wasteful way. – Jon Bentley Jun 4 at 8:47
  • 1
    I worked on a military project and had to demonstrate maximum coverage. To do this there was the memorable day when they modified the hardware on a running rocket launcher just to test my fault detection exceptions. And (this is C++) the exceptions I couldn't trigger - only happens on fifth Monday of a month at 48 degrees latitude - I tricked the system by putting the exception and all its code in one long line of code. Having said that, it is the most robust code I ever wrote. – RedSonja Jun 5 at 11:52
  • 100% coverage in unit tests is always achievable. In fact, unit tests are the only place you can achieve 100% code coverage. That's why we write mocks. But 100% coverage, doesn't imply anything close to 100% of code path permutations, and that's where you get most of the interesting behaviors in software. Was the exception thrown when the fault bit was set in a mock register? Check! Dose it prevent the rocket from exploding on the pad, on the fifth Monday of a month at 48 degrees latitude? Only one way to know for sure. But that doesn't imply UT is counter productive. – jwdonahue Jun 5 at 18:19
0

My company often used a metric they called "velocity". How this works is that developers assign points to each story that delivers user value before the story is worked on. The points are an estimate of how much time or complexity is involved in the story and are usually done on a Fibonacci scale, 1, 2, 3, 5, 8.

We use a tool called Pivotal Tracker that tracks how many points worth of work are completed in a sprint and that ends up being your velocity. You can technically game it by pointing things higher than they are, but we would point stories together as a group, which mostly kept people honest.

Here's a blog post from Pivotal Tracker about how you can use velocity https://www.pivotaltracker.com/help/articles/planning_with_velocity/.

2
  • 1
    It sounds like Scrum. (The Fibonacci scale is close to exponential - it very quickly converges to a ratio of the golden ratio (about 1.618).) – Peter Mortensen Jun 5 at 20:15
  • If management evaluates individuals on their velocity, they're doing it wrong. The team's velocity is what matters, and only for planning purposes. Any artificial metric used for evaluation will be gamed. – BryanH Jun 14 at 23:07
-1

This to me smells of a product owner, executive BA or non-tech-savvy director who really wants to see more productivity, while ignoring the true meaning of productivity in the software industry (some devs like to make a few, significant commits while others make loads of tiny commits of little impact).

Let me just address the many issues with each "demand" one at a time:

Number of commits per week

Lines of code changed per week

This tells you nothing, anyone can make 100 commits in half a day while having no impact on the overall system. If anything it encourages the less-than-engaged developers to just make loads of tiny commits like reformatting, quality control, removing white spaces, changing the target framework and then changing it back... a bored developer gets really creative if they only need to "make more commits".

I wonder if my manager thinks I am an idiot that I cannot figure that out.

Like you said, this is a VERY hackable way of boosting your review points, if you think the new policy is stupid (you're not the only one) then by all means hack it. If your manager is code-savvy then you just need a reasonable explanation/description of each commit. Otherwise, you can commit whatever you want, however you want so long as it doesn't break the build or pipeline.

PR review times

This tells you nothing on it's own. Yes, it's generally bad to need loads of PR reviews and revisions for one piece of work, however it can be very circumstantial. What if your trying to PR a major breaking change with integration tests but the system changes very frequently? You can either ask the rest of the devs to stop working for a bit or you just have to tweak, tweak and tweak that PR until it's ready for release. Sure, silly mistakes are a red flag but I've made dozens of PRs where they would have passed I raised the PR but then, hours later, someone else change the master branch and I have to tweak my PR in order to let it build. Happens all the time. This is why it's imperative to have a team lead who can orchestrate PRs in a way that will never break the build.

Error log entries generated

On face value, this seems appropriate. I.e. more error logs means your making more mistakes, but where do these errors come from? Compile-time errors are 100% unacceptable (even if they're possible to commit) but run-time errors and exceptions usually get bubbled up through the many layers in a large scale system. E.g. there may be a low-level programmer who pushes a core fundamental change to the system, which still works fine. But then there may be code in a much higher layer like the front-end which could malfunction for a multitude of reasons (sometimes it's down to the user's device, sometimes it's down to a bloat of 3rd party frameworks, sometimes it's down to other developers using different IDEs, the list goes on). In all honesty, this shouldn't even be part of a performance review process, it should be an ongoing discussion between Devops and developers. In today's standard of software development, it can be a real pain to even find where an error is logged (server, client application, browser window, cloud host instance) for this reason I would always have a central log repository like Splunk with a dedicated log watcher who knows exactly where each message originated.

Second, these metrics are going to cause me to actively harm the codebase in the hunt for a better score. #NotMyProblem long term and I will do it if required, but I would rather not.

I couldn't agree more! Your company has basically imposed a competition between the developers, except this competition has no real reward, no guarantee of improving company performance and honestly it's more likely to do the opposite.

Are there some other numerical (required by the guy on top of the pile) metrics that are easy to collect that we could use instead?

Yes, but you need a very experienced software engineer to keep track of these metrics (and the developers who do the work, which areas are they focusing on and how much control they have over the underlying system/legacy code).

Some useful examples of productivity metrics:

  • (PR review time-to-number of changes) ratio, if a small PR needs loads of revisions then it's a bad sign, but it's normal for larger PRs to have
  • Customer Feedback/ Bug Reports If you give customers the opportunity to rate your software then it's very beneficial to know what works and what needs work. If a certain dev gets repeatedly poor feedback on most of their work it's a bad sign.
  • Gantt Chart Accuracy Dunno if your team uses Gantt charts, they are very helpful for long term goal tracking, If your developers can plan work 8 weeks ahead of time and finish said work on time you can rest assured they are not only reliable but also realistic in their goals. Those are the two most important factors in my opinion when it comes to having a solid, productive team. You can't ask developers to achieve any goal in any time frame, but if they can consistently achieve goals in the time frame they estimate, then THAT'S AS GOOD AS IT GETS (without hiring some Rockstar contractors)
  • Story Board Points This is like a more agile way to keep track of Gantt charts, so long as your team is realistic when setting the points of each story board item, you can use the points per week metric as an indicator of progress.
1
  • Points as a measure of progress is a misuse of points. Gantt Charts are the wrong tool to use for most software development projects. Customer feedback will be on design and use, not on code. – BryanH Jun 14 at 23:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .