We all know it: the best programmers are at least ten times as productive as the worst ones. How not to believe it, when it is preached on us by the holiest of the evangelists:
Brooks, Mythical Man-Month:
In one of their studies, Sackman, Erikson, and Grant were measuring performances of a group of experienced programmers. Within just this group the ratios between best and worst performances averaged about 10:1 on productivity measurements and an amazing 5:1 on program speed and space measurements!
DeMarco and Lister, Peopleware:
Count on the best people outperforming the worst by about 10:1.
McConnell, Code Complete:
Although specific ratios such as 25 to 1 aren’t particularly meaningful, more general statements such as “There are order-of-magnitude differences among programmers” are meaningful and have been confirmed by many other studies of professional programmers.
For a long time, I accepted this as the truth, although there was something uneasy about it. Then one day I was talking about recruiting with my boss and I evoked the “let’s only hire tens and not the ones” -argument and she said:
“It is true that the best programmers are more than ten-times as productive as the worst ones. It is true, because the worst ones don’t produce anything at all.”
And you know, it is true. Somewhere, there is a Wally who never produces anything. If you want to outperform him tenfold, you only need to produce something, anything at all.
So the problem with the 10:1 statement is not that it is false, but that it is so obviously true that it doesn’t say anything interesting.
How to fix it? The original question — how large is the variance in productivity between programmers — is still an interesting one. The way to say something meaningful about it is to step up from individual comparisons to statistical comparisons. If, instead of comparing THE worst programmer with THE best programmer, we compare the group of the worst (let’s say bottom 25%) with the group of the best (to 25%), the ratio should be more meaningful.
What is that ratio? A compilation study by Lutz Prechelt gives a ratio of 7.1, but that applies only to pure programming tasks. On debugging, the ratio is 3.2, and on other development activities, it is lower still.
Nowadays, instead of saying “the ratio in productivity between the best programmer and the worst is about 10:1”, I say:
“The top 25% of programmers are in most programming tasks more than five times as productive as the bottom 25%.”
It would be interesting to know how good the top 10% or the top 5% are compared to the rest. Do you know any papers answering that?