Choose Your Units of Measure Wisely

I have a pet metric that I call Time Estimation Error. It’s the percentage of time that a person is over or under, on average, for all of their time estimates over a given period. For example, if over 10 tasks, a person estimated 20 days, but took 30 days to do, their Time Estimation Error is 50% (overrun).

This kind of tracking is built into Devshop so that it helps you get to know people over time and build better schedules by automatically adjusting for these kinds of factors. So eventually, when someone tells you something is 10 days work, you’d know that it’s likely 12 or 14, depending on that person’s accuracy rate.

It occured to me a few weeks ago that perhaps a percentage is the wrong unit of measure for a factor like this. What’s in a unit, you ask? Well, for one thing, perceived boundaries. Sure, we’ve all seen numbers like 300% and 400% (figures that are more than the soft maximum of 100%), but if you blindy ask someone what they think a good percentage is for something, anything, I bet there’s a statistically better chance that they’ll give you something within the bounds of 0% to 100%, not even considering anything over 100% – unless you really push them to think hard.

Now with estimating time on a software project, typically in clumps of days per feature, my experience has been that people are off by as much as several times their original estimate. Like a 2 day task that actually ends up taking 5 days (= 150% overrun or a 2.5 times multiple of the original estimate).

I’m beginning to think that a multiple (of x times) or decimal value would be better at getting the point across, than a percentage which tends to “lead” people into giving certain ranges as a response to a question.

Of course, if Devshop’s tracking it for you then it doesn’t matter, you’ll get an accurate figure anyway, but I think the unit of measure chosen to track a metric like this has a bearing on the results you get, if you’re asking human beings for the answer (guestimates).

By the same reasoning, when a lot managers “buffer” for unforseen difficulties during a software project, you’ll often see them use percentages – like a buffer of 20% for some particularly risky component. I think in many cases people would be better served by thinking of risky parts of software projects in terms of multiples, as if the not so risky parts of a software project (an oxymoron) aren’t tough enough to nail time estimates for.