Posted on September 5, 2012 by Colin

Measuring Measurement

Gathering demographic information is a common marketing problem, and for a long time the latest and greatest tool was mailing out surveys and having people fill them out in exchange for some kind of reward.

The problem with that method of data collection is that it only measures the demographic information of a niche group of people: folks who are willing to fill out surveys that arrive in the mail. But the measurers were confident the figures were legit, so numbers were continuously extrapolated into representations of entire audiences, leading to a lot of false-starts and failed enterprises in the business world.

Although technology has evolved, our targeting tactics have not kept up. Our measurements today are still generally quite skew.

It’s not that we don’t have systems that accurately measure things — we’ve got lots of those — it’s that we are measuring the wrong things. We’re only measuring things we know how to measure.

The consequences of this are far reaching and incredibly well embedded.

It’s not just marketers who fall prey to slanted information. Standardized tests are perhaps the worst possible solution for determining if a student has learned something. They are perfect, however, for quantification of students and subsequent analysis of that quantification.

It’s far more difficult to give students rank-worthy numbers based on essays, non-standardized projects, or other, more complex expressions of their knowledge. The result of this can be seen in several generations of graduates who are excellent at taking tests and getting good grades, but not as great as they could be at learning.

The professional world is also part of this system.

Consider freelance writing, which is generally paid by the word. Consider, too, that a piece of writing can be five hundred words long and utterly worthless, while another piece could contain only fifty and be the most amazing collection of words ever devised. The most common method of paying freelance writers is based on the numerical, measurable number of words they submit, not the value of those words. This allows publications to easily measure if they’re getting their money’s worth out of a writer, but in doing so they’re putting measurement ahead of value. They may be overpaying or underpaying and they have no way of statistically telling the difference.

Take a look around and you’ll see variations on this same system everywhere. What will fix it — leading to more focus on value creation and less focus on arbitrary numbers — is the advent of standards that allow us to measure value, not just numbers that are sometimes associated with it.

Developing such methods won’t be easy, of course, but it has to be done. The only alternative is to continue measuring for measurement’s sake, and all that tells us is how well we’re spinning our wheels.