Measuring Measurement

Gathering demographic information is a common marketing problem, and for a long time the latest and greatest tool was mailing surveys and having people fill them out in exchange for some kind of reward.

The problem with that method of data collection is that it only measures the demographic information of a niche group of people: folks who are willing to fill out surveys which arrive in the mail. But the measurers were confident the figures were legit, so these numbers were extrapolated into representations of entire populations, leading to a lot of false-starts and failed enterprises in the business world.

Although technology has evolved, our targeting tactics have not kept up. Our measurements today are still generally quite skew. It’s not that we don’t have systems which accurately measure things — we’ve got lots of those — it’s that we are measuring the wrong things. We’re only measuring things we know how to measure.

The consequences of this are far reaching and well-embedded.

Standardized tests are perhaps the worst possible solution for determining if a student has learned something. They are perfect, however, for the quantification of students and subsequent analysis of that quantification. It’s more difficult to give students rank-worthy numbers based on essays, non-standardized projects, or other, more complex expressions of their knowledge. The result of this test-focused approach can be seen in the generations of graduates who became skilled at taking tests and achieving passing grades, but not so great at learning non-school-related things.

The professional world is also part of this system.

Consider freelance writing, which is generally paid by the word. Consider, too, that a piece of writing can be five hundred words long and utterly worthless, while another piece could contain only fifty and be the most amazing collection of words ever devised. This common method of paying freelance writers is based on the numerical, measurable number of words they submit, not the value of those words, which allows publications to easily measure if they’re getting their money’s worth out of a writer, but in doing so they’re putting measurement ahead of value. They may be overpaying or underpaying and they have no way of numerically telling the difference.

Take a look around and you’ll see variations on this same theme everywhere. What will fix it, leading to more focus on value creation and less focus on arbitrary numbers, is the advent of standards which allow us to measure value, not just numbers that are sometimes associated with it.

Developing such methods won’t be easy, of course, but it has to be done. The only alternative is to continue measuring for measurement’s sake, and all that tells us is how well we’re spinning our wheels.

Update: February 20, 2017

Another foundational aspect of life that’s been saturated by this way of thinking: business and self-worth. We use the money metric to judge how well we’re doing in life, and how well a business is doing compared to other businesses. In both cases, though, this comparison doesn’t tell anywhere near the whole story.