Measure, test, measure, compare

Pfeffer and Sutton on “managing by facts”

Part of the problem with metrics is that so much of what we do seems so fuzzy.

Strategy and Business suggests “using hard facts, such as qualitative or quantitative data, to make strategic decisions is the clearest path to the best business choices. Yet many executives ignore the facts and make ‘gut’ decisions based on fads or hunches.” The article by Jeffrey Pfeffer and Robert Sutton, “Why managing by facts works”, misdirects the conversation towards data: “Although there’s great value in keen intuition and fresh ideas, evidence-based management leads to competitive advantage.”

Unfortunately, data means squat.

You only reap an advantage when you use quantitative or qualitative data to help evaluate your actions, to help you learn. Overall, Pfeffer and Sutton’s examples imply the need to evaluate and learn, but by focusing on data, they miss the key piece of the equation.

They’re focusing on the task (gather data), but missing the goal (to learn).

And because of that, they miss the opportunity to explain how to evaluate abstract, fuzzy notions like “intuition and innovation” To be fair, they do offer hints:

By emphasizing the importance of evidence and knowledge, we do not mean to dismiss the value of intuition and innovation. But even hunches, fresh ideas, and inventions should be measured against logical and empirical benchmarks to determine whether they are efficacious ideas or just momentarily exciting thoughts better off abandoned.

Reading between the lines, it’s clear intuition and hunches are fine as long as you document your assumptions so you can evaluate them later. Secondly, evaluate frequently. Do decisions that stem from these assumptions continue to offer the results you expect? And if not (here’s the evaluation) why?

This might seem like a lot of hair-splitting, but read wrong, Pfeffer and Sutton’s article explains why you can’t try innovative, new, and fresh ideas. Since that’s what designers do, I figure it’s best we understand exactly how to evaluate (and justify) new and fresh. And just as importantly, we also need to understand that new and fresh — dare I say it — Innovation should also be measured, evaluated, and justified.