Surrogation: When an Accurate Number Answers the Wrong Question

My daughter Khadija came to me, very pleased with herself: “Dad, did you know a giraffe can handle thirst better than a camel?!”

That surprised me. It goes against almost everything we know about camels.

So I looked it up and asked Gemini, which confirmed it, at least on the surface. But when I dug a little deeper, I found a lovely little fallacy.

There’s a difference between two questions:

Which animal can go longer without drinking?

And which one is better at coping with dehydration and losing body water?

Giraffes get a lot of the water they need from the leaves they eat, so they can go a long time without drinking.

Camels, on the other hand, have remarkable physiological adaptations that help them cope with dehydration and fluid loss.

Watch the two from a distance and you would indeed see the giraffe go longer without going near the water. That doesn’t mean it isn’t drinking.

Drinking less isn’t necessarily proof of coping better with drought!

The fact itself isn’t necessarily wrong. The problem is that we used a measure of one thing to judge something else, and ended up with a misleading conclusion.

The surrogation trap

It reminded me of an important idea in management accounting called surrogation.

We pick a metric to measure something, and after a while we forget it’s only a stand-in. We start treating the metric as if it were the thing itself.

This isn’t just a theoretical point. Three management accounting researchers gave the trap its name in a 2012 paper in The Accounting Review, and later research showed it can happen even when no pay is tied to the metric. Having the number in front of us is enough.

Great number, right?

Let me give you a clearer example.

A company announces that 99% of its client fund settlements were completed successfully.

Great number, right?

Now suppose the remaining 1% accounts for 40% of the total value that needed to be settled!

The 99% is perfectly accurate. But does it mean the company settled 99% of the money?

Of course not.

We measured the number of transactions and used it to draw conclusions about value and risk.

Credit portfolios work the same way.

In one portfolio, 98% of borrowers are paying on time. In another, it’s 95%.

Is the first one necessarily less risky?

Not until we know the size of the exposures, the risk grades, the concentrations, the effect of rescheduling and so on.

Which shows that choosing a KPI is never just a reporting decision.

You aren’t only choosing a number to show management. You’re choosing the angle from which management will see reality.

Not Goodhart’s Law

This trap is different from Goodhart’s Law, which we covered in the fourth trap of this series.

With Goodhart’s Law, turning a measure into a target can push people to improve the number at the expense of reality.

Here, nobody has to manipulate anything. The data is clean and the calculations are right, yet the conclusion is wrong, because we’re using the measure to answer a question it was never designed to answer.

So before celebrating a KPI or building a decision on it, it’s worth asking ourselves:

What are we actually measuring? And what are we assuming this number proves?

Because sometimes the most dangerous financial report isn’t the one with the wrong numbers.

It’s the one with perfectly accurate numbers that answer the wrong question.

Surrogation: When an Accurate Number Answers the Wrong Question

The surrogation trap

Great number, right?

Not Goodhart’s Law

Further reading