The Paradox of Measurement: When More Data Leads to Less Understanding

We have more ways to measure success than ever before. So why can it still be so difficult to know if we’re making a difference?

I spend a lot of time helping organizations measure things.

Attendance. Engagement. Giving. Participation. Retention. Growth.

We build dashboards, define metrics, establish benchmarks, and track trends. We compare this month to last month, this year to last year, and actual performance against our goals.

And all of that can be incredibly valuable.

But I’ve noticed something interesting after 20 years of helping organizations use data to better understand their impact.

Sometimes the more we measure, the less we understand.

Not because the numbers are wrong.

But because somewhere along the way, we start confusing what we can measure with what actually matters.

When the measure becomes the mission

Imagine a nonprofit whose mission is to help families achieve long-term stability.

They measure the number of families served, the number of appointments completed, and the number of resources distributed.

Every month, those numbers go up.

By every metric on the dashboard, things are getting better.

But here’s a question:

Are the families actually becoming more stable?

We don’t necessarily know.

We know the organization is doing more. We don’t yet know whether that activity is producing the change it exists to create.

Now imagine a church measuring attendance, volunteer participation, and small group enrollment.

Those numbers may tell us something important about the health and activity of the organization.

But do they tell us whether people are experiencing belonging? Whether relationships are deepening? Whether lives are changing?

Not necessarily.

And that distinction matters.

Because when we become overly attached to a measurement, something subtle can happen.

The measure stops being a way to understand the mission and starts becoming the mission itself.

The illusion of understanding

There’s something comforting about a number.

Numbers feel objective. Concrete. Certain.

A dashboard with green arrows pointing upward can reassure us that things are going well. A downward trend can create urgency to act.

But a number is not the same thing as understanding.

An increase in attendance might indicate growing community engagement. Or it might reflect a change in how attendance is counted.

A decline in program participation might signal a problem. Or it might mean people no longer need the program because it accomplished what it was intended to do.

The same number can tell very different stories.

And without curiosity, context, and conversation, we may never discover which story is true.

That’s the paradox.

We collect more information in pursuit of clarity, yet the information itself can give us a false sense that we already understand.

And once we believe we understand, we stop asking questions.

When a measure becomes a target

There’s a principle known as Goodhart’s Law, commonly expressed this way:

When a measure becomes a target, it ceases to be a good measure.

Think about that for a moment.

When we choose a metric to represent success, we naturally begin organizing our behavior around improving that metric.

If success means serving more people, we’ll find ways to serve more people.

If success means increasing participation, we’ll focus on increasing participation.

If success means growing attendance, we’ll develop strategies to attract more attendees.

None of those things are inherently bad.

But here’s where it gets interesting.

It’s possible to improve the measurement without improving the thing the measurement was intended to represent.

A nonprofit could increase the number of families served by reducing the amount of time spent with each family.

A church could increase volunteer participation while failing to notice that its most committed volunteers are becoming exhausted.

A program could celebrate rising enrollment while overlooking declining outcomes for the people it serves.

The metric improves.

The mission might not.

And if we’re not careful, we can become so focused on achieving the target that we stop questioning whether it’s still pointing us toward the outcome we actually care about.

What gets lost when we only measure what we can count?

Some of the most meaningful things organizations hope to accomplish are also the hardest to measure.

Trust.

Belonging.

Hope.

Transformation.

Human flourishing.

That doesn’t mean we shouldn’t try to understand them. It means we need to recognize the limitations of any single metric.

A number can be an indicator of something meaningful without being the thing itself.

A person’s participation in a program might suggest engagement. It doesn’t tell us everything about their experience.

A family’s continued involvement might suggest they feel supported. It doesn’t necessarily tell us whether their circumstances are improving.

And sometimes the most significant impact won’t show up in our reporting systems at all.

I think this presents an interesting challenge for leaders.

How do we remain accountable to outcomes without reducing those outcomes to whatever fits neatly on a dashboard?

How do we use measurement to deepen our understanding rather than narrow our attention?

And what happens when the things that are easiest to measure begin receiving more attention than the things that matter most?

What if measurement was the beginning of the conversation?

I’ve come to believe that the greatest value of data isn’t always the answer it provides.

Sometimes it’s the question it makes possible.

Instead of seeing a 15% decline in participation and immediately asking how to reverse it, what if we asked what the decline might be telling us?

Instead of celebrating a record number of people served, what if we asked what changed for those people because we showed up?

Instead of asking which metrics should be on our next dashboard, what if we started with a different question?

What are we trying to understand?

That question changes the relationship between data and leadership.

Data becomes less about proving that we’re succeeding and more about helping us discover what’s actually happening.

It becomes an invitation to curiosity rather than a declaration of certainty.

And perhaps most importantly, it gives us permission to find something we weren’t expecting.

We might discover that a program we thought was thriving isn’t creating the impact we imagined.

Or that something we’ve barely been measuring is producing some of our most meaningful outcomes.

We might even discover that we’ve been asking the wrong question all along.

That’s not a failure of measurement.

That’s measurement doing exactly what it should.

The courage to question the numbers

I’m not suggesting we stop measuring.

Quite the opposite.

After 20 years working with data, I believe more than ever in its power to help organizations create meaningful change.

But I also believe that the purpose of measurement is not to make us feel certain. It’s to help us see more clearly.

And seeing clearly sometimes means discovering that our assumptions were wrong.

It means being willing to question a metric we’ve celebrated for years.

It means acknowledging that an impressive result might not tell the whole story.

It means asking whether we’re getting better at accomplishing our mission or simply getting better at producing the numbers we’ve chosen to track.

Those aren’t always comfortable questions.

But they might be some of the most important ones we can ask.

Because the goal was never to have the best dashboard.

The goal was to make a difference.

And those are not necessarily the same thing.

A question for your leadership team

If every metric on your dashboard improved next year, how would you know whether your organization had actually made a greater impact?

And perhaps the more uncomfortable question:

What could be getting worse while all your numbers are getting better?