What Changed?

The hidden risk of measuring success

For a long time, I believed the hardest part of impact was measuring it.

Looking back, I'm not sure that's true.

I think the harder challenge is remembering what we're actually trying to measure in the first place.

During my time working across education, product development and social impact, I spent a lot of time looking at numbers.

We tracked downloads, active users, unique visitors, dwell time, course completions, educators trained, confidence uplifts, educators delivering learning, and young people reached and more.

These metrics mattered.

They helped us understand whether people were finding value in what we were building. They helped us test assumptions, demonstrate progress and make decisions.

But over time I found myself returning to the same question:

And then what happened/so what?

If an educator completed a course and went on to teach young people, what changed?

If a young person engaged with content about refugees and migration, what changed?

If somebody learned about preparedness, resilience or humanitarian issues, what changed?

The more I looked for answers, the more I realised that the outcomes I cared most about were often the hardest ones to evidence.

Not because they couldn't be measured.

Because they took time.

Sometimes months.

Sometimes years.

Sometimes they appeared as a shift in behaviour, a conversation, a decision or an action that happened long after the original learning experience.

The further you move from activity towards impact, the messier things become.

And that's where organisations can face a challenge.

Outputs are easier to count, easier to report, easier to place on a dashboard and easier to hold people accountable for.

So naturally, we spend a lot of time talking about them.

The risk isn't that outputs are wrong.

The risk is that we slowly start treating them as outcomes.

I don't think the answer is fewer metrics.

I also don't think the answer is abandoning evidence in favour of stories.

The answer is understanding the role each type of evidence plays.

Downloads matter.

Reach matters.

Engagement matters.

Participation matters.

Validation matters.

But they're often the beginning of the story rather than the end of it.

Some of the most meaningful outcomes require patience, curiosity and a willingness to keep asking questions after the reporting period has ended.

Looking back, I don't think we were ignoring impact.

In many cases, we were actively trying to understand it.

The challenge was often having enough time, evidence and organisational patience to follow the chain far enough to see what had actually changed.

In one area, we recognised this challenge from the outset and built a longer-term Social Return on Investment (SROI) evaluation approach into the work. Even though my time with the organisation has ended, that work continues today.

I'm genuinely interested to see what the findings reveal.

Not because they will provide a definitive answer, but because some of the most important questions about impact can only be answered over time.

And perhaps that's the lesson I've carried with me.

The most useful question I learned to ask wasn't:

How many?

It was:

What changed?

Reflection

Looking back, one of the most important things I've learned is that outputs and outcomes are not competing ideas.

Outputs help us understand whether something is working.

Outcomes help us understand whether it mattered.

The challenge for leaders isn't choosing between them.

It's creating the conditions to understand both.

This reflection was inspired by a discussion on impact measurement and draws on my experiences working across education, product development, organisational learning and social impact.