Mike's Take: "You have to come up with a hypothesis before you look at the data... Everybody tends to look at the data after the fact and then come up with a story after the fact. But you have to figure out how to separate the thing where you can come up with a story after the fact."
This is Mike's marketing hill to die on. And it's brilliant.
Here's the problem: Humans are AMAZING at pattern recognition. Too amazing.
At SpyFu, they do this in interviews. They generate a completely random chart with random axes. Then ask candidates to explain it.
Good candidates can tell a compelling story about the chart. Even though it's completely random. No meaning whatsoever.
That's a strength in storytelling. But it's a weakness in analysis.
We can look at ANY data and create a narrative that makes sense. Even when there's no actual signal.
Vanity metrics are the obvious example. But there's deeper stuff. Cause and effect. "This campaign worked because..." when really you're just making up a story to fit the data.
Mike's rule: Write your hypothesis BEFORE you look at the data. What do you think will happen? Why?
THEN look at the results. Did your hypothesis hold? If not, why not?
This is the scientific method. And it's how actual testing works.
Otherwise you're just storytelling with numbers. And marketers are GREAT storytellers. Which means we can convince ourselves (and our bosses) that anything worked.
Also: AI has the same problem. Give ChatGPT data and ask what it means? It'll make up a story. Just like humans.
So if you're using AI to analyze marketing data, you need to be even MORE skeptical.
Takeaway: Before your next campaign, write down: What do I think will happen? What metric will move? By how much? Why? Put it in a doc. Date it.
THEN run the campaign. THEN look at results. Compare to your hypothesis.
This forces you to think clearly about what you're testing and why. It also prevents you from retroactively justifying anything that happens. Do this for every test. Every campaign.
Every major initiative. It's the only way to actually LEARN instead of just telling stories.