The argument in brief
- The question
- What makes a report worth sending, rather than just a summary of numbers everyone could look up?
- The claim
- A report that only restates the data has no actionable insight. A good report digs into the data before it is sent, reports the cause rather than the symptom, proposes what to do next and checks what last cycle's actions achieved. Confidence: A practitioner's view from my own experience of e-commerce reporting, stated firmly. No studies are cited, and the example figures are made up.
- The evidence
- My own experience of sending and receiving email reports on PPC, organic search, content and e-commerce performance.
- An illustrative example report and an illustrative 5 whys chain. Neither is real data.
- The 5 whys comes from the Toyota Production System, where Taiichi Ohno described asking why five times as the basis of its approach to problems.
- The best counter
- Asking why repeatedly can lead to one tidy story when several causes are at work, and an analyst can stop wherever their existing beliefs say the cause is. Some reports exist to keep people informed, not to drive action.
- What would change my mind
- Evidence that teams who receive cause-and-action reports make no better decisions than teams who receive plain summaries, or that root-cause chains built from analytics data are usually wrong when tested.
The question: What makes a report worth sending, rather than just a summary of numbers everyone could look up?Do you ever send email reports to colleagues where you’ve pulled together data from various systems and summarised it for management and the wider team? Do you receive email reports from your team? Let me tell you the biggest mistake I’ve experienced time and time again.
You see, this is quite a frequent occurrence in my experience. Maybe it’s the results of last month’s PPC campaigns or an organic search performance update. Maybe you’re publishing blog posts and sharing what impact they’re having. Or it could be a report pulled out of Google Analytics looking at e-commerce conversion rates and revenue for a sales period.
More often than not, these reports give a warm fuzzy feeling about whether something is good or something is bad. But when you stop and look, you’ll discover there is a fundamental flaw with these reports.
An example report
Illustrative: These figures are made up to show the shape of a typical report. They aren’t from any real business.Let’s take an example. You’re comparing this month’s Google Analytics figures to last month’s for your e-commerce store. You may have the following figures:
Revenue: £1,323,222 (up from £1,192,331) +11%
E-commerce conversion rate: 2.8% (down from 3.1%) -9.7%
Average order value: £168 (up from £145) +15.6%
On the face of it, these figures look great: you’re 11% up on revenue and better on average order value. Conversion rate is down though, but that’s okay, we’re making it up in revenue.
And that’s it. Maybe a congratulatory “well done team” email from management and we all get on with our day.
The biggest flaw
Claim, confidence:firmly held, from my own experience of reporting: The biggest flaw in this type of report is that there are no actionable insights. It’s just reporting the data (which, let’s not forget, Google Analytics does automatically). All we’ve done is use a human as the notification of that data.
I would look at this data and immediately have questions.
- Was the increase in revenue seasonal, or due to an external promotional push?
- Did any channels contribute to the extra revenue? For example, did we increase spend in PPC to generate the extra revenue, which could be masking a downward trend in organic traffic?
- What caused the decrease in e-commerce conversion rate?
- If average order value is up, is that because we are bundling more items per order, or did our customers buy a more expensive product?
An assumption: I’m assuming the person writing the report has the time, access and permission to dig into the data. Often the report is a summary precisely because nobody was given time to do more.The list of questions could go on, and results in a list of hypotheses that you need to prove or disprove. It’s guesswork, and we’re not letting the data reveal the true story.
Report the cause, not the symptom
Every time I look at a report I ask myself: is this an actionable insight of the cause, or is it just the symptom of the real insight? Claim, confidence:a working rule I hold firmly; not tested against alternatives: For a report to be great, it needs to do two things.
- Report on the cause, not the symptom.
- Provide actionable insights about what we must do next, or whether to keep doing the same.
It is critical to empower your team to dig into the data before reporting it. They should naturally be able to segment by channel and device to highlight any sub-trends that are occurring.
The 5 whys
The 5 whys principle gives you the opportunity to really find the root cause. It’s used in lean manufacturing to make sure the root cause of issues is identified and fixed, not just the symptoms. This prevents the same thing happening over and over.Source: Ries says he learned asking why five times from Taiichi Ohno’s book on the Toyota Production System, which Ohno helped invent.The Simple, Productivity-Boosting Art of Asking Why, Eric Ries, Next Big Idea Club, 2015-12-19Source: Says the technique was developed at Toyota, that Ohno called it the basis of Toyota’s scientific approach, and that it is used in lean manufacturing. It also lists criticisms, including its arbitrary depth of five.Five whys, Wikipedia
Illustrative: This chain is invented to show the method. In a real report, each answer would need checking against the data before you moved on to the next why.Let’s apply this to our example report:
E-commerce conversion rate is down 9.7%.
Why?
Because our bounce rate has increased and customers are leaving.
Why are they leaving?
Because we are seeing an increase in visitors that bounce on our product page, so the product page isn’t good enough.
Why is the product page not good enough?
It’s happening on mobile, but not desktop and tablet, so the product page doesn’t work as well on mobile.
Why does it not work well on mobile?
Because the product image is below the fold.
Why is the product image below the fold?
Because we launched a sales campaign with a big countdown banner at the top, pushing the product image down.
A competing explanation: Asking why in a single line tends to produce one neat story, when a drop in conversion rate usually has several causes at once, and people tend to stop at a cause they already suspected.As you can see, the data can drive a great narrative on where to find some actionable insight. In this example a product image could be the cause of an increase in bounce rate, and thus a decrease in e-commerce conversion rate. This would have been masked because the cause was also pushing revenue up, thanks to the sales campaign.
Actionable insights
By performing the root cause analysis, you will discover some interesting questions along the way. In this example, it may be “does the product image below the fold impact conversion rate?”
Each report should then have a reasonable set of actionable insights that can be carried out in the next cycle. In this case it would be to propose an A/B test on the product image.
Measure what last month’s actions did
What would change my mind?: Evidence that teams who receive cause-and-action reports make no better decisions than teams who get plain summaries, or that root-cause chains built from analytics data usually turn out wrong when they’re tested.Finally, these actionable insights should not be fire-and-forget. You need to make sure the action taken was worthwhile. This may be improving things such as the conversion rate, or learning from your actions. The report should summarise the outcome of any previous actions and what impact they have had on the current report.
If you don’t take action from the data, and you don’t measure the impact of that action, then there is no point reporting on the data, unless you just want to be a spectator and see what happens through natural causes.
What are your thoughts? Do you agree with this perspective, or do you have a better way to approach actionable data?
Sources
- The Simple, Productivity-Boosting Art of Asking Why, Eric Ries, Next Big Idea Club, 2015-12-19
- Five whys, Wikipedia
Books behind this essay
Not sources for any claim here. These are books I have finished that shaped how I think about this question. Where I rely on evidence, it is cited in the text.
The Lean Startup (opens audible.co.uk, external site)
Shaped how I think about asking why repeatedly and about measuring whether what we changed actually worked.
Black Box Thinking (opens audible.co.uk, external site)
Shaped how I think about looking for the real cause of a result instead of settling for the first comfortable explanation.
Thinking in Systems (opens audible.co.uk, external site)
Shaped how I think about one change, like a sales banner, rippling through a system in ways a headline number hides.
Measure What Matters (opens audible.co.uk, external site)
Shaped how I think about tying the numbers we report to the actions we commit to next.
