Doing metrics right

How I make it effortless for decision-makers to read and act on metrics as an analyst

Recently while working on a major data problem, the solution I came up with was going to seriously change the data. So to give my stakeholders a glimpse of how hot this sauce(data) is before I smother it on their wings(dashboards), I decided to run a test to get the BEFORE v/s AFTER picture. After the test, I was excited to see what changed so I created a comprehensive 5 page report with all kinds of metrics and how they changed. To my surprise, this wasn't very well received, people looked at it and then looked away and even after 2 whole days I had no feedback.

"The opposite of love is not hate, it's indifference."

I knew something had to change in the way I report metrics which led me on a journey on how to do so. Here are the 7 things I learned:

• Don't show all your cards

As opposed to your internal urge to show everything you worked for, leaving out a few is the smart thing when it comes to metrics. Now I don't mean to keep information, what I do mean is to give the decision-maker a clear picture to act on by sifting through the dirt yourself. Anything that doesn't add value to the decision being made can be taken out. In my case, I had a bunch of metrics that were merely a sanity check for things that shouldn't change. If the test doesn't break anything, they don't provide any information. If it does, reporting metrics doesn't make sense anyway. So I kept these for my eyes only.

• Choose your north star

Before I was reporting the same metric before and after in multiple ways - absolute value, difference, and percentages. However, most of these didn't have much significance on their own. I realized as the goal was to see how the test impacted the data, the only thing worth looking at would be the 'Percent Change' for different metrics. Picking a north star metric like this makes life simpler.

• Influence attention

This may seem silly at first but has a ton of value. Looking at a report with many numbers can be intimidating for anyone, especially if you don't know where to start. As an analyst, your aim should be to get your reader's attention to the most important thing right away. The easiest way to do this is by using colors! In my case, I highlighted the percentage change for the most important metric in yellow and the most alarming one in red to give an overview and a starting point. Color coding tables using a heatmap is another good way to do this.

• Address rebuttals

I got this one I got from X(Twitter) and have been using it every time I send out an email or share results. It's to take a minute before sharing to think about what counter questions the reader might have like "Why is this metric so high?" and address it before they ask. Revealing the skeletons in your closet is not a bad thing in any relationship. I did this by annotating certain metrics with comments that explained the deeper context behind the change.

• Show the path

Most people just want the conclusion right away rather than going through an entire report and making it for themselves. As an analyst with some experience, you generally have an idea about what could be done with your results. So while reporting results, I just add a few sentences for plausible NEXT STEPS. Many times folks would just take your word for it, which is why it's important to be careful here and only make suggestions if you feel confident about the results.

• Make it actionable

How you present your results sets the mindset for what happens next. Your goal is to put the decision-makers in a mindset where they look at the metric and start thinking about how to influence it positively for your business. To do so you should pick actionable metrics. For example in email marketing, the click-through rate is an actionable metric as opposed to the number of emails sent(a sanity check).

• Don't just say it, show it

Lastly, we all know a picture is worth a thousand words, so why not use a picture over a table to report results? A nice hack I've been doing lately is asking ChatGPT to visualize metrics for me. It will do the work to find the best way to show these metrics and it's so much easier to digest for others.

Here's a quick summary:

• Don't Show All Your Cards: Sift through the data yourself and present only what's valuable for decision-making.

• Choose Your North Star: Select a 'north star' metric to simplify and focus your analysis.

• Influence Attention: Draw attention to critical data points using color coding to simplify comprehension.

• Address Rebuttals: Proactively tackle potential queries by adding explanatory annotations to your metrics.

• Show the Path: Include suggestions for action to direct the reader towards productive outcomes.

• Make It Actionable: Present actionable metrics to prompt decision-makers to think strategically.

• Don't Just Say It, Show It: Enhance data comprehension and retention with visual representations over text-based tables.

If you’re into this, you can find me on X @abhishek27297 where I talk data.