Vanity Metrics vs. Actionable Metrics

Vanity metrics: good for feeling awesome, bad for action. (photo source: UK Guardian)

This is a post by serial entrepreneur Eric Ries. He was most recently co-founder and CTO of IMVU, which has more than 20 million registered users and generates $1,000,000+ in revenue per month. Eric is also a venture advisor to Kleiner Perkins.

How do you get to $1,000,000 per month in sales? By testing the right things. Eric is a metrics man.

Here is just one business-changing example, taken from the outstanding “How IMVU Learned its way to $10M a year” on Venture Hacks…

IMVU learned its way to product/market fit. They threw away their first product (40,000 lines of code that implemented an IM add-on) as they learned customers didn’t want it. They used customer development and agile software development to eventually discover customers who would pay for 3D animated chat software ($10M in revenue in 2007). IMVU learned to test their assumptions instead of executing them as if they were passed down from God.

Enter Eric Ries…

The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do.

When you hear companies doing PR about the billions of messages sent using their product, or the total GDP of their economy, think vanity metrics. But there are examples closer to home. Consider the most basic of all reports: the total number of “hits” to your website. Let’s say you have 10,000. Now what? Do you really know what actions you took in the past that drove those visitors to you, and do you really know which actions to take next? In most cases, I don’t think it’s very helpful.

Now consider the case of an Actionable Metric. Imagine you add a new feature to your website, and you do it using an A/B split-test in which 50% of customers see the new feature and the other 50% don’t. A few days later, you take a look at the revenue you’ve earned from each set of customers, noticing that group B has 20% higher revenue per-customer. Think of all the decisions you can make: obviously, roll out the feature to 100% of your customers; continue to experiment with more features like this one; and realize that you’ve probably learned something that’s particular valuable to your customers.

//

Read the rest of the post for more, one of the best articles I've read recently. I pasted a bare bones summary below:

1. Split-tests
2. Per-customer metrics.
3. Funnel metrics and cohort analysis.
4. Keyword (SEM/SEO) metrics.

Summary:
1. Measure what matters. It’s tempting to think that, because some metrics is good, more metrics is better. That’s why vendors routinely list the thousands of reports they are capable of generating as a feature. The truth is, the key to actionable metrics is having as few as possible. Detailed reports are useful when we’ve diagnosed a problem and are looking for clues as to what’s gone wrong. But where does that diagnosis come from in the first place? Actionable metrics help us realize we have a problem and point us in the right direction to start solving it.

2. Metrics are people, too. Great metrics tools allow us to audit their accuracy by tracing reports back to the individual people who generated their data. This improves accuracy, but its more important effect is that it lets us use the same customers for in-depth qualitative research. Not sure what the numbers mean? Get the customers on the phone and ask them.

3. Measure the Macro. Lastly, even when we’re split testing the impact of a minor change, like a wording or a new button, it’s important not to get distracted by intermediate metrics like the click-through rate of the button itself. We don’t care about click-through rates, we only care about the customer behaviors that lead to something useful,