Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Do you know how much money you’ve made over your entire lifetime?
Probably not.
Or how many miles you’ve ran since the day you were born?
Maybe. If you aren’t a runner.
I can only think of one person who might know the answers to both of these questions. And it’s probably not you.
But I bet every single one of you know how much money you made last year.
And if you run, you probably know how many miles you ran last week.
It turns out, all-time metrics are rarely useful.
What’s more important is finding the meaningful time increment that communicates progress.
Here in the United States, many professionals are paid annual salaries, making how much money you make each year, a good measure of progress.
You might start out making $35,000 per year and work your way up to a six-figure salary. You can track your progress year-over-year.
As a runner, I need to make sure that I don’t grow my mileage too fast, week over week, otherwise I risk injury. So I usually know how many miles I ran last week, as it’s a useful gauge of how far I should run this week.
So what does this have to do with building product?
Eric Ries, in The Lean Startup, talks about vanity metrics. Vanity metrics make us feel good, but they don’t give us anything valuable to act on.
A great example of a vanity metric is tracking registered users of all time. This is a lot like tracking how much money you’ve made in your whole life. The number always goes up. So it feels good. But it doesn’t give you anything you can act on. It doesn’t tell you if you are you improving. Or if you are declining.
It’s easy to think you are improving when the number always goes up. But what if you signed up 3,000 people last week and only 2 people this week. That doesn’t look like an improvement.
Instead, you need to ask yourself, what’s a meaningful increment of time that you can measure that gives you data you can act on?
With registered users, it’s far better to track weekly or even daily sign ups. This lets you see what happens to the rate over time. Are your signups increasing? Decreasing?
Week to week, day to day, this data gives you something you can act on.
Thanks in part to The Lean Startup, many internet companies are moving away from all-time registered users as their key measure of success. There have been too many companies like BranchOut where they rapidly scaled their registered users, but had little engagement.
Many companies are now turning to monthly active users. This is a better measure. But for most companies and for those of us making product decisions, it’s still not good.
Who wants to wait a month before getting feedback on whether you are improving or not? That’s forever in internet time. We release product changes almost every day. I don’t want to wait 30 days before I know what I’m impacting.
You might argue, what about weekly active users or daily active users. These might be better. But it depends on the product.
You might not have a service that warrants weekly or daily usage. Consider textbook rental sites. You don’t rent textbooks every day, every week, or even every month. Most students rent textbooks 2 or maybe 3 times a year. For these products monthly, weekly, and daily active users are useless most of the year. Maybe the better measure is book-buying-season-active-users, whatever that means.
So first you have to ask yourself, what’s the time increment that communicates the most meaning that gives you something to act on.
Remember, the goal is to design your metrics so that you can take action on what you learn. Ask yourself, how will measuring this change my behavior?
At my company, we help people find jobs. We don’t yet have a handle on how long it takes someone to find a job. We also don’t have an idea yet of how active they need to be to get a job. So we have no idea if monthly, weekly, or daily active users are what we should be measuring.
So instead, while we are learning, we look at each week’s new signups as a cohort of users and we compare their performance to the prior week’s performance, and so on.
Right now, this is the most meaningful measure we have. It allows us to make product decisions. So for now, it’s good enough.
What’s your meaningful increment of time?
