The Ladder of Evidence: Get More Value From Your Customer Interviews and Product Experiments

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

One of the best signs of an effective product team is a regular cadence of customer interviews and product experiments.

But it’s not enough to simply check the box on these activities; we need to make sure that we do them effectively.

Without a strong background in research methods, it’s hard to know how you are doing. I’ve developed a simple framework to help teams evaluate the effectiveness of their methods. It’s called the Ladder of Evidence and I explain it in the video below.

Can’t see the video? Click here.

Watched the video and want to learn more? See these related articles.

Full Transcript

The Ladder of Evidence will help you get more value out of your customer interviews and your product experiments.

Hi Everyone, I’m Teresa Torres with Product Talk and I want to take ten minutes to walk you through the Ladder of Evidence. Many of us have learned that focus groups and surveys are not the best tools for figuring out what to build, but do you know why? The Ladder of Evidence will tell us. It also helps us get more value out of our customer interviews and our product experiments. Let’s take a look.

The more effort you are willing to put into climbing the Ladder of Evidence, the more value you’ll get out of what you learn.

When we want to learn something, we tend to ask about it directly. We ask questions like, “Would you use this feature?” or “Would you pay for this service?” The problem with these types of questions is that we aren’t very good at speculating about our future behavior. The answers to these types of questions aren’t reliable. It’s easy for me to say I’ll go to the gym next week. That doesn’t mean I will go to the gym next week. It’s important when asking customers for feedback that we do our best to collect reliable feedback. The Ladder of Evidence is going to help us do just that.

You can see here on the left that as we climb the ladder, the time and effort to collect the feedback goes up, but so does the value of what we learn. This is a delicate balance. We want to learn as much as we can, while moving fast, and continuously shipping value to our customers. We don’t want our engineers sitting around while we conduct perfect research, but we also don’t want to build the wrong functionality because we got unreliable feedback. Instead, we want to climb as high up the ladder as we can, given the time constraints in which we have to work.

You don’t need to do perfect research, but you do need to collect reliable feedback. – Tweet This

Focus groups and surveys are often used to ask people what they might do in the future. This isn’t the only way to use these tools, but it’s the most common, and it’s why they have a bad reputation. You can see that asking people about what they would do is the lowest rung on the ladder. It’s easy to do, but it doesn’t lead to valuable feedback.

Asking people what they would do is easy to do, but doesn’t lead to reliable feedback. – Tweet This

Instead, we want to climb higher. The next rung is to ask people about what they’ve done in the past. Our past behavior is a better indicator of our future behavior than our speculation about our future behavior. In other words, I’m more likely to go the gym next week if I went to the gym last week than if I’ve never been to the gym but think I’ll go next week.

But we can do even better. If you are going to ask about my past behavior, you might as well climb one more rung and ask me for specific stories about my past behavior. Here’s why: If you ask me if I work out regularly (a question about my past behavior), I will say yes. If you then ask me what I do, I would say yoga, CrossFit, and running. You might also ask how often I work out. Again, I have quick answers. I do yoga twice a week and CrossFit three times a week. But all you are doing is collecting facts. Insights are rarely generated from simple facts.

Rather than asking me for facts, ask me for specific stories about my workouts. Stories convey context like where the workouts happen, with whom, the emotion associated with the workout (did I love it, was it a grind), how I felt beforehand (was it hard for me to go, was I excited to go), how I felt afterwards (was I glad I went, did I regret it), and so much more.

Ask for specific stories about the past to generate insights. – Tweet This

Here’s an example: Yesterday, I went to yoga. At 11:30am, I was wrapping up my last coaching call. It ran a little long and I had to rush out the door. Because I was in a rush, I forgot my water bottle. Halfway there I remembered that I had a 1:30pm phone call and that I should have brought my phone just in case class went long. I laughed at my own busyness as I raced to get to yoga on time. As soon as I walked into the yoga studio, I relaxed. Jill, my favorite instructor, was smiling at the front of the room. Michael, my brother-in-law, had saved a space for me. I sat down on my mat, took a deep breath and was ready to sink into yoga. It was a hard class. I’ve been dealing with a foot injury and it was hard to stay present while attending to an injury. I got frustrated a few times but just kept coming back to my breath. At the end, I was mentally and physically exhausted, but I felt great. I walked home, calm, quiet, and smiling. I laughed at the difference an hour of yoga makes.

It took me less than three minutes to tell this story. And it’s packed with so much rich context. If you were designing a product or service for me, you learned so much more than the quick facts that I rattled off earlier. There’s so much to explore here. The role yoga plays during my work day, the people that I see there, the mental and physical benefits, the scurry out the door, the amazing feeling at the end. There is so much context, emotion, and richness that you aren’t likely to forget this story.

And it gets even better. Because this is a specific story about the past, it’s much more reliable than the factual answers I gave earlier. If you asked me why I go to yoga, I would probably say because I care about my fitness, but if you look at this story, I go to yoga for many reasons where fitness is only one of them. Others include: it’s a break during my workday, it makes me feel great, I see friends and family there, it relaxes me, it’s a challenge. There’s so much more to work with.

Asking people for specific stories about the past is one of the richest ways to generate insights about which opportunities to go after and which solutions might deliver on those opportunities.

But no matter how good you get at collecting specific stories, you still have to test those insights. And that brings us to the top two rungs of our ladder.

Too often, when we are ready to test a solution, we put it in front of people and ask, “What do you think?” This is the wrong question to ask. Again, we want to stay away from speculation and look for action. Instead, we want to simulate an experience and prompt them to take action.

The key is to try to simulate the experience your customer would have if your product or service was real. This can be challenging, but it’s worth the effort. You don’t want to learn after you build it that people like it, but won’t use it. We want to discover whether or not they will use it as early in the process as possible.

Evaluate solutions by simulating an experience and then measuring action. – Tweet This

If based on the stories I collected, I learn that people view yoga as a good break during their work day, I might look to offer lunchtime yoga programs to big companies. To test this idea, I don’t want to ask people if they would go to lunchtime yoga at their office. This is an easy question to say yes to, but it’s pure speculation. Instead, I want to simulate the experience. I might start by offering one lunchtime yoga class at a company and measure how many people come. If I’m worried about attendance over time, I might offer lunchtime yoga for a couple of weeks, before evaluating the results.

Of course, I won’t know for sure if this program will work until I actually offer it (the top rung of the ladder). But the closer I can simulate the experience without doing all the work to make it scalable or sustainable, the more I’ll learn about my idea.

You’ll notice that the top two rungs each have two options. We’ve already talked about the options on the right. The options on the left are for when you are building a product or service that is similar to a competitor’s product. In this case, you can use existing solutions to test your idea. Instead of simulating an experience, you can ask a user to show you what they do using your competitor’s product. And instead of waiting until your product or service is live, you can observe them during a real-life experience using a competitor’s product.

Here are the key takeaways from the ladder of evidence:

1. Avoid speculation.
2. Ask for specific stories about the past to generate insights.
3. Look for evidence of action to evaluate solutions.

I’d love to hear your thoughts or feedback on the Ladder of Evidence. Leave a comment below, reach out on Twitter, or send me an email.

Thanks for your time. If you have questions or comments about The Ladder of Evidence please reach out at ProductTalk.org or on Twitter. Until next time, keep embracing your doubt and questioning your assumptions.

Related Articles

• Ask About the Past Rather Than the Future
• Why You Are Asking the Wrong Interview Questions (And What To Do Instead)