Why Testing Everything Doesn’t Work

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

Please stop throwing spaghetti at the wall.

It doesn’t work.

I know you can launch a landing page in five minutes. I know it’s easy to toss up a quick A/B test. And I know how easy it is to chat with the 3 people sitting next to you in the coffee shop about your hot idea.

This isn’t experimenting. Nor is it validation.

In fact, it’s a waste of time.

Now don’t get me wrong. I’m not saying don’t experiment. I’m saying stop experimenting at random.

Be Strategic About What You Test

Over the last couple of weeks, we looked at how to generate and evaluate ideas. You should have a list of ideas that are consistent with your vision that you suspect will help drive progress toward your goal.

Most teams will take this list, prioritize it, and start building.

Don’t do that.

Go ahead and prioritize your list based on your best guess at expected outcomes. How will this idea impact your goal? By how much?

But do realize that these are guesses at best and are likely wrong.

This is hard for a lot of people to swallow. I’ve heard countless product folks fall back on their many years of experience or espouse the art of building products. They don’t need to test, they claim. They’ve done this before.

The Internet changes quickly. New products are launched at a rate we’ve never seen before. User behavior adapts.

What worked last year may not work this year. What worked in one context won’t work in another context.

So yes, you do have to test.

But don’t test everything.

The “test everything” camp has similar flaws in its logic. It is true that everything changes. It is possible that what didn’t work last week might work this week. But there are infinite possibilities and you can’t test them all.

You want to accelerate your learning. If you test every possibility, all you’ll do is get good at testing. But you won’t build the right product faster.

There is a role for human judgment. Now by that I don’t mean your personal preference or opinion. Don’t pick the ideas you like the best. I mean judgment informed by data and user context.

You can use judgment to estimate the expected outcome. And you can use expected outcome to rank your ideas. Involve other people and follow these estimation tips.

And of course, never forget the most important rule of building products: Prepare to be wrong. – Tweet This

You don’t want to just prioritize and build. Instead, assume you are wrong and go out and look for evidence to the contrary.

Test Your Assumptions, Not Your Ideas

You shouldn’t test your ideas. You should test the assumptions behind your ideas.

That’s an important distinction. So much so, it’s worth saying again.

Don’t test your ideas. Test the assumptions that have to be true to make your ideas work. – Tweet This

This may seem like an obvious point. But it’s often overlooked.

Let’s look at a simple example. Suppose your goal is to drive email signups. You suspect that if you drive enough people to the sign up page, sign ups will go up.

You have 3 different ideas designed to drive traffic to your sign up form.. You test all three ideas. Should you build the one that drives the most traffic?

It depends.

Do the people who land on that sign up page sign up?

Each idea is designed to drive traffic. But each idea also has an impact on what happens after the user arrives at your sign up page. Does the idea that drives the most traffic, also lead to the most sign ups?

Your assumption is that more traffic will lead to more sign ups. That assumption may be true for one idea but not for another.

This may seem like a silly idea. Of course, you should go with the idea that drives the most sign ups. But what if your funnel was more complicated?

How many stores optimize for adding more items to the shopping cart and then lose the sale when the shopper is scared away by the high price? The idea is “get shoppers to add more products to their cart.” The underlying assumption is, “more items in the cart means more revenue per shopper.” But if the shopper abandons the cart, this assumption is false.

Don’t make this mistake. Don’t just test your ideas. Make sure the underlying assumptions that need to be true are true.

Run the Right Experiments

Once you’ve identified the assumptions that need to be true for your ideas to work, you need to design experiments to test them.

Throwing something up to see what happens is not an experiment. It’s a waste of time.  – Tweet This

No matter what happens, you will convince yourself it was good. You might as well skip the test all together.

Different assumptions require different types of tests

Are you trying to figure out why someone is doing something? Talk to them.

Are they not doing something you want them to do? Observe them, while they think aloud.

Are you trying to figure out which layout works better? Split test them.

Are you curious about whether what you hear from one or two people is prevalent across your entire population? Run a survey.

Too many people are only familiar with one testing method. Of late, everyone is hot on customer development. I’m never going to argue with talking to your customers more. But when all you have in your toolbox is a hammer, everything starts looking like a nail.

Add to your repertoire. Learn a variety of testing methods and know when to use them. Laura Klein does a great job of explaining what to do when in her book UX for Lean Startups.

Draw Lines in the Sand

You’ve done your homework. You know what your assumptions are. You’ve designed good experiments to test them. You are feeling pretty good about yourself.

But you have one more step to do before you run the experiment.

You need to draw a line in the sand.

What does that mean? You need to decide before you run the experiment what a good result looks like.

This is critical. Why?

Your brain is wired to see what you want to see. If you don’t draw a line in the sand before you run the experiment, it doesn’t matter what results you get. You will conclude the results are good.

You know what I’m talking about. You’ve done it before. You get a mediocre result and you start to rationalize to yourself why it’s okay. The next thing you know you are looking for reasons why it is good.

Don’t do this. You are cheating yourself out of a learning experience.

Instead, take the time to ask yourself before you run the experiment, how will I act on these results? Set a threshold.  If the results are above this threshold, I will do x. If the results are below this threshold, I will do y.

Be Honest With Yourself

Setting this threshold is easy once you know to do it. Sticking to your outcomes based on that threshold is challenging. Your brain is going to work hard to convince you the results are good.

Be ruthless. Work with someone who will hold you accountable to that threshold.

Did the results clear your threshold? Do x. Did they miss the mark? Do y.

It’s that simple. And yet, it’s difficult to do.

Do it anyway.

Many teams do everything right up until this point. But when they see the result they lack the discipline to act on it. They start to second-guess the results. They question the experiment itself.

Don’t fall into this trap. If you do, you will waste your experimentation efforts. Instead, setup the systems to ensure that you trust and act on the results. Make the hard decisions.

On Thursday, we’ll move on to prioritization based on experimentation results. Don’t miss it. Subscribe to the Product Talk mailing list.