What Product Assumptions Are You Making?

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

When a team sets out to develop a product, they generate a series of hypotheses. In my experience, good teams are explicit about these hypotheses and test them before they invest too much time into a particular product plan. However, more often than not, I see that development teams are not even aware that these hypotheses exist, instead building their products as if they were fact. The problem with this approach is that rarely does a team get all, if any, of these hypotheses right from the get-go. If they don’t first test them, they run the risk of building a product that nobody wants. Let’s take a look at an example to illustrate how this works.

Explicitly Enumerating Product Assumptions as Hypotheses

I have an idea for a Caltrain mobile app. Caltrain is the local train that runs from San Jose to San Francisco. Riders include daily commuters and occasional train riders. As a daily commuter, I witness a number of problems experienced by occasional riders. They don’t know how to buy tickets, they don’t know how to pay for parking, they don’t know which platform to stand on. They don’t know how to tell what stop they are at or when and where to get off the train.

I suspect that if Caltrain tackled some of these problems, they might convert more of these occasional riders into daily commuters, growing their ridership. I’ve often considered building a Caltrain mobile app that addresses some of these usability problems.

The app would walk the occasional train rider through each step of the process of riding the train, starting with where to park, how to buy a ticket, updates on when the next train is coming, updates on where you currently are relative to where you want to get off the train. My goal would be to make it as easy as possible to ride the train.

I ride the train every day. I know how to do all of these things. I’ve observed these problems first hand for years. Should I just get started and build the app? How do I know that occasional riders will want it?

Let’s take a look at some of my assumptions or hypotheses underlying my app idea.

• H1: Occasional train riders will download a Caltrain mobile app before they ride the train.
• H2: The problems that occasional train riders experience are big enough that they will remember to use the app they downloaded earlier to help solve their problems.
• H3: The desire to ride the train is great enough that if occasional train riders had help they would ride the train more frequently.

It’s quite possible that occasional train riders don’t anticipate having problems. If they did, they would probably choose to drive rather than take the train. So H1 could be a big hurdle.

H2 may not be as big of a hurdle. It’s possible that if I did download the app and I run into a problem, the problem itself may act as a trigger to remind me I have the app. But that still needs to be tested. It might be easier for me to just ask somebody nearby. Or I might just give up, something I see people do daily.

With H3, if my goal with the app is to help grow Caltrain ridership, then I am assuming that the reason people don’t ride the train more often is because it’s hard. This might be part of the problem. But there are a number of other reasons why occasional riders might not take the train more often – the big one being that it is often slower than driving. Even if people had all the help they needed, they might still choose to drive over taking the train because they simply want to get to their destination sooner.

Testing Each Hypothesis

Before I decide whether or not it’s worth it to build this app, I want to test each of these three hypotheses. But how do I do that?

H1 is fairly easy to test. It requires that riders take action on their phones or computers before they ride the train. Using the Google keyword tool, I can easily find out how many people search for keywords like Caltrain schedule, Caltrain tickets, Caltrain parking, and other phrases I identify as related to some of the problems I want to tackle. If the numbers are high enough, it will validate that people do in fact seek out this type of information.

But that’s only part of the story. I also want to validate that the riders seeking this information would download a mobile app to solve their problems. How do I do this without actually building a mobile app?

Again, Google can help. I can build a simple splash page for my mobile app – a webpage that talks about the features and benefits of the app and ask visitors to download it. To get visitors to this page, I can buy Google Ads for the keywords I uncovered in my first test. But wait, there’s no app to download? That’s okay. I’m just testing whether or not visitors will click on the “Download” button. If they do, I can always apologize that the app is not yet ready and let them sign up to be notified when it is. This does two things, 1) it validates that people will take action to download an app and 2) it lets me build up a potential user list before I start building.

H2 might be a little bit harder to test. But here’s where some creativity can help. How do I test whether or not someone will remember they have an app when they don’t in fact have an app. I can fake it, just like I did for my H1 test. Here are a couple of ideas:

• I can ask friends who are occasional train riders to call me when they run into any of these problems. Do they remember to call me? Do they figure it out on their own? Do they just give up?
• Better yet, for the folks who clicked on my “Download” button when testing H1, I can ask them to bookmark a page that acts like an app, so it’s available on their phone. The page could be as simple as an FAQ, or could allow the user to enter their own questions that I directly answer. Either of these would be simpler to manage in the short-run than the full app and either gives me the data I need to know whether or not riders will remember to use the app.

H3 is even tougher yet. But still not impossible. I can ask my early users (again, I have users even though I don’t yet have an app), to tell me how often they ride the train. After they’ve accessed the fake app a certain number of times, I can ask them again how often they ride the train. If their ridership goes up over time, I’ve validated my hypothesis.

I already know what’s going through your head. If I’m solving the problem with an FAQ or by answering one-off questions, why would I build the app at all? That’s exactly the point, I wouldn’t necessarily have to. I would have all the data I need to know exactly how much product to build to solve the problem I set out to tackle. If I’m answering one-off questions, I probably still want to build a product that automates this. If a FAQ is getting the job done, I might learn my problem was much easier to solve than I originally thought. It’s not about my original product vision, it’s about solving a problem, meeting a market need.

It’s An Iterative Process

Even if I can prove all three of these hypotheses and I decide to build the app, I’m not done testing hypotheses. Each step along the way, as I build out the app, I’m going to keep layering more hypotheses onto the ones I’ve already tested. Product development is an iterative process. But if I keep testing each assumption, I’m very unlikely to build too much of a product that nobody wants.

Next Steps

In a future post, I’ll explore how to get better at surfacing assumptions and turning them into explicit hypotheses. In the meantime, what do you do to make sure you aren’t building products on too many untested assumptions?