Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Every time I speak about hypothesis testing, I get a series of questions about how to run experiments from people working on enterprise products.
The assumption behind these questions is that hypothesis testing only works with consumer Internet products.
And the assumption behind that belief is that the only way to experiment is to run A/B tests.
But these are both false assumptions.
There Are Many Ways to Experiment
An experiment is simply a procedure designed to test a hypothesis. – Tweet This
A hypothesis is a conjecture that you hope is true and is specific enough to be refuted or supported by an experiment.
Many of the activities that we undertake as product managers are experiments.
Yes, A/B tests are experiments. But so are usability studies, customer development interviews, surveys, user diary studies, and so on.
The reason why we don’t think of these methods as experiments is because we often don’t start with a good hypothesis.
Instead, we misappropriate experimental methods in the search for knowledge about our customers.
If we improve our methods, we can gain the benefits of experimentation in any type of environment.
A/B Testing Might Not Be Right For Your Environment
Are you currently thinking, I can’t experiment because …
- Our software is deployed onsite.
- Customers don’t want constant change.
- We don’t release new code very often.
- Customers should decide when they want to upgrade.
Every single one of these objections hinges on an underlying assumption – that the only way to experiment is to push changes to your product.
Don’t assume that experimenting necessitates making changes to your product. – Tweet This
This is a false assumption. It stems from the fact that you are assuming that experimentation equals A/B testing.
It is hard to A/B test when your software is updated infrequently, whether it’s because your software is deployed onsite or your development cycle is long.
It’s true that most enterprise customers don’t want constant change. They’ve invested time and energy into learning your product and they expect it to keep working the way it always has. Product changes are disruptive.
And customers should decide if and when they want to upgrade. Again, they pay good money for your product. Shouldn’t they have some say in what they get for their money?
While these objections are true, they aren’t reasons why you can’t experiment. They are reasons why you may not be able to A/B test.
That Doesn’t Mean Experimentation Isn’t Critical
However, while each of these reasons might rule out A/B testing, each of them is also a strong argument for why you should be integrating experimentation into your product development process.
If your code is deployed onsite and your updates are infrequent, isn’t it even more critical that you release code that matters?
In this type of environment, software updates are an annoyance. You need to offset that annoyance with code changes that your customers want. There is even more pressure to get your product changes right.
If I have to take time out of my day to upgrade software or if I have to wait months for the upgrade to be ready and the new software doesn’t do anything for me, I’m going to be disappointed and frustrated.
The fact that customers don’t want constant change makes it even more critical to test your ideas before you build them. You don’t have many chances to get it right.
And since customers want to decide if and when they upgrade, the pressure is on you to build compelling product changes that encourages people to upgrade.
The objections that make A/B testing challenging in enterprise environments are also why experimentation is so critical in such environments.
Expand Your Experimentation Toolbox
If an experiment is a procedure that helps you to support or refute a hypothesis, then you have many ways to experiment within the constraints of an enterprise environment.
You can interview customers, conduct usability studies, analyze usage trends, test rapid prototypes, learn from surveys, ask customers to keep usage diaries, observe your customers in action, and so much more.
Your enterprise product will thrive or fail based on your ability to repeatedly deliver functionality that your customer base cares about.
Stop focusing on the experiments you can’t run and start conducting the experiments you can. – Tweet This
Do you want to get better at running good product experiments? Subscribe to the Product Talk mailing list so that you don’t miss a post.
