Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. The report is wrong.
This was the last thing I wanted to hear.
It was March or April of 2013. We had launched Explore at AfterCollege a couple of months earlier.
In the past week, we had circulated and celebrated a report that indicated that Explore (our new product) was outperforming Search (our old product).
This had been our goal from day one. I had never dreamed that we would get there so quickly. It was a huge win.
There was only one problem.
The report was wrong.
We assumed that the funnel we were looking at was a page to page funnel. It wasn’t. It was a goal funnel. That difference changed everything. It meant people could be going from Explore to Search and then having success and the success would be attributed to Explore. And that’s exactly what was happening.
It took us some time to dig in and understand the data. We had to build some of our own analytics tracking and reporting tools because we couldn’t get what we needed out of any of the 3rd party tools that we were using. But we eventually got to the bottom of it.
We weren’t quite there yet. Explore was improving. But it would take us about 4 more months before we actually hit our goal.
As much as I wanted to believe that report, I knew something was off. I don’t remember what tipped me off. But you know the feeling. There was an outlier. Something didn’t quite add up. Things looked too good to be true.
I had to investigate. And I’m glad I did. The last thing I want to do is celebrate a success that isn’t based in reality. And neither should you.
The Research and Experimentation You Are Doing Doesn’t Matter Without Intellectual Honesty
I first heard of the concept of “intellectual honesty” from Ben Yoskovitz.
It might have been this post.
Or maybe it was in Lean Analytics.
It doesn’t matter.
Intellectual honesty is a critical mindset that all product teams must foster. – Tweet This
I love that as an industry, we have shifted toward experimentation and research.
Thanks to Steve Blank, Eric Ries and many other advocates, we are obsessed with customer development interviews and hypothesis testing.
We are becoming more data-driven.
We are debating how we should measure success and what is or isn’t a vanity metric.
I love it. We are getting better.
But no matter how many A/B tests you conduct, customers you interview, or traffic patterns you analyze, none of it matters, if you don’t build the discipline and rigor of intellectual honesty.
What do I mean by intellectual honesty?
To me it means you:
Use good research methods. You understand which tools are good for which types of questions. You take care to get the methods right. You script your usability tasks. You test with an appropriate target segment. You don’t peak at your A/B test results before they are done. And so on.
Understand if your data are reliable. You only trust statistically significant split test results. You understand that one user does not represent all users. You dig deep to get to the true meaning behind a request. You run the appropriate analysis.
Don’t leave out data just because it doesn’t fit your story. You don’t use data to support the story you already want to tell. Instead, you tell the story supported by the data – all of it. You don’t ignore the outliers.
Separate interpretation from data. You distinguish between the exact words that you heard and what you think it means. You understand the difference between the raw data and how you might interpret it. You describe what you saw and withhold judgment until all the data has been collected.
Get multiple perspectives on the same data. You ask others to provide their own interoperation of the data. You consider multiple perspectives. Most importantly, you ask for input from people who disagree with you.
Ask about what’s missing. You actively look for missing data. You fill in the gaps. You look for what you don’t know.
Separate the researcher from the invested party. You have someone other than the designer tally the results of the usability study. You ask an engineer who wasn’t involved to evaluate the data from the landing page test. You understand that confirmation bias is working against you and that you need to seek out people who are more objective than you on this particular matter to help you understand your results.
Admit when you are wrong. It’s going to happen. Often. What matters is how you handle it. If you are the first to admit when you are wrong, others will feel comfortable doing the same.
It’s okay to be wrong. It’s not okay to hide it. Acknowledge it, course correct, and move on. – Tweet This
Do the work required to get to the truth. Too often, people aren’t willing to put the work in to get to the truth. Ignorance may be bliss, but there’s no room for it if you want to drive product success. Dig in. Get to the root of the issue. Get beyond the surface level.
How You Can Develop Intellectual Honesty
Most of us want to do the right thing.
We want to know what’s really happening with our products. We want to get to the truth.
The problem is we have a competing interest.
We want to be right. It feels good to be right.
The problem with this competing interest is that few of us are willing to admit this to ourselves.
You are probably shaking your head right now and thinking to yourself, not me.
But cognitive science / behavioral economics / psychology is not on your side.
No matter how much we don’t want to believe it, it’s true. Our brain is wired to work against us.
It’s causing us to only notice the things we want to see.
It’s linking our ideas to our ego and sense of self-worth encouraging us to react defensively and stubbornly.
It’s interfering with our ability to listen, to empathize, and to be curious.
Which means, if we want to be build better products, we need to fight against the desire to be right and develop our sense of intellectual honesty.
I rely on the following to do so:
1. Assume you are wrong.
I do my best to operate from the position of assuming that I’m wrong. In other words, I take the stance, “I think this, but I’m probably wrong, how can we test it?” Or, “Let’s try this, but it probably won’t work, so if it doesn’t, we’ll try this, this and this.”
This might sound depressing, but it’s actually the opposite. Just like Stoics reflect on losing the things they love most to create a deeper sense of appreciation for them, assuming and planning for being wrong, takes the sting out of being wrong. It allows you to move on to the next idea, the next test.
When your test fails, instead of being disappointed, it is expected and you are prepared to move on to the next test.
2. Reflect.
Take the time to reflect on what went well and what you can do differently next time. Get feedback from other people.
The startup community has a strong bias toward action and that’s great. But we don’t learn just by doing. We learn by doing and then reflecting on that doing. So don’t skip that step. Make it a daily practice. You won’t regret it.
This can be simple. Take 5-10 minutes at the end of your day to write about what happened that day. If every day is too much, try it once a week. It might feel silly at first, but it will grow on you.
When you are ready, share some of your insights with others. Getting another perspective can help you break through barriers you don’t even see. It will help you think about things differently and push your own thinking.
Intellectual honesty requires self-awareness. You can build that awareness through reflection. – Tweet This
That’s it. There might not be half a dozen items on this list for you to experiment with like some of my other posts. But these two items are life-changers. They take a lot of work to get good at. So hurry up and get started.
Stay tuned. Up next, we’ll be taking a look at what a product leaders needs to know about statistics. Don’t worry, it won’t be a math lesson. Instead, we’ll look at key concepts that will ensure you can trust the results of your experiments. Don’t miss it. Subscribe to the Product Talk mailing list.
