Every time I speak about hypothesis testing, I get a series of questions about how to run experiments from people working on enterprise products. The assumption behind these questions is […]
Run Experiments Before You Write Code
The Lean Startup has a flaw. It’s a simple one. It advocates the feedback loop: Build -> Measure -> Learn. I agree wholeheartedly with this loop. The flaw is in […]
Don’t Rely On Confidence Alone
You find yourself in a conference room with your product team. The white board is covered with sketches. You have an idea that everyone is excited about. You mapped it […]
How to Estimate the Expected Impact of a Product Change
Your design team is excited about a new signup process. They are convinced it will increase registrations. Your sales team keeps asking for the same feature over and over again. […]
What to Do When You Don’t Have Enough Traffic To A/B Test
You’ve read the Lean Startup. You’ve drank the Kool-Aid. You’re ready to start experimenting. There’s only one problem. You don’t have any web traffic. If the only tool in your […]
The 5 Components of a Good Hypothesis
Update: I’ve since revised this hypothesis format. You can find the most current version in this article: How to Improve Your Experiment Design (And Build Trust in Your Product Experiments) […]
Putting the 4 Levels of Product Analysis Into Practice: A Halloween-Themed Example
It’s Halloween week! The pumpkins are carved, the candy bowl is full, and the neighborhood is about to be crawling with princesses, ninja turtles, and pirates. You’ve only got one […]
Hypothesis Testing Mistake #1: Not Knowing What You Want To Learn
Have you ran an experiment in the past day, week, month? Can you clearly articulate your hypothesis? Do you know under which conditions your hypothesis will pass or fail? Do you know […]
