Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Welcome to the latest installment of Product in Practice! For this post, we spoke with a product team from Simply Business about some of the major lessons they’ve learned since adopting continuous discovery habits like interviewing their customers, questioning their assumptions, and using the opportunity solution tree to guide their work.
Want to check out the other posts in this series? You can find them here.
Seasoned Product Talk readers know how much Teresa believes in regular customer interviews. It’s a topic she feels so strongly about, she’s named it a keystone habit, incorporated it into her definition of continuous discovery, and designed an entire course to help people improve their customer interviewing skills.
Conducting regular interviews allows you to better understand your customers and their needs.
But it’s not enough to merely listen to everything your customers say.
Sometimes, people will talk about something that’s a genuine pain point for them. It’s something that irks them. And they get satisfaction out of complaining about it. But that doesn’t always mean that it’s a pain point that’s important or worth solving.
So how do we know the difference? How do we know when a pain point is worth pursuing?
The Simply Business product team recently had this experience. They chose an opportunity based on feedback from customers, but quickly realized it wasn’t worth pursuing. They then used other continuous discovery practices to choose a new target opportunity, identify solutions, and launch experiments—all within a week.
We caught up with the team to hear their take on the entire experience.
Have your own Product in Practice story you’d like to share? Submit your story here.
Can you provide a quick introduction to your team? What is each person’s role? How would you describe your team’s overall purpose and goals?
Rosie: Our team’s overall purpose is to increase customer retention and net promoter score (NPS). We aim to surprise and delight our customers through a seamless post-purchase journey.
We are an empowered cross-functional team consisting of experts in engineering, design, user research, data analytics, and product. The members of our team are Mina Kasherova, Senior Product Manager, Raya Raycheva, Senior User Researcher, Rosie Kipling, Insight Analyst, Fernando Leite, Senior UX Designer, and Giulio Ambrogi, Front-End Developer & Tech Lead.
The Simply Business product team, clockwise from left: Giulio Ambrogi, Front-End Developer & Tech Lead, Raya Raycheva, Senior User Researcher, Rosie Kipling, Insight Analyst, Mina Kasherova, Senior Product Manager, and Fernando Leite, Senior UX Designer
Please give us a quick overview of Simply Business. What does the company do and who are your main customers?
Mina: Simply Business is the UK’s largest business insurance broker, with over 600,000 small businesses, landlords, and not-for-profit organizations protected through us. Our award-winning insurance broker proposition covers over 1,000 different types of businesses, from plumbers and consultants to dog walkers and community groups. We’ve also recently expanded our product proposition to the US.
What was your team’s approach to product management like before you began working with Teresa?
Mina: Simply Business’s approach to product management is heavily influenced by the principles of The Lean Startup, Silicon Valley Product Group, and design thinking. Our product team members are firm believers in dual-track discovery and delivery and we’ve had some great successes in identifying valuable ways of solving customer needs through continuous experimentation in the past. Some of our biggest challenges have been delivering multiple test iterations on a weekly basis and killing ideas fast.
Raya: While our initial reading and research was a great introduction to us becoming a product team and forming what we call our Product Discovery Unit or PDU (product manager, designer, insight analyst, researcher, engineer), it became slow and clunky quite quickly and more often than not left us wondering what to do next when the idea didn’t survive the initial research stage. This also led us to over-test ideas in the quest to definitively prove they’re not good—with all the research and evidence in the world—instead of cutting our losses and moving on early on.
Rosie: One of the key traits of our team that became clear working with Teresa was that we are quite heavy on over-thinkers. Two-way door decisions were a revelation.
[Editor’s note: The phrase “two-way door decisions” is inspired by the 2016 Amazon shareholder letter written by Jeff Bezos. In this letter, Bezos refers to changeable, reversible decisions as “two-way door decisions.” Read more about this concept in Teresa’s post, How Much Time Should You Spend in Product Discovery?]
The realization that we’d learn more by choosing the wrong solution and running a quick demand test rather than spending hours debating each decision meant that not only were we learning much faster, but we were also less hard on ourselves when it didn’t work out. We were less invested in each solution, which has allowed us to make more data-driven decisions without feeling like we’d sunk too much time on any one solution.
Choosing the wrong solution and running a quick demand test vs. spending hours debating each decision meant we were learning much faster and we were less hard on ourselves when it didn’t work out. – Tweet This
Early on in our coaching journey, we chose the opportunity that customers have trouble with late payments. Since most of our customers are small businesses, when their clients are late to pay them, it can become a financial strain and it’s something they often mention during user interviews. Ordinarily, we’d have agonized over this first decision of which opportunity to go after. We’d have sized our market and done some user research before agonizing over which solution to build and A/B test. This would have taken weeks—if not months—and by the end we’d become so invested in the solution we wouldn’t want to give it up.
Teresa helped us out with this first decision by putting us on the spot and asking, “Which one do you think will have the most impact on your outcome?” We made a call then and there, and killed it within two weeks. We felt comfortable about the decision, and knew exactly where to go next because we’d learned so much by making the ‘wrong’ decision.
How the Simply Business product team approached customer journey mapping—with lots of sketches and Post-its!
How did you find out about and begin working with Teresa?
Raya: I’ve been following Teresa’s blog and talks on YouTube. We started trying to use opportunity solution trees early in our formation as a product team, but we were lacking the full context of the continuous discovery toolbox in order to utilize this specific tool to its full potential. While starting every quarter with an opportunity tree mapping session has been really helpful, we were still falling into old habits immediately afterwards, namely long periods of in-depth user research, elaborate prototyping, statistically significant A/B tests, etc.
Mina: Last year we identified a gap when it comes to a consistent discovery framework across our product organization. Our Head of Product spent the summer interviewing a number of discovery coaches before finally settling on Teresa. I’m personally a big fan of Teresa’s—I follow her blog and saw her speak at Mind the Product a few years ago—so it was great to have a chance to work with her.
Let’s talk about your continuous discovery journey. Why did you choose your first opportunity of late payments?
Raya: We were very fortunate that we already had ideas about what our customers’ problems are. We have been speaking to customers from the day we formed as a team, but in a more traditional user research setting. We already had a sense of our customers’ top problems and once we started our continuous interviewing practice, we kept hearing the same things again and again.
As Rosie mentioned earlier, the issue of late payments was something that we had been hearing a lot about from a variety of customers. Many small business owners and freelancers have trouble receiving on-time payments from their customers and clients. In addition to that, we had a lot of market stats and government research data suggesting this is a top problem for small businesses. So it felt like the right place to start evidence-wise.
Rosie: Knowing that our decisions are two-way door decisions really sped up the process of picking that first opportunity.
What did you learn in your first tests to realize customers talked about this need a lot but didn’t really care about it?
Rosie: We decided on three solutions to test out: tools and articles, invoice discounting, and automated payment collections. We could provide the tools and articles based on our existing website content, but the other two solutions involved partnering with existing “late payment” solutions. A small business owner would need to involve a third-party to take care of invoice discounting or automated payment collection.
For tools and articles, we created a demand test within our existing customer account product. This was relatively straightforward—it just involved pulling existing content from our website and putting it within the customer pages to test whether anyone demonstrated interest by clicking on it. You can see what this looked like in the screenshot below.
The Simply Business team created the “Get help with late payments” section shown in this screenshot as a demand test to see whether customers demonstrated interest in learning more about this topic.
For invoice discounting and automated payment collections, we ran remote tests with mockups of different solutions, asking open-ended questions to gauge users’ understanding and interest. We quickly discovered that many of the solutions in the market are too complex for customers to understand. And solutions that involved introducing a third party were also problematic. Most small business owners pride themselves on managing relations with their clients and don’t feel comfortable handing over invoicing or payment collection to someone else.
Raya: Our key learning was that while this is a real pain point and it’s a big one for our customers, they already had their own solutions in place to deal with it! It wasn’t a problem that they were feeling helpless about and it wasn’t a problem that they were looking for solutions for. It’s just a topic they bring up all the time because they want to talk about it. We compared and contrasted three test concepts for solutions to the problem, pulled out this insight from remote test participants, and ultimately killed the opportunity. We learned that late payments are a problem, but not an opportunity for us.
We learned that late payments are a problem, but not an opportunity for us. – Tweet This
What happened next? What did you do after you realized late payments were a problem but not an opportunity?
Raya: Continuous interviewing was a real game-changer when it came to picking a new opportunity. Because we had been speaking to two to three customers every week, we were regularly contributing to the opportunity solution tree.
Continuous interviewing was a real game-changer when it came to picking a new opportunity. – Tweet This
So when that opportunity died, it was a matter of going through our interview snapshots, picking out the pains that kept coming up repeatedly, mapping them against the tree, and voilà—a cluster of pain revealed itself to us in a new opportunity leaf!
Killing our first opportunity was a blessing in disguise because it really brought to life the meaning of two-way door decisions and “just enough” evidence. We totally overanalyzed and spent too much time on every step of the process.
Killing our first opportunity was a blessing in disguise because it really brought to life the meaning of two-way door decisions and ‘just enough’ evidence. – Tweet This
So when it came to doing it all over again with a new opportunity, our thinking had changed completely and we were much quicker and straight to the point with our activities. We timeboxed every task and were very clear what we wanted to have achieved by the end of every block of time we had together, and we stuck to this schedule. We thought small when it came to the experiments and re-used all the learnings on how to run them quickly and cheaply.
Let’s talk about the process of vetting ideas. How were you able to quickly vet and throw out ideas?
Raya: We used a combination of unmoderated remote tests, website surveys, and fake door tests to de-risk our riskiest assumptions for our top three solutions. We had the tools in place to set these up very quickly and, crucially, we had agreed on clear success criteria for that kill/keep decision, so we didn’t dwell on it much. We learned from Teresa to ask, ‘What data would convince you this assumption is or isn’t risky?’
Rosie: Being very clear on our riskiest assumptions really helped us to design the smallest test possible to de-risk these ideas. We often tested customer behavior out of 100 or 1,000 customers in a matter of hours. In these cases, threshold success criteria didn’t need to be precise, just indicative of potential.
Being very clear on our riskiest assumptions really helped us to design the smallest test possible to de-risk these ideas. – Tweet This
An example of how the Simply Business team quickly tested a very high concept of a solution (invoice discounting) to see if users understand what the solution does and if they relate it to their problem at all.
How often were you experimenting and what was that like for your team?
Fernando: We were sustaining multiple experiments each week for several weeks in a row. It was quite an intense pace. However, as soon as we got more familiar with the tools and frameworks, we started to understand how to better use them and reduced the time we spent on each part of the process.
Giulio: Finding ways to test our assumptions that don’t always require engineering work really helped our pace of learning as we had different members of the team using their expertise to design and launch tests in parallel. While I was creating a live demand experiment, Raya and Fernando would be designing and testing a set of prototypes with remote participants.
Finding ways to test our assumptions that don’t always require engineering work really helped our pace of learning as we had different members of the team using their expertise to design and launch tests in parallel. – Tweet This
Why was it valuable to have an engineer participate in the process?
The Simply Business product team uses Miro to work in the solutions space, creating one row for each opportunity and one frame for each different solution. Inside the frames they also work on customer journey mapping and assumptions mapping.
Giulio: It was useful to assess the complexity of tests and solution ideas before we’ve considered building any of them.
Mina: We’ve learned that we deliver a much better end-product when all members of the team have the right customer and business context to work with. Our engineers come up with solutions or testing approaches we wouldn’t know are possible, so we can’t afford to leave them out of the earlier stages of discovery.
Our engineers come up with solutions or testing approaches we wouldn’t know are possible, so we can’t afford to leave them out of the earlier stages of discovery. – Tweet This
Fernando: Besides bringing a different point of view, it made a big difference to the team’s pace as we answered key questions around each idea early and restricted scope to the questions that really needed an answer first.
Can you describe in detail how you are using unmoderated tests to quickly test your ideas?
Raya: In the example of the late payments opportunity, we created a one-pager Canva prototype for each solution explaining the high-level concept (we lifted those from existing providers of these solutions).
We put them on Validately, the remote testing tool we used, and screened for small business owners who self-report late payments to be a top business challenge.
Before showing any of the concepts, we asked participants to talk about this problem and how they solve it at the moment. Then they saw each concept and were asked to explain what they understood from it, how they imagine it would work, how it applies to their business and their problem, if at all.
This told us all we needed to know about the problem on an opportunity level as none of the solutions resonated with the reality of the problem and, most importantly, we learned that business owners had no real interest in third-party solutions and help on the topic. They were actually handling the problem themselves.
If you were offering advice to another team that was interested in making a similar transformation to continuous discovery, what would you tell them? What have you learned from this process so far?
Fernando: I think regularity is a key element for continuous discovery. Not only regularity in speaking with customers, but also regularity in meeting and working with your team to progress within the process. We have two-hour sessions at least three times a week and they’ve been really important in sustaining our speed. We start on Friday by planning for the following week. The structure of the sessions will depend on what we have planned for that week—we might be working on new solutions for an opportunity or reviewing test results. We have a shared Google doc where we list all the activities we want to cover that week and outline the top three priorities. This doc helps guide our sessions, where we always try to have at least three team members present.
I think regularity is a key element for continuous discovery. Not only regularity in speaking with customers, but also regularity in meeting and working with your team to progress within the process. – Tweet This
Raya: Continuous interviewing is the first thing I started recommending to other teams—and I’m happy to say several have picked up the habit and love it—because we were so used to waiting for something to test or waiting for something specific to ask and then lining up five to ten people to talk to.
Continuous interviewing is the first thing I started recommending to other teams—and I’m happy to say several have picked up the habit and love it! – Tweet This
I’ve also been talking to my fellow user researchers about compare and contrast testing and very high-level concept testing instead of what we were used to doing, which was prototyping whole experiences before knowing if the experience is worth it at all.
The concepts of ‘just enough’ evidence and ‘two-way door decisions’ are ones that I’ve fallen in love with. I think product people tend to be overanalyzers, we want to learn all the things and we want to do things properly, so switching to a more ‘learning first things first and just enough’ approach to tell if it’s worth learning more has really accelerated our pace of discovery.
Mina: Adopting or creating a discovery framework takes so much pressure away from wondering which technique to use when. Following Teresa’s framework has freed us up to focus on the problem rather than to think what process we need to invent. Treating discovery like a daily practice we dedicate team time to has dramatically improved our team discipline—we generate a lot more ideas (especially if we set some brainwriting time aside) and have improved the quality of our thinking. We have an agenda for our daily discovery team time and use a Time Timer religiously to help us make decisions at pace.
Adopting or creating a discovery framework takes so much pressure away from wondering which technique to use when. – Tweet This
Rosie: I’ve learned that A/B tests are not a one-size-fits-all, but more of an impact-measuring device, once you know which solution you’re invested in. We now reserve A/B tests for the rollout, rather than investing weeks in tests on MVPs that we will end up killing. We are running several tests a week now and killing ideas at pace, which is great because when an idea does make it past these initial ‘just enough’ demand tests, we have a lot more context to inform our longer tests.
Is there anything else that you’d like to add?
Fernando: The freedom and trust from the company is essential. If you have to get approval for every test you want to run, you’ll never have speed and you will end up doing fewer and more complex tests that aim to learn one million things at once.
Mina: As the product function in Simply Business scales up, we risk generating valuable learnings that don’t reach all relevant teams. With a number of product teams looking to solve the needs of our customers, we also risk re-learning the same insight because we are fundamentally trying to meet the needs of the same target audience. We’re now experimenting with a cross-team opportunity solution tree which we use to communicate that information and educate others’ decisions on which opportunities to be prioritizing as we grow our wealth of knowledge about the customer.
The Simply Business story offers several valuable lessons. While yes, it is important to talk to your customers regularly and listen to them, it’s also important to assess if what they’re telling you is a true opportunity. Sometimes they may just like to complain about something because it’s a topic they feel passionately about.
Regular customer interviews in tandem with other continuous discovery best practices helped the Simply Business team overcome their tendencies to overthink and take quick and decisive actions. Reminding themselves that they needed just enough evidence and many of their decisions were two-way door decisions also facilitated this process.
Do you have a story about how you and your team adopted continuous discovery best practices? Get in touch to let us know and you may be featured in a future post!
