Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Back in March, I shared a case study about one of the product teams I worked with at Snagajob (now Snag). I’m excited to share another one with you today.
This case study is about a team at CarMax that I just finished working with. You’ll see that they already had good discovery skills before I worked with them, and that allowed us to really dive deep into what it means to be a continuous discovery team.
Since the team already had strong discovery skills, we were able to explore the value of compare and contrast decisions, to run several experiments in parallel, and to really solidify habits that will make continuous discovery sustainable over time.
It was a ton of fun to watch this team excel. They are a great example of how a continuous mindset both saves you time and drives more results in the long run. I hope you enjoy reading about their experience.
Melissa Suzuno, my blog editor, conducted the interview with Victoria Lawson, Product Manager, Jake Mitchell, Designer, and Sri Arcot, Tech Lead.
The product team at CarMax: Jake Mitchell, Victoria Lawson, and Sri Arcot
Melissa: Tell us a little bit about your team. What are you working on? What are you trying to accomplish?
Victoria: Our team’s focus is on creating exceptional digital merchandising experiences for our customers. Our mission is to help people fall in love with a car that meets their needs. You could think of us as building the dating profile for a CarMax car—we want to make sure we are giving car shoppers all the information they need to get to know the car, and ultimately decide whether it is the right one for them. CarMax empowers us to work as a truly independent, durable product team led by a cross-functional core team: a lead developer (Sri), a product designer (Jake), and a product manager (me).
Sri: Car-buying is often an exciting experience but can be overwhelming or even frustrating at times. At Digital Merchandising, we see ourselves as strong advocates for our customers seeking their next car. It is our daily endeavor to ensure they are empowered to make sound decisions comfortably and confidently.
The car page the CarMax product team works on
What was your life like before coaching with Teresa?
Victoria: We were following product discovery and customer-centric product development practices prior to our engagement with Teresa, but Teresa was able to take us to the next level. She provided us with new ways to think about product discovery and several new tools to help us organize our processes and ensure we were working towards the right discovery at the right time.
Jake: With being a mature product team, we naturally developed a full backlog of ongoing discovery opportunities we had built over the years from interviewing customers, working with stakeholders, and improving the experiences we already had. Through our partnership with Teresa, we identified the opportunity to take a step back to organize our thinking to understand how the solutions we were considering laddered up to larger opportunities to drive our desired outcomes.
How did you decide what to build?
Victoria: We have an annual goal set by leadership. Based off that goal, the team would meet each quarter to review data from the site and qualitative insights from customers to determine what opportunities in our space would drive the biggest value for CarMax. We would then start prototyping and experimenting around different solutions. I think we had all the right ingredients, we just weren’t always organized and deliberate in how we were using them.
How often did you talk to customers?
Victoria: We would aim to talk to customers at least once a week, but often would defer to talking to customers only when we had something to validate.
Jake: CarMax has invested in several tools to make it easy to talk to customers, so we’ve always been free to find a customer when we want to show them something.
What did you think about conducting customer interviews?
Victoria: We knew customer interviews were critical to success but they were also time consuming and it was hard to synthesize the results from them. A lot of the time you would just end the interview with 100 Post-its but no clear takeaways.
How often did you run product experiments?
Victoria: We tried to run at least one experiment per month, ranging from basic prototype testing to live data tests.
What was the process of working with Teresa like? What were some moments that stood out to you?
Victoria: Overall, working with Teresa was amazing. We felt like we were already fairly well-versed in product discovery practices but working with Teresa proved that there is always room to grow. From the very beginning it forced our team to take a step back and give a very thoughtful examination of our customer journey and where key opportunities are within that journey, something we hadn’t done in some time.
We were able to re-align our thinking and revisit some old assumptions. There were aha moments throughout the entire process, including how to think about opportunities, how to brainstorm solutions, and how to conduct thoughtful and useful experiments.
Some of the biggest learnings we had were around experiment design—Teresa pushed us to design experiments around individual assumptions vs. trying to test all of our assumptions and risks within one large experiment. It really helped us iterate quicker and move towards what the customer values most.
Designing experiments around individual assumptions vs. trying to test everything with one large experiment helped us iterate quicker and move towards what the customer values most.– Tweet This
Sri: As Victoria mentioned, I too had many aha moments throughout the process. One that stands out to me is “to test for desirability before usability.” If there is no desire for a product, however usable we make it, it is of no value. On the other hand, if we have confirmed our customers’ desire for a product, then we can figure out how to make it usable.
The CarMax product team now makes usability tests a regular part of their weekly schedule.
What are things like for you now? How often do you engage with customers?
Victoria: We now engage with 2–4 customers per week, regardless of what we have to show them. We have a consistent cadence of talking to customers and ensuring that their perspective is always a part of our perspective.
Jake: Teresa was great in getting us to shift our mindset in how we engage with customers, and to develop some clever productivity hacks to make finding and talking to customers very easy.
How often do you run experiments?
Victoria: Our experiments now are simpler but at the same time much more sophisticated. We are more thoughtful in what we are trying to learn. We align the team and stakeholders on what success looks like and the next steps we’ll take based off a predetermined threshold. This allows us to also have more confidence in our results and ultimately make better decisions for the customer and for the business.
Jake: A big aha moment we learned from Teresa was that an experiment doesn’t mean just a live A/B test. We now see experiments as a method of gathering data that supports or challenges an individual assumption. That means we can conduct many more experiments at once in several ways.
We now see experiments as a method of gathering data that supports or challenges an individual assumption, so we can conduct many more experiments at once in different ways. – Tweet This
Are you using the opportunity solution tree?
Victoria: Yes! It is a great tool to think about opportunities big and small—we just pulled our main tree out recently to discuss which next opportunity we should go after.
Jake: What’s great about the opportunity solution tree is that it visualizes the collective mind of the product team—instead of trying to keep it all in our heads, we have a living diagram that shows where we are focused and where our opportunities could take us.
The opportunity solution tree is a living diagram that shows where we are focused and where our opportunities could take us. – Tweet This
Sri: As an engineer, my instinctive approach led me to quickly jump to the solutioning step. The opportunity solution tree not only helped us organize and prioritize our backlog, it also disciplined us to not think of solutions until we identified and deliberated on our opportunities thoroughly.
What have the results of this shift been?
Victoria: We are more in tune with our customers than we have ever been before and are able to take their stories and needs into every decision we make. We are also much more efficient in determining what are valuable solutions that are worth pursuing and how to identify and work through risks within a potential solution.
We are more in tune with our customers than we have ever been before and are able to take their stories and needs into every decision we make. – Tweet This
Jake: Our biggest shift has been going from a team that only learns when we build a high-fidelity prototype, to a team that is constantly learning and improving through deliberate testing of the biggest assumptions in our proposed solution. This style of working has allowed us to move much quicker and zero in on what exactly is driving the metrics we want to change.
Sri: We have been more methodical and structured in identifying opportunities/solutions. Since we now test against assumptions and risks, and desirability before usability, we are able to fully vet our hypothesis quicker. We are now able to “fail fast” or iterate forward in a much more informed manner and with much more confidence.
Now that we’re more methodical and structured in identifying opportunities and solutions, we’re able to “fail fast” or iterate forward in a much more informed manner with much more confidence. – Tweet This
The CarMax product team regularly conducts simple experiments like this A/B test.
What would you say to another team who was considering working with Teresa?
Victoria: Teresa goes above and beyond the theory of product discovery and truly gets her hands dirty, working one-on-one with your team, adjusting and flexing based off your specific business, team, and customers, and giving you valuable tools that will help you run useful and effective discovery well beyond your coaching.
Jake: Where Teresa really shines is the level of engagement she brings every week, and how she tailors her program to the needs of the team. She is a coach in the true sense of the word, where she identifies weak spots and works with the team to develop habits that will allow them to strengthen over time.
Sri: Teresa has my double thumbs up as a product methodology coach and mentor. She has the patience and experience to thoroughly understand and accurately assess your team’s needs and will prescribe a customized approach that you can follow as a daily routine.
Final note from Teresa: I had a blast working with Victoria, Jake, and Sri. It was fun to watch them grow and evolve into a continuous discovery team, adopting many of the tools and methods you’ve read about here on Product Talk. If you or your team is interested in learning more about my coaching, please start here and then feel free to get in touch.
