What a Good Continuous Discovery Team Looks Like [Case Study]

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

One of my favorite parts of Mind the Product London this past fall was meeting a product manager and a designer who I had coached in the prior year.

We met for afternoon tea and to my surprise they gushed about our time working together. They had so much they wanted to share about how their work had changed. They described how much time they were spending with customers and what impact it was having on their work.

It made me so happy. Immediately, I knew that I wanted to share their story. One of the challenges of teaching continuous discovery is that we don’t have very many public examples of what “good” looks like. So I asked Amy O’Callaghan (the product manager) and Jenn Atkins (the product designer) if they would be willing to share their story and they agreed.

One of the challenges of teaching continuous discovery is that we don’t have many public examples of what ‘good’ looks like. – Tweet This

Amy and Jenn work at Snagajob, a company that is well on its way to becoming a continuous discovery powerhouse. I feel incredibly honored to have played a role in this transformation and I can’t wait to see how it turns out.

I’m going to let Amy and Jenn take it from here.

(By the way, if you are looking for women product leaders to speak at your conference, these two have a phenomenal story to share.)

Melissa Suzuno, my blog editor, conducted this interview.

Melissa: Tell me a little bit about your team. What are you working on? What are you trying to accomplish?

Amy: Our team has gone through a number of evolutions, but the core problem has always remained connecting people who need work with people who need workers in a satisfying way for both. We are currently fully staffed (hooray!) with 2 API devs (wizards), 2 UI devs (magicians), 1 QA (a saint), 1 product designer (unicorn) and a product manager (me).

We are working on some really cool stuff that creates transparency in the job search and hiring process that benefits applicants and hiring managers.

What was your life like before coaching with Teresa? How did you decide what to build?

Amy: Hunches, directives from leadership, and sometimes things we uncovered in discovery with our users.

Jenn: I was on a few different teams before we reorged and received training from Teresa. Within two years I had worked with three product managers. All were great, passionate and talented managers, however we never really had time to establish a true partnership and journey through discovery together. We were reorged every quarter and tasked with building out a feature. We had a quarter to get the MVP out and then never had a chance to make incremental improvements because it was on to the next feature. So you can imagine we had a lot of bloat and a lot of unfinished products we wish we could have made better.

How often did you talk to customers?

Amy: Once every few weeks seemed acceptable since our customers are traditionally very busy and hard to get ahold of.

Jenn: I typically talked with customers on a weekly basis, mostly through online screening (UserTesting and Ethnio) and sometimes in person at our office. We had a problem with no-shows for in-person interviews and scheduled calls for both employers and job seekers.

What did you think about conducting customer interviews?

Amy: They were awesome, but scheduling was a complete crap shoot. More often than not they’d have to reschedule or cancel at the last minute due to conflicts.

Jenn: That was my favorite part of my job, but also frustrating because the problems we would uncover through research didn’t align or translate into things that would help us achieve what we were told to build. Also, there were many times when the product manager was pulled into meetings that trumped research. So there wasn’t as much shared understanding going on. I would create research reports (8 hours worth of work) that no one would ever look at.

Customer interviews were my favorite part of my job, but also frustrating because the problems we uncovered didn’t align with what we were told to build. – Tweet This

How often did you run product experiments?

Amy: Not often—we typically ran usability testing and then launched without much lightweight testing in the actual product itself.

Just before we started working with Teresa, a consultant from a well-known company came in to talk about agile product processes. He asked us to think of our time as a pie chart and what it would look like. Naturally I actually drew it (image left, below). Then he described what he believed it should look like, and I drew that, too (image right, below).

Amy sketched out how she spent her time and then how a consultant suggested she should be spending it.

That moment I felt a mix of frustration and disillusionment. Yeah, we all want to be doing more research, but it’s not realistic. With all the stuff we have to get done, no one can do that much discovery. I’d heard it before, and it was starting to feel like something consultants say to set a goal you can never reach, but that sounds good to your boss.

[Note from Teresa: I love how honest Amy is about this. I bet you can guess what’s coming…]

No one can do that much discovery. It was starting to feel like something consultants say to set a goal you can never reach, but that sounds good to your boss. – Tweet This

Jenn: From a designer perspective, we were heavier on delivery (prototyping and usability testing) and meetings than true discovery within the problem space. It was more discovery in terms of trying to find the problem within the solution we were told to build. As a designer, we naturally want to find the problem and then solve for it; so this process felt a bit backwards. Plus it was hard to find passion in what we were delivering because we didn’t always agree this was the right problem to solve or the right way to go about solving for it.

It was hard to find passion in what we were delivering because we didn’t always agree this was the right problem to solve or the right way to go about solving for it. – Tweet This

What was the process of working with Teresa like? What were some moments that stood out to you ?

Amy: Being on the hook for talking to at least one customer per week initially felt overwhelming, but it made us get scrappy and realize that there were more effective ways to reach people than those we were using. This was a huge step in developing the habit of getting to really know our users and their problems.

The opportunity tree was a BIG one. We use it at different places in the discovery and delivery cycle—sometimes daily, sometimes monthly. But at all times the tree is irreplaceable as a way to visually communicate all the discovery and thinking that goes into development and decision making. Stakeholders that used to throw curveballs into our sprints can now truly grasp the level of thinking (and testing) that has gone against our opportunities and are much less likely to interrupt our course.

[Note from Teresa: Yes! The tree is a great tool for communicating with your leaders.]

The opportunity solution tree is irreplaceable as a way to visually communicate all the discovery and thinking that goes into development and decision making. – Tweet This

Learning to assess, test, and manage risks against your solutions was a lightbulb moment for me. I’d known it was part of the job and tried to execute it in the past, but having a systematic approach made all the difference to doing it effectively.

Jenn: The foundation of the triad going through the coaching process was key. It was refreshing and exciting to finally feel like I had a partner in product discovery. I truly believe this is what helped push us to the next level. Our lead dev at the time wasn’t able to be as engaged but it’s okay, we iterated on the process and just made sure to find ways to bring the devs on our squad along with us on the discovery journey.

It was refreshing and exciting to finally feel like I had a partner in product discovery. I truly believe this is what helped push us to the next level. – Tweet This

Interview snapshots were a great way for us to quickly find similarities in problems our customers had. This became an epiphany for us as soon as we realized what we needed to focus on. For the first time we figured out what customers truly needed, and it was because we were able to read through what they were telling us and see what they actually meant. This started us down a course of discovery that has been truly invaluable to our squad’s product development.

Interview snapshots were a great way for us to quickly find similarities in problems our customers had. – Tweet This

Opportunity Tree!!! I echo everything Amy said. For me, the concept was confusing at first, but I caught on once we started building it out. In hindsight I think it was also confusing because at the time we didn’t really have an outcome that was driven out of our discovery process. If it had been introduced after we had our epiphany about where to focus I think I would have caught on faster. Also, Teresa’s talk from Mind the Product in London was the best explanation of the process and it really clicked for me there in a way that I was able to start sharing the concept with other squads.

Kano model and opportunity assessments were key in helping us understand where to focus our efforts and took away the feeling of being overwhelmed with all the opportunities and solutions. We combined the Kano model with Jared Spool’s analogy of hot water and cookies to help us with our decision making as well.

What are things like for you now? What are you doing differently that you weren’t doing before? How has this impacted your work?

Amy: We’ve developed a belief that truly understanding the problem is more important than thinking of and pitching solutions.

[Note from Teresa: Nothing in the world makes me happier than reading that sentence.]

We’ve developed a belief that truly understanding the problem is more important than thinking of and pitching solutions. – Tweet This

For the first time in my career I truly feel like a subject matter expert. If I don’t have qualitative or quantitative data to answer a question, I can get my hands on it. I feel confident pushing for things that we’ve prioritized because we know they will bring value—we aren’t taking nearly as many risks with development time as we used to, and the dev team appreciates that.

Jenn: Ditto to what Amy said! Also, I feel like a true product designer—not just an order taker. We started gaining trust and buy-in from stakeholders because they realize we have a deep understanding of our customers. Leadership sees the importance of keeping our squad intact and instead of moving us to different teams they’re giving us different problems to solve.

I feel like a true product designer—not just an order taker. We started gaining trust and buy-in from stakeholders because they realize we have a deep understanding of our customers. – Tweet This

The entire squad is excited, INCLUDED, and passionate about what we’re delivering. They’re very aware of what’s going on by the various ways we include discovery in our squad. Amy and I post updates in Slack. My part of daily standup is the after party where I show them prototypes of what we’re working on and get their feedback. They’ll pitch in ideas and sketches. I’ll start Slack calls so they can listen in on research and usability testing. They’ve come out to the field with us. We have quarterly two-hour Discovery Zone days where Amy provides breakfast/lunch and we present all of what we learned in discovery.

For the first time we are actually doing dual discovery and delivery. We’re incrementally improving our MVPs.

For the first time we are actually doing dual discovery and delivery. We’re incrementally improving our MVPs. – Tweet This

Bonus: We recently joined forces with another tribe to tackle a problem similar to our current problem-space and have been able to quickly fall into our rhythm of dual discovery and delivery. We’re humming along and it feels great.

[Note from Teresa: Heck, yeah! This is what a continuous discovery team looks like—empowered, engaged, and aligned!]

How often do you engage with customers?

Amy: Weekly, or we start to get itchy and just call random people from our NPS screener.

How often do you run experiments?

Amy: Little manual experiments are run frequently, I’d say several times a month to several times a week depending on where we are in a cycle. Alongside that we also deploy fake door tests and other things that require dev work, but less often—more like a couple times a quarter.

[Note from Teresa: Note how Amy distinguishes between experiments that require code changes and those that don’t.]

Are you using the opportunity solution tree?

Amy: Oh heck yes! You can pry it from my hands when I’m no longer interested in solving problems. We’ve been iterating on it; weaving in journey maps, pain points, quotes, and all sorts of fun stuff, but it’s still the same core tree.

You can pry the opportunity solution tree from my hands when I’m no longer interested in solving problems! – Tweet This

Jenn: YUP! Not only are we using it, we’re evangelizing it throughout our product org. I may or may not have called myself a Teresa Torres Ambassador and started planting opportunity trees throughout the product org. Amy and I have referenced it several times in our discovery share-outs and I’ve most recently presented alongside another researcher on the topic of observational research to our product org. It’s a one-hour session in which the first 30 minutes are about how to do observational research and the last 30 is a workshop on how to use the opportunity tree to get the most out of your discovery. We’ve done two so far within the last three weeks across all the offices and squads/tribes. What’s been really awesome is the response from the other teams sharing their trees and asking for feedback.

What have the results of this shift been?

Amy: More than a year later, my time frequently looks like that ‘ideal’ pie chart. Our sprints only include features and tasks if they are validated through discovery and usability first. Our developers’ morale is amazing and the team works together beautifully. We have time for tech debt since we are no longer furiously releasing half-baked features praying we find something that sticks. I’ve started to sound like one of those consultants selling impossible dreams when I talk about my time and my team. I’ve never had a more fulfilling time in product.

I’ve started to sound like one of those consultants selling impossible dreams when I talk about my time and my team. I’ve never had a more fulfilling time in product. – Tweet This

I also measure Teresa’s impact on us in dead trees. I’ve never used so many sticky notes or notebooks before. (I’ve been through 2.5 notebooks in the year and change since we finished with Teresa—a personal record).

Amy measures Teresa’s impact on her team in the number of notebooks and stickies she now goes through.

Jenn: Our OKRs are formed around the opportunities and outcomes that come from our discovery and there are SO many opportunities for us to build products people love. I have input into the OKRs and it feels good to have earned a seat at the table.

Our favorite type of research is immersive, where we have the opportunity to live their life, understand what is most painful about the hiring/finding work process. We have deep empathy for them and can speak their language, which opens them up to working with us. We have developed a group of charter customers who are excited when we meet with them to see what we’re working on. They love seeing themselves in the product, where they’ve influenced us. They understand when we don’t build something exactly as they wanted because we’re building for the persona and not them specifically.

We delivered MVPs for three features that were born out of our discovery—which customers are actually using without us putting tours, callouts, tooltips on the page. We’re iterating, building and scaling.

Jennifer and Amy spent a month doing immersive discovery with Snagajob customer Chick-fil-A—and used this discovery to fuel three new features.

What would you say to another team who was considering working with me? Why should they do it? What should they know?

Amy: I’ve heard from some teams that the time commitment is hard. It is, but it’s strategically important. Our team trained with Teresa for three months. If it were significantly less time or less rigorous, we’d have been able to fake it without truly having to form new habits that would stick.

Be flexible depending on your own circumstances—we wound up including our developers in a different way than we learned with Teresa, but for the team we have today it’s the right fit.

Stick with it. Communicate frequently back to the organization to share your learnings so that other people can see the change and start to get excited too. If your leadership is bought in and you can deliver results from this training to back their confidence, incredible things can happen.

Jenn: I would also add that even if the team doesn’t end up adopting the opportunity tree, the fundamentals behind it are reason enough to work with Teresa. The coaching reset our team and everyone gained a shared understanding of how discovery and product development works. We were all on the same page and from that we were able to form a partnership. The left side of the brain came together with the right side of the brain (or you can slice it horizontally—either way, we need each other to be present and passionate for the magic to happen).

She opens your mind to think above and beyond the solutions; and that’s where your product can become a market differentiator and have true, valuable impact.

Final note from Teresa: This Q&A blew me away. I’m so proud of this team. They answered these questions a year and a half after going through coaching and they’ve continued to practice everything they learned. They have graduated from just being a highly effective continuous discovery team to also being advocates for continuous discovery within their organization. They are real change agents doing awesome work. Congratulations Amy and Jenn! And don’t let up now. I can’t wait to see what you do next.