Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Last Wednesday I put out this call to product managers to participate in a research study about the life of a product manager.
I am overwhelmed by the response. In the first 48 hours, I have 90 responses.
I was initially thinking that I would interview 9–12 people. But now, I’m excited to make this an ongoing part of my work.
After all, if I was working in a corporate product role, I’d be doing customer interviews on a regular basis, so why should it be any different as a consultant?
I’m surprised it’s taken me this long to do this.
Time and time again, I’ve found that the things I know to do for someone else’s company, I often forget to do for my own.
But better late than never.
I’m also excited to write about this process as I go.
I know that customer research is on a lot of product managers’ minds and doing research well is not an easy task.
So I’ll use my own research to expose how the sausage is made.
The Life of a Product Manager
Beyond process, I’m also excited to share what I learn from these interviews. It can feel like product management is so different from place to place, but as a consultant I see many more similarities than differences.
Too much of what is shared in the public sphere is the ideal view of product management. – Tweet This
What is left unsaid is that most teams are struggling to get there. Product management is hard.
I hope that by sharing some of the common themes and threads that come out of your stories, it will help each of you feel like you are part of a bigger product management community.
But before we get to all of that, I have a problem to solve.
When I set up this study, I never thought I’d get 90 responses. By the way, now that this has turned into an ongoing research process, it’s not too late to add your name to the list.
While it’s a good problem to have, it does mean that I now have to figure out who to interview and when.
Which brings us to the topic of participant selection.
How to Choose Interview Participants
You might start by thinking you need to select a representative sample.
A representative sample is a subset of a population that statistically reflects the whole population.
You were probably introduced to the idea of representative samples through surveying. Or maybe through split testing.
But these methods are quantitative methods and interviewing is a qualitative method.
Quantitative methods live in the realm of statistics. Qualitative methods do not.
The mindset and goals behind quantitative and qualitative research are different. – Tweet This
And these differences impact the way that we can and should choose participants.
Qualitative Research is Descriptive
Qualitative research includes methods like interviewing, field observations, diary studies, and usability studies.
Our goal with qualitative research is to describe the participant’s experience. – Tweet This
Research deliverables are descriptive and come in the form of customer journey maps, empathy maps, and customer personas.
We can’t possibly interview or observe everyone, but we can get a deep understanding and a rich description of the experiences of the people we do interview and observe.
We don’t presume after interviewing ten people that we’ll be able to predict the behavior of all of our customers.
But we do presume that after interviewing ten people that we’ll have a richer understanding of those ten customers’ experiences.
Quantitative Research is Predictive
Quantitative research, on the other hand, includes methods like surveys, A/B testing, multivariate testing, and traffic analysis.
Our goal with quantitative research is to predict the attributes or behavior of our audience. – Tweet This
Deliverables are statistical in nature and include charts describing the attributes of a sample or cohorts that demonstrate a trend over time.
When we conduct surveys, only a sample of our audience responds. But we want to extrapolate from those responses to predict the behavior or attributes of the rest of our audience.
The same is true with A/B testing and multivariate analysis.
If you have enough traffic, you probably run your A/B tests with a small percentage of your audience, perhaps starting with 1–10%. You grow the percentage over time as the data warrants.
The small sample is meant to predict the behavior of your larger audience.
Because with quantitative research, we want to predict behavior, it’s important that we are working with a representative sample.
We need to make sure that the survey respondents aren’t statistically different from our larger population. Otherwise our results wouldn’t be predictive.
The most common way to do this is with random sampling. This is why you randomly divert a percentage of your audience to your split test.
Using Qualitative and Quantitative Research Together
Teams tend to skew toward qualitative or quantitative methods, but not both.
User researchers tend to favor qualitative methods. Growth hackers and Lean Startup advocates tend to favor quantitative methods.
But the real power comes from combining both.
You can use qualitative research to get a deep understanding of some of your customers. This deep understanding helps you generate insights.
You can then test these insights quantitatively to understand whether or not they predict the behavior of your larger audience.
For example, with my interviews, I’m trying to understand your experience as a product manager.
Specifically, I want to know how you invest in and develop your skills given that you have a hectic job in a fast-changing field.
By interviewing you, I will learn a lot about your unique experience. If I want to know if your experience is representative of the product management community at large, I’ll need to pair my interviews with quantitative research.
But if I started with quantitative research, I would just be guessing at your experience. Unless I’m lucky, it would take me a long time to find a successful quantitative test.
I don’t like to rely on luck. So I’m starting with qualitative research.
Qualitative research generates insights. Quantitative research predicts if those insights apply to a larger audience. – Tweet This
Select a Variety of Qualitative Participants
Now that you understand the descriptive nature of qualitative research (versus the predictive nature of quantitative research), you can throw out the idea of selecting a representative sample for your interviews.
However, you still need to choose your interview participants wisely.
When you select qualitative research participants, select for variation. – Tweet This
A good variety helps you understand more of the experiences related to your topic.
The experience of a product manager at a ten-person startup is likely to be different from that of a product manager at Google or Facebook.
If I want to understand the experience of product managers in general, I need to talk to both.
If you look at my screener, you can get a hint at where I’m looking for variation.
I asked you to tell me your:
- Job title
- Years of experience
- Company size
- How many people you manage
- How important your own career development and growth is to you
- (For leaders) How important your team’s career development and growth is to you
Now there are no right or wrong answers to these questions. My research isn’t focused on product managers with 3–5 years of experience or product managers from large companies.
Instead I’m interested in breadth as I’m looking for patterns in behavior and experience across all product managers.
I want to make sure that I interview people from a variety of company sizes, with different years of experience, different job titles, and eventually different levels of the importance they put on their career development and growth.
And this is reflected in my first set of scheduled interviews.
You can see I have decent variation in years of experience. That was my priority for this first set. And I have some variation in company size and slightly less variation in the number of people managed.
Some of this is a reflection of the pool that I’m selecting from. I have very few responses from people who manage large teams. If you manage a large team and are willing to participate, I’d love to hear from you
Down the road, I might find that an interesting pattern emerges among product managers with 6–9 years of experience. At that time, I can choose to narrow my research to just that audience.
If I do this, then the years of experience question becomes a qualifying question that will rule out participants, whereas now it’s there to ensure variation.
Focus on Extreme Users
The d.school at Stanford and design thinking advocates everywhere encourage us to focus on extreme users.
A common example they use is that of designing carry-on luggage. They encourage you to explore how the needs of a child might differ from the needs of a senior citizen.
You might misinterpret this exploration as trying to find a target customer segment. But that’s not the intent of this exercise.
The goal is to uncover more of the experiences with carry-on luggage. Even if your target customer is business travelers, you will still benefit from exploring the experiences of extreme users like children and senior citizens.
An insight gained from observing these users may apply equally to business travelers.
For my study, I’m interested in exploring how people develop their product management skills. I’m less interested in how people become product managers.
But I might still benefit from interviewing people who want to become product managers. They have the most to learn and might provide valuable insights that apply equally to experienced product managers.
How to Know if You’ve Interviewed Enough People
One of the advantages of quantitative research is there’s an element of certainty to it.
You can do math to determine whether or not your sample is representative of your larger audience.
This makes us feel good. It helps us feel like we are doing the right thing.
However, this feeling of certainty, even with quantitative research, is often overblown. See the accuracy of the Iowa Caucus polls for a great example of how even large scale polls by good statisticians are uncertain.
But regardless of how uncertain quantitative samples are, qualitative samples are even more uncertain.
And this makes us uncomfortable.
The general rule of thumb in qualitative research is to interview enough people so that you get enough variation and until you start to see the same patterns or hear the same stories over and over again.
Jakob Nielsen argued this in the context of usability studies
I’ve now read it in three different books about academic qualitative research (here, here, and here) and one industry book on qualitative research.
You’ve interviewed enough when you have both variety and you hear the same patterns again and again. – Tweet This
But most people’s reaction to this rule of thumb is one of fear.
It’s messy. It’s ambiguous.
It forces us to question, “will I really know it when I see it?”
But the answer is yes, you will.
You might overdo it at first. You might interview more people than you need to. You might run a few extra participants through your usability study. But there’s no harm in that.
With experience, you’ll build a good intuition for when enough interview participants is enough. – Tweet This
If you want to keep following along as I conduct this research, please join the Product Talk mailing list so that you don’t miss the next article.
Always interesting, any thoughts on exploring mixed method research?