How Compare and Contrast Decisions Lead to Better Product Outcomes

This past week I was in London speaking at Mind the Product. As usual, the Mind the Product team hosted a phenomenal event. You can watch the video here:

Full Transcript

This transcript is the script of the talk as it was written. It is not word-for-word the talk as given.

Hi everyone. I’m excited to be here. I’ve been developing a visual critical thinking tool that I want to share with you today. It’s a bit hard to explain outside the context of a specific product challenge, so I’m going to start by sharing a story that I think will resonate with many of you, and then we’ll get into the tool itself.

Back in 2008, I was a product manager at a startup that operated online communities for university alumni associations. Like many product teams, we had some tough challenges to address.

While alumni associations (our customers) loved our product, alumni (our end-users) did not. When we launched a new community, we’d get a rush of activity as alumni came to check out their new site. But over time, engagement would dwindle to a predictable trickle.

Our user research told us that alumni loved sending messages to their community: They asked for advice on everything from how to find their next job to what neighborhood to live in in their new city. It was exactly the type of engagement we were hoping for.

There was only one problem. Nobody wanted to receive these messages. We had alumni in Dallas receiving emails about items sold in Chicago, houses for rent in Boston, and internships in San Francisco.

We were making it easy for people to spam their entire alumni network. If we wanted to increase alumni engagement, we needed to reduce the number of unwanted messages in our communities.

Now if you are like me, your brain is already starting to think about how to solve this problem. But when I turned to my team and said let’s brainstorm, I got a surprising response. Seth, one of our engineers piped in with, “Let’s integrate Google maps!” Seth wanted to use the Google Maps API to integrate a map that showed where alumni lived around the world.

I was shocked. This idea came out of nowhere. I was trying to figure out how Google Maps was going to help us address the spam problem. So I asked Seth. And he responded, “Oh, it won’t, but it will drive engagement because it’s cool.” I looked to the rest of the team for help. And sadly (from my point of view), they agreed with Seth. Maps would be cool.

At the time, I didn’t have the words to express my frustration, but intuitively, I knew that building cool stuff wasn’t good enough. Knowing where people lived didn’t feel like a big enough need. And adding a Google map felt like a gimmick.

Now this story isn’t about, “I’m right and Seth is wrong.” We’ll see in a minute, it’s more complex than that. It’s a story about me as a product manager wanting to include my team in deciding what to build, but not knowing how to do so in a productive way.

Today, as a product discovery coach, I see this play out on team after team. We don’t know how to effectively go from a stated outcome like increase engagement to executing on solutions that will drive that outcome.

So I started to deconstruct the problem and here’s what I found.

We Fall in Love with Our Ideas

It’s easy for us to generate an idea. We hear about a need, we immediately think of a solution. It’s almost automatic.

And because it feels good to close that loop, we tend to fall in love with that idea.

And when we fall in love with our idea, we don’t think to examine it. We don’t pause and reflect. We don’t ask, “Is this idea any good?”

This is what happened to Seth. He learned about the Google Maps API and got excited. He wanted to try it out. He shared his idea with the rest of our team and they quickly fell in love with the idea, too.

We Don’t Consider Enough Ideas

When we fall in love with our ideas, we don’t consider enough ideas.

My team was so enamored with the Google Maps idea, they wanted to dive in and start creating. They wanted to do something that would drive engagement right now.

And don’t get me wrong, the Google Maps idea may be a good idea. But research on brainstorming shows that when we generate more ideas, we generate better ideas.

When we generate more ideas, we generate better ideas. – Tweet This

And more importantly, when we consider other ideas, we set ourselves up to make a “compare and contrast” decision rather than a “whether or not” decision.

A “whether or not” decision is a decision where we ask, “Is this idea good (or not)?” This is a hard question to answer because it treats “good” as an absolute trait.

Instead, we want to ask a “compare and contrast” question: “Which of these ideas looks best?” This is easier to answer because it treats good as the relative trait that it is.

Imagine Usain Bolt running around a track on his own. Is he fast? It’s hard to say. Now imagine him running around a track with other runners. Is he fast? Absolutely. A “compare and contrast” decision makes it easier to evaluate a relative trait.

Ask “compare and contrast” questions, not “whether or not” questions. – Tweet This

Now for those of you who are thinking you already consider a lot of ideas, you probably do. Many of us have way too many ideas. I’ll come back to this problem in a few minutes.

But first, let’s return to my team’s challenges. Not only did we fall in love with our first idea and therefore didn’t consider enough ideas…

We Don’t Align Around a Target Opportunity

… we also didn’t align around a target opportunity (or problem that we were trying to solve).

Seth’s Google Maps idea drove me nuts. Not because I thought it was a bad idea, but because I thought it was irrelevant. It didn’t solve the problem I wanted to solve.

But I didn’t take the time to make sure that my team was aligned around the problem we were solving before we jumped into idea generation. As a result, Seth was thinking about our engagement goal, but he wasn’t thinking about reducing spam, the problem that I was focused on.

Even when teams do align around an opportunity…

We Rarely Consider Enough Opportunities

… we rarely consider enough ideas.

I walked into the brainstorming session assuming that reducing spam was the right opportunity. Seth walked into the session wanting to help people connect with alumni who lived near them. Both of us were only considering one opportunity.

Just like we want to avoid “whether or not” questions with ideas, the same is true with opportunities. We don’t want to ask, “Is this opportunity worth pursuing?” We want to ask, “Which of these opportunities looks best?” And that requires that we have a set of opportunities to choose from. If we don’t do this, we run the risk of solving unimportant problems.

What we should have done was taken a step back and asked, “What are all the opportunities that might drive alumni engagement?”

Product teams rarely consider enough opportunities before jumping into solutions. – Tweet This

So how do we prevent these mistakes?

Visualize Your Thinking with an Opportunity Solution Tree

I want to introduce you to Anders Ericsson. He wrote the book Peak, which is a summary of his life’s work trying to understand the differences between novices and experts. He argues that experts use more sophisticated mental representations than novices.

He defines a mental representation as follows:

“… representations are preexisting patterns of information—facts, images, rules, relationships, and so on—that are held in long-term memory and that can be used to respond quickly and effectively in certain types of situations.”

And he argues, “The key benefit of mental representations lies in how they help us deal with information: understanding and interpreting it, holding it in memory, organizing it, analyzing it, and making decisions with it.”

That’s great. Isn’t this exactly what we need? Something that allows us to understand, interpret, organize, and analyze all the information we collect so that we can make better product decisions with it?

Looking back, here’s how I would diagnose my team’s challenges. I came to the brainstorming session with a depth of knowledge about our users. I had just completed an extensive round of user research. Seth came to the brainstorming session with a depth of knowledge about new technology. He had just read about the Google Maps API.

We each came to our brainstorming session with different patterns of information. And we were each relying on our own mental representations to make fast decisions.

The only problem is product teams need to make fast decisions from a shared mental representation of their combined knowledge.

Product teams need to make decisions based on their combined knowledge. – Tweet This

This challenge is what led to what I call the opportunity solution tree.

Start with a Clear Desired Outcome

A product team needs to be clear about what they are trying to achieve.

My team had a clear desired outcome—to increase alumni engagement.

But as we saw with my team, a clear outcome wasn’t enough. We needed to ask ourselves, “What will increase engagement?” And before we jumped to solutions, we should have mapped out the opportunity space.

It’s easy to think about opportunities as customer needs or pain points, but we also want to consider opportunities to delight, and opportunities to replicate success.

Opportunities Should Emerge from Research

These opportunities should emerge from generative research—customer interviews and customer observations. To make sure we stay user-centered, I like to frame opportunities as something a customer might actually say.

Opportunities should emerge from generative research—customer interviews and customer observations. – Tweet This

Now my team had just wrapped up a series of alumni interviews and we could have easily generated the following list of opportunities:

• I get too much email. – My opportunity
• I’m moving to a new city and want to know who lives there. – Seth’s opportunity.
• I need help finding a job.
• I want to stay connected to my alma mater.
• I want to know what my college friends are up to.
• I’m looking for something interesting to read / learn.
• I want to keep up with my school’s sports.
• I want to hire a recent grad.
• I’m willing to donate but want to know the impact.
• I’d like to give back to the community.
• I’d enjoy mentoring a student or recent grad.

This list represents what we heard in our alumni interviews.

Now what? Most of us would start prioritizing this list by asking which of these opportunities is most important for us to address next.

But it doesn’t make sense to compare an aspirational opportunity like “I’d like to give back to the community” with a more specific opportunity like “I want to hire a recent grad.” It’s hard to prioritize a list when it isn’t made up of similar items.

The items on this list also aren’t all distinct from each other. Hiring a recent grad or mentoring a student are both ways in which an alum might give back to the community. We would be comparing apples with fruit. It doesn’t make sense.

Don’t prioritize a list of unlike items that aren’t distinct from each other. – Tweet This

An Opportunity Solution Tree Simplifies Prioritization

If I group similar opportunities together, the list becomes more manageable. I end up with three distinct groups.

• I need help.
• I want to stay connected to my alma mater.
• I want to give back to the community.

Instead of prioritizing the longer list, I can start by prioritizing these three groups.

I can tell you from our research that the most prevalent opportunity we heard was “I need help.”

But notice where my “I get too much email” opportunity lives.

Suddenly, it looks like Seth might be on the right track.

If Seth and I were having this conversation today, the tree would help us move up one level. Rather than arguing about “too much email” vs. “who lives near me,” we would want to start by asking which of these top three opportunities are more important. It would have been easy for us to agree that the “I need help” opportunity was the most important for alumni. We should have then shifted our focus to prioritizing the children of the “I need help” opportunity.

Now I would have had a problem with this. I would have argued that the “I need help” opportunity is intrinsically linked to the “I get too much email” opportunity. We need to connect the person who needs help with someone who can offer help. And if we are overwhelming people with too much email, they will be less likely to help.

This tells me that we didn’t get the structure of our opportunities quite right. If two opportunities are inherently linked then I want them closer to each other on the tree to reflect that connection. Let’s try again.

Play with the Structure of Your Opportunities

First, I merged the “I want help” opportunity with the “I want to give back” opportunity into one opportunity called “I want to connect with other alumni.”

Underneath that we have:

• I want to connect with alumni professionally.
• I want to connect with people near me.
• I don’t know who to connect with.

This new structure brings the two sides of the market—people who need help and people who can offer help—closer together under the same opportunities. This makes it easier to prioritize the different types of connections (e.g. professional, location, etc.) without favoring one side of the market over the other.

It also minimizes the disagreement between me and Seth. We can both agree that we should be working in this left branch. We just disagree on which sub-opportunity is more important. This makes it easier to look at data to resolve our differences. We might ask, “How many alumni want to connect with people near them?” vs. “How many alumni don’t know who to connect with?”

Sometimes it takes a few tries to find a structure that works. The key is to make sure that it reflects what you are hearing and seeing from your customers and that it helps you make good decisions about what to prioritize.

The structure or your opportunities should reflect what you are hearing in your customer interviews. – Tweet This

Remember, there are no right or wrong answers here. Your tree structure will continue to evolve as your team learns more about your customers. The value of visually externalizing your thinking is it will help your team resolve differences and align around a shared perspective.

Once you have an opportunity structure that you are happy with, you are ready to consider solutions.

When You Have Too Many Ideas

Earlier I said that we don’t consider enough ideas and I mentioned that that might not resonate with some of you. That’s because most of us have too many ideas. So let me clarify what I meant.

We don’t consider enough ideas for the same target opportunity. We may have many ideas, but they tend to be spread across our tree, like this:

The value of generating a lot of ideas is to push past the first obvious ideas, to get our creative juices flowing, to get to the truly innovative solutions. When we brainstorm across the tree, we might generate a lot of ideas, but we generate a lot of first ideas.

Not only do we lose the creative benefit of brainstorming, but we also end up with a lot of solutions that we don’t know how to prioritize. Remember it’s hard to prioritize a list of unlike things.

Focus Ideation on a Target Opportunity

Instead, we want to choose a target opportunity by prioritizing row by row and then generate solutions for just that one opportunity. Like this:

 

I want to see lots of ideas for how to solve a single opportunity.

Generate ideas for one target opportunity rather than across opportunities. – Tweet This

Now after you’ve generated these ideas, your inclination might be to prioritize the list and start experimenting with the top idea. But this leads to another “whether or not” decision. Is our top idea good or not? Instead we want to set up a “compare and contrast” decision. I want you to ask, “Which of these solutions looks most promising?” When we start with a lot of ideas, I recommend two ways to compare and contrast solutions.

Use Dot Voting to Go from Lots to Some

First use dot voting to whittle down a large list to 3–5. Research shows that groups are better than individuals at evaluating ideas and dot voting is a quick way to poll the team.

Use Experiments to Compare and Contrast Your Smaller Set

Then use experiments to identify which of the remaining 3–5 ideas looks most promising.

Use dot voting to whittle a list from lots to some. Use experiments to evaluate the smaller set. – Tweet This

I want to highlight a key point here. When most teams experiment, they experiment to determine if a single idea is good or not, another “whether or not” decision. I want you to experiment to help you choose amongst a set of good ideas, setting up a “compare and contrast” decision.

The easiest way to do that is to identify the key assumption that needs to be true for each idea to work and then run experiments to test each of those assumptions.

For example:

If our three ideas are recommending recipient criteria like where people live, auto-matching messages with recipients, and sending messages to friends of friends, the key questions for each might be:

• Will people asking for help trust our recipient recommendations?
  • To test this: We can prototype the user interface and see how people respond.
• Will we be able to predict who should be able to receive a message?
  • To test this: Our machine learning team can run feasibility experiments.
• Are friends of friends more likely to help?
  • To test this: We can look in our database and see if friends of friends are more likely to respond to previous messages.

I often get asked, “How do I know if my experiment results are good?” For example, is 15% conversion rate high enough? This is akin to asking, “Is Usain Bolt fast?” when he’s running alone on a track. It’s hard to say. But if instead, you experiment with multiple ideas, you can ask, “Which of these solutions looks best based on the data I’m collecting?” which is much easier to answer, just like it’s clear that Usain Bolt is fast when he’s running with other runners.

Run experiments to choose amongst a set of solutions, not to evaluate a single solution. – Tweet This

The Benefits of An Opportunity Solution Tree

If like me, you want to engage your entire team in deciding what to build, but you keep getting caught by your own version of my Google Maps challenge or your team gets bogged down in opinion battles, then I’d recommend you start building an opportunity solution tree.

Taking the time to visually map out your thinking, will help your team catch common critical thinking errors. Errors like creating “whether or not” decisions instead of “compare and contrast” decisions.

The tree also acts as a discovery roadmap, helping your team align around a shared understanding of the opportunity space and the potential paths to reaching your desired outcome.

And like traditional roadmaps, it will help you communicate what you are learning to your leaders and the rest of the company.

Your opportunity solution tree serves as a discovery roadmap—helping your team align and communicate. – Tweet This

Get Started with Your Own Opportunity Solution Tree

Start with a clear desired outcome.

Then map out the opportunity space. Remember opportunities should emerge from generative research: customer interviews and customer observations.

Play with the structure of your opportunities. Different structures lead to different possibilities.

Choose a target opportunity by prioritizing row by row.

Limit idea generation to your target opportunity.

And finally, run experiments to evaluate your set of solutions.

If you build your own opportunity solution tree, I’d love to hear how it goes. Please send me an email or tweet at me. And if you want to know more about some of the research I referenced today, I put together a list of sources for you.

Thank you!

Get the latest from Product Talk right in your inbox.

Join 41,000+ product people. Never miss an article.

Subscribe