Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. A few months ago, I sat down with Hope Gurion, Chief Product Officer at CareerBuilder, to talk about her team, how they work, and how she stays current in the fast-moving world of product management.
I’ve had the pleasure of knowing Hope for the past two years and I am thoroughly impressed by her leadership and vision. She sets a great example for the rest of us.
Over the course of this 45 minute conversation, we cover a lot of ground. Hope started in business development at CareerBuilder and introduced both product management and user experience design to the company.
We spend much of the conversation talking about her team’s move to dual track Agile, the challenges with adopting product discovery practices, and how they continue to iterate on their delivery practices.
CareerBuilder has both a consumer and an enterprise component to their business and we talk about the challenges of product discovery in both environments.
For more details including timestamps for specific topics, see the show notes below the video.
Can’t see the video? View it here.
Show Notes:
- 0.40 About her team
- 1:34 How she got to where she is
- 4:55 How she introduced product management and user experience to CareerBuilder
- 7:29 Who does product strategy at CareerBuilder
- 10:03 Building good habits / Managing ideas
- 11:28 Why product managers should prioritize with their cross-functional team
- 13:05 Working with Marty Cagan, the value of Collaborative Gain
- 15:00 Dual track Agile, learning before writing code
- 17:14 Setting the expectation of doing product discovery
- 18:44 How discovery happens for new product ideas
- 22:25 Why discovery is worth it
- 23;32 What happens to sales requests and executive ideas
- 27:40 Why her team doesn’t use roadmaps and what they do instead
- 29:30 How discovery works with enterprise clients
- 31:15 The biggest challenge in adopting product discovery practices
- 33:07 On getting her team to focus on what they can do vs. what they can’t do
- 35:10 Creating a shared backlog across product managers to get better focus
- 38:40 The value of weekly demos across self-organizing teams
- 44:05 On making a focus on outcomes not output habitual
- 45:09 On the challenges ahead
Resources Mentioned
People
Companies
Tools
Video Transcript
[Edited for brevity and clarity]
Teresa Torres: Hi, I’m Teresa Torres with Product Talk and I’m here today with Hope Gurion, Chief Product Officer at CareerBuilder and we’re going to talk a little bit about the evolution of product management at CareerBuilder. Thanks for being here Hope.
Hope Gurion: Thanks Teresa, I’m excited for this.
Teresa: I’m excited too. Tell me a little bit about your role at CareerBuilder.
Hope: I run the product group at CareerBuilder and that’s a mix of some of the product management functions, UX, analytics, and then also digital marketing to help draw in new users to our sites.
Teresa: Okay, so digital marketing fits in product?
Hope: In our organization it does.
Teresa: Okay, perfect. Tell me a little bit about your team. How many product managers do you have, how many user experience folks do you have?
Hope: I don’t have exact counts. Roughly we have a team of about 15 to 20 product managers and about a similar ratio of interaction designers and a couple of visual designers.
Teresa: Do you have user researchers?
Hope: We have a couple user researchers, and within our analytics groups—we recently shifted this—we have two people who are responsible for more of our user analytics. They’re paired as part of our analytics team, but they originally were part of our UX team. We’re really making sure that all the product teams have access to Google Analytics data about their users and the site behaviors.
Teresa: Yeah, that’s great. You oversee almost everybody but engineering that’s related to getting your product out the door.
Hope: Yes.
Teresa: Okay, tell me a little about how you got to where you are.
Hope: My first roles in product management were when I worked at AOL. This is more than a decade ago. I ran the jobs and real estate verticals on AOL and then ran our shopping channel on AOL. It’s funny to think about the contrast between the type of product management that I was doing then and what I’m trying to enable our teams to do now, because that was definitely in the world of PRDs that were requested but never read and trying to launch things very slowly and largely about managing stakeholder and advertiser needs as opposed to user needs. It’s definitely a contrast to what I’m doing now.
Teresa: Yeah, that’s great. I definitely want to get into what you are doing now because I think if I didn’t know about you or CareerBuilder I would assume that CareerBuilder was building products like AOL.
Hope: No. Not at all.
Teresa: Excellent. I think that’s exactly what’s going to make this interview interesting. So you didn’t start out in product management at CareerBuilder, you started out in business development?
Hope: Yeah, when I first started I was VP of business development and that was a mix of our ad, sales and operations, and a few other things that were largely operational or sales related. The way that I interpreted my role was what do we need to do to help ensure that our company grows, so the business actually develops. What that led to was how do we actually expand beyond our two core products, which were jobs and resume database. At the time I started back in 2006, those were really our primary two products.
What that led to was creating a lot of additional products and services that were closely related to the technology and products and where we really had expertise and were directly related to things that our customers needed and our sales force would be capable of selling. And so that is what led into the creation of a product management function at CareerBuilder because we started expanding our products.
Teresa: Were there product managers at the company before that?
Hope: No. This is very recent.
Teresa: Yeah, so you really introduced product management as a function to CareerBuilder.
Hope: Absolutely.
Teresa: What about interaction design and the analytics piece?
Hope: That’s even newer. I remember when I first joined there was a single UX person who was largely trying to make the case for the need to do UX and we were working with an outside agency to create all of our interaction designs. It wasn’t a very successful strategy, needless to say.
And then actually because that person wasn’t particularly successful we eliminated the role and literally had zero interaction design function. It was something that we thought the engineers could just handle as well. It was over the last couple of years that we recognized the deficiencies of not making that investment and we’ve worked to correct it. Now we have a really robust team of about 15 people who are a part of the product team with the engineering lead and our product leads, and we also have a UX lead as part of each product team.
Teresa: That’s a really large organizational change. Tell me a little bit about how that came about.
Hope: It was really born out of necessity—we really had unusable products in a lot of ways. We saw it evidenced in the need for a lot of training from our customer care team to invest in helping our customers be able to use the products. A lot of inbound requests that came in, in terms of our consumer site and people finding things unusable, or confusing. So over time we recognized it’s a worthwhile investment to make sure the value to the customer doesn’t gets lost because it’s just not that intuitive or usable. So once we started making that investment I think that it snowballed and people wanted to make sure that every product had the luxury of that investment and that’s what led to the expansion of the team.
Teresa: Yeah, I just realized we both have a lot of context for CareerBuilder and sort of the recruiting space. Let’s take a minute and tell me a little bit about who your customers are, versus your consumer site.
Hope: Yes.
Teresa: Do you want to just lay the foundation for the rest of the interview?
Hope: Sure, CareerBuilder is a classic two-sided marketplace. We have consumers or candidates, as we refer to them. These are the people who are often using not just the CareerBuilder site, it could be our niche sites, it could be some of the career sites that we create for our employer customers, but typically most of our products have that consumer or candidate side to the marketplace and then we have a suite of other products. Most notable would be job postings, resume, database, our talent networks, or other career sites or ATS [applicant tracking system] functionality that are servicing a more corporate customer. It could be a staffing agency, could be an employer, could be the recruiting or acquisition function at an employer. Those are the backend users of our products and we really want to make sure that they get the quantity and quality of people that they need—typically so they can fill open roles that they have at their organizations.
Teresa: Okay so very simply your job seekers are consumers, your consumers are people that are actively looking for jobs?
Hope: Yes.
Teresa: Or maybe passively looking for jobs. Then your customer is an employer looking to hire or an organization helping companies hire.
Hope: Yes, correct.
Teresa: Okay, excellent. Let’s get a little bit into how product management works at CareerBuilder. As a chief product officer I imagine you’re responsible for product strategy at CareerBuilder. What does that look like across your team? What do you do, what do the other people on your team do?
Hope: I think that in a lot of cases what I want to do is enable the different product teams to formulate their own strategies that they think best serve the needs of their customers and users. I feel like it would almost be inappropriate and misplaced to say that I’m going to determine every strategy for every product. I just feel like that’s too disconnected from the reality of the products and the users. I really want to make sure that the product teams are the ones that are closest to the users and the customers so that they can formulate strategies and I’m just figuring out how to remove the obstacles that prevent them from delivering on the value that they’ve identified that they want to deliver.
Hope explains her job is ‘to enable my product teams to formulate their own strategies …[they] are closest to the users and customers.’ – Tweet This
Teresa: Yeah, so what does that look like in terms of what you are doing to enable your team or where they need help?
Hope: Right, so over the last several years once we recognized the need to have a product management function, that it couldn’t just be a customer or sales person calling the engineer and getting whatever they asked for built and then we recognized the gaps. We didn’t have an investment in the UX. We also recognized that our analytics were not as robust as we needed them to be. Again, we’re trying to help the product teams make good decisions on behalf of the company and on behalf of our customers.
So we’ve made significant investments in the analytics, the user research function, understanding how to do discovery so that we can actually really elicit what the true pain points or delight opportunities are, and then the thing that I spent most of my time doing is asking, “How do I make it a habit?” How do I make sure that they’re doing it not sometimes, not when I ask them to, not when I ask them to deliver a presentation about what they’ve done—but that it is part of what they expect from themselves and for the team every single day so that they are constantly moving forward,constantly evolving the products in a way that make them easier to use and more valuable for our customers and therefore more valuable for the company.
Hope asks, ‘How do I make good product discovery a habit for my teams?’ – Tweet This
Teresa: Let’s unpack that a little bit. It seems on the surface level this is very obvious, isn’t that what product managers do? We make our products better incrementally over time. But I love what you said about how do we make it a habit, because I think it’s really easy to get bogged down in day to day pushing paper, making sure engineers have things to build. What are the things that you’re doing or your team is doing to build those habits?
Hope: Part of it is making sure that the teams themselves have created an agreed upon scorecard. A process, a mechanism by which they can truly objectively evaluate all the good ideas. They’ve got the ideas, stakeholders have the ideas, customers have ideas, users have ideas. There’s no shortage of ideas. I think that’s what causes people to get bogged down in a lot of cases. There’s so many ideas and you don’t know which ones are really important and impactful. How do we take all the rich ideas and synthesize and work through the ones to get to the few that really matter?
I feel like that’s where a lot of the time goes, is let’s look at what are the top of your lists for the greatest ideas right now? How do you know that those are great ideas? How feasible are they? How do you know that they’re going to have the intended impact? Once we sort of get to a short list, and it’s constantly weeding through that list, then we get to, okay now how are we going to look to validate or invalidate that these are good ideas or that the design or the feasibility of them is going to be accomplished in a short period of time.
Teresa: So is each product manager doing that within their scope? Each product manager has a long list of ideas, they’re doing the first cut at identifying these are the ones that I think are most promising—
Hope: Yes.
Teresa: And they’re focusing on vetting those ideas?
Hope: Yes. Sometimes that’s the case. What I have found is that when product managers work alone, they go faster, but where they tend to make the most progress is when they’re actually working as a true product team and they’re working through the long list of ideas and what is going to be the way in which we prioritize these ideas and decide that the engineering lead, the UX lead, and the product lead together. Otherwise, the next thing that will distract them and take up a lot of time is that constant debate about whether or not they’re working on the most important thing. And so I find it helpful if they can come up with their own scoring criteria and once they all agree that these are the most important things, then let’s look at it and figure out what are the things that we need to do to help you execute on those ideas, or at least get to a place where you’ve invalidated, or validated the ones that are actually worthwhile.
When product managers work alone, they go faster. But when they work as a team, they make the most progress. – Tweet This
Teresa: Yeah, when you say product team, you mean that cross-functional tech lead product manager UX person.
Hope: Exactly.
Teresa: How long has CareerBuilder been working that way?
Hope: We’re not consistently working that way across all product teams. We’ve acquired some companies, there have been offshoots. I would say it’s not as consistent, but within my team that is absolutely the expectation.
Teresa: Okay. That sounds very influenced by Marty Cagan’s school of thought and I know you’ve done some work with him. How did you hear about Marty Cagan, how did you bring him in? How did that come about?
Hope: One of the groups that I’m involved in that I value tremendously from a professional development standpoint is this Collaborative Gain group. Do you want me to describe it?
Teresa: Please, Phil Terry’s cheering somewhere.
Hope: I’m sure he is. It was transformational for me. It’s a group of product, tech, UX executives at a lot of digital companies, some marketing as well and there are these non-competitive counsels where people meet twice a year and we have a lot of interactions throughout the year. We share and collaborate around how do we solve the problems that make product management really hard? How do we actually help our teams be successful to deliver great products that serve user needs and create those products with the users?
Marty Cagan obviously has done lots of work with lots of those companies and after the eighth person told me that they had been trained by Marty Cagan I said, “Well, I don’t know what I’m waiting for.” We’d read the book, we said okay let’s just bring him in and really have him do a diagnosis and prescription on the needs of our organization. He identified our lack of UX, he identified lack of QA, identified that we weren’t working through rich product backlogs and we didn’t have a good cadence for discovery and delivery. We’ve done a lot of work to correct for those deficiencies.
Teresa: This was pretty early on at product management at CareerBuilder when Marty Cagan came in?
Hope: That’s true.
Teresa: One of the things he talks about is not just cross-functional teams, really owning the whole process, but also this idea of dual track agile where you have your discovery process and your delivery process. I know that you guys are working under that philosophy. Tell me a little bit about how that works.
Hope: We weren’t doing sufficient discovery—or when we would do it, we were building and then figuring out what works. The way he coined it is that we were doing all of our discovery in delivery. We were being data driven, we were making sure that we validated with the users, unfortunately we were just doing it in the slowest, most resource-intensive way.
We were doing all of our discovery in delivery… the slowest, most resource-intensive way. – Tweet This
So how do we have leaner methods and make it more habitual for people to be figuring that out before we have to write any code? That’s been part of the team’s evolution is recognizing that that is A) important to do, and B) thinking through what are the problems that we should actually be doing any sort of discovery effort for. It takes time, if you’re going to have a really well thought out objective validation process—you can’t just wing it. You have to be intentional about how you plan that out. It’s taken different people different learning curves to be able to get to that point. Then people are human. I have humans working in product management and they aren’t always objective and they fall in love with their ideas and they only want to be proven right. Having that objectivity when you have a list of ideas, to really have that discipline to know that really only 10% are worth doing when you’re so excited to do all of them requires this habit to help people really get good at discovery. Help them recognize that it’s okay to invalidate.
Our process with dual track agile has been doing more discovery than we’ve ever done and our evolution is learning to ask ourselves how do we make sure that we are only focusing on the 10% of things that matter.
Teresa: Yeah, you mentioned that everybody has sort of a different rate of learning, but it hasn’t prevented your team from moving forward with this new methodology?
Hope: Right.
Teresa: Right, so across your 15 to 20 product managers they might be each in a different place in terms of their skills or even their habits around discovery, but they’re all doing it.
Hope: They’re all doing it, yes.
Teresa: I know a lot of teams get stuck on, I don’t know how, I don’t know where to start, it’s not perfect. Your team seems to have balanced very well, we’re just going to do it and we’ll get better at it over time.
Hope: That’s right.
Teresa: How did you instill that?
Hope: It was the expectation. That is what I spend most of my time talking to my team about. Again, what’s on the list? How are you going to invalidate or validate it? That is the majority of the discussions that we have. It’s one of those inspect what you expect, traditional management techniques. Like I said, there were other things that we had to put in place. I couldn’t expect them to do a lot of true validation at scale without an investment in Google Analytics and some of the testing and optimization platforms. We put a lot of that in place, but now it’s there and so now it’s more exciting because now we’re really, it’s back to the ideas and which of those ideas have the most merit and which are the ones that are going to deliver the most value for the least amount of effort? That is why it is the majority of the discussions that I have with my product managers.
Teresa: That’s great. Walk me through what that looks like in practice. Let’s say that I have a new product idea as a product manager, it goes on my long list, it makes the cut to my handful that I’m going to focus on this quarter, what happens next?
Hope: It depends on the nature of the idea. Assuming it is a traditional sort of feature that is going to be manifested in some sort of UI on the site,then really what we would do from that point is consider the expected impact. Typically, that would lead to, if it’s a great idea and you haven’t done any of the qualitative research then ideally that’s the place to start. Why is this the most important idea? Why does this bubble up to the top and outrank all the other ideas? Because it’s solving a really frustrating pain point for a user or it has tremendous potential to really elevate the experience. That’s what we’re usually trying to figure out. Are we solving for pain or are we trying to really increase delight and engagement with the site?
Then, what is the expected impact? Almost everything that we would do, we really try to tie it back to some sort of business metric. Either it’s solving a need for customer, it’s going to drive up one of our KPIs… I would say it’s very rare that we do something just to minimize a pain point for a user. It really has to tie back to a metric for one of our customers or for our company.
And then we want to get quickly to assessing the relative importance of that feature and we do that by trying to size the potential impact. Typically, what that leads us to do is more of a quantitative experiment of some sort where we can get to a number. It may not be a great projection of what the future outcome will be, but it helps us come up with what’s the relative value of that versus all the other ideas.
That’s typically the process. And that process also helps us work through the UI of the solution. During that process we can prototype potential UIs of that proposed solution and we can validate that again both qualitatively and ultimately quantitatively when we run the experiment.
Teresa: So give me a sense of what you mean by qualitative tests and what you mean by quantitative tests.
Hope: Okay, so qualitative tests we’ve been doing in a few different formats. It could be that we have users come into the office and we’ll have them actually use our services and we’ll just strictly observe them to see if we’ve correctly identified the pain points or how they’re using the service.
Other times, if it’s a mobile app, we’re out in the street watching people use our app or use another competitor’s services. And that’s really just to make sure that we correctly identified the pain point or the delight opportunity. That’s typically what it is. Or if we feel like there’s a good solution, typically if it’s a competitor’s solution and we think they might be onto something, we want to actually validate that it is working as users expect.
That’s really the types of qualitative research. Sometimes it’s UserTesting.com or some other types of research, but that’s generally very user observational based. And then when we shift over to quantitative, that’s typically more of some sort of onsite experiment or an email test or some other way that we’re going to be able to use to get a data point that helps us assess incremental impact.
Teresa: If I have an idea and I have to go through qualitative and then quantitative before it goes to my engineers, it sounds like a lot of work.
Hope: It sounds like a lot of work, yes.
Teresa: What’s the argument? Why does your team do all of that before it goes to engineering?
Hope: I think again, this comes back to we’ve experienced not doing it. We’ve absolutely been in a mode at points in our past—we’re trying to prevent it in the future—where we fall in love with an idea, we’re so sure about the solution, we think ‘Why waste all that time when we can just rush it into delivery? It won’t take very long, it’s completely feasible, and then let’s just launch it.’ We’ve seen the error of that way of thinking. It always takes longer. Your first design is rarely the right design execution and you don’t have any basis on which to forecast your impact if you haven’t even done an experiment. We’ve lived through the fallacy of that thinking and I like to learn from our mistakes so we don’t do that anymore.
Your first design is rarely the right design execution. – Tweet This
Teresa: That’s great. I know CareerBuilder has a really large sales force, I know there’s other executives at the company besides you.
Hope: Of course.
Teresa: What happens when an executive has an idea, or a sales rep has an idea that comes from a customer? Do they go through this process as well?
Hope: Not consistently. We have a lot of different approaches. We have a cultural belief that ideas should and can come from anywhere and we really want to foster the ideas. Where I try to draw the boundaries, is we’re not just going to build it because one person strongly believes in this idea. What we try to do again is go through that process of validation. If it’s say a new product that we believe that there’s a market opportunity around that, what we’ve done—and again this is textbook Cagan—we want to have a set of referenceable customers that we have as a cohort of beta customers.
What we’ll do is, if we can figure out a way to have a very lean first execution of that idea. This is again, assuming we’ve gone through all this qualitative and quantitative research. If we really believe that there is a prospective group of target customers for it, then we will have them be part of a beta group and we will figure out if they’re going to accept this solution at the end of the trial. If they don’t, then clearly we haven’t solved the problem in the way that we think we have and we learn throughout that process. We’ve done that with a couple of products recently. One of which we chose to continue releasing, one we had good traction with the customers, but the market size wasn’t big enough and the investment was too large for us to continue.
Teresa: Oh interesting. Do you mind if we dive into that a little bit?
Hope: Yeah, that’s fine.
Teresa: One of the things I’ve run into, particularly with sales requests, because it’s really easy to think, ‘My customer wants it, we should build it.’ There’s a natural tension between how a sales rep is focused on closing this customer, whereas the product manager should be focused on meeting the needs of an entire market. I can especially see how discovery might pull out those differences. You have a customer who wants it, but it doesn’t meet the needs of the market. How are you navigating that? Is your whole organization on board with this product methodology?
Hope: I would like to say yes, but of course, no. Again, people fall in love with their ideas. It’s tricky. I would say this is why there are levels of management and leadership in our organization to navigate these discussions. I try to protect my team under an umbrella of let me or let other managers help navigate these discussions so that you can stay focused on what we know is important and already validated and getting to release that value. I would say I don’t have the secret magic answer. It’s a lot of negotiation and discussion and how do we come up with an objective way to validate and then invalidate that idea with a group of customers or with a prototype. Again, I would just want to make an objective decision and I think at their heart most other executives and leaders want to make a good decision, I don’t think they want to send us on a collision course to unhappy customers or missed expectations. Trying to orient everybody around how do we get to an objective decision is how I try to navigate that.
Teresa: I think that’s great. I think I would define product management as how do you get an organization to make objective product decisions. Right? I think a lot of people define it as they make the decisions, but really that’s never going to be the case.
Hope: Right, I want lots of people making good decisions.
Strong product orgs have many teams making good product decisions – Tweet This
Teresa: That’s great. This is a topic that comes up a lot. I get asked about on my blog on a regular basis. Tell me, does your team build a product roadmap?
Hope: No.
Teresa: Not at all?
Hope: In the traditional sense of there’s a formal document that’s going to have release dates and features I would say no, we do not. What we do is really try to constantly communicate. We do it in many ways and I would say most people would consider them insufficient, because there is no perfect way to have shared understanding across an entire organization about something that is constantly changing and moving. I just feel like it’s a bit of an impossible ask. Every quarter we invite all the leaders and we make it very transparent. Lots of people in the company can participate where each product team is presenting what is going to happen that quarter. The ones who do it the best also anchor that towards a longer term vision of what problems they’re trying to solve, what they see as the market opportunity, who are their target customers, what are the personas that they’ve already identified and validated, and then link all the releases to the problems or opportunities that they see related to that product.
The timing will change. The order of what’s released will change. Something new may come in and replace something. But keeping people constantly reminded that these are the problems that we’re trying to solve, this is the market opportunity that we’re pursuing, and then the details. We certainly report out on them and we try to keep people updated outside of those meetings through blog posts and meetings and emails and many other forms of communication.
Teresa: Okay, I want to go back to something you said earlier. You have a large sales team.
Hope: Yes.
Teresa: You have enterprise products.
Hope: Yes.
Teresa: … For your employer customers. I can’t tell you how many times people tell me I have to have a road map, my customers demand it, my CEO demands it, the rest of the business requires it. That only works for consumer companies.
Hope: Yeah, right.
Teresa: How is it working in your enterprise company?
Hope: We learn from decisions that we’ve made and things that we’ve tried and maybe not done as well as we’d like to. In the past we used to feel like, “Oh we should just be releasing. ” We wanted to release so that we could learn, but it wreaks havoc on customers, especially customers who are paying you to use the service. What we have instead tried to do is release, but not deploy. Have a more gentle deployment approach to our customers where we might release to a test group or have a beta test group for certain products—and we’ve done that successfully. Or we have the opportunity to test and measure that has the intended impact and then we can invite people to upgrade to that new product.
Teresa: Your enterprise teams are adopting all of the same discovery practices as your consumer team—you’re just gating the deployment.
Hope: We gate the deployment and because the product management and technology and UX teams are filled with humans, there are different rates at which they’re understanding and adopting and figuring out ways they can do that discovery so they’re releasing the most important impactful items.
Teresa: Tell me a little bit about the biggest challenges your product managers are facing as they try to adopt these discovery practices.
Hope: It depends on the teams. I think some teams, especially the ones that have a higher volume of internal stakeholders—so typically our more B2B products—are constantly balancing the questions of: Whose voice should I be listening to? Is it the end user? Is it staffing? Is it employers? Is it small businesses, medium size businesses, large businesses? Really being clear about who they’re developing for can sometimes be a challenge for them when there are so many customers that they’re trying to serve.
Then there’s the other internal stakeholder, the sales group, sales leaders that frequently request features based on an ability to get the sale. I want to remove the pain point—a feature that a competitor product has—I want to be able to tick that box for my customer to help them make the sale. And try to balance that with the fact that there are also a set of existing customers that we want to make sure they have a successful experience and renew and are getting the intended value out of the product. How do you balance the acquiring new customer needs, the existing customer needs, make it easy for people to sell,make it easy for people to renew? I think it’s difficult for them to balance a lot of voices in addition to their own ideas or the team’s ideas about the things they should be investing in for the evolution of their products.
Teresa: Yeah, it sounds like you’ve managed to get your teams to focus on what they can do rather than what they can’t do. Especially at an enterprise environment it’s really easy to get caught up in the worry that we’re disrupting our customers, it’s too much change. It sounds like your team is really looking at, okay yes we have all of these constraints, what can we still do? I’m sure that varies product manager to product manager—what are you doing to reinforce that across a team?
Hope: One of the things that I try to do is to have the shared understanding of examples that people have done. I feel like it is easy to make excuses or to be biased in terms of the ideas that you’re validating because they’re your ideas and you want to see them come to fruition. I think the more examples that we have internally of people who are doing really great work and are advancing their product and seeing the results, the collective happiness of both users and internal stakeholders and external customers is a fuel for more investment and people continuing to make good product decisions.
The collective happiness of users and stakeholders is fuel for making good product decisions. – Tweet This
One of the things that I’m trying to do is have people share their stories internally: What was their idea, how did they approach, it and then what were the results? Share that out to the rest of the organization and to the rest of the product teams. We have an internal product blog so when I hear about different people’s experiences I’m constantly encouraging them to write a blog post about it and share it with the rest of the organization because people have the opportunity to learn from that. As people start to see more and more of that and activities really picked up there, I think it fuels people’s desire to be really great at what they’re doing and to do more of them.
[Positive examples] fuel people’s desire to be really great at what they’re doing and to do more. – Tweet This
Teresa: Yeah, that’s great. Another big Marty Cagan tenet is this idea of cross-functional teams, and getting cross-functional teams together. I know you recently changed the way your teams work.
Hope: Yeah, on our main consumer website we did change the way our teams work.
Teresa: Tell me about that. What did you change?
Hope: At the beginning of Q2, we had actually just come out of a couple of quarters where the majority focus had been on a replatforming effort. There was a good pent-up desire to release some value to users. We had been spending a lot of time working through the idea backlog and we said let’s whittle it down to ten things. Ten things that are important and impactful for users, for our customers, for the company.
Teresa: Across the whole organization.
Hope: Just for the consumer website team. Just for one product team. What do we need to do to release those items by the end of the quarter? It was really saying we’re going to be very focused, we’ve already been through a rigorous prioritization process. On April 1st we went through a day where we had basically all the teams look at those items and I really wanted there to be consensus that these are the most important, impactful items. I actually had the entire group. The developers, the UX, even our scrum masters and the product managers and analysts, they all looked at these ten items and they collectively sorted them to what would be the most impactful, important items. If we were going to start at the top and work our way down, we were really focused on the items that the teams wanted to deliver the most.
Teresa: How did they do that? That’s a lot of people to be involved in ranking ideas.
Hope: Yeah, part of it was we had really crisp user stories for each one of them. Part of the criteria was they had to be delivery ready. So that means that the validation had been done, we had to have done experiments to understand what the impact was going to be. It assumed that we had done the qualitative research, and we needed to make sure that they were delivery ready, ready to go.
The reason that we had the whole teams involved is because I also am sensitive to a dynamic where the product manager is CEO, you will do what I say. These are the right ideas, don’t question me. That is not the environment that I want to cultivate. I really want there to be this shared accountability that we are working on the most important, impactful items. With a big group we basically had teams, we did this fist of five exercise, but essentially what it means is we had all the items up and people were sorting them back and forth like a card sorting exercise and then ultimately the entire team voted whether they agreed or disagreed with the sorting, and they could advocate for items to be moved up or down the line.
I’m sensitive to the dynamic where the product manager is CEO … that’s not the environment that I want to cultivate. – Tweet This
It took 30 minutes. Eventually we got to a stack rank that everybody could live with. It was great, because the next part of the expectation was we’re going to demo our progress every week so that if we’re running into obstacles, if it wasn’t as delivery ready as we thought, maybe we thought we had the right design execution, but it’s not performing as we thought. All of that would be transparent to the entire organization.
Teresa: So you started at the beginning of the quarter with a force ranked list—
Hope: Of ten ideas.
Teresa: Of the top ten ideas for the quarter. Then as you developed you did demos every week.
Hope: Every week.
Teresa: Didn’t you also change how you assigned people to teams?
Hope: We let them self organize. That was also part of this what I consider to be true collective accountability. If there’s an idea or a user story that’s expected to have impact, but no one really wants to work on it, or they don’t want to work with that product manager, or they don’t think the idea’s that impactful, I want to flush that out as fast as possible. So that’s why I wanted teams to self organize. I wanted to witness the interplay of what ideas people wanted to work on, what ideas they thought were impactful and watch that debate play out.
If there’s an idea that’s expected to have impact, but no one wants to work on it, I want to flush that out. – Tweet This
Largely, people hold each other accountable to really what’s a validated impactful idea and what isn’t. If they organize or choose not to organize around a particular idea, clearly there is some doubt. Let’s not kid ourselves that it’s going to be worked on one way or another. If people don’t want to work on it or don’t think it’s impactful, they’re not going to work on it. So let’s just flush that out as early as possible.
Teresa: How many engineers were a part of this process?
Hope: Typically, we have two to three engineers per team. We had four concurrent squads working on each one of those ideas. We probably had about 20 total engineers.
Teresa: Okay and did all ten ideas get resourced?
Hope: Eventually. In fact, we just released the last item last Thursday or Friday, so almost at the end of the quarter. So eventually they did. Midway through the quarter most of the teams on the first four squads were completed, and it was released to 100%. It might have been released to 10, 20, or 50%, but once they were released to 100% then people would come off of those squads and then we could self organize again for the remaining items. So that’s how it played out. It wasn’t without bumps in the road, but it largely worked.
Teresa: Tell me a little about how that helped. Why was that better than what you were doing before? And tell me a little about those bumps in the road.
Hope: What we had done prior is there would be the cross-functional product team and they would be working through their backlog and releasing, but there wasn’t this shared understanding across other teams about what was being released, about the expected impact, there just wasn’t enough transparency. As much as people can and should communicate to one another, people get mired in the day to day of their own jobs and so they just didn’t share that information.
We did the weekly demos every Monday, and now we’re going to do it every Tuesday. This sort of event that happened was a big change because people were held accountable to delivering results. People were held accountable to releasing value. People were held accountable to measuring the impact that they made. It was really fun. People wanted to see what else was going on, what value was being created, how well the teams were doing, and one of the best things about it was when people were struggling—because sometimes that happened—they were coaching one another. There was really a collective desire for us all to succeed as a team and have a meaningful result. Have a new and improved site that delivered more value, was more competitive, that delivered on users’ expectations better—and people wanted to get there quickly. The demos were about the most important, impactful change that we made.
Teresa: That’s great and I think again, especially for people that work at smaller companies some of this may sound like that’s just common sense.
Hope: Right.
Teresa: I think at large companies—I’ve seen it over and over again—you get teams that are off in their corners, no one really quite knows what’s happening over there, they report out every once in awhile, but it sounds like what you did was you brought your teams together and almost working like a startup would of these are the ten most important things we can do, how do we just get them done, rather than it’s your team versus my team.
Hope: Right. I think it was something we reflected on throughout the quarter. What do we like about what changed? What would we want to change? One of the bumps is that people felt like if people are moving in and out of working on different projects maybe they won’t put the quality into how they write the code, then somebody else has to pick up that code that they didn’t really write in a way that helps it be easily consumed by another team, then there’s the knowledge transfer. Definitely it’s not without its imperfections, but by and large people were excited to be releasing value, watching the needles move on the KPIs as it was being released.
Also, recognizing that there are some things that we released or tried to release and the teams would often be so excited that they wanted to quickly get to 100% and sometimes they would actually forget to look at whether the metrics were actually improving. So sometimes they would overvalue the qualitative research and then when we’re asking how it moved a particular KPI, usually a conversion KPI, they’d say, “I didn’t have a chance to look at the data.” Or whatever. It just wasn’t habitual. No, the expectation is not just that we release the UI—it is that we release the UI and it had the intended impact. That all sort of came out in the wash when we would do these weekly demos.
Teresa: People fall in love with their ideas in delivery as well.
Hope: Yeah, of course.
Teresa: During your nine years at CareerBuilder, you’ve seen a lot of change, you’ve been a big advocate for product management and definitely keeping it current. What would you say are the challenges ahead for product management and CareerBuilder?
Hope: I think they’re more of what we’ve experienced. It’s human nature to be biased, it’s human nature to fall in love with ideas. For me, the thing that will make us more successful as a company is when we get a lot of great ideas, but can quickly synthesize them and figure out the ones that are the most important and impactful. If we have a culture where we are willing to objectively look at the situation and the needs and make investment decisions that will actually give us the best return on that investment.
I just think that is the reality of what we’re facing every day. It manifests in different products, in different ideas, different people, but every day it’s what are we choosing to work on and where are we going to invest our precious time and people resources. That’s what I spend my time thinking about.
Teresa: It’s a never-ending problem, right?
Hope: It is, it doesn’t change. That problem doesn’t change.
Teresa: Yeah, excellent. Well thank you Hope.
Hope: Thank you Teresa—this was fun.
Thanks Teresa! Welcome other product leaders leave comments on how they’ve handled some of these challenges as well.