Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Hello, Product Talk readers! It’s time for another installment of Product in Practice. In this series, we highlight the impressive work that forward-thinking product teams are doing. Want to check out the other people and companies we’ve featured? Find our other Product in Practice posts here.
Ask 100 product managers how they ended up in their roles and you’re unlikely to hear the same career trajectory repeated twice. This field draws people with diverse backgrounds and skill sets, and Lisa Orr is the perfect example of this.
Ask 100 product managers how they ended up in their roles and you’re unlikely to hear the same career trajectory repeated twice. This field draws people with diverse backgrounds and skill sets. – Tweet This
Lisa began her career in data science and spent four years as a data scientist at Airship, a marketing and messaging automation company. During her tenure as a data scientist, Lisa built two predictive products. Her exposure to the product team—and the departure of a previous product manager—sparked Lisa’s interest and led to her stepping into the product manager role right around the time that Teresa began working with the product team as a discovery coach.
Teresa was impressed by how Lisa took the big, hairy product idea of orchestration (more on that in a moment) and turned it into a tamable, iterative product strategy. Lisa is also making bold bets on a more innovative visual interface for her product and backing it up with good discovery—all in a company that’s just starting its continuous discovery journey.
We caught up with Lisa to learn more about what she’s accomplished in her time as product manager at Airship so far and her key takeaways for implementing continuous discovery.
Have your own Product in Practice story you’d like to share? Submit your story here.
Meet Lisa Orr, Senior Product Manager at Airship
Delivery Before Discovery: Lisa’s Early Experience with Product Management
During her tenure as a data scientist, Lisa was able to see how the product team approached building predictive products. Lisa says, “We led with ‘What can we predict?’ and spent the majority of the effort in creating the model. We then jammed those models into our platform so that customers could access the outputs in many ways.” While these products have been successful on the market, Lisa has learned that “For data science in particular, building the trust of the marketer into the product has to be a top priority. We’ve onboarded many customers onto our predictive suite and every time we get the same question: ‘How can I test that it works?’ Asking the customer to use a service that makes decisions for them means asking them to trust you. We realized we needed to build that trust into the product itself.”
For data science in particular, building the trust of the marketer into the product has to be a top priority. Asking the customer to use a service that makes decisions for them means asking them to trust you. – Tweet This
Lisa officially joined the product team just as they were adopting a continuous discovery approach. But from her earlier work with the product team as a data scientist, she has a sense of how the team used to operate. Lisa says, “Looking back, I can see how discovery would have helped us build a better product.”
Understanding Airship’s Products: An Overview of Orchestration
Airship engaged Teresa to coach Lisa’s product team, which is responsible for orchestration. Lisa leads discovery, helps set up design sprints, recruits customers for research, and keeps the ball moving forward on all the discovery tasks the trio needs to accomplish to go from opportunity to good solutions under the umbrella of orchestration. But what exactly is orchestration? Lisa explains that “Orchestration is a catch-all term that refers to orchestrating messages across different channels—sometimes referred to as cross-channel orchestration.” Within marketing, a “channel” is a medium through which to deliver messages such as push notifications or email.
To dig into this concept a bit more, imagine you’re a retailer and your marketing team sends promotional emails tied to things like holidays, purchases, and coupons. You also have a mobile app that sends similar messages via push notifications. And web notifications also pop up whenever a customer visits your website. You also occasionally send people physical coupons or discount codes via postal mail. How do marketers coordinate across these channels and ensure customers aren’t being overwhelmed by too many messages? How do they know which channel is best for each customer? How do they match the message to the channel? And how do they know that they got it right? These are the challenges involved in orchestration.
However, as Lisa’s team dug into orchestration with some of the continuous discovery practices mentioned above, they realized that the problem was broader than they had realized initially: “We believed going in that orchestrating messages across channels was the challenge we needed to solve. But when we dove into the topic, we realized that marketers struggle with creating user journeys in general, whether it’s for one channel or spans multiple channels.”
We believed going in that orchestrating messages across channels was the challenge we needed to solve. But we learned that marketers struggle with creating user journeys in general, regardless of the number of channels. – Tweet This
If we continue with the retail example mentioned earlier, a marketing user journey might be designed around getting someone to take a specific action, like using a coupon. In order to achieve this outcome, the marketing team starts by emailing the coupon. If the customer doesn’t use it, they’ll send a reminder email. If the customer walks into the store, the marketing team sends them a text to remind them about the coupon. And once the customer has used the coupon, the marketing team sends them an email receipt to congratulate them on the discount they got. This is just one example, and you can imagine that with a number of user journeys and communication channels, there’s a lot for marketers to manage.
Lisa continues, “The first breakthrough we had was including the dimension of time into orchestration: marketers are tasked with coordinating messages across channels over time.” And adding the time dimension brings other nuances like incorporating reactions from previous messages. For example, if a user doesn’t click on the first message, you can send a follow-up message with similar content on an alternative channel. Lisa explains, “We’ve found that conversations about cross-channel orchestration become richer if we include other concepts such as the goal of the messaging and keeping track of relevant context across channels and across time.”
Lisa’s team started the project thinking of orchestration as primarily a cross-channel need—how to help customers coordinate their messages across channels—but as they dug into the discovery, they found that the needs were far more complex. Marketers do need help managing messages across channels, but they also need help managing messages across time. They need help constructing messages that support a user journey and can respond appropriately when a user engages—or more importantly doesn’t engage—with messages in that journey.
When Teresa started working with the team at Airship, she led them through the “How to Make Toast” exercise to help clarify orchestration. Principal Engineer Todd Johnson walks the team through orchestration based on the results of the exercise.
Widening the Scope from Solutions to Opportunities
As Lisa began to focus more on discovery, she quickly learned some important lessons about her customers. Since a marketer’s average tenure in a role is just two years, Airship needs to be able to make—and demonstrate—immediate impact. Lisa says, “When a newfangled concept comes into play like predictive tools, it’s not good enough to just turn the system on. Marketers are looking to track the impact of the new solution on their KPIs, understand the types of decisions the predictive tool is making on their behalf, what the best way to use the new tool is, and—when possible—modify the predictive tool to fine-tune how and when the tool gets deployed.”
Widening the scope beyond just what a feature might accomplish, Lisa focused on what impact that feature might have on the customer’s role and business. Developing this deep understanding of her customers’ needs has also led to a few key decisions on the product team. First, they need to ensure that testing the tool is easy and can be used in a well-controlled environment. Next, they aim to create a straightforward way of tracking the impact of the tool on important key performance indicators (KPIs). Finally, they want customers to easily understand the predictive tool and how to get the most out of it. To put this in Product Talk terms, these are three key opportunities that Lisa uncovered.
Developing a deep understanding of customers’ needs has also led to a few key decisions on the product team and to uncovering key opportunities. – Tweet This
And while Lisa made these observations in the context of discovery for predictive tools, she says, “These three principles apply to our general non-predictive solutions as well since they are relevant to trying any new thing as a marketer.”
Making Changes: Adopting Continuous Discovery Best Practices
Part 1: The Design Sprint
Lisa soon realized that her team was turning to her for guidance on the orchestration roadmap. But after focusing intensely on the orchestration opportunity map, Lisa says, “I was without answers as to what we’d build next and what our general plan was for the next six months.”
The materials that would fuel Airship’s first-ever design sprint: masking tape, whiteboard markers, dot stickers, and plenty of Post-its!
In order to find these answers—and involve the team in the process—Lisa says she decided to run a full week design sprint “so we could turn our customer research into a cross-channel orchestration vision. This vision could then be broken down into parts that could be prioritized and fed into the roadmap. It also worked as a way to get leadership buy-in to where the team was heading overall.”
Running a design sprint leads to a vision that can be broken down into parts that can be prioritized and fed into the roadmap. It can also get leadership buy-in to where a team is heading overall.”– Tweet This
Lead Back-End Engineer Russell Mayhew, Senior Product Designer Nicole Mors, Lead Designer Brett Heckman, Lead Front-End Engineer Adam Coddington, and Lisa gathered for a storyboarding session in the middle of their sprint.
To prepare for the sprint, Lisa first gained support from stakeholders and managers since spending one week on non-core activities required leadership approval. Lisa also talked with Teresa about how to best blend continuous discovery principles into the sprint. Teresa recommended fitting the loop of discovery into the week. Day 1 is map creation and selecting an opportunity, Day 2 and 3 are brainstorming solutions and story mapping, Day 4 is assumption-finding and designing experiments to test assumptions, and Day 5 is executing the tests and evolving the solution from what you’ve learned.
Russell Mayhew, Lead Back-End Engineer shares his drawings based on the information gathered in the first two days of the sprint. Lisa says this activity was particularly challenging for the non-designers in the room.
“The biggest takeaway from Teresa’s input,” says Lisa, “was to focus on assumption testing as the goal for the Day 5 customer prototype tests. This helped us identify key concepts we needed to test out for the storyboard we came up with. The overall goal of the sprint was to coalesce much of the research we had done up to that point and come up with an overarching vision of where the orchestration product line was heading. With this sprint we could identify several major themes of orchestration, rank them in order of importance, and begin drawing up plans for executing on the first couple themes.”
The end of the sprint means a lot of used up Post-it notes and paper representing all of the work over the course of the week.
Part 2: Running an Intensive Beta for Airship Journeys
Airship Journeys is an interface for coordinating messaging across important lifecycle milestones. Milestones such as making a purchase, signing up for subscription services, or upgrading to VIP status are important lifecycle events that marketing teams can create campaigns around through Airship Journeys. The solution provides a visualization of the user journey with the messages being sent and the impact those messages have on conversions and engagement as well as the audience filters and channel selection applied to each message.
To prepare for the initial launch of Journeys, Lisa made the case to company leadership to run a one-month-long beta program before making the product generally available. During that month, she was able to onboard 18 customers and gather ongoing feedback from 10 customers. These customers participated in mini design sprints and helped evolve the Journeys reporting page.
In the course of her early interviews, Lisa learned that most marketers struggle with the existing journey tools already out there on the market. They are complex and require a lot of management by marketers. She even started asking to see the journeys that marketers were creating and she realized they were needlessly complicated. The existing tools were creating a ton of work for marketers and were error-prone.
Lisa and her team found a way to simplify the UI in a way that reduced the amount of work required by marketers. But they had to make sure that a marketer could understand it and that it truly was better, so they iteratively prototype tested it. They also ran into some internal pushback. Salespeople were uncomfortable with how different it was. They wanted what customers were asking for—the broken way everyone else was doing it. But Lisa got great feedback during her beta and that helped get the rest of the organization on board.
Sales wanted what customers were asking for—the broken way everyone else was doing it. But Lisa got great feedback during her beta and that helped get the rest of the organization on board. – Tweet This
During a second design sprint on optimization in journeys, Lisa’s team created “How might we” Post-its based on interviews with internal stakeholders.
Involving customers in beta testing helped the product team gather valuable insights to inform the product’s development, but it also meant that they could formally launch Journeys with two customer case studies, two customer speakers who agreed to share their experiences in the upcoming Airship Forum series (a tour of digital marketing leaders and Airship customers across the US and Europe), and official quotes that were key to the launch both in external press and bolstering internal enthusiasm. Lisa says, “The product marketing manager I worked with went so far as to say she’s never heard so much enthusiasm come out of a customer about a solution. Our beta customers are more than happy to volunteer all the ways Journeys is making their jobs easier and how it’s presenting new opportunities to solve previously unsolvable problems.”
Getting Everyone Bought in to Continuous Discovery
One of the major results of the beta Journeys launch, according to Lisa, was “the company’s growing belief in the discovery process.” The executive staff had assumed that a beta was simply to debug the product and de-risk the launch. “Now on the other side of it with customer quotes and case studies, the value of a beta speaks for itself,” she adds.
So what’s next for Lisa and Airship? As continuous discovery becomes more ingrained in Airship’s company culture, Lisa says she’d like to continue working towards smaller and quicker iterations of a product, especially when making a first step in a new direction. “Quick iteration means quick feedback and potentially capturing new customers to do discovery with,” says Lisa.
Quick iteration means quick feedback and potentially capturing new customers to do discovery with. – Tweet This
And since the value of speaking with customers is clear, Lisa is hoping to continue building out the research program and making customer research more accessible. She’s seen a lot of value in sharing recordings or having people drop in and listen in on a prototype test, so this is something she’d like to see Airship continue to prioritize.
Nicole Mors presenting the outcomes of the first design sprint at Airship’s weekly Friday happy hour.
Lisa’s story shows that when you adopt continuous discovery best practices, everyone wins. Your product team gathers valuable insights that inform the direction of your product and your company. Your customers become trusted resources (and maybe even superfans who can’t wait to tell the world about you). Your marketing team gains a greater understanding of your customers’ pain points and needs while gathering valuable customer stories and testimonials. And your executives see the value in your work and support your ongoing discovery activities.
Do you have a story about adopting continuous discovery at your company? Get in touch to let us know, and we may end up featuring your story in a future Product Talk post.
