Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery. Hello Product Talk readers, we’re excited to share the latest Product in Practice with you! For this story, we caught up with Sonja Martin, Product Manager at tails.com. Sonja shares how her cross-functional team has changed their approach to defining their desired outcome over time.
Want to check out the other posts in this series? You can find them here.
A clear, desired outcome is the foundation for good discovery.
But how do you go about defining your outcome?
Hint: It’s simple, but rarely easy. And your work is never really done.
As Hope Gurion discussed in a recent Product Talk post, product teams are often assigned business outcomes rather than product outcomes. This tends to happen because business leaders think in terms of their own priorities, like growing revenue or market share.
The problem? Business outcomes are lagging indicators and often influenced by many factors that have nothing to do with the product team.
Business outcomes are lagging indicators and often influenced by many factors that have nothing to do with the product team. – Tweet This
Product manager Sonja Martin at tails.com experienced this firsthand.
Her team recognized that retention was critical, especially at the 90-day mark. But when they tried to use retention as their outcome, they discovered it was a little too broad to guide their day-to-day work.
Here’s the story of how Sonja and her team gradually refined and iterated on their outcome until they came up with a metric that was meaningful and actionable.
Have your own Product in Practice story you’d like to share? Submit your story here.
Sonja Martin
What does tails.com do and who are your main customers?
At tails.com, we create tailor-made dog food to help dogs be happier and healthier. Customers answer a series of questions online about their dog—including their weight, age, activity levels, any health issues, and so on—and we create a blend specifically for them out of over a million different combinations, which we then deliver to their door. It’s a monthly subscription service, and our blend will change over time as the dog ages, responding to their specific needs. We launched in the UK in 2014 and we’re now in nine different countries across Europe.
Tell us about yourself and your role. How would you describe your team’s overall purpose and goals?
I’m a product manager and I work within a cross-functional team focused on improving our retention. Our current focus is on our early-journey retention as that’s where we see the most churn. Our aim is to create an onboarding experience for our customers that sets them up well both through their trial period and for life.
We have a wide cross-functional team, comprising a product manager, product designer, an engineering squad, QA, data analysis and commercial finance, user research, and loyalty. Within that team, we have a smaller ‘core’ of just four of us: Product Manager (me), Squad Lead (Valerio Sarasini), Head of Data (Alan Cruickshank), and Product Designer (Maureen O’Connor Eibeler).
A team meeting with (clockwise from top left) Valerio Sarasini, Andrew Healey, Stefan Nowak, and Sonja. QA Samir Thomas is an essential member of the squad but not pictured here.
How does your team go about defining your desired outcome?
Across tails.com, we set all our goals on a quarterly basis, with each team proposing their goals and working with their stakeholders and the leadership team to ensure they’ve got the right focus. We use OKRs, and they’re shared centrally and reported on weekly, meaning the whole company is pretty familiar with the process.
While the process of setting and reporting upon our goals is fairly formalized, it’s more fluid within the team itself, where we’re continually talking about our progress against our goals. The aim is for this to be an ongoing conversation. What we’re learning each week informs the direction of our work and the way we measure it.
What we’re learning each week informs the direction of our work and the way we measure it. – Tweet This
At tails.com, cross-functional teams like ours have a “sponsor.” This is a member of our leadership team whose role is to challenge and support the team to make sure our strategies are aligned with our overall business goals and we have the resources we need.
We have a weekly core team check-in and a fortnightly touchpoint with our sponsor and other partners in our loyalty and customer teams. We use these to assess our progress against goals, as well as share challenges, insights, blockers, and ideas, creating a continual conversation that’s punctuated by our quarterly goal-setting.
How has your approach to defining your desired outcome changed over time?
Right from the get-go, we were really clear on the overall business outcome we wanted to drive: an increase in the proportion of signups that were still with us 90 days later (we call it 90-day retention). This is a great North Star goal that gives us autonomy to find the best solution, but it’s not a particularly useful metric for driving our day-to-day work as it’s a) extremely lagging and b) affected by multiple other factors, such as the channel that signups come from.
We’ve played around with a few different ways of measuring our impact on a more immediate scale than our 90-day retention North Star, and learned a lot along the way!
We knew right away that 90 days was too long—we wouldn’t be able to wait an entire quarter before concluding the results of any tests we launched. So we kicked off by using 30-day retention. Our plan was to watch over the course of the year if increases in 30-day retention could then be mapped to 90-day retention.
We knew right away that 90 days was too long for a product metric—we wouldn’t be able to wait an entire quarter before concluding the results of any tests we launched. – Tweet This
Our assumption was that if we could improve retention early on, that relative improvement would continue throughout the full lifecycle.
What happened when you used 30-day retention as your metric?
We were focusing on a couple of themes that came out of our discovery: We were sending customers their second order before they were ready for it, and customers were often unaware of the need to switch from old to new food slowly, over several meals. Most dogs take time to adjust to new food. This is a process we call ‘transition.’
We ran a few tests relating to these two discovery areas using the 30-day timeframe. This meant a test would run, we’d wait for that cohort of signups to have been with us for 30 days, and then run our next iteration after that. At this cadence, the rate at which we’d get through live A/B tests was achingly slow. The results of those two tests convinced us that it was worth doubling down on transition as a theme, but ultimately it took us longer than it needed to make that call.
Once you realized it was taking you too long to gather results, what did you do?
We wanted to speed up, and thought we could by shortening our timeframe to five days. But we soon realized that a result at five days isn’t reliable enough. A test can show an improvement at five-day retention that then disappears if we continue to watch that same metric over time (or vice versa).
Another decision we were making at the same time was to reduce the noise in our data by focusing solely on our most stable signup channel.
Can you walk us through that decision?
Whenever looking at retention, we always filter through different signup channels. There’s a big difference in how they perform, so we can get quite a lot of fluctuation in the overall retention number if we’ve had better- or worse- performing channels over-index in a given month. Our historical data shows us that customers who sign up through affiliate links or Facebook tend to fluctuate more than people who directly searched for our website. We’re already fairly used to talking through those differences.
We were still rolling our changes out across the board (i.e. customers from all signup channels would experience them), but just using one as the deciding metric. At the time, it felt like an easy way to insulate ourselves from some of the noise inherent in our overall retention. And then COVID-19 changed everything.
I bet it did! How did COVID-19 change the way you were thinking about your outcome?
We were starting to use the five-day metric and focus more of our weekly reporting around it just as the UK and Europe went into lockdown and people were less able to go to the shops. We do home delivery, so all of a sudden one of our key benefits became more important to people. We had both a significant step-up in the number of signups and saw customers ordering on a slightly higher frequency.
Our normal signup offer was that customers would receive a two-week free trial of their dog’s unique recipe. When the lockdown went into effect, we decided to move to a different signup offer of 50% off one month in order to ensure that we could meet demand (i.e. to prevent opportunistic stockpiling).
We essentially had two factors changing the way customers acted at this time: lockdown and our promo.
In terms of the impact this had on our metrics: within a couple of weeks of starting to focus on the five-day mark, deactivations by this day dropped to less than half the rate they had been at.
At the time, we were unsure whether this behavior would continue or not. Since then, lockdown has eased, but our default promo has stayed at 50% off one month, and the deactivations have increased, but are still less than their pre-COVID levels. It’s now stabilized—we’re seeing similar figures week on week—so we believe we have our new baseline.
Ultimately, we recognized that measuring retention on any timescale and with any specific channel is always going to be noisy. It was time to find a new metric that maps more neatly to our span of control.
Measuring retention on any timescale and with any specific channel is always going to be noisy. That’s why finding a metric within our span of control is critical. – Tweet This
How did you decide what that metric would be?
This was when we turned to the two key problem areas we’ve identified during our discovery work.
Through regular interviews with customers and survey data, we’ve developed a strong understanding of the opportunities within our early journey experience. These are:
- Customers feel more reassured they’ve made the right choice for their dog because they understand the benefits of tails.com for their dog. We tailor each and every blend to the specific dog, yet this wasn’t resonating with customers and they didn’t always understand what this actually meant.
- More dogs love the food. We know that when a dog doesn’t like the food, the customer churns. But it’s normal for a dog to need time to adjust to a new food. And if they really don’t like the food, we can help with a different recipe. We wanted to get in front of this and see if we could save some of these customers with better education. We realized we weren’t doing enough to help customers in this situation, whether by reassuring them that it can take a while, or by helping them get a new recipe.
And just to provide a bit more context for the non dog owners out there: It’s really important for dog owners to feel that their dog eats and enjoys the food. Labradors, for example, famously eat anything and everything, but there are plenty of owners who struggle to get their dogs to eat.
There’s often a window of time in which owners are watching their dog to see how they react, to see how their poop changes (TMI perhaps, but talking about poop is a mainstay of our jobs at tails.com!), in order to work out whether it’s the right food. Rather than waiting for a customer to get to the end of this period and decide the food isn’t right, we want to support them through this change, helping them introduce the food in the best way, and if the dog doesn’t like the food, helping them with a different recipe.
Rather than waiting for a customer to get to the end of the trial period and decide our product isn’t right, we want to support them through this change. – Tweet This
Having a design sprint focused on dog poop is all in a day’s work at tails.com.
Now that we’re clear on these opportunity areas, we can create metrics that map neatly to them.
For our first theme, we need to turn the slippery concept of customer perception into something quantifiable. Our Senior User Researcher Andrea Burris has been formulating a ‘quality of experience’ metric, where we ask our customers to agree or disagree with a series of statements about tails.com, such as “tails.com is made with the right ingredients and nutrition for my dog.” We then take the percentage of people who strongly agree with these statements over time, and try to push that number upwards.
A screenshot of some of the questions tails.com is asking customers to measure the quality of experience metric.
This is still in early stages. Our next step is to prove that an increase in it will map back up to retention. We are planning to run A/B tests that still reference retention. We’ll be looking for an increase in both our quality metric and retention within a given cohort to give us the confidence that these two are linked.
For our second theme, we’ve been experimenting with creating an interactive experience for customers to tell us early on whether their dog is enjoying the food or not. Rather than waiting for them to cancel before we try to help them, we want to support them early on. This helps us surface tips and solutions to the customers who need them, and potentially provides us with a metric for understanding how many dogs love (and don’t love) the food. Our early tests have shown really encouraging levels of engagement from our customers. Our next step is to surface the most useful solutions to them so we see more dogs enjoying their dinners!
The message new customers see when they log into their tails.com account.
What have you learned from this process? What advice do you have for other teams who’d like to take a similar approach?
Our experimentation with different metrics has happened at the same time as we’ve embraced continuous discovery to a much greater degree than before. We now talk to customers almost every single week, our engineers are deeply involved in discovery, we’re speeding up our learning loops, and we have a much deeper understanding of our customers’ needs.
Experimenting with different metrics and embracing continuous discovery mean we can build our confidence in placing bigger bets. – Tweet This
Ultimately, this means we can build our confidence in placing bigger bets. It’s definitely not plain sailing all the time just yet (and probably won’t ever be!) but we’ve learned a lot along the way, including:
- Bringing engineers right into the discovery process has been really rewarding. They speak in terms of our customer needs (and write blog posts about them). They feel more empowered and more able to write code that will solve the right thing. Sharing the problem space across the team empowers us all to create better solutions for our customers.
- Humility and patience! It takes time to learn what might move a metric when you’re entering a new problem space and doing so with a new team. It felt stop-starty to begin with. We’re getting into our stride now to a much greater degree because we’ve had some continuity and are getting to grips with the problem area and what’s worked well and what hasn’t.
It takes time to learn what might move a metric when you’re entering a new problem space and doing so with a new team. – Tweet This
Sonja’s story demonstrates the iterative nature of defining your product team’s outcomes. You’ll always be gathering new information about your customers’ behavior and the levers you can tweak to have the biggest impact. And, of course, there may be unpredictable external factors at play. But when you take a continuous improvement mindset, it’s much easier to adapt to the changes and new information you gather and adjust your outcomes accordingly.
Do you have a story about how you and your team adopted continuous discovery best practices? Get in touch to let us know and you may be featured in a future post!
