Measuring The Impact Of Each User Story

Have you heard? My new book Continuous Discovery Habits is now available. Get the product trio's guide to a structured and sustainable approach to continuous discovery.

On Thursday, I wrote about how to use themes to stay focused on your product goals. But using themes is not enough. You also need to identify how you are going to predict and measure the impact of each user story.

Let’s return to our previous example and suppose you are building an event site like eVite or Facebook Events with the following goal:

• Increase # of Events

Now for each user story, you should be able to measure how the story impacts the goal. Remember, we looked at the following user stories:

• As an event host, I am able to import my address book so that it is easier for me to invite my contacts to my events.
• As an event host, I am able to look up a venue address, so that I am sure to get the location correct.

Let’s look at how you might measure the impact of each of these stories. For most user stories, you are going to want to measure the effectiveness of the mechanics as well as the overall impact. Let’s take a look at the first story.

To measure the mechanics of the feature, you might want to measure:

• the number of people who start the import process
• the number of people who make it to each step in the import process
• the number of people who complete the import process

You want to measure the whole funnel. But this only tells you whether or not your visitor is able to complete the process you designed. It doesn’t tell you whether or not the functionality is having an impact.

This feature is based on the assumption that people who import contacts will be more likely to create events. You also need to measure whether or not this is true. This tells you the true impact of the story.

So you also want to measure something like:

• the number of events that are started
• the number of events that are completed

You want to compare this data between the two sets: people who have imported contacts vs. people who have not imported contacts. If and only if the people with imported contacts outperform people who have not imported contacts (in other words, they create more events), do you want to focus on improving or keeping this feature around. You are looking to verify that this new feature is actually having an impact on your goals, not just that people are successfully using it.

For the second user story, you want to do something similar. First, you want to measure: the funnel associated with this feature:

• number of people who start the venue lookup process
• number of people who make it to each step along the way
• number of people who complete the process.

But you also want to measure the impact. You want to know, are people more likely to finish the event creation process if they use the venue lookup functionality? So again, you want to compare the results of people who use the feature against people who don’t to truly understand whether or not this story is impacting our overall product goals.

For each story, you want to make sure that you are measuring both 1) the ability of a visitor to use the feature as it was intended and 2) the impact of that feature on your product goals. The former is easier to measure and can tell you where people are running into problems, and while the second one might be harder to define, it really measures the why behind the functionality in the first place.

On Thursday, we’ll look at how to predict the impact of a story before implementing it. Do you have other tips for measuring impact of a user story? Please share in the comments.