Helping other people use metrics frameworks

Somehow the past week I was thrown into a last minute planning group for a design workshop. For those who haven't been to a design workshop before, while formats and styles vary, the general gist is that teams get together, they're presented the problems they're trying to solve, theres some presentations of relevant research and other information, then they engage in various design exercises meant to help come up with creative ideas before working on a focused design.

While there's a ton of work involved in getting one of these workshops organized, probably the part that is UX researchers put the most energy into is in preparing the presentation of relevant research and information. My parts obviously involved talking about relevant data related to the thing we're working on. But another thing that I had to present was on metrics frameworks since any discussion about what work is "impactful" involves talking about different ways to measure and compare success. While data scientists are used to working with tons of metrics and keeping them straight in our heads, the same can't be expected of everyone else. Applying them takes a certain amount of preparation and forethought, and that's what I wanted to write about.

As a super quick example as to what I'm talking about, I usually like pointing to "Pirate Metrics" which is a common framework used out at various businesses and startups and is popular enough to quickly pull up in a web search. The framework recommends that businesses look at "AARRR" which largely consists of:

• Acquisition (or awareness) – How to people learn about the product?
• Activation – How many people we convince to take action (aka, purchase)
• Retention – How many customers come back?
• Referral – Do users recommend the product to others
• Revenue – How much money are we making from all this?

The idea is that businesses can set metrics and goals against these five major categories, and they'll cover the most important things that a business needs to monitor to be successful. The specifics of what to measure are left as an exercise to the reader.

Overall, my opinion is that a framework like this one, or the myriad other ones that are used and popularized in the popular business literature are useful to have and lean on. They're awesome mnemonics and reminders to keep an eye on important aspects of the business that might not immediately be relevant in the present. Even I'm likely to forget about measuring for one aspect of the business when I'm deep in the weeds that focus on another aspect.

But as well all good things, I've occasionally seen teams going down weird paths when trying to use them.

People can confuse frameworks for checklists

Over the years, I've noticed that people can become confused as to what a framework is for when they're busy. Overall, a framework is scaffolding, it's a reminder of things you should be paying attention to while staying very high level so they're able to be applied across a broad array of business models and companies without change.

The problem is when teams that are pressured to "be data driven" from above and are told that they must use some executive's preferred framework instead of whatever it is they're familiar with. Teams can sometimes treat the framework more like a checklist to tick off. "Yes, boss, we did the pirate metrics. See how we have a metric associated with each item?" In highly volatile situations, overworked teams will grab any metric they have handy and shove it into the framework just to check the box and get on with more important work.

Obviously, no complex business process can easily be summarized by a single number. Even if the team has to pick one metric out of many to put things on the dashboard, there's a whole complicated web of metrics and measurements that should be underpinning that declaration. That's research work that an overburdened team would be unlikely to have time to take on.

So if a team declares that they have a revenue metric by... declaring that their top line revenue is their metric... we need to put quite a bit of energy into helping them work out that there is a huge difference between having the number and understanding how the number works – you can't exactly just shout "Make revenue go up!" and have it happen without understanding anything else.

People forget frameworks are a viewpoint

We're in the business of data, so we know that every metric that is measured is a choice. Metrics are a reflection of what we choose to be important, as well as how we want to view and organize the world around us. We can measure success for our company in a myriad number of ways that all can be valid so long as we all agree they align with what we care about. So, obviously, metrics frameworks that are nothing more than a prescriptive list of metrics suffer from the same issue – frameworks are one potential viewpoint out of many.

But for people who aren't thinking about data all the time, this insight might not be immediately obvious. Imagine if you're in the business of selling one-way rocket trips to Venus. A traditional product metrics framework would likely be telling you to measure customer retention because that's usually very important for a business. Except you're in the business of selling one-way trips to a guaranteed lethal murderscape of a planet whose surface temperature can melt lead, by definition there shouldn't be repeat customers unless you do some very creative definition manipulation.

The hint to people should be that there exist multiple metrics frameworks that people can choose to adopt. Each have different opinions about what is important to a business. The overall contours can be similar, like for example getting customers, making sales, and keeping repeat customers are generally considered "good things to monitor" in a business, but they can disagree on what's important.

In the same way, we have to remind all our stakeholders that just because we've picked a framework doesn't mean we should be abandoning the opinions about what we think are important. Just because something isn't on a framework doesn't necessarily mean it needs to be cut out, nor does it mean we have to adopt a metric that doesn't make too much sense in our specific context.

Stuff rarely affects one metric at a time

This is more a complaint about metrics in general instead of frameworks, but it pops up quite a bit. People are focused on moving the needle for one specific metric and they can forget that metrics and the decisions that cause them to move rarely affect a single metric alone.

As an example, offering a free trial is often a great way to acquire more new users, but the result is that those new users are often much less likely to become long term repeat customers who willingly paid the full price the first time around. So the acquisition and retention metrics can move in opposite directions. Discussion about frameworks or metrics rarely discuss these tensions and conflicts, and so teams forget to look at their products in a more holistic sense. I think it's part of our jobs as the people guiding them to use metrics to also be aware of these tradeoffs before hand.

Despite the issues, metrics frameworks are great. Use whatever framework makes sense to you. Just make sure everyone else is following along.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!