Metrics trees and other mental frameworks

A couple of months ago, I don't exactly remember when, an unfamiliar term floated across my eyes and made me jot down a note in my drafts pile – "metrics tree". (For example, Ergest happened to mention them recently in a couple of posts.) At a glance it seemed at least related to what I work on all the time (typical metrics setting stuff), so I made a note to look into it later and here we are.

The heck is a metrics tree?

As best as I can tell, this term is relatively new and goes by a couple of names: metrics tree, drivers tree, KPI tree, etc.. They're not to be confused with a similarly named data structure called a "metric tree" that's used to index and store information in a metric space.

Using my Super Accurate research method of "see if any hits come up in a Google search restricted to a single year" the terms seemed to start popping up occasionally around 2013-ish – quite young in terms of terminology. But obviously, it didn't become popular enough because I'm sure many of you have no idea what it is either.

Whatever the term, the concept behind a metrics tree should actually be quite familiar to most practitioners. The basic idea is this: take a business and write the top most important metric(s) for the business – often this is revenue and you might hear it being called a "North Star metric". Draw a box, and put the KPI in it. Then, one level below the [revenue] box, you write down the one or more factors that causally feed into revenue, for example, goods purchased by customers. Go down another level, and split up goods purchased by customers into purchases by new customers, purchases by repeat customers. Next level down, you might include the steps required to convert a new person visiting the page into becoming a new customer. You might put in the different ways new visitors get to your store.

The "tree" part of metrics tree is that you're supposed to build this causal map of what the business is, what drives and feeds into what metric. If you do this thoroughly, you're supposed to get this sprawling map of the business, with either absolute volumes or conversion rates linking everything step by step in a causal chain. Then, with this map in hand, people should be able to use it very much like a map to find inefficiencies for improvement, and otherwise find actions to take in order to increase your primary metric of making money.

If something goes wrong and sales tank you can use the same map to help you find out what's broken by checking which factor "broke". Imagine if you had a continuously updating dashboard that has this whole map on the wall and a box will turn red if it's going off the rails and affecting the main KPI.

Or so goes the sales pitch for the idea. I think I've seen some people claim that creating, vetting out, building, and reporting against the metrics tree is the raison d'être for metrics teams. I think claim is utterly ridiculous, and thankfully the idea didn't catch on enough we have people blindly parroting that point.

A framework taken too far

So what I'm seeing with the metrics tree idea is that some people might be taking a useful framework and forgetting that it's supposed to be scaffolding to build upon and not some inviolable truth.

I think every one of us who works with metrics intuitively agree with the fundamental tenant of metrics trees. Metrics usually have causal explanations and drivers that in turn depend on other factors in a causally connected fashion. If you trace these causal links deeply enough, you will eventually find items that you can influence, whether it's intervening with a new process, or increasing marketing spend, or improving a conversion rate. With a lot of work, you can figure out which of these levers can be pulled to what degree of effectiveness.

A lot of a data analyst's job is actually exploring and identifying such causally linked connections. We run experiments and do analysis to find factors that drive sales all the time. When revenue drops suddenly, we're going through in our minds an internalized metrics tree of the business in order to guess at where the problem is before verifying it with analysis. All of this probably comes naturally without even having to slap a name to it.

But even though we use the concept in our day-to-day work, we also know that it is nigh impossible to create a single complete causal model that explains the majority of the variation of an entire business. It can sometimes work okay-ish for revenue numbers because accounting practices let you trace those metrics back quite deeply. But I personally find it much more difficult to draw a causal graph that explains something like "retention" or "engagement". There are always going to be competing alternative models that are just as plausible. There's going to be multiple factors that we don't think of, some obscured by dominating influences we aren't expecting. New factors can come into existence, while existing factors can become irrelevant thanks to the world constantly changing around us. To claim that there is a platonic ideal metric tree that truly represents how the business works and will lead to decision-making nirvana once we have find it is... setting ourselves up for disappointment if not unemployment.

I think people are willing to entertain the notion of metrics trees being something to work towards up to a point, recognizing that the ideal vision is likely unobtainable. And that's probably a healthy enough position to take about a work framework.

There's many mental frameworks out there

Out in the world, there are many different times of frameworks that have been developed to apply to all sorts of situations. Frameworks for analyzing a set of problems, like for example the HEART framework that came out of Google years ago that lists out a series of metrics categories – Happiness, Engagement, Adoption, Retention, Task success – that teams making products should remember to measure. There's also Pirate metrics, AARRR – Acquisition, Activation, Retention, Referral, Revenue – that offer a similar idea.

All these frameworks act as useful mnemonics and reminders to teams completing a project because it's honestly quite easy to forget to measure entire classes of items during the final rush to finalize a product launch. You're supposed to sit down and convert the vague guidelines into concrete measurements for your unique situation. Go ahead and pick what looks like a reasonable fit for whatever situation you're dealing with.

Just remember that relying on frameworks can be taken too far. If memory serves, the thing that would become "Agile Development" was at some point referred to as a framework too. Then a bunch of people hyped it up, took it way too seriously, bolted on all sorts of things, used it to overpromise, and now there's a certain amount of justified backlash against it. Business in general tends have ridiculous fads (hi, NPS) and we always have to be on guard against adopting nonsense.

The tipping point is probably this: frameworks are great to use as signposts and reminders that help make thinking about complex things easier. It's infinitely easier to walk a trail with markings than to wander in wilderness. Frameworks become dangerous and fall apart when they become used to replace thinking, Look at all the instances where people follow a methodology without understanding why and apply it poorly, like A/B tests, or agile, and so forth.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted, so support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!