Sometimes, stuff's not worth measuring

This newsletter is literally named after the act of measuring things, and yet it's worth telling people time to time – it's sometimes not worth the trouble measuring certain things.

In most industry organizations, people tend to look towards the people who work daily with data, the data scientists, researchers, analysts, marketing experts, for our expertise in measuring things within our respective domains. Since we're the ones who are supposed to be using the data to do things at the end, it's perfectly reasonable to at least seek our input for how to measure new things. After all, failure to do so can often lead to collecting unusable data and that is a painful enough mistake that even people who aren't experts in using data quickly learn the lesson.

Those folks who have lots of questions they want answered with data are all well meaning in thinking that many of the questions they have can be answered through quantitative methods if only they collected the right data to give to us to "work our magic on". Therefore, a large part of our job then is to reconcile their desire for insights with the harsh realities of actual measurement and inference. It's no surprise that the starting part of a lot of data science work is asking people a polite variation of the question "So what is it you're trying to understand?" Seeing past any direct request for a data point or a methodology and getting clarity on the actual research question of interest is a fundamental part doing data work. I dare say that developing this habit is one of the first signs that you're becoming a bit of a veteran in the field.

I think that most writing on the internet that discuss "getting the research question right so we can measure the correct thing in the correct way" all tend to take as a given that we know what is best for a situation using our magical powers of expertise. As someone who's been around long enough, it feels simultaneously obvious that this is true while also being really hard to articulate the actual process that goes on in my head. So how is anyone new to the field of data work going to learn this other than painful trial and error? What IS this expertise anyway?

It's always a balancing act

Questions about feasibility are always subtle questions about balancing conflicting concerns enough to get to a goal, whether it's for solving engineering problems or research methodology ones.

Experience is very useful in these situations because out of the near infinite combinations of different balance points in the decision space, experience is very helpful in telling us what combinations worked in a specific context and also what failed to work in other contexts. The extremes can be somewhat self evident, like how impractical it is to directly interview 10,000 people in a week instead of sending out a survey to the same 10,000 people. The hard part is these decisions become increasingly difficult to judge when they're not extreme – is interviewing 100 people too much? How about 25? Is sending a survey to 1000 people useful? How about 100?

But even when evaluating the extreme cases, the axes which you're making tradeoffs is important too. Seasoned researchers know that they're making a complex trade between the richness of data being collected, the overall cost to do the data collection, the time needed to complete the work, the extent and confidence that results could be stated, the availability of willing study subjects, the technical feasibility of the data collection, the reliability and repeatability of the data collection. That's not even a complete list, and on top of it all there's potentially ethical, social, legal, and political concerns related to collecting data and doing the research.

All of which is to say, the answer to whether we can possibly collect some data to answer a question is usually going to be "yes, but...". Yes, but it's going to cost you a lot. Yes, but it's not going to tell you anything you can use. Yes, but it's got no informational content and is a complete waste of everyone's time, especially mine. The only time the answer is "no" involves some kind of physical impossibility, like measuring the exact position and speed of an electron, or "what a user intended to do when clicking that button".

So what sorts of things make me say that it's not worth the trouble measuring and we should go back to the drawing board?

When there's no causal mechanism that wil move things we care about

This is usually the big one. Plenty of requests for arbitrary metrics can come from people, and while they might sound like useful bits of information to know, they aren't tied in any causal way to things we want to achieve.

Vanity metrics like "total users" that only go up and to the right fall into this broader category because they look useful, but aren't practical levers to anything involving growing or running the business. At most if these metrics flatline and stop moving, then something is already critically broken.

Measuring how much revenue is coming in today is somewhat useful to know if you're worried about it becoming zero suddenly. But measuring how many orders are coming in successfully, how many customers are visiting, are much more forward-looking and useful and gives you more control over the situation.

When the measurement can't be used for anything

If a measurement is made but never used to make a decision, does it even matter if we measured it? Can I measure how many people hover over a button on the page for more than 3 seconds before clicking on it? Probably. Would knowing that information realistically ever matter for a decision of consequence? Probably never.

I'll get requests for these sorts metrics every once in a while, and they're usually so unexpected that I have to pause and think through why it sounds like a bad idea. Useful measurements are usually a step in a chain of reasoning and narrative that is clear from the start how it fits into the bigger picture. That means it's usually easy to predict at least what the most obvious use case of a measurement will be at the time a request is being made. The prediction might have the wrong sign, or we later find it's uncorrelated, but at least it passes a sniff test.

Sometimes, these requests are a bit more subtle in that while IF we had perfect measurement abilities we would be able to get a useful measurement, but limitations in methodologies and physics stop us.

For example, every marketer in the world would love measurement of "intends to immediately purchase the exact item I'm selling" – in theory we could totally measure that exact metric by sending you a survey question that asks you that, then send you a link to buy the product the moment you say 'yes!'. Except, that sort of survey question has all sorts of self selection bias, sampling bias, and even potential user error mixed in. The answer to the survey question is almost meaningless, and we could've just sent you the link directly and saved everyone extra steps. Instead, in the real world we have much more complex models and instruments to try to measure intent to purchase and even those are a bit of guesswork.

Cost, LOL

Most things in the business world can be measured – for a price. But in the current climate of high interest rates, layoffs, and budget cuts, most teams want things on the cheap, and can't afford much beyond that. So there's a ton of things can't measure in practice just by considering cost alone.

By way of examples, when I worked in e-commerce, if we wanted an fresh re-count of all our stuff in the warehouse, we had to pay the warehouse vendor a set fee to have it done. Instead of paying the price all the time, we only did it every quarter or so, and had processes in place to pay extra attention to modeling items going out (sales) and going in (new stock or returns). It took up some of the time from the data and finance teams, but we had a working system going that was only off slightly from the true value.

At my current work, we could run an on-screen survey for essentially the cost of researcher time, or we could run a complex email survey campaign with incentives. The quality of the data in the later surveys is often much better, but obviously the cost is also higher. Guess which one we tend to do a lot more of?

Want to know how many people enter a store? How many observers can you hire? Want to count trees in a location? Send people to count or maybe buy satellite imagery and use fancy AI. All of that costs usually money that your team doesn't have. Let's not get into what the high energy particle physicists want to spend on their measurement devices.

Eventually, you'll learn what kinds of measurements your particular organization is willing (or can be convinced) to spend money on. Sometimes the answer may surprise you because an executive currently values a piece of information much more than expected. For all the other times, it is sometimes possible to come up with hacky proxy measurements that are much cheaper to execute. But the proxy makes you lose fidelity, and very often it costs you personally a bunch of time to finagle that proxy measurement. It's up to you to decide if that price is worth paying.

When even the "quick" way takes too long

Sometimes, good work, or even sloppy work, takes time. If you want to send a survey you need to write the survey, send it out, wait for results to come back, and analyze it. Every step takes time that you might not have. If you need data on weekend customers, you must wait until the weekend to observe them. There used to be a time where you had to BUY A DATA CD FROM A VENDOR to access some special dataset. If you need the answer before you can place the order, and have the courier come over, you're going to have to wait.

This is true even if you're willing to cut a bunch of corners and sacrifice data quality to speed things up. There are fundamental limits on speed beyond which you're just guessing. Being aware of these constraints up front lets you put a stop to requests that would be asking you to violate the laws of time and space.

Keep saying no to silly work

There's plenty of other reasons to say no to some kind of request for measuring something. The ones I listed today mostly center around the cost in terms of money and time, but you can easily come up with unrelated situations where saying no to a request is the correct thing to do.

So keep saying no, not because we merely would like to, but because many times the rules of the universe tell us we must.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!