What the heck is "Insight"?

TopPost Feb 20, 2024

Once, a UX writer pulled me aside for an interesting discussion. They were working on a feature that would help users better use the product – either suggest ways to save money, or surface features that would be useful in a given situation. The system will also show other potentially useful information. The writer asked me this deep question: "we're thinking of calling this feature 'product insights'... But what counts as an insight?"

At the time, we talked about what the term meant within the context of my technical work and we decided that "insights" was about as close a word to use to name the feature as anything else we could come up with. But the question always stuck in the back of my mind. After all, for most of my career as a data analyst and UX researcher, one of the things that I very casually say about my work is that I "deliver insights". That's one of the fundamental ways that I bring value and justify my paycheck. Despite that, I never really thought about what this "very valuable service" I provide is.

So recently I asked myself just what counts as an insight for work. And at the same time I asked on Bluesky what other people felt "insights" meant within the context of doing data work. Mashing all those thoughts and responses together, I had this list of potential properties that might help pin down a definition of the word "insight" as applied to data work.

Insights can be any piece of information. They are very much an eye-of-the-beholder phenomenon.
Insights have an element of novelty to the receiver. The information needs to be new to the receiver, or at least it needs to reminds them of something they have forgotten.
Insights seem to need an element of utility. Being able to use the information for something is what distinguishes them from trivia. They can serve to help make a decision, or inspire new thoughts.
Insights also aren't supposed to be obvious, they're supposed to come from a deeper understanding of a topic. Dictionary definitions of "insight" mention having deep understanding into the true nature of things.

The problem with all of these aspects is that they are all subjective and external to me. Just like how some people will think that it is insightful to hear that during a sale, customers often pay attention to the size of a discount (50% off!) than the absolute price. Other people will scoff that it is obvious because that topic has been studied and known in business/marketing research for ages now. But those same scoffing people may find it insightful if you can demonstrate that such a well-known sale effect exists within their specific problem space. There is little predicting how any given stakeholder would react to any given piece of information you provide.

So just from these surface details alone, you would think that consistently providing insight is an exercise in flinging idea spaghetti at the wall until you find something sticks. It's hard to argue that providing insights is not a numbers game of grinding out facts. It sounds almost mechanical and depressingly devoid of humanity or even intelligence – essentially, a dumb "AI" bot should be able to do at least some of this work.

But as practitioners, we all know that it's never that simple. Many of us have seen "insight systems" that just spam us with a barrage of notifications that are usually irrelevant and impossible to act upon. Most importantly, as users of such systems, we very quickly lose trust in the system, grow frustrated at the spammy uselessness, then we stop using such systems. Just think of how many anomaly detection systems died within weeks of launch after everyone learned to ignore its useless alerts. We know such systems take a lot of tuning to get right.

So then, we're back to the question of how do successful analysts and researchers seem to deliver "insights" on a consistent enough basis that they continue to be employed and valued, while also not annoying the hell out of everyone around them?

The thing that I think separates good analysts from a spam bot is that we have access to much more context and find ways to leverage that context to ask the right research questions and pick out good facts to highlight as being potential insights. I'm making the word "context" to a lot of heavy lifting here. Context encompasses all the things that we think about when doing our work.

There's the immediate context surrounding the analysis we're doing. What business problem caused people to ask the question to begin with? What's the decision that's being made? Who's actually asking the question and what's their background and quirks? What do they already know? Are they someone who wants to dive deep into the details or do they want a single tight paragraph with "the answer"? What's the potential decision landscape of the organization – what actions can/can't be taken due to existing constraints? Has this question been asked before, how was that received and what's different now? For any bit of data analysis work, we have at least partial answers to every one of those questions and we use them to guide both what we look for and what we're going to report back. The report is crafted to work under all those nebulous constraints. I think this is why good analysts take a year or more of work before they become truly effective at their job – it takes time and direct experience to absorb all this context.

But that's just the immediate surrounding context of the problem and business environment, there's always more out there. There's the broader academic and business landscape context to consider – it's why we do lit reviews and competitive research. There's research that's in other fields on other topics that aren't directly relevant but sometimes can be inspiration for hypotheses. This adds more context for deciding what hypotheses are worth considering and exploring with our finite time.

Put all this rich context together and you can really filter out the majority of "so then what?" items off of a presentation. That doesn't mean we're guaranteed to churn out bangers, of course, but the chances are significantly more in our favor.

So circling back, what the heck is this "insight" that us data people claim to provide? It's seems to me like it's curated knowledge that is timely and useful. Like the saying goes, the best time to plant a tree is twenty years ago, the second best time is today. The knowledge that get uncovered through analysis and research are often durable effects all would have been just as effective in the past – an improved scheduling system would've started saving money any day it gets implemented – but the surrounding organizational context may mean people are more or less willing to listen. It seems part of our job is to "read the room" well enough that we bring up the things that people either are willing to listen to, or need to hear.

It's definitely a weird way to look at things. Truth obviously doesn't speak for itself in such a view. But at the same time, in an age where tech is constantly trimming teams down for not providing enough value, I think it's quite important to understand the nature of the value we claim to be providing so that we can foccus and do it better.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

All Tuesday posts to Counting Stuff are free and will remain so. Generous support from subscribers keeps the servers running and makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Share posts you like with other people
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Consider a paid subscription to help pay for the servers and get occasional supporter-only posts
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!