How does one find "signal"?

data-culture Oct 12, 2026

I maaaaay have burnt the majority of my tiny sleep budget on Ace Combat 8 last week... but it was totally worth it. That said, apparently if I'm sleep deprived I start musing on what the heck I do at work...

There's a number of valid ways to describe what a data scientist does, and the answer mostly just depends on who you're talking to and what lens you want to apply tot he discussion. One of the many things that we wind up having to do consistently is this vague idea of "finding signal" with respect to some question.

At its core, "finding signal" for something is a question about information – given a situation, is there some bit of information that can be extracted from the environment that would offer better-than-guessing odds of coming to a correct decision. Very often we just frame things as a prediction problem of "can we find some factor X that lets us predict Y". Other times we call it a classification problem, or some other form. Regardless of the surface form, the method for finding and showing the relationship between X and Y is relatively uncontroversial, it's all the common tools of correlation, regression, curve fitting, hill combing, and all the other familiar tools we wield. If anything, the model part is "the easy step" because since most of us aren't statisticians or methodologists, we treat the models as a largely mechanical concern.

Instead, the hard work is figuring out "where the signal is". I've seen this take a couple of forms.

First, there's just science

The first form of the signal problem is when you have some phenomenon you want to explain or predict, and you're given an essentially clean slate as to figuring out what to measure to help predict it. This fundamentally the realm science and the scientific method.

Of the nearly infinite measurements we have available to plug into the X variable, as well as the countless transformations and filters that could be applied to X, it's impractical to expect to blindly pick some measurement at random and expect it to have anything to do with Y. The solution we've developed to narrow the intractable volume of candidates is usually to falling back to some kind of theory, as I've written about before. If you've got a working theory (and I use the term super loosely here), narrative, or even just a hunch that there's a causal relationship, then at narrows the infinite field down to a few things that can be put to a test.

In the actual sciences, most theories are deeply researched things that have already been subjected to multiple studies that all work together to figure out the boundaries of where the theory appears to work and where it doesn't. The more well developed a theory, the more refined tools and methods exist that extract the important signals the theory needs to work. For example, our theories of temperature started with measuring with simple glass tubes filled with mercury or alcohol, but now with more refined theories we can measure temperature through things like inspecting light wavelengths or measuring voltage traveling through certain materials.

I have to admit that very little of my data science work ever comes this close to this version of signal finding. The closest is whenever I have to work on a survey and wind up having to remind myself that certain constructs have validated questionnaire items and if I want to measure them it's best to just use those instead of making random crap up on my own.

What's nice about this mode of thinking is that you start without constraints. You can list out all your theories and all the potential measurements that can help prove that the theories are true. Only afterwards do you have to take that list and bring it down to reality where you figure out the messy details of whether those theoretical constructs can be operationalized into an actual measurement. The hope is that "the signal", the magical theoretical thing you're chasing, is reflected enough in your measurements. Expertise in the domain manifests by knowing much of the existing theories and measures and knowing which ones are the most relevant for the situation at hand.

Then, there's the sorta-science

The other way that I get asked to "find signal" is when I'm given a preexisting dataset or data collection system and a business question and am asked to find if there's anything in the data set that could help shed light on the business question. I consider it different because by starting with the hard constraints of existing reality, you wind up asking a different set of questions.

The most important question in this situation is, given the data points we are collecting, does the data contain any information that can be used to shed light on the problem. Looking at the data points together, if the data moves in a certain way, does that carry any useful meaning to it.

As an example, let's say I was collecting data on which way people go when they enter the main entrance of a large space, like the entrance of a supermarket or department store. People can enter, pause, go left, right, straight, or variations thereof. I remember seeing an anecdote decades ago that the majority of people tend to turn right upon passing through an entrance into the decompression zone at an entrance. This is a weird piece of information buried in the data points – what is it capable of telling me? Is it showing some kind of quirky human bias? Is it a local cultural phenomena? Does it have something to do with handedness? Maybe the layout of the building or its signage is causing it? It's not clear, but there's "something" there that warrants investigation. Moreover, regardless of why that curious signal exists, can I use this information to help with my business question?

The same goes for other data points – a bunch of users are taking this click path, a specific feature is only used in this context despite it being available in other places, and so on. The data appears to be saying (or seems to have the capability of saying) something, and its up to your interpretation to figure out if it is useful or not. If usage clusters along a certain demographic or time, then that tells a different story. The candidate "signals" in this situation are the myriad potential interpretations of the data's behavior itself. They are guideposts that direct your attention to a particular bit of data in hopes that this is a correct X to go with your Y. Very often the ideas are parallels to existing theories, but are novel, weird ways of approximating the same constructs.

Finding such signals is very much like playing the role of a detective in a B-tier novel – you're essentially fabricating a whole story out of a few clues and overlooked details. It's an act of creative generation from largely fixed points and constraints. If anything, it's these weird happenstance observations that provide the seeds for actual hypotheses that allow actual research to happen. Luckily we can go back and then test against our data to see if such imagined relationships actually exist in the data. Given how I love linking weird details together into a story, you can tell this is one part of the job that I like.

Domain expertise is helpful here because it can provide other data points that help flesh out the more likely hypotheses and stories that can be pulled from the data points. Being in a space for longer means you're naturally able to better imagine the different potential stories a data set can tell than someone who is new.

In some ways, this sounds like a weakened form of the "data mining" boom that happened in the 90's. Back then, the idea was that if you just threw "algorithms" at massive data sets, then you'd somehow strike magical causation gold that transforms your business. We all know that most of those hopes turned to dust in the face of spurious correlations and the lack of theory. So how is spamming hypotheses against a data set any better? Probably because our narratives are a weak form of theory, insofar that we're at least constrained by positing plausible stories instead of complete nonsense. We're still at the mercy of spurious correlations, but in industry where everything's a chaotic unknown anyways, having a direction to move in is better than nothing. We can always course correct as we learn more.

So as I see it, "finding signal" at work is about creatively considering the boundaries of the data and data collection methods we have available. We can isolate signal with the help of preexisting theory telling us where to look, which is admittedly rare in my line of work. We can also leverage the knowledge we have about what's in the data and how it's collected to hypothesize ways in which it may potentially contain signals relevant to the question. Often that process is guided by theory, but because of the limitations we work under we have to invent poor imitations and proxies to get the job done. However we arrive at our measurements, the hope is to subject the ideas to tests to see if they hold water.

Subscribe!

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

Counting Stuff Official Forums: Discuss posts, or other data topics with the community.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
Send a one time tip (feel free to change the amount)
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!