How social scientists measure the abstract

data-culture Nov 10, 2026

I recently realized that I write a lot about measuring stuff, but had pretty much avoided this one because... I guess I didn't want to revisit my old textbooks. 🙃

I once had a friend tell me a work story when we met and traded war stories – once, in a large tech organization that had years of expertise in measuring user preferences like satisfaction with their product, there was a debate going on about what metric they should be targeting in the future. The debate itself isn't the horrible part, but one of the data scientists had declared that they thought all metrics should be based off behavioral logs data and none of the "fake survey stuff". Apparently this DS was of the opinion that self-reported survey responses are biased, while behavioral telemetry was impartial, objective, and I suppose "real". Luckily, the higher levels of management decided not to listen to that particular DS's opinions, and they resumed doing the usual mix of social science and data science research that the organization was known for.

At the time, myself, and almost everyone else I had shared this story with expressed shock that someone who ostensibly works with data held the opinion that somehow a whole class of methods, one that underpins a vast majority of social science research, could be casually brushed off. It's not like there isn't litany of issues with logs-based telemetry data either. Pots, kettles and all that.

But, if I want to be charitable, I do have to admit that the theoretical foundations of the validity of social science measurements aren't immediately obvious to people who don't either come from social science or a scientific method nerd.

So I figure now's a good time as any for me to go refresh my somewhat rusty knowledge of my ~20 year old graduate Communications training and look into how "asking people stuff" can be pulled into a scientific framework.

It's all about validity

If there's one thing I remember from my first year research methods course was how we spent a huge amount of time discussing different forms of validity. In short though, the discussion of validity of a metric is about "does the proposed measurement actually measure the thing it says it is measuring". Interestingly, Wikipedia lists a lot of the different types of validity we had discussed under "Validity (statistics)" since the term itself is used in a lot of contexts in various ways.

The biggest takeaway out of all the classes I took was that the validity of a given measure as discussed isn't some magical on/off property, but a complex discussion along many dimensions. We could discuss whether a measurement or study had "ecological validity" when a lab study seemed so artificial that maybe it may not apply in real life. We may say that a measure had low "face validity" because it didn't seem to make sense on the surface, but find that it was consistently reproduced in lab studies.

Often the validity that comes to the forefront in social science methods is the notion of "construct validity", which is basically to what extent are our measurements good operationalizations of a theoretical construct like "user satisfaction" or "happiness". In essence, construct validity provides a framework from which abstract theories can be transformed into concrete measurements that can then be analyzed to show that our theories and measurements align.

Bridging theory and measurement

While there's been different proposals of how this bridging is done, one way comes from building nomological networks, proposed by Cronbach and Meehl in 1955 in "Construct Validity in Psychological Tests". I find it relatively straightforward, but it's about as convoluted as any meta-scientific process description.

The basic idea is you take a theory of some sort that expresses a relationship between two constructs – for example, users with higher customer satisfaction are more likely to retain as ongoing customers. Then you operationalize the construct of "customer satisfaction" and "retained customer", collect data, and test the hypothesis of whether your operationalized measures follow the prediction of the theory. If your theory and measurements are valid, then you should see a relationship where satisfaction going up or down predicts customer retention in the future. You may later expand on the theory by adding new constructs to help explain more of the world, like perhaps adding demographic factors.

Ultimately the whole apparatus hangs on this inter-connected network of abstract constructs and their relationship to other constructs as dictated by theory. While we may start by measuring "user satisfaction" with a simple question of "Are you satisfied with this product?" we may find that the measure doesn't behave as our theory expects and we have to revise. Maybe we think that the correct measure of user satisfaction with respect to retention is their most recent interaction with the company, whether it be last sale or last support call.

What's also important in evaluating construct validity is what doesn't cause a measurement to change. In our user satisfaction example, maybe you don't expect user satisfaction to increase when someone receives their ordered item on time, as promised, but you'd expect them to be dissatisfied if the item is extremely late for no discernible reason. That relationship is yet another thing that your "theory of user satisfaction" would say that becomes part of your operationalization. It's this ongoing process of defining the borders and relationships to everything that gives us confidence that we're measuring something that is related to our construct.

The main "problem" of this entire process is that none of it feels solidly attached to anything. It's literally definitions and relationships that simply point to each other. People love to point out how there are infinite ways to word a questionnaire item, or define and redefine a theoretical relationship to fit the data. They feel like the lack of a concrete physical anchoring to the world means that social scientists can continuously make stuff up, all while overlooking the decades of research that goes into validating important measures like the Big Five personality traits. There's a reason why, for at least the well-studied constructs, there are standardized, validated questions used to measure things.

In industry, we of course don't have as much standardization since we don't publish nearly as much. At best we have to use shared measures we hate like NPS. But even then, we still share basic rough theories about things like how user satisfaction behaves, how user engagement can manifest in telemetry, and many other measures. That said, if you've noticed that I occasionally bang the drum about using theory at work more (see: We should talk about theory to teams more), this is where that habit comes from.

I know that a lot of people point at the physical sciences as some example of "measuring real stuff" to contrast to how we measure abstract concepts, but if you look at how things are actually measured, it's the same process. Voltage way back in the early days of the study of electricity was essentially the deflection of a bits of metal in a particular experimental apparatus. Eventually, the theory of electromagnetism got refined and methods for measuring properties of the theory got better. But at the core, the only reason any of us believe the value displayed on our digital multimeter is because of a long chain of theoretical constructs and operationalizations that led to that tool being made in a specific way. It all hangs together within a theoretical framework just like how "happiness" measurements hang together. It only feels like shoving two probes onto some metal contact is directly "measuring" something "real" because people don't think too hard about the complicated metrology involved. They've taken the measurement process completely for granted.

Meanwhile, everyone has opinions about how we can "measure satisfaction" because it's "easy" to ask people questions.

*sigh

Subscribe!

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

Counting Stuff Official Forums: Discuss posts, or other data topics with the community.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
Send a one time tip (feel free to change the amount)
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!