We should talk about theory to teams more

TopPost Jun 11, 2024

Short post this week because house stuff and talk writing. This week is Quant UX Con 2024! Please register come watch a bunch of talks about the quantitative side of UX research. A bunch of topics are usually quite relevant to data scientists working on products. Incidentally my talk is June 13th, 7pm EDT, "Designing experiments for maximizing getting things done".

My talk at Quant UX Con 2024 is broadly about avoiding common bad applications of experiments. That is to say, while the experiment itself might be valid from a purely mechanical standpoint, for many other reasons the whole endeavor is wasted energy that could have been used anywhere else.

One of the topics that I go over is how we need to talk about "theory" a bit more.

By the term "theory", I'm specifically talking about the theoretical frameworks, the mechanisms and causal linkages, with which we predict what will happen for a given situation. For example a theory of social proof leads us to think that people are more willing to do something if they see other people do them, and this theory leads to product features like star ratings and "best selling" tags.

Industry is a-theoretic

While probably an unfair overgeneralization, industry does not particularly care about theory the same way academics (and recovering academics) do. If we give industry a black box that endlessly prints business value, industry would happily use it while only sparing the barest of thoughts on whether a black swan event might break the box in the future. Pretty sure a bunch of sci-fi is based on this premise. Theory is optional. It is a nice to have. But results speak a language of its own.

Outside of the smaller circles of people with researcher backgrounds, industry doesn't really talk about theory. I've primarily seen the more academic people (myself included!) try to push theoretical discussions out to the broader company with rather limited success. People can be interested and polite, but there's few lasting effects on how people do things. It's all too easy to forget about such abstractions when you're busy testing out new implementations of the same abstraction month after month.

Within an industry experimental context, not caring about theory is quite convenient. If we can build something on the website, we can test it. It doesn't matter if our changes are "grounded in theory" or not. We can build just as much with principled reasoning as with vibes. Every engineer, designer, product person, or CEO can propose something and have it stand before the relatively impartial judge of an A/B test. There's no ivory tower gatekeeper anywhere in sight slowing things down.

Velocity is good. Getting design ideas from a more diverse population is very good. The lack of the use of abstract theories is correlated with these observed benefits. Some might want to take a leap into making a causal claim that it's the removal of such tedious topics that provides such benefits, while I want to say that the two things are merely correlated.

Unspoken theory

What many in industry don't realize is that a lot of what is being done does actually align with various theories of how the world works. The things that "work" eventually get studied and eventually get worked into theories somehow, so there's always a relationship. The difference is that such abstractions are discussed with less intimidating words like "concept" or "design pattern" instead of the T-word.

A designer might arrange widgets according to their internalized sense of information seeking theory. They might include social proof mechanisms simply because they think the feature is useful from past experience. They may or may not be able to articulate what they're utilizing in the same academic terms. But if you take a step back, you can abstract slightly and arrive at the "why do we think this works" reasoning all the same. Many of these ideas can find parallels in academic literature if you're willing to dig just a bit. More importantly, these concepts yield testable hypotheses, designs, and also links to ideas you may have never considered before.

It's the fact that we don't articulate these abstractions out loud that's the problem with much of industry experimental methodology. We treat the lack of explicit theory as the same as the irrelevance of theory – when that's not true at all. If we put zero discussion into what class of action we are trying to take, we are probably putting zero thought into it too. Without some awareness of the abstract, we can't map our ideas in relation to all the other possible variations we may want to test out. If we try five strong variations of the same abstract idea and they all fail miserably, then maybe it's time we learned to give up on that abstract idea instead of blindly trying a sixth test.

Talking about theory without saying "theory"

I think there's something fundamentally off-putting about having our coworkers engage with theory on a daily basis. Our daily work has little to do with the formal scientific work of testing, evaluating, and possibly even rejecting actual theory. We're not searching for a higher truth. We're just pushing pixels on a screen hoping to make some money. It's pretentious to pretend otherwise.

At the same time, since people are used to working with abstract design concepts already, it seems best to just run with that. Using design concepts and similar abstractions in place of theory gives up the stronger predictive powers of having a theory. You lose the "If we believe this theory is true, then these other things should be true". But in exchange, you at least get people to work at a higher abstraction level than just "test this design we made". If a test is framed as being one test of many of a given concept, eventually you're going to build a viewpoint as to whether that concept works or not. Memory of past successes and failures is super important, and I find lots of organizations just leak experimental experience like a sieve.

Building stores of organization memory is always a hard task. A lot of times I've personally become a single point of memory failure because of just my tenure at a team. I've had precious little success making other people an alternate group historian. But some of the little success I do have involve meeting people where they are with design concept ideas and reminding them that we've tested X/Y/Z aspects of them. It seems to stick a bit better. And even such a small improvement should be a big change in quality of work being done.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted, so support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
Share posts you like with other people!
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!