"Working" with inconclusive results

When I first started my job as a Quantitative UX Researcher, my UX manager asked me this: "So, normally the other (qualitative) UX Researchers write up and share out a research plan before they go do a study. Does that really apply to quant work?"

I had to stop and think about it for a solid moment. At that time I had already been a data analyst for over ten years so even if I was unfamiliar with the role, I wasn't new to doing quantitative product work. My answer was basically "3/4ths no, 1/4th maybe". The overall gist was that a quantitative research plan tends to be very different in length and quality than a qualitative research plan.

More specifically, for a lot of the work we do, there's a significant risk that the results of the study will be inconclusive. We can make all sorts of lofty plans about methods and data sources and theory, but none of us can really predict if we'll walk away with a result at the end. Maybe the collected data was garbage or didn't even exist to begin with. Maybe the effect we're looking for is too small to detect, or our hypothesis led us astray. Or maybe the effect flat out doesn't exist and we just can't tell. For a million reasons, we can spend six week digging into a pile of data and walk away learning practically nothing besides how terrible our data is.

This is very different from a typical qualitative UX research plan. The general format of those is the researcher will detail the research question, the population they're going to study, the protocols and the survey or interview questions they might use, and what they hope to get. Very reasonable and straightforward things that you'd think would be reasonable to ask quantitative researchers to produce regularly. But there are two big fundamental differences that gives them more confidence that things will go well.

First is that they're going to directly collect new data. They might have already done a lit review, but they're not going to rehash existing data. Going off to collect new bespoke data requires planning ahead and writing up a research plan is part of that process. This of course translates to taking a lot of time and work to make sure things go well. But despite the costs involved, collecting bespoke data mitigates a lot of the reasons quant research work can fly off the rails – you know the data is suitable for the study because you control how it's collected. It's specifically designed to answer the question at hand. Data science and quantitative researchers very often don't have this luxury and instead rely on pulling findings out of pre-existing data.

But second reason is the kind of results that are expected from such studies are completely different. Qualitative results are usually in the form of "here's patterns of things we observed from our sessions." Due to the general low volume, high detail nature of qualitative work, they're not expected to do statistical inference. They are primarily looking to find patterns and interesting threads and then describe them. It is already known and accepted that they do not seek to describe the full breadth of experience in a space, but to highlight the most prominent and important-seeming ones.

Such studies are something that is almost guaranteed to generate some kind of result so long as the study was designed well. The findings don't even have to be particularly novel or surprising because confirmation about the state of the world is still very useful for product development work in industry. While you might not get the specific results that you were expecting, you can at least expect to get results that you can put to use. It's quite rare to walk away with "inconclusive" results from such studies. The worst I've seen is we find some unexpectedly important thing that we feel is important to do follow-up studies on.

So, against this long history of existing user research, management and executives that are very familiar with qualitative research work might mistakenly believe that quantitative research work would follow similar processes. Research is research after all. But then those people become confused by how our research plan tends to boil down to "get access to Table X, use P,Q,R methods to search for useful relationships related to the research question, hope we find something".

Why hope? Well even with the extremely p-hacky nature of industrial research ("kitchen sink" regression models anyone?), all of the social sciences is littered with the dead husks of extremely plausible hypotheses that completely fail to manifest when put to rigorous statistical analysis. The effect sizes are tiny if they even exist at all. There's competing theories of why things happen. There's often no telling whether the data itself or the theory is the cause of our null results. So any attempt at finding an honest significant result is an unpredictable mix of good technique and blind luck.

I think even fairly un-savvy managers can understand that we might fail to find statistically significant results in pre-existing data. After all, "significant results" means things must meet some supposedly rigorous standard (in a layperson's eyes at least). If it were so easy to find significant results, we would be churning them out day after day. So I don't think that part is the problem.

I think the problem some people have is they don't expect that it can takes days, weeks, possibly months of work to come to even come to the conclusion that we found nothing. To make things even more frustrating, our result isn't that there is NO effect and we've ruled it out completely – it's that we are unable to answer whether an effect exists or not.

You and I know that the time spent coming to the conclusion is all from wrestling to explore the data, find out something's missing, find the people who own the data, and run a dozen exploratory analyses that lead to dead ends. Maybe we find a bug or three in the data collection for good measure. Maybe we even build some new systems to collect better data in an effort to find the effect we're chasing. And despite all of this effort, it all can just... not work. Because that's just how this line of scientific inquiry is.

To be very clear, I'm not trying to say that qualitative work isn't science, because it is. Detailed collection of rich data and the careful synthesis of that data is a very important part of the scientific process. Nor am I saying that qualitative work always gets results in the sense that they'll always find what they're out to look for, because they don't. It's just I see product teams manage to pull at least some meaning out of qualitative data to continue moving forward more often than is possible with quantitative research. The methodologies just yield different bits of information.

So how can we keep our stakeholders on board with how our research plans contain a nontrivial amount of luck after a long dev period? A lot lot lot of stakeholder management.

If people aren't aware that statistical methods only yield any kind of result, positive or negative, after a very lengthy process of understanding, cleaning, and modeling the data, they're going to become frustrated at the process. So it's important to keep people informed about issues are coming up.

Obviously, if there is missing data or a need to change data collection practices, those have to be escalated to the appropriate teams to be dealt with. Those blockers are pretty easy to explain and understand. What's less accessible are the things in the exploratory data analysis phase where you're trying out hypotheses and they don't pan out. We usually don't want to bore stakeholders with these little details, but at the same time it's incredibly easy to just disappear into a rabbit hole for three weeks while chasing those ideas. I've seen lots more people get frustrated at those silent periods because there's no steady progress, just big step function jumps when we stumble upon a solution. Learn to communicate the status of those when you can.

But another thing that is tricky for stakeholders to appreciate is how the scientific method can't prove a negative. If you mention it they'll pause and think about it and say it makes sense, but the implications of the fact are a bit remote from their experience. So they can get confused when faced with the fact that our work ending in not in a negative or positive result but just "inconclusive".

They often try to push to get some kind of partial result they can use, something "directional" perhaps, or permission to rule out the existence of an effect. But inconclusive means exactly that, we just don't have the information to come to a conclusion. We get almost nothing.

For people who've done research before, this is just part of the process. But in the eyes of a stakeholder, do it often enough and it's a sign of failure and wasted effort. The only consolation prize is that we might have better data policies and collection methods to show for it thanks to all the issues we uncovered along the way.

One cynical way of looking at the situation is this explains why a lot of early (for a given organization) data scientists tend to lean towards data engineering as they work. Their lofty attempts at finding a result slam them headfirst into all sorts of technical challenges that prevent them from ever running a successful model. So we sell the tech benefits that were generated on the way to our inconclusive results. Look, our reports are faster, our metrics are clearer, we can even explain some things. We just still can't quite detect that lofty goal just yet.

But hey, it's progress.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!