New data analysis can be faster, but MUST be grounded in slowness

data-culture Aug 4, 2026

For over a year and a half, on our fun little Discord server, I had been giving updates on one of the silliest home projects I've had to work on – the removal of a hole. It's actually an old, dead, pond, lined with concrete and stones, but for various reasons it needed to be drains and turned into a level patch of dirt.

Pictures have trouble expressing scale, but very roughly speaking, it was about 40 square feet of area. So over the course of two years, I had to do all sorts of menial tasks. First was draining and shoveling out the many years of accumulated muck and loose stones at the bottom. Then getting an electric demolition hammer (think, 35lb mini jackjammer) and punch out the bottom to improve drainage and remove the stones along the edge. The final phase of work was to fill the thing with a pile of sandy dirt which, at ~40sqft times the average depth, came out to something in the ballpark of 3-5 tons of sand. Manually, with a shovel.

Did I mention that I'm a literal office worker who doesn't work out and the heaviest thing I lift on a regular basis is my kid that weighs a bit more than 50 pounds?

Now, I certainly could have hired someone to do this work at any phase to come in with either machinery or laborers who can work 3x faster and longer than I ever could. But given the quirks of the surroundings and how my plans to safely do the job kept evolving, it was never really a valid option in my mind.

So instead, when time, weather, and (most importantly) motivation allowed, I'd spend about an hour here or there hammering at concrete, lugging heavy stones, or in the very end moving shovel loads of sand from one part of the yard into the hole. I don't honestly remember how many days it took, but just the shoveling alone took about three whole weeks of almost meditative toil. In my mind, it was a fixed amount of work that needed to be done eventually. There was no fixed timeline, but every stone, every shovel load of dirt would be inexorably closer to the end goal in my mind. The proverbial journey of a thousand miles was just taken in thousands of tiny little steps.

But the capitalistic world we live in here doesn't particularly reward toil and labor. Any laborer I (or a contractor) would hire to do the exact same job would be paid a relative pittance compared to the amount of sweating involved. It is a job to be mechanized away with the help of expensive labor-saving tools. (I seriously considered renting a little bobcat earth mover for a day at one point.) Either way, the reaction I got from a bunch of various friends and others I shared the project with expressed incredulity at how someone with my very spindly arms and zero visible muscle tone managed to move literal tons of sand over the course of a month.

This brings be back to the data topic of the week. At work I'm in the middle of a research project, and I'm essentially analyzing about 2200 rows of text data. The problem itself is the frustratingly vague research question that comes up constantly at my level of work – "here's a pile of data, it seems useful, do something useful with it". Given the data set and what I know of the important questions people needed answer, I decided to analyze how language was being used within the 2200 data points.

An example of what I was doing would be something like "was the first couple of sentences stated in the form of a question or statement?" Or, "what topic was this conversation about?". So very ambiguous stuff that doesn't quite lend itself to the more simple methods of keyword counting.

Now, depending on exactly what you're doing with text analysis, the NLP bag of tricks you have access to can be pretty deep, or pretty pretty shallow. The particular questions I was trying to answer could be answered in one of two easily predictable ways. I could do it the traditional way, which was to hand-code the 2200 rows of text data with the aid of whatever humans I could recruit for the task. Or, in the modern LLM world, I could try to leverage LLM-generated code, embeddings, and straight up LLM-based classifications, to get at my research questions.

Now, again, because capitalism, I certainly did not have any resources to hand code all that text. And even my "roll up the sleeves and grind through the work" tendances that led me to move tons of sand with nothing but a shovel was not about to dive into hand-coding the whole mess either. I know with the lead-in story, you were expecting that I just sat down and slowly lost my mind reading 2200 giant text blobs coding it all. But no, I'd expect to be fired for non-existent productivity long before I got through such a large data set. So in the end, despite my reservations about the accuracy of LLM-based NLP measures, it was the only reasonable way to work through the data quickly enough.

Confirmation bias everywhere

Can I yell at a LLM coding tool long enough to generate the code I needed to run a fancypants "embeddings + HDBSCAN" clustering analysis on these arbitrary bags of words? Sure! It only took day or so to convince the machine to give me an answer. The problem was that I was that I knew that whatever answer the initial pass gave back was going to be something I wanted to hear. LLMs are, if nothing else, giant confirmation bias generating machines. I'd argue that is their primary function – they'll string together symbols that on average meets our expectations. That makes the extremely dangerous "tools" for research use.

The initial results gave me a bunch of clusters of text that were supposed to be related. Further refinement and analysis attached labels to those clusters. At a very high level, the analysis concept was "sound" in the sense that it was going down the steps I thought were reasonable for the problem at hand. It was so damn tempting to just run with these numbers and go on. But hey, that same feeling of "oh I can run with this because it sounds good!" has come up countless times in my career and it always means the same thing – I must have screwed up somewhere.

So what's a seasoned analyst to do? Go into the clusters and see exactly what got clustered and labeled together. Now, we all know that cluster analysis is fairly artificial in that the number of clusters and specifics of what gets clustered together are largely fictions of model parameters. For my purposes, I don't care too much if we wound up with 5 clusters or 50, so long as they help me apply a lens to what is going on with the text inside. So it is here that I rolled my sleeves up and carefully started inspecting the clusters, the labels, the underlying text. I do this because the effort of verifying the output of an LLM in this case is easier than doing the actual job itself.

And, boy, did I find issues. There were outright bugs in how the pipeline was executing. There were issues with the LLM prompts that were being used to classify for certain properties. There were data quirks that needed very specific handling and cleanup to make the output make sense. Some things were stupid bugs that the LLM coding agent created while trying to do the task I had given it. Others were just data issues that only revealed themselves when subjected to analysis that made me think "this can't possibly be right".

I'm sure people will ask, given how much time and effort you put into checking the damned output of this Frankenstein analysis model, did you save any time doing it by using LLM tools? The answer to that is, definitely yes. I probably would have written different kinds of bugs in the analysis, but I absolutely would have written bugs that would've needed to be fixed. Plus, it would have taken me days or weeks to write everything since a lot of this was bespoke stuff.

I'm also sure that people will ask whether constantly checking on the output is itself a trap because there's theoretically no end to it. There could always be yet another bug or missed caveat that I didn't manage to catch despite my efforts. And to that, I have to say that it isn't a trap. As I whittle away at the bugs and many issues that came out of the analysis, the story the data started to tell became more cohesive. Statements you could make about one part of the data wouldn't be contradicted by statements about other parts. It's the same sense you get when you finish a manual analysis and the various numbers, ratios, and distributions generally start agreeing with each other. Even for 100% human analysis, there could always be a critical detail overlooked, but we eventually ship those results too. I've found the process to eventually lead to the same state of "I've done my best to poke holes in the arguments and can't find any more obvious ones".

And so, after all of this, I think that the need to grit our teeth and dive into our data has evolved somewhat but hasn't fundamentally changed. While now the demands of modern data work are such that we don't literally have to roll up our sleeves and hand-code every single piece of text any more, we still have to inevitably do extremely tedious checking passes on output. And for problems where human judgement is still "state of the art", there's no getting around being that human judgement.

There will be huge amounts of pressure from employers and stakeholders to push faster and just give in to the confirmation bias machines to use the first few answers they push out. These machines are practically designed to tempt us with that choice. But if we value being the voice of data, being the voice of end users and not just "whatever the PM wants to hear", we must find ways to call out the nonsense that the machines spit out. Despite it looking like we're manually shoveling sand into a hole out in the suburbs.

Subscribe!

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

Counting Stuff Official Forums: Discuss posts, or other data topics with the community.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
Send a one time tip (feel free to change the amount)
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!