Exploratory Data Analysis Can't Be Automated Away

data-quality Nov 3, 2026

It will be Election Day in the US when this post goes out. Everyone go do that if you're able. As for myself, I don' t need the unnecessary stress of watching the moment-by-moment over-analysis of everything at the polls and will actively hide from the news until the morning.

If there's one thing LLMs have done for my professional life, it's that they can finally write a lot of tedious repetitive nonsense for me. I don't quite have to re-invent the "send a report on a cron job" thing every time I switch jobs, I don't have to write that janky helper module that lets me quickly convert a SQL query into a CSV file by saving the fussy login details and machine names. And if I ever have to write an oauth2 flow again in my life that will still be one lifetime too soon.

While I don't have to worry to much about the regular readership of this newsletter, because most of you are data nerds who actually don't mind doing analysis work, I think it bears reiteration in the modern age – when doing an analysis, it is still required to personally, manually, look through the data with your own eye balls. Exploratory Data Analysis is simply not something that LLMs can make completely go away, despite how much AI frontier companies, startups, and overly enthusiastic stakeholders want you to believe. For all sorts of reasons. Even in my attempt to actively force the bots to do my job, the end results were just so frustratingly bad that I'm confident in saying that, as of 2026, we are very much not there yet, if we ever are even going to get "there".

Most importantly, your intentionality is missing

Exploratory data analysis exists as a way for us to poke around a data to get an idea of what it can and cannot say, especially for a dataset that is unfamiliar. That means that we aren't aware of all the potential little gotchas, the quirks, the bugs that potentially lurk within.

On the surface, this sounds very much like a "math" thing. Surely you can "find outliers" or "do descriptive statistics" to examine the shape of the data at a high level. You can even find very obvious issues with certain data sets by doing basic stuff like that. LLMs with coding loops can iterate and run through checklists and stuff. But the problem is that what looks correct versus weird is entirely depends on what you intend to do. Since math formulae will happily calculate with any values you shove into them, the interpretation comes from us. Why are we looking at this data? Do we know how that data was generated and collected? Do we have an idea of where we should be directing our attention? All of those questions, and more, should be bouncing around your head as you poke and prod and explore what's going on in different fields. No LLM can, in a vacuum, answer any of these questions because they're purely an invention of our individual human minds.

Doing EDA not only let's you narrow down your thinking to what can be possibly done with a dataset, it also expands your ability to generate new and interesting questions and hypotheses to ask of the dataset. For example, you can be scanning a bunch of fields and notice there seems to be a correlation that may be interesting – you can then go and check specifically to see if such a correlation exists. That is likely not a question you would even think to ask without accidentally stumbling upon it.

And sure, you could try to give an LLM enough context to do this, in theory it should be possible to write a giant essay explaining the business context, what decisions are pending, what current knowns and unknowns are, all the data collection details, etc.. But it's a herculean writing exercise to do, and there's no guarantee that you actually provided all the needed context. Plus, given how LLMs can struggle with longer and longer inputs, I can't see a situation where this doesn't eventually go badly.

LLMs continue to make really stupid analysis decisions

While I'm fine offloading specific tasks to an LLM, I find they're just really terrible when given any leeway with decision-making. Since there's no absolute positive/negative feedback in an analysis, typical coding agent harnesses have no way to tell if their analysis has completely flown off the rails. While toying with the idea, I've seen Claude get a peculiar idea that obviously takes assumptions that doesn't apply to the data and attempt to build a giant testing harness around that analysis idea. I watched it run to completion and asked some pointed questions about why it was doing that and it did the "oh! my bad!" routine. You can jump through all sorts of hoops and harnesses to try to get a handle on things, but the only creativity I've encountered with this process is how it comes up with new and unimaginable ways to do the wrong thing.

It's not good enough to give it vague analytics instructions like "go check if X and Y are correlated", because that leaves it room to decide how to handle edge cases and other details. You have to baby-sit it down to the minute details about how it's going to check, and what. Either you do it before-hand (which is tedious and hard since you're exploring an unfamiliar data landscape), or at the least you take the analysis plan and turn right around to check to see whether it coded up the thing as explained. I've lost count of the times when what was 'promised' wasn't what made it out.

While all these shenanigans saves us the actual tedium of writing the specific lines of code, we are still the giant bottleneck of the system because we need to catch all the dumb decisions these models produce. Sometimes, I'm not even sure how it manages to come up with some of its crazy ideas.

Even if we manually do everything in EDA, like back in the old days, we would still find bugs in our work that require us to go back to rewrite a bunch of stuff. Adding LLMs to the mess does not seem to make that time appreciably faster in most cases once the "review what happened and actually mentally process it" part is taken into account.

Hypothesis generation is an organic process within you

Everyone who does analysis knows that even if you come to a dataset with clear questions and goals, there's no telling where you will wind up. The "real story" that you want to tell can be largely unrelated to what you were originally looking for. What's interesting is that this change isn't just about "getting a better grasp of the data" like I mentioned just above – it's also you clarifying your own thinking as you work through the multiple iterations of figuring poking and prodding at the data. Your hypotheses get better and sharper because you've spent time refining them as you go. Offloading that mental work wholesale to an agent pretty much gives up on that entire process.

There's probably ways you can balance the process of exploring thinking your way through an analysis with avoiding the tedium of implementing such an analysis, but it is a fine line to walk and you'll have to figure out for yourself. In my own work I've learned that while it's cute and entertaining to just throw in a dataset and dictate a bunch of "does the data say this? how often does this condition exist? what's the distributions along this axis" questions at a bot, I'm still missing something from the process. I can't make interesting connections and leaps of pattern-matching this way. I used to have trouble doing so for large datasets that a typical data scientist would throw into pandas or R-studio to poke around with too, so it's not a new thing for me. I need to "hold" the data in my hands to mess with to hit that point.

I'm sure you have your own way of doing things that works, go do that. The new way isn't ready.

Subscribe!

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

Counting Stuff Official Forums: Discuss posts, or other data topics with the community.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
Send a one time tip (feel free to change the amount)
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!