The era of unscientific management

measurement Apr 7, 2026

Back when I was much younger and still taking business management classes as an undergrad, I inevitably had to learn about the overall history of work and management. Overall, if you zoom out far enough, everything involving humans seem to happen in waves of popularity and this includes the messy history of the various forms of "scientific management" that has come and gone over the past century.

At the very core, the various flavors of scientific management, be it Taylorism, lean manufacturing, operations research, or even data science, all make a similar family of claims – that aspects of "work/business" can be studied, understood, and optimized through the application of the scientific method and analysis. That is, it's not completely the domain of chance, craftsmanship, nor heroic genius figures. The various flavors all have their own views on what can and cannot be studied and optimized.

Since the topic touches deeply upon how work is understood, viewed, and measured, as well as how entire organizations and societies can attempt to align (or not) with the core concept, it gets touchy pretty quickly. When taken to extremes, there's valid arguments on all sides. For example, one of the classic critiques of Taylorism was that it reduced workers to mere numbers, replaceable cogs in machine. At the same time, while more modern formulations recognize the complex human factors involved, many of us to this day (and certainly almost everyone who is reading this newsletter) believe that there are aspects about work and productivity that can be measured and improved upon with data and analysis. The truth is probably some complicated mix of all of the above and then some.

If you trace the arc of management philosophies into the modern age and the rise of "Data Science" as a field, which at the time of its incept was literal shorthand for "apply the scientific method on the data that you have, and make better decisions", you can clearly see that the entire field stands quite firmly rooted in that scientific way of viewing the world. Since the mid 2000s, after the smoking crater of the dot-com bust, companies started hiring more and more data scientists because the general belief was "This Actually Works!". Big successful companies like Google and Amazon were doing "cutting edge" stuff like running an A/B test to see what designs worked better. The whole idea got so hyped up the job was declared one of the "sexiest jobs" of the 2010s by a bunch of people who clearly weren't living in the trenches of that era.

But as with all ideas that take hold in the zeitgeist of the management class, eventually the idea will sound old, and some new shiny thing will take its place. Because once everyone who aspires to be the next Steve Jobs has been indoctrinated to take measurement without understanding why, to report on vanity metrics to "check the metrics box" so they can ship, then the "science" part of the whole "scientific method" thing becomes an inconvenience. Think of how many meetings you've had that could be summarized as "can we just juice this number so I can ship and get my promotion?"

And conveniently, the current New Super Hyped Thing is a universal hammer that anyone can wield with natural language and it will generally tell you whatever you want to hear! I already hear of executives, product managers and engineering leads straight up say they have asked LLM-based tools to do analysis for them and they just go with the results. I'm going to charitably just ignore the people who decide that "silicon samples" aren't anything but nonsense for my own sanity's sake today. This behavior is so common that I don't even have to point as a specific company because I've heard the story so often from so many people in so many places.

It's wholesale laundering decisions through a confirmation bias machine.

It's a pretty amazing implicit rejection of the whole idea of "we believe something because we rigorously studied it" in favor of "the machine told me an answer that sounded plausible, YOLO".

I've been doing AI-assisted analysis stuff for a couple of months now and while it's really convenient to not have to write a bunch of regex or list comprehensions to manipulate columns, I've had to watch details like a hawk to keep things from making ridiculous decisions every step of the way. I once requested some dataset be clustered via correlations between the items and somehow the LLM came up with a ridiculous metric to cluster against instead of doing literally anything that used the word 'correlation'. The whole thing is a horrifying exercise in asking yourself how do we actually know what we claim to know.

This ... vibe-analysis behavior is dangerous because people have had such (surface level) success vibe-coding they think it somehow extends. (Yes, I'm aware SWEs are learning in real time the dangers and un-maintainability of massive vibe coded codebases as I write this.) What sucks for analysts is that unlike code, which has a clear "run/doesn't run" signal that is used as feedback, all analysis code will eventually print a result, regardless of whether the analysis is correct or not. There's no magical compiler, no runtime error, that will call you out. It's that very tight feedback cycle that makes vibe-coding even remotely doable to begin with.

The people who lean the hardest into this whole vibe-analysis, vibe-product-development thing are the folks who believe they really are the next Steve Jobs and want a yes-robot to confirm their beliefs all the way to glory. The more self-aware, cautious folks will often stop and ask some critical questions. The best folks I've met will actually have the same ingrained warning system of "I'm seeing something I want to see, I must be careful here" that many data scientists develop. But guess who ships the most in the least amount of time?

What's happening is that we're now currently presented with an competing theory for product development. Instead of the "Science-based" methods that most of the industry has adopted being gold standard for how an unremarkable product manager can iterate on something to make things better, it is a Vibes-based theory that if we let an random average person, be it product lead, engineer, UXer, or intern ship as many things as quickly as possible, something has to stick and the waste will somehow sort itself out. It's a claim that all the things we fret about damaging that are hard to measure like user trust, user learning and the stability of their experience are just nits that can be papered over because we're gonna ship something so awesome user won't even think to complain.

My greatest worry is that the industry is currently not set up to stop the people who are completely throwing rigor and skepticism out in the name of velocity. The current system was created in an environment where shipping a major feature was so slow and expensive that people wanted research and analysis to give them confidence that they were investing into a decision that had a "better than chance" odds of success. The current system isn't built for ... whatever this is.

To make things more... exciting... the interesting bit is yet to come. These two opposing viewpoints of the world are currently in the process of clashing and figuring out a new equilibrium. Neither extreme is going to survive the moment unchanged. At some point, reality is going to rear its ugly head and provide feedback as to what extent that each viewpoint is working out and everyone is going to have to adapt to some new, uncomfortable, state of the world.

If our claims for the scientific method are supposed to bear out, then the wholesale ignoring of the scientific method is supposed to lead to poorer decisions that affect a company's bottom line. The problem is whether we can measure and show these effects. If launching crappy features and mounting outages from LLM generated code actually generates reputational damage that hurts sales, it's going to be a pretty arduous case to show with data. Same for launching a feature and seeing the eventual wave of people complaining that it doesn't work and/or stop using it. It's like the old adage about how disinformation travels faster and wider than the truth, and by the time the fact checks come people's attention has moved on. There's an asymmetry in how this is going to fold out.

At the same time, the slow pace of the scientific method side we stand on isn't immune to criticism either. Do all the metrics, all the experiments, and all the friction involved in understanding users and products actually needed to make good decisions? At what point is a lot of the work a very elaborate game of corporate cover-your-ass? I've criticized organizations before for letting the appearance of being "data driven" lead to pathological risk minimization. There's a set of metrics and checkpoints that we can't measure effectively enough, quick enough, to satisfy the needs of the business and we need to learn which things can be thrown out as being ineffective, and which must be kept.

At the end of the day, I think after the dust settles between all viewpoints, the people building product need to be taking calculated risks balanced by measuring important metrics including important things that necessarily lag. And we're going to have to have conversations that tie decisions to outcomes despite all the AI-driven decision churn mucking up the waters. I don't expect it to be easy. I honestly am expecting that some series of giant explosions of epic proportions, like the weird situation where AWS blamed its SWEs for allowing AI tools to introduce code that caused outages instead of the AI tools themselves (?!).

Until things get sorted out down the line, all we can do is keep asking how do we know if what we're doing is correct. Luckily, we've been training ourselves to ask this same question for a long time now.

Subscribe!

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

randyau.com — homepage, contact info, etc.

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
Send a one time tip (feel free to change the amount)
Share posts you like with other people!
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!