Don't expect data to change everything like magic

TopPost Apr 9, 2024

O_o the trip to see the eclipse upstate was largely a success... except for the part where I'm driving >9 hours in a single day. So this week's post is on the shorter side because I need rest.

One of the things that's quite persistent when companies and people want to talk about their successes in using their data is how they "transformed their business" with data. Case studies and marketing copy abound with an almost infomercial-esque before-and-after snapshots of how things were terrible before new data practices were implemented and then the massive changes in cost, profit, whatever, benefits that materialized after. Today, I'm not here to say those claims are lies, but that they are generally very rare, drawn out events – so rare that anyone working in data will honestly be lucky to participate in a handful over their entire careers. Expectations should be set accordingly.

There seems to be two major reasons why participating in a big data revolution is rare. First, step function type changes from a single piece of insight are extremely rare. Second, other transformations are the result of steady work and iteration.

Step function changes are rare

This is usually the scenario we dream about – data person walks into a company, pulls some data queries and analysis, finds something odd and reports on it, maybe builds a model. That bit of information takes hold and inspires massive changes across a company. Charts go WAY up and to the right.

For one thing it plays into the massive ego trip of a random data person coming in and being smarter than all the people who were working full time on the business just by using "algorithms". Reality is much different. The chances are extremely low that tons of staff who have been working on a shared problem for years will have systematically missed something about how the business functions that a simple data process can pick up. Betting your career on doing "that one weird trick" is akin to those articles about how a physicist or economist has "discovered" some aspect of another field by applying a method from their home field and think they found a super novel result while the actual field has already known about that effect for decades.

Usually, if you're going to find these sorts of eureka moment insights, there needs to be some kind of systematic blind spot in the organization. For example the one time I can make a claim of helping transform a business significantly, it involved looking at user behavior that was a largely neglected priority for 10 years and amounted to saying "hey we shouldn't ignore the needs of this very large user segment". At the time, I had already been working there about four years and stumbling upon that nugget of wisdom was entirely by chance.

It's always a possibility that any given organization has a systematic blind spot. Humans tunnel vision all the time, and there's always competing priorities to juggle. The question is more a matter of degree – just how much bang are you going to find from unpicked low hanging fruit that other people haven't noticed? Does that amount of "transformational change" or just mere "improvement". Can you even predict what the impact will be before hand?

Building programs is slow work

If finding miracle silver bullets is rare and unreliable, then the only other way that data will change an organization is through stacking on improvements. In some aspects, this is a lot of the work that we normally do – an experiment here, a new process there, a new bit of infrastructure as needs arise, a refactor when things break. The individual effects may be small, but if you get new processes to stick, you might find compounding effects.

But at the same time, as you're probably familiar with, the impact of every little project will vary a lot. Sometimes you get a big effect, other times nothing changes. Other times the business goes up or down completely unrelated to whatever is going on with data work and that shift can noise out anything else you're doing.

Unless you have experience with how things were before data programs were initiated, and then remember to compare the present with that past, it's impossible to tell a improvement story let alone a transformation one. This places a time constraint on things – you need to have a big enough impact quickly enough that it's memorable.

Unless you are in a position to push projects through and not busy doing front line analysis work all the time, it's really hard to get velocity since all improvements gets interwoven with other work. That spreads out the projects over more and more time. Plus, how many opportunities do you have to embark on an unbroken 2-4 year program in your career?

So yeah, it is extremely hard to do any of these things on purpose. My theory is that very often organizations spend time making small data improvements – fostering a data-driven culture, infrastructure, learning from research and experiments... Then somewhere along the line the stars align where everything clicks together around a project and things are noticeably different. It's just hard to know when things are going to click.

So in sum, we really need to tell people joining the data field that they're not going to magically change everything by just arriving and doing simple stuff. There's no free gold just lying around on the streets. But instead, the steady work that we do every day can contribute to big things over time. If we keep good records long enough to notice the difference.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted, so support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
Share posts you like with other people!
Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!