Muscle memory in data work

Supporters of the newsletter get these off-the-cuff posts every other Thursday where I write about whatever's on my mind.

For much of my early career working as a data analyst, I had a peculiar habit – for daily ad hoc DB queries for random questions I'd get throughout the workday, I always rewrote the queries from scratch whenever possible. I pretty much did that for everything unless I was trying to exactly reproduce an analysis where I needed to remember the exact details of a nuanced query. I'm sure people who regularly use saved queries, cookbooks, or dbt are gasping in horror right now.

But hear me out. I had a pretty simple reason to keep doing this for the relatively simple queries I was getting – I knew I needed practice to drill the most common data patterns into my head. That is to say, I really know myself. I knew that if I just had to copy and paste snippets of queries together, my brain would very quickly shut down and just stop trying to understand the underlying data that I was working with every day. I needed to stay on top of things because in the small startup I was working for during that time, being able to bang out new and accurate queries on the fly was valuable. It was extremely useful to have a product lead or exec come to ask me a question and I just have them sit down beside me while we go explore the data together.

And so, in my opinion, having muscle memory from constant deliberate practice is very useful. And it doesn't apply to super mundane, repetitive things like remembering which fields to check for in a query. There's other aspects about doing creative work that benefit from practice.

For example, for the first 13 years or so of my career, I simply could not come up with a useful talk proposal to ever send to a conference or even internal work event. I'd just stare at a sheet of paper while filled with doubt and constantly draw blanks or think in useless circles. But after practicing thinking up long-form topics to write about for the past 150 weeks, this is not a problem any more. To my surprise, the act of "being creative within a space" is also a skill that improves dramatically with practice. I sure wish I learned that lesson earlier in life. I'm also sure every single artist in the universe just rolled their eyes at me for this seemingly obvious 'revelation'.

Another skill is writing decent business emails that people are willing to read. Think of how many years you spent writing emails where the recipient answered the wrong question, ignored action items, or just flat out misunderstood everything. I had occasion to stumble onto some business-y conversations buried deep in my email archives and the difference a decade makes is pretty amazing.

Amusingly, my presentation style has benefitted from practice insofar as my content is tighter and my delivery better. But my design skills are still absolute garbage and at this point I expect to be using mostly black-text-on-white-background until it's outlawed. It's very telling that continue to put almost no effort into getting better at making slides visually appealing.

Returning back to practicing at data work, one thing it does is put you in an extremely awkward position over another major aspect of our jobs – automation. I am very obviously making a decision to avoid automating a certain class of "easy tasks" so that I can eventually be faster at writing queries on the fly. I admit that make decisions on what to automate and what to manually rewrite pretty much entirely by vibes. Things that become too frequent and annoying get automated, things that don't remain manual.

So anyways, I don't have a grand thesis on whether my way is a better way to work than anything else. Tools like dbt solve very well-documented problems in the analytics space. Obviously manual query writing introduces a fair amount of variation and risk of errors. For the context I worked in, the cost of errors in ad-hoc analyses were cheap and easily ignored. This doesn't quite apply to my current job and so I find myself writing queries unaided a bit less often.

But it's a tension I always revisit and think about whether I need to shift more towards one direction or the other.