(Not) Building for the long term

My whole 15+ year career is grounded in startups. Even my current position at a giant megacorp working for one of the major cloud providers, the vibe and pace of work is most definitely more startup-y than "mature corporate". This is somewhat a deliberate choice on my part because my hyper-generalist skillet and high tolerance for fast-paced chaos means I enjoy having lots of opportunities to work closely across job functions. It's a personal character flaw that some people share, while many others don't.

Last week, a teammate asked the group what methods/tools we used for long term planning when a lot of our day-to-day work can be measured in mere sprints or shorter. While long, multi-year plans are difficult to execute on no matter what organization you're in, it's especially difficult in startup environments where strategic goals might shift over might without any warning.

So if you ask me what my long term project plans for work is... I don't know. Do I have projects that I am currently working on that will probably stretch across multiple quarters or even years? Yes, but I don't have actual "plans" and "visions" for where they'll be in 2027.

I'm not a long term planner at heart and my life journey made of patchwork decisions reflects that. It's not "optimal" in any sense and I'm sure anyone who does know how to plan even a modicum for the future is somewhat appaled. But for better or worse, it keeps me moving forward.

Think rock climbing, not driving

Most typical advice that you find about planning and goals involves some form of setting a goal, like "build a giant new data pipeline infrastructure thing" and then work backwards into figuring out the things you need to get to that point. A methodical planner/leader type person would then put together a grand design of the thing, use their persuasive abilities to convince other people of the need and value of their proposal, put together a plan and timeline, and get the organization to put the resources behind executing that plan to completion.

This is obviously takes a lot of different, complex skills and the ability to imagine a future and the journey needed to get there. It is the equivalent of planning out your roadtrip by reserving all your hotels a month ahead of time. I know some people somehow manage to do it this way and succeed (even if I'm exaggerating a bit for effect), I do not understand how.

Obviously, these big plans aren't set in stone because things always change and adjustments need to be made. It's just that if you subscribe to the "write the whole route down" philosophy, then you're also going to commit to writing down new updated plans that account for the changes.

Since I'm not good at doing any of that, the way I tend to approach "big projects" is to break them down into useful self-contained chunks that have these properties:

• The chunk of work is small enough to deliver sorta/kinda quickly so that people don't get annoyed that I've run off for a year
• The chunk is valuable on its own merits, even if it subsequent chunks don't ever get built
• The chunk is potentially useful for other projects, it's not solely tied within the big project

As an example, imagine if I thought the company e-commerce site could use some kinda recommendation engine. I could dream up one that pulls in data from all internal sources, external data vendors, social media, and it all passed into a model that recommends the perfect bar of soap to sell the customer. I could do a full sized plan for all of this scope, but probably the most important thing to build first is the "put a recommendation in front of the user" part. If I build that part right, we could feed it data from my complex model infrastructure later. Or, if we change priorities in a few months, having a recommendation-aware widget is likely to be useful at some point. Having a working example of how the pipelines work will be useful. So completing that chunk of work would get me closer to my big plan, while being tolerant of the risk that we lose interest in the near future. It's useful both now and later.

It's something like more similar to what I think rock climbing is like? (Disclaimer: I've never done rock climbed before and dislike heights). You might have an overall plan of where to put your hands and feet to climb up a cliff, but you'll likely discover the plans don't work out and you need to make adjustments. Every step, every foothold, hopefully gets you closer to the goal, but you never know as you make adjustments based on existing conditions.

The point of breaking things down this way is because fast moving organizations change so quickly, I have a whole graveyard of abandoned or "de-prioritized" projects in my past. But few of those projects are considered total failures because we got use out of the small pieces that I did manage to complete. And if opportunities arise later on, these pieces often yield some value to build upon, so the time wasn't completely wasted. A preliminary study about how a certain user segment showed that they're not that important today, but since we have a clear method for identifying those people now, we stand ready to use it when they become relevant. A set of slides to teach UXers how to write good bug reports will be good reference regardless of whether we run training classes or not. My crontab based email reports were all the infrastructure we had time to implement for years but everyone read the things and the logic could easily be reused in a proper system.

Not all work lends itself to chunking in this way. Sometimes you just gotta do that from-scratch rewrite that takes a while to do. But a lot of things can be rearranged into useful, semi-independent modules, from research studies, to code and infrastructure, to migrations, to organizational processes.

There is a balance that needs to be struck between executing on a well-defined plan and taking things step by step as they come. It makes the most sense to adopt an adaptive strategy when you are less able to control your own destiny – like if you're subject to market whims every other quarter. But as your experience and leadership skills grow (along with the social capital needed to make bigger organizational changes) you should be able to do more of that top-down planned style of work because you'll have the ability to convince other people to build and deliver things. That sort of far-reaching coordination requires actual clearly defined plans because all the people cooperating need to all agree on what they're working towards. It is much more difficult to just "wing it" with multiple teams who don't all fully understand what needs to be done.

But if you're on most data teams of size 1-5, you are most likely going to be doing much more rock-climbing style work where often pays off to take a flexible approach.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!