The basics of project management for data folk

Whoa, time flies. I'm supposed to be giving a talk (more like a rant) about doing experiments at Quant UX Con 2024 on June 12-13. Tickets are available, including a free option for those who need it. Please attend if you're interested!

Work in data long enough and you're going to wind up taking ownership of a project. It might be a small little one that takes a couple of hours, or a giant multi-year effort with different phases and large teams. While "project management" can be a whole career path by itself, just about every data person essentially has to learn how to manage projects on their own through trial and error. That's of course less than ideal.

The problem is that projects are all different from one another. There's no way to teach solutions to every situation, every organizational need. Even the professional project management community seems to provide education as much by teaching common tools and techniques as well as studying a patchwork of case studies.

I'm not going to do the field justice with just one or two little posts, but I think I can at least summarize some of the major guiding principles that help me fiddle through my own projects.

Projects vs programs

It might sound silly to spell it out, but projects are defined as being distinct, temporary actions – they have a distinct beginning and, most importantly, an end. This is in contrast to ongoing efforts that some people call "programs" which are intended to keep going for indefinite periods.

The reason this detail is important is because it lets you work backwards from the end result. The ability to work backwards is probably the most important skill in handling projects. It's what lets you plan sequences of necessary steps and get estimates mid timelines. It keeps you focused on the goal and let's you cut out unnecessary scope.

Projects exist within a context

Every one of us has managed individual and group projects in our school days. We may have been sloppy about it, but we at least fundamentally understand how to order and execute on actions until a project is done.

So why is it that work projects feel completely different and so much more difficult?

My opinion is because school projects primarily focus on the basics of execution – getting tasks completed in the correct order. School projects all exists within a vacuum. There's no reason why a project must be done other than because the class requires it. There's almost never any competing concerns, no market forces, no changes in strategy, no people upstream blocking you, not people downstream waiting on your results.

But the instant we start doing such projects at work, we have to deal with these issues even if we aren't even aware those issues exist. The reason we get bombarded by frustrating things like status meetings, get nagged about timelines, get our scope changed, or have projects entirely cut with little apparent warning, is all because of the context surrounding the context.

Managing the context

Let's take executing on a project to be table stakes for making a project a success. What is this context that we're supposed to manage?

From my experience, a lot of the outside work involved in managing a project to completion at work is keeping your project moving in formation with everything else going on. That is to say, your project is just one effort in a disorganized fleet of other efforts going on in an organization and you're paying the price of coordination. Just like you are likely relying upon someone else to give out an input to your project, your project output is going to be used by someone else. Zoom out a little and your project is just one entry in someone else's own large project plans. Thus, everyone out of self interest wants to be informed of the status of things. This is especially true when a potential delay is going to impact their timelines.

The vast majority of the annoyance and overhead of running a project at work is the work of maintaining this coordination. Why spend weeks gathering requirements and making sure everyone agrees on what work is to be done? The seemingly endless status reports to various levels of management? The people asking you for time estimates? The varied reactions when delays crop up? Why you eventually get in trouble when you shirk these duties? All that is identical to a bunch of migrating birds honking at each other mid-flight to remind everyone else where they are in the formation. Unless you have a huge amount of trust and social capital, you can't just take a project, disappear for a few months without a trace, and then expect show up to a hero's welcome.

I'd argue that while the execution and delivery of a project's results is important, the coordination aspect of things becomes increasingly more important as you work on things that are more central to the organization in general. Big projects tend to need more resources and cooperation to get done. You're eventually going to have to delegate work to others. Contracts, vendors, clients might all be relying on the proposed timelines. With every expansion, the cost of failure keeps ramping up and important people are going to want to know if there are problems.

So given this reality, good project management means getting good at working in coordination with everything else that is going on. It's going to mean being proactive in communicating things that are changing. It means paying attention to and anticipating problems that are outside your scope of control – hence why you see people who run projects paying attention to the status of other team efforts. It definitely means learning how to manage expectations and negotiating what work needs to be delivered. Everyone knows that projects face difficulties and require course corrections as details become clearer, they just want to know how that affects everything else so they adapt their project plans as necessary.

It's all this context stuff that also determines whether projects get spun up or shut down. Some opportunity (or need) turns up that nothing is addressing currently, so a project is spun up to handle the issue. If at any point during execution, the organization collectively decides that a project isn't worth the investment in resources, it gets killed off. That determination is also done with a lot of bird-like honking.

You can imagine how muddled everything gets when everyone else is juggling the same considerations for their own work. It's projects all around. Everything's interconnected. Everything cascades.

I use this very high level understanding of how projects exist to help me understand just why I need to spend so much energy doing "not-execution" things.

And don't worry about PM Tooling

Managing a project tends to require organization and keeping track of details. If the main goal is to keep up to date on everything that is going on, it doesn't usually take complex bespoke software. For tiny projects where there's not going to be a lot of complexity and coordination burden, it's easy enough to keep all the details in your head, or use simple notes and a spreadsheet.

It's when things get too complex to reliably fit into memory that the much disliked ticketing systems get used. Many data folk are probably familiar with at least some of the basic work tracking software out there... the ubiquitous Jira, the lightweight Trello, or older software like MS Project, and even things like Bugzilla. All these tools are well known, and almost everyone hates them because of the huge amount of overhead needed to make them work. Few people like filling out forms with status updates, time estimations, and priority codes. Even fewer like going back to keep them up to date as things rapidly change.

As data science people, we are usually the ones who dictate which tools are being used. We just use whatever's made available by other teams. Regardless of WHAT is being used, the tools are only as valuable as the energy you put into making them useful. They're all varying degrees of terrible, just make do with what you have.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something a data-related post to either show off work, share an experience, or need help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted, so support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!