We're all carriers of software memes

I got to inflict AWK on a group of people today! The inner nerd in me had a complete field day at the first logs class. By the sense of how things are going, I'm going to work on adapting the overall class into more compact, easily digestible chunks that I can share more broadly to subscribers since in person I can voice over all the flaws and inconsistencies of the slides. Subscribers can look forward to this in the near-ish future.

One of the topics of discussion that came up among during the day was... how the heck does someone wind up learning this crazy sort of tooling way back in the late 2000s? Even in the late 2000s, awk was definitely not a preferred or even a candidate solution for most problems that ever came up. You'd have to have had some exposure to it in some Computer Science course or been browsing the O'Reilly zoo. Why am I even bothering to teach, even at a surface level, the existence of this stuff?

I think for me, these weird little topics are essentially the result of history and path dependence. Our computers and software is this huge, vast, complex space that is far larger than any one person can comprehend. That means that solutions found for any particular problem are unlikely to be optimal. If you're a complete weirdo like myself and have browsed Wikipedia pages of various Unix commands, like for more, patch, the bourne shell, and so on, you'll notice that a significant portion of the page is dedicated to the history of the program. Same goes for programming languages, operating systems, and all sorts of other things from that early era of computing. This pattern carries on for other major bits of important software.

More interesting, very early discussions of software will show how the stuff can trace lineage. The history of the terminal rogue-like game Nethack shows this complex history of the very small group of people who had privileged access to computing resources back in the day.

NetHack is a software derivative of Hack, which itself was inspired by Rogue. Hack was created by students Jay Fenlason, Kenny Woodland, Mike Thome, and Jonathan Payne at Lincoln-Sudbury Regional High School as part of a computer class, after seeing and playing Rogue at the University of California, Berkeley computer labs. The group had tried to get the source code of Rogue from Glenn Wichman and Michael Toy to build upon, but Wichman and Toy had refused, forcing the students to build the dungeon-creation routines on their own. As such, the game was named Hack in part for the hack-and-slash gameplay and that the code to generate the dungeons was considered a programming hack. After their classes ended, the students' work on the program also ended, though they had a working game. Fenlason provided the source code to a local USENIX conference, and eventually it was uploaded to USENET newsgroups. The code drew the attention of many players who started working to modify and improve the game as well as port it to other computer systems. – History section of Nethack on Wikipedia

So, the short of it is that I'm teaching awk... to do analysis of data logs from the command line, because I had learned it from an engineer, who likely learned it at school or self study. We both wound up using it occasionally because it was more suitable for the task at hand than any other alternative. And now, I'm passing that same knowledge on (albeit in abbreviated form) because I feel that knowing bits of that rare knowledge will prove to be useful in some unpredictable way in the future. We are just a bunch of humans doing what worked for us, and sharing it with other humans who may later find it useful enough to transmit to yet another human. That's sorta how society works.

Even if you're not going out of your way to teach quirky niche programming languages, you're very likely doing something similar in your domain of expertise. I've noticed that the SQL people write is generally very unique. There's so much room to write equivalent code in SQL that my way of writing a lengthy query will significantly differ from someone else's way. Someone familiar with both our code would likely be able to make a guess as to who wrote it. From how the code is styled or even indented, to what kinds of joins are used and whether I tend to put equivalent logic conditions in the WHERE clause versus the ON clause. Whenever I teach someone to write SQL, they inevitably will pick up at least some of the habits I have due to all the exposure they get from my code. I'm effectively a vector for SQL styling memes.

Even now I know of the occasional developer teams that use Perl, Haskell, or Lua, or some other niche thing for a lot of their production work, when the vast majority of developers nowadays would consider it pretty nuts. But if you effectively came of age in such an environment, people can obviously make it work.

All this ranting here is to say is that, we all are a nexus of tooling quirks that are wholly unique to us thanks to our personal history and experience. We take our patchwork of skills and use it to do things we want to do. The stuff that's successful gets reinforced over time, and then we inevitably will share our tips and tricks to folks in the next generation. For me that included sed and awk today. Tomorrow it might be my general avoidance of using too many CTEs when in-line strategic subqueries would do. Another day I might dig out my VBA skills and help someone commit data horrors in Excel.

I have no idea what quirky combination of useful knowledge you have within you that only your specific history will allow to blossom, but I do know it's there. You very likely don't think it's particularly interesting or glorious – I'm not super proud of the things I did with VBA Excel for example – but if you ever share it to with someone you'll get very fascinated reactions from people. And I'm pretty confident that you're going to wind up sharing it without even being conscious of the fact – because, the endeavor of software and data is, despite all the technobabble and appearances, still human. Sharing is human. As is history and funny traditions.

There's no guarantee, or even promise, that "the best practice way" is all that much better anyway.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — Curated archive of evergreen posts. Under re-construction thanks to *waves at everything

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!