Logs analysis from a bygone era, the command line

A continuation of the logging series. I just needed to this out of my head and onto paper first. I expect to be on other topics next week. See the Day1 slides available in the supporter's area in the site top nav for more specific technical bits about CLI-based analysis that won't fit in a short post.

Back in the mid 2010s, I was just starting to learn how to do "Big Data" at Bitly, the link shortening service. It was a great training ground for that because SQL interfaces for the Hadoop ecosystem was barely getting off the ground. This meant that in order to answer a bunch of simple questions I needed to write MapReduce jobs manually using Python, golang, or shell scripts. Obviously that was a bit of a pain in the butt.

So naturally, being lazy ... I mean, energy efficient – for smaller and simpler problems, I would do basic event counting and analysis using command line tools against one or two compressed log files. It was often faster that way for the small tasks, and certainly required less brain power.

Nowadays doing analysis using the CLI has become something of a party trick because basic SQL interfaces can do pretty much all the same operations with a more flexible syntax.

But there are two places I can think of where learning how to do it is helpful.