Dysfunction is still dysfunction, regardless of how it got there.

Just some habits I've built up

Employers are being very picky right now and it sucks

Things are gonna be okay

A rant about failure and magical thinking

Don't see "Full-stack" flying around that often any more

I always told my wife this newsletter was my backup plan...

The first steps at making things work, in my usual scrappy way

I didn't see them everywhere until I really started looking. I had forgotten they existed.

Apparently we're not in a field that makes it easy to get rich quick =\

No one ever thinks of data science as "customer facing", but that doesn't mean we're never...

Thoughts on how people should teach and learn analytics based on the question at hand, and not the other way around.

It'll be in person, and it's gonna be fun! I promise at least that much.

An update for an evolving job title.

An update for a year of changes. In hopes anyone wanting to write more can get something out of it

Planning is hard ot do, especially at fast-paced startups. So how about you spend less time planning and more time breaking things down into useful chunks?

Probably a lifesaver if you're in this academic field, especially if you've never collected data before. BUT the later chapters are relevant to people of all skill levels and experience, even in industry.

A guide to things that aren't worth our time and we need to bap the requests that try to make us work on them.

Data work tends to mean getting inconclusive results, which are really inconvenient in industry because they're... nothing... and it's quite inconvenient

I admire good dataviz, I also know that I'm hopeless at those. So instead I've found a place where it's good enough for the work I do.

There are no shortcuts to writing something worth reading

If you use it all the time at work, you're probably already in the upper percentiles in skill.

Domain knowledge is critical to dat awork, but we really do get thrown into the deep end at times. Why does that even work at all?

More fields are getting on the whole LLM synthetic data generation bandwagon. Honestly at this point the best career move seems to be in learning how to pick apart and vet LLM systems instead of fighting them all the time...

It's impossible to explain project management in a single post, but the general framework for understanding why work projects feel so objectively different from school ones is helpful to make sense of it all.

Consumer products are fairly straightforward and interesting, but I guess I like the weirder stuff.

Many people joining the data science field harbor the belief that they're gonna help change how things work, make stuff magically better. Sadly, we all learn that it's much harder than that.

Making transitions smoother

Good for people starting their analyst careers

TL;DR: Spread love, joy, or at least helpfulness

Go out, take a breath, and see what other industries are doing

I'll do it. You should do it too!

Working around myself

FINALLY

During slow stretches

On Nov 02, 1:30pm

Maybe it's a thing

Despite it being critical to our job

Gotta learn to handle it

Because its different

But it's not just me

Before things spin out of control

But you can easily become one if you're not careful

If you like execution-type work like me

Just like code reviews

The corollary to this, of course, is that you know something that they don't!

Going with the flow

Can take a lot of unexpected paths

In case "Data Twitter" becomes just "Data"

It's either a bit of work, or a lot of work

Because the hardest part is starting

Instead of mastering a thing

Life keeps happening

For something more general

One day you're going to have to

It's a mess because everything's undefined

Of at least something

There's soooo much overlap

Mirrors upon mirrors

Not all of us sits at the big table

And stop obsessing on surface technical skills

It can be done... with effort

And that's a good thing

Practice practicing!

Does one even come before another?

Because, surprisingly, it can

Looking at things from the other side of the table

A mini case study

A bit of reflection

Is something of a niche. Heh.

Because you'll be buried otherwise.

We rarely talk about this to newbies

Sorry, brain's overloaded

This job will use everything you are

The Human Experience

Not everyone's on the bleeding edge

It's hard until you learn to handle yourself, and then it's still hard

Generating value upwards

A past experience dump

For (your) sanity's sake

Everyone tends to complain about their work

I still can't find it at times

It's complicated

It's a sliding scale, not an end goal

Work has conditioned us to

It's dangerous to go alone! Take this.

Don't expect to land the title easily, heck, don't even go for the title

We seriously don't need that sort of nonsense

Be Aware of the Power You Wield as a Data Person

It needs to be said regularly —  it’s OK to be nothing like them

Not all AI/ML ethical dilemmas involve life and death…

Not exactly Data Scientist, but close enough you can barely tell the difference. It’s a fairly obscure position. This is, in broad strokes…

I think it's time DATA got a FORUM. Party like it's the mid 2000s!

TL;DR is, your job as a reviewer is to help the reviewee come up with better code than they initially did.

Updates!

The audience for messy, one-off examples seems larger than it initially appears. If we are only willing to listen

It's what the internet needs, and we're the only people who can create it for ourselves and our peers.

Social media has got plenty of positive publication bias, and it's a good thing data folks are slightly more open to talking about failures. Slightly.

Because I don't know what else to do...

A reminder that a community is made of all of us together, including you. We all need to act out the community that we want to be in.

Dare I hope Data-bluesky will become a thing?

Old person revives ancient ritual that connects related sites from data writers. Asks you to submit PR and join.

A brief update on what's been going on at the newsletter. Nothing crazy.

A full summary of all the talks from Data Mishaps Night 2024, lightly anonymized for privacy.

Who would've thought scheduling random meetings would be hard?

TL;DR: Spread love, joy, or at least helpfulness

Go out, take a breath, and see what other industries are doing

However much I hate them, you can't avoid them

Talking about using our tools instead of the tools themselves

The corollary to this, of course, is that you know something that they don't!

Or just for yourself

Yes, you too

The Newsletter Version!

In case "Data Twitter" becomes just "Data"

Because the hardest part is starting

Because knowledge gaps are hard to spot

A bit of dogfood reporting

A mini crash course

Of at least something

and it's no one's and everyone's fault

Nerding out on counting

And I'd like to help

Not a "What happened?" post no one needs to read

Despite it being our primary job

And we're going to absorb even more. MORE!

Sorry, brain's overloaded

Last post of the year!

But worth it

This job will use everything you are

Some of them, at least

An 3-month dogfooding update

I want more people to talk to dangit!

What else can you do =\

And as a teaching tool

And other stuff about teaching stuff

It'll be fun, I promise!

A fun little (online) event for data folk

It needs to be said regularly —  it’s OK to be nothing like them

And how I use it to build partnerships with cross-functional teams

Some post travel reflections on urban infrastructure and age and how changes happen to infrastructure

Many would describe setting metrics as a art and a science. This is about the artsy part.

It's not just the "blessed" few who get to work on novel data problems. Every one of us likely does so at least a couple of times in our careers. We just don't realize it.

I think it's time DATA got a FORUM. Party like it's the mid 2000s!

Sometimes, people can actually tell when vanity metrics aren't useful

A useful tool that's mostly unused because people usually don't like what it has to say.

TL;DR is, your job as a reviewer is to help the reviewee come up with better code than they initially did.

Dysfunction is still dysfunction, regardless of how it got there.

It really shows when people don't do this

Warp speeeeeeeeed

Just some habits I've built up

Starting the year with some philosophical wandering!

Nothing wrong with building sandcastles, right?

Employers are being very picky right now and it sucks

Things are gonna be okay

Don't see "Full-stack" flying around that often any more

We're all just stumbling around doing our best, I guess

Also the first batch of confirmed speakers listed

Goodbye BLS as we knew you

Somehow google trends for the -topic- of "real-time data" spiked up in the past year and it's just really odd

The first steps at making things work, in my usual scrappy way

We can do that?

Please, let's celebrate doing things the ugly way. For our peers that do things justly as ugly.

Apparently we're not in a field that makes it easy to get rich quick =\

No one ever thinks of data science as "customer facing", but that doesn't mean we're never...

While we usually want to avoid touching daylight savings time changes in our datasets, when you're working with people, local time actually does matter.

Moving fast and breaking things is the name of the startup game. And as data folk we're called upon to provide guidance while neither we nor anyone else have any real idea what we're doing.

Metrics frameworks are great tools to help teams think more about how their decisions potentially affect important parts of a business. But we often have to help teams get the finer details.

As analysts, we push data systems until they creak and groan. Then we modify them to not do that. Then repeat. Here I'm just thinking through the process.

Social media has got plenty of positive publication bias, and it's a good thing data folks are slightly more open to talking about failures. Slightly.

The default state of data systems is to assume they're cooperating with our goals, if not just neutral. It's worth remembering that there are systems that are downright hostile to our use.

A reminder that a community is made of all of us together, including you. We all need to act out the community that we want to be in.

Dare I hope Data-bluesky will become a thing?

A guide to things that aren't worth our time and we need to bap the requests that try to make us work on them.

Sometimes product teams constantly hear a feature request and then think that everyone will adopt it upon launch. They are then very very disappointed.

Old person revives ancient ritual that connects related sites from data writers. Asks you to submit PR and join.

Big shoe brand crashes in value due to a big strategy shift that relies heavily on data... so maybe we should pay attention to it a bit.

A text transcript of my Quant UX Con 2024 talk for those who prefer text over video.

I've been on data democratization efforts countless times now, and they've usually failed despite a lot of coaching and guardails... so maybe I've been completely missing the point?

For this Thursday's Subscriber post, some thoughts bouncing around my head that might eventually turn into a sane post but needs some thinking-out-loud to see where it actually goes.

Post for subscribers -- I think the way we still want our many data tools to interop via common file formats is a sign of dynamism and optimism of future innovation

Many people joining the data science field harbor the belief that they're gonna help change how things work, make stuff magically better. Sadly, we all learn that it's much harder than that.

"How hard can it be?" – these are the famous last words of many a data science project. We try not to underestimate the difficulty, but can go too far. Sometimes we just need to know the simple version.

Not everything is rational =(

It's something we "deliver" but what is it anyway?

Working around myself

Where are the limits? Edge cases?

In ways that I want

Data products, all products

Gamedev, Counting, and Ethics, oh my!

Took me a while to learn this one

And your product folk might not know that

Because we're not selling products

Because its different

Or a month? Averaged?

It can have it. Please.

Why are we even effective at anything?

It's all vibes

If you like execution-type work like me

Because of course it'll happen some day

And then catching flak for it

And they look nothing like the ones I make...

That we need to balance out

Yes, you too

It's hard to describe and very branchy

In case "Data Twitter" becomes just "Data"

It's easy!

Everywhere! EVERYWHERE!

Would it be so bad?

Beep Beep~~~

Because what choice do we have?

Humans break them

Life keeps happening

The insignificance of significance

There's a lot

A mini crash course

It's a mess because everything's undefined

Of at least something

Nerding out on counting

There's soooo much overlap

Flashbacks of past pain

Os different from a lot of other work

Mirrors upon mirrors

It's messy because we're messy

It's not ALL waste. Just mostly waste

Not a "What happened?" post no one needs to read

It's not all about big, complicated systems

Despite it being our primary job

And we're going to absorb even more. MORE!

And it's a giant mess

Out of all the things we can do, so far, working with deep, buried assumptions is probably the hardest to learn and teach

Every so often I just have to indulge in this topic

It's gotten easier to screw up someone else's survey!

I should know better, but even I catch myself doing this.

Why is the real world so ... messy =\

Gyah deadlinessssss!

How cloudy is a sample of water or air? Well, someone's figured it out, right?

The bad habits of data over-collection run deep.

The hot stuff right now is labeling with LLMs, but y'know, maybe consider not?

The default state of data systems is to assume they're cooperating with our goals, if not just neutral. It's worth remembering that there are systems that are downright hostile to our use.

Probably a lifesaver if you're in this academic field, especially if you've never collected data before. BUT the later chapters are relevant to people of all skill levels and experience, even in industry.

Data work tends to mean getting inconclusive results, which are really inconvenient in industry because they're... nothing... and it's quite inconvenient

A guest post about actually collecting data points in the field and coming up with categories that make sense for collected items.

A short post looking at a 2015 paper that tracked households over a year, with PCR swabs and symptom diaries, to see how often households had certain respiratory infections.

More fields are getting on the whole LLM synthetic data generation bandwagon. Honestly at this point the best career move seems to be in learning how to pick apart and vet LLM systems instead of fighting them all the time...

The most instinctive reaction to wanting data is to collect and store as much of it as you can, figure it out later. For many reasons, that's probably a bad idea.

Pulling, aggregating, categorizing, and interpreting public data is always a hairy task with lots of details. Here, Howe shows an example of pulling startup funding data w/ the help of AI classifiers.

I've been on data democratization efforts countless times now, and they've usually failed despite a lot of coaching and guardails... so maybe I've been completely missing the point?

April 5th 2024 brought a rare earthquake to the Northeast US. That opens up a big rabbit hole for looking at the roots of the USGS "Did You Feel It" survey.

Good for people starting their analyst careers

Mis-care of plants and miscommunication of information

It's dangerous to go alone...

Working with Data Quality

And some examples on how to pick

And other duplicated row situations

Involves forming an often bad opinions

We can't help but sell this one

Because of course it'll happen some day

Data nerd bait

Because it is/was perf season

It's easy!

And no (public) resources for working with the data yet

Humans break them

Because it's so easy

There's a lot

We are destroyers and preservers

As a data nerd, pretty interesting stuff

A reminder to us all

It's just bad news for everyone involved

It's sorta by definition

Taken to extremes!

You need to. Seriously.

Shooting slightly less of your foot off

Cleaning data is important!

In all sorts of ways. Except the butter kind.

Metrology!

At what point are you completely useless?

The dream...

Failures ahoy!

It's fairly realistic, albeit messy, which is the point

Also, most data cleaning articles suck

There's no escape

It's always been important. It's just a bit more important these days.

With sketchy charts!

It's dangerous to go alone! Take this.

What you can do when you actually collect email data

It’s not every day we can watch an expert explaining how hard counting is

Beware! They will not agree! The question is to what degree

This is supposed to save us all, it’s merely the best we’ve got so far

How we got here, how we’re not getting out yet, and dealing with it

Sure it’s a pain in the butt and has its own issues, but human brains are still amazing

In Production, things inevitably get wonky, and that’s normal

“Democratizing” data and insight, open access, data transparency, it goes by a lot of names, many want it, but the correct decision for…

99% chance you’re not ready

Know your data, where it comes from, what’s in it, what it means. It all starts from there.

Why is the real world so ... messy =\

The hot stuff right now is labeling with LLMs, but y'know, maybe consider not?

Subscriber post with thoughts about some of the dangers of giving people data and having them feel like they "know" how things work when there's plenty they don't know

A text transcript of my Quant UX Con 2024 talk for those who prefer text over video.

Industry tends to not care about theory, but we should talk about it a bit more because many aren't adverse to it when we talk about it right.

There once was a time where working with lots of data was so awkward, it became a giant hype cycle. Now, almost nothing is "big" any more.

There was a brief period in the early 2010s where data scientists were interested in bandit models... and then you barely hear about them any more.

Bi-weekly Thursday posts are primarily for subscribers and feature more in-progress type work and thoughts from Randy. This week is about experiments.

Nor vice-versa. We need the right tools for the job.

And how it connects to everyday life. sorta.

Or a month? Averaged?

It all starts looking the same

Os different from a lot of other work

Wait, what?

Modeling for incorrectness

With lots of photos

At what point are you completely useless?

The dream...

Doing things the incorrect way, for science(?)

But only in my brain

A lot of it involves making sure fears are addressed. The fear of the repercussions of failure. The fear of finding out they spent a…

Somewhere, someone’s having this conversation. Right. Now.

It’s nothing like at a big mature company.

A guest post about actually collecting data points in the field and coming up with categories that make sense for collected items.

There are no shortcuts to writing something worth reading

Learning new programing languages is always tricky. But using tutorials from a language you already know to learn another one makes things surprisingly easier.

The most instinctive reaction to wanting data is to collect and store as much of it as you can, figure it out later. For many reasons, that's probably a bad idea.

Pulling, aggregating, categorizing, and interpreting public data is always a hairy task with lots of details. Here, Howe shows an example of pulling startup funding data w/ the help of AI classifiers.

Or just for yourself

The weird ins and outs of procedurally generating a literal world for simulation purposes

Some pure fun to ring in the new year on

Seasonal trends! With CHARTS!

Through the magic of RADIO WAVES~~~~

So excited to be able to write about a random fun project again.

Every tool I can remember owning, and what I use it for.

I got nerd-sniped after being asked if it's possible to fly a plane for a day and keep up with the night side of the Earth.

Using tools to figure out where to go in all the chaos

Mis-care of plants and miscommunication of information

Including from the JWST and Hubble!

For nerds who need to see the whole process

HARD. WARES.

So. Much. Weird. Artwork.

I love hobbies. This is no surprise to any of you.

Doing things for the fun of it

With lots of photos

But worth it

Seasonal trends! With CHARTS!

Don't see "Full-stack" flying around that often any more

I didn't see them everywhere until I really started looking. I had forgotten they existed.

Please, let's celebrate doing things the ugly way. For our peers that do things justly as ugly.

The audience for messy, one-off examples seems larger than it initially appears. If we are only willing to listen

In every one of us, there's a little gremlin that learned something cool, silly, weird, but oddly useful.

Gyah deadlinessssss!

Thoughts on how people should teach and learn analytics based on the question at hand, and not the other way around.

There are ways to measure how much visible light something is giving off. It's not simple.

Data scientists have got "one weird trick" and it's called optimization. Luckily there's a surprising amount of depth to the trick.

It'll be in person, and it's gonna be fun! I promise at least that much.

The surprising depth needed to analyze bits of looped string

It's what the internet needs, and we're the only people who can create it for ourselves and our peers.

Social media has got plenty of positive publication bias, and it's a good thing data folks are slightly more open to talking about failures. Slightly.

Probably a lifesaver if you're in this academic field, especially if you've never collected data before. BUT the later chapters are relevant to people of all skill levels and experience, even in industry.

Says guy who writes more than most of the population... but hear me out.

I got nerd-sniped after being asked if it's possible to fly a plane for a day and keep up with the night side of the Earth.

A short post looking at a 2015 paper that tracked households over a year, with PCR swabs and symptom diaries, to see how often households had certain respiratory infections.

There are no shortcuts to writing something worth reading

If you use it all the time at work, you're probably already in the upper percentiles in skill.

Learning new programing languages is always tricky. But using tutorials from a language you already know to learn another one makes things surprisingly easier.

Domain knowledge is critical to dat awork, but we really do get thrown into the deep end at times. Why does that even work at all?

It's impossible to explain project management in a single post, but the general framework for understanding why work projects feel so objectively different from school ones is helpful to make sense of it all.

For this Thursday's Subscriber post, some thoughts bouncing around my head that might eventually turn into a sane post but needs some thinking-out-loud to see where it actually goes.

Not everything is rational =(

A full summary of all the talks from Data Mishaps Night 2024, lightly anonymized for privacy.

More venting out product ideas, but finding some live examples and course correcting

Making transitions smoother

It's something we "deliver" but what is it anyway?

Good for people starting their analyst careers

TL;DR: Spread love, joy, or at least helpfulness

Go out, take a breath, and see what other industries are doing

Just enough networking knowledge to do data work and debug your way out of tight spots.

Working around myself

And is it "enough"?

A PyData NYC 2023 talk

And how it connects to everyday life. sorta.

On Nov 02, 1:30pm

Including from the JWST and Hubble!

Took me a while to learn this one

But it's optional

Because thinking about LLMs

And arithmetic!

A just what the heck is it anyway?

Gotta learn to handle it

For the education!

Because its different

It's a scoring function!

For nerds who need to see the whole process

Involves forming an often bad opinions

Because we learn a lot that way

It's all vibes

Just like code reviews

Most of us probably don't even work at it

HARD. WARES.

The corollary to this, of course, is that you know something that they don't!

Yes, you too

The Newsletter Version!

It's hard to describe and very branchy

It all starts looking the same

Can take a lot of unexpected paths

Everywhere! EVERYWHERE!

Because the hardest part is starting

Before GPS or even lasers

Because knowledge gaps are hard to spot

So it's ok to crack things open to see

A bit of dogfood reporting

So. Much. Weird. Artwork.

For something more general

One day you're going to have to

There's a lot

I love hobbies. This is no surprise to any of you.

It's complicated

Nerding out on counting

And I'd like to help

Psst... run a survey

With tons of math going over my head

Doing things for the fun of it

Despite it being our primary job

And we're going to absorb even more. MORE!

And stop obsessing on surface technical skills

You need to. Seriously.

It can be done... with effort

Practice practicing!

For Science!

Does one even come before another?

Because, surprisingly, it can

Through the magic of RADIO WAVES~~~~

It does go on and on and on ...

Some thoughts about the intersection that keeps me up at night a surprising amount

Also the first batch of confirmed speakers listed

Every few years I forget and have to remind myself

It's easy to not realize we manage a DAG surrounding... ourselves

The first steps at making things work, in my usual scrappy way

I didn't see them everywhere until I really started looking. I had forgotten they existed.

So excited to be able to write about a random fun project again.

Gyah deadlinessssss!

As analysts, we push data systems until they creak and groan. Then we modify them to not do that. Then repeat. Here I'm just thinking through the process.

Every tool I can remember owning, and what I use it for.

It's what the internet needs, and we're the only people who can create it for ourselves and our peers.

Planning is hard ot do, especially at fast-paced startups. So how about you spend less time planning and more time breaking things down into useful chunks?

Old person revives ancient ritual that connects related sites from data writers. Asks you to submit PR and join.

Real world measurement means making compromises in the name of work speed. Because precision is time is money.

A guest post about actually collecting data points in the field and coming up with categories that make sense for collected items.

Just some thoughts about how refactoring software is nowhere near as complex as refactoring physical spaces.

I admire good dataviz, I also know that I'm hopeless at those. So instead I've found a place where it's good enough for the work I do.

Using tools to figure out where to go in all the chaos

More venting out product ideas, but finding some live examples and course correcting

Who would've thought scheduling random meetings would be hard?

Working on it!

Where are the limits? Edge cases?

And is it "enough"?

And some examples on how to pick

On Nov 02, 1:30pm

Including from the JWST and Hubble!

Maybe it's a thing

But it's optional

And arithmetic!

Because we're not selling products

For the education!

Talking about using our tools instead of the tools themselves

For nerds who need to see the whole process

Because we learn a lot that way

This is SO HARD!!!!

HARD. WARES.

It all starts looking the same

And no (public) resources for working with the data yet

So it's ok to crack things open to see

And I'd like to help

With tons of math going over my head

Doing things for the fun of it

And learn about learning about data sets

During a pandemic?!?!?

Failures ahoy!

But worth it

It's fairly realistic, albeit messy, which is the point

Some of them, at least

Because I doubt most people, myself included, is

But only in my brain

Mic mic revolution

Fun is hard

It’ll save your butt. Lots.

Many would describe setting metrics as a art and a science. This is about the artsy part.

When it comes to perception, our ears are as complicated as our eyes.

It's a really annoying turn of events but here we are...

Dysfunction is still dysfunction, regardless of how it got there.

I should know better, but even I catch myself doing this.

I've done enough shoveling this weekend. zzzz.

Why is the real world so ... messy =\

Starting the year with some philosophical wandering!

Nothing wrong with building sandcastles, right?

A peek into the frankly mind-melting complexity surrounding food energy measurement

There's a whole legend about how the carob became the carat and I wanted to chase that down with my own hands.

We're all just stumbling around doing our best, I guess

Goodbye BLS as we knew you

Another episode in the ongoing logs analysis primer series

Heatwave writing means overthinking heatwave measurements

How cloudy is a sample of water or air? Well, someone's figured it out, right?

While we usually want to avoid touching daylight savings time changes in our datasets, when you're working with people, local time actually does matter.

Moving fast and breaking things is the name of the startup game. And as data folk we're called upon to provide guidance while neither we nor anyone else have any real idea what we're doing.

There are ways to measure how much visible light something is giving off. It's not simple.

The bad habits of data over-collection run deep.

The surprising depth needed to analyze bits of looped string

The default state of data systems is to assume they're cooperating with our goals, if not just neutral. It's worth remembering that there are systems that are downright hostile to our use.

A guide to things that aren't worth our time and we need to bap the requests that try to make us work on them.

Sometimes product teams constantly hear a feature request and then think that everyone will adopt it upon launch. They are then very very disappointed.

Real world measurement means making compromises in the name of work speed. Because precision is time is money.

Meteorologists have a curious little metric called "accumulated cyclone energy" that they use to compare hurricanes and entire hurricane seasons. This is a dive into what that's about.

Subscriber post with thoughts about some of the dangers of giving people data and having them feel like they "know" how things work when there's plenty they don't know

A short post looking at a 2015 paper that tracked households over a year, with PCR swabs and symptom diaries, to see how often households had certain respiratory infections.

It's never been easier, and cheaper, to measure things in our world, just to actually do it and learn how annoying it can be.

A text transcript of my Quant UX Con 2024 talk for those who prefer text over video.

The most instinctive reaction to wanting data is to collect and store as much of it as you can, figure it out later. For many reasons, that's probably a bad idea.

I've been on data democratization efforts countless times now, and they've usually failed despite a lot of coaching and guardails... so maybe I've been completely missing the point?

Frameworks for doing things like setting metrics or identifying issues abound. They're useful tools but it's worth knowing now to take them too far.

There once was a time where working with lots of data was so awkward, it became a giant hype cycle. Now, almost nothing is "big" any more.

April 5th 2024 brought a rare earthquake to the Northeast US. That opens up a big rabbit hole for looking at the roots of the USGS "Did You Feel It" survey.

It's something we "deliver" but what is it anyway?

Mis-care of plants and miscommunication of information

And how it connects to everyday life. sorta.

Gamedev, Counting, and Ethics, oh my!

Took me a while to learn this one

A just what the heck is it anyway?

However much I hate them, you can't avoid them

It's a scoring function!

Involves forming an often bad opinions

We can't help but sell this one

This is SO HARD!!!!

It's all vibes

Data nerd bait

And then catching flak for it

And they look nothing like the ones I make...

Because it is/was perf season

For 100 years anyways...

Are they just wishful thinking?

It's easy!

A career of chasing ghosts

And then we'll have to work harder =O

Before GPS or even lasers

And it's everywhere...

Humans break them

Because it's so easy

The insignificance of significance

It's a lot

As a data nerd, pretty interesting stuff

It's complicated

Mirrors upon mirrors

A reminder to us all

It's messy because we're messy

It's not ALL waste. Just mostly waste

It's just bad news for everyone involved

It's sorta by definition

Taken to extremes!

Modeling for incorrectness

Because the possibility just got slightly more plausible

Almost bizarrely so

Apparently a huge amount

And learn about learning about data sets

Traditional units ahoy!

Metrology!

Learning from self inflicted pain

And collecting data to do that...

A past experience dump

There's no escape

Shocker: Stretching a painful thing out doesn't make it less painful!?

Not how we deal with it ourselves

The History of a little cup

If you're not good at Viz, like me, there's hope for us

Fun is hard

What you can do when you actually collect email data

Black boxes are hot again

It’s not every day we can watch an expert explaining how hard counting is

It’s surprising to think we rely on this as much as we do

Beware! They will not agree! The question is to what degree

The art is finding a good cadence for your metrics

It's a really annoying turn of events but here we are...

Warp speeeeeeeeed

Somehow google trends for the -topic- of "real-time data" spiked up in the past year and it's just really odd

We can do that?

Metrics frameworks are great tools to help teams think more about how their decisions potentially affect important parts of a business. But we often have to help teams get the finer details.

Big shoe brand crashes in value due to a big strategy shift that relies heavily on data... so maybe we should pay attention to it a bit.

Frameworks for doing things like setting metrics or identifying issues abound. They're useful tools but it's worth knowing now to take them too far.

Nor vice-versa. We need the right tools for the job.

In ways that I want

Gamedev, Counting, and Ethics, oh my!

And your product folk might not know that

It's a scoring function!

We can't help but sell this one

Data nerd bait

Most of us probably don't even work at it

Are they just wishful thinking?

Beep Beep~~~

Because it's so easy

Wait, what?

It's messy because we're messy

It's sorta by definition

Having an opinion takes a lot of extra work

Almost bizarrely so

For a certain class of problems

In all sorts of ways. Except the butter kind.

Metrology!

Learning from self inflicted pain

There's no escape

Black boxes are hot again

Beware! They will not agree! The question is to what degree

The art is finding a good cadence for your metrics

As the stewards of good data practice within a company, we data professionals are often asked to help set metrics, there’s a whole…

There's probably more upsides for us than SWEs

It could be better, it could be worse

The fact that it CAN work, but only with a ton of aggravation is the worst part of the whole thing...

So excited to be able to write about a random fun project again.

The hot stuff right now is labeling with LLMs, but y'know, maybe consider not?

I tried to run some ML stuff for the first time in ages! And I failed! r-o-f-l

More fields are getting on the whole LLM synthetic data generation bandwagon. Honestly at this point the best career move seems to be in learning how to pick apart and vet LLM systems instead of fighting them all the time...

Pulling, aggregating, categorizing, and interpreting public data is always a hairy task with lots of details. Here, Howe shows an example of pulling startup funding data w/ the help of AI classifiers.

There was a brief period in the early 2010s where data scientists were interested in bandit models... and then you barely hear about them any more.

FINALLY

It can have it. Please.

So. Much. Weird. Artwork.

and it's no one's and everyone's fault

Sure it’s a pain in the butt and has its own issues, but human brains are still amazing

Not all AI/ML ethical dilemmas involve life and death…

99% chance you’re not ready

An update for a year of changes. In hopes anyone wanting to write more can get something out of it

Because I don't know what else to do...

A brief update on what's been going on at the newsletter. Nothing crazy.

Bi-weekly Thursday posts are primarily for subscribers and feature more in-progress type work and thoughts from Randy. This week is about experiments.

A full summary of all the talks from Data Mishaps Night 2024, lightly anonymized for privacy.

Good for people starting their analyst careers

Mis-care of plants and miscommunication of information

TL;DR: Spread love, joy, or at least helpfulness

Go out, take a breath, and see what other industries are doing

Normcore self-hosting a newsletter

I'll do it. You should do it too!

Working on it!

It's dangerous to go alone...

Working with Data Quality

Working around myself

A brief reminder

Where are the limits? Edge cases?

And is it "enough"?

FINALLY

During slow stretches

In ways that I want

I seriously have no clue

A PyData NYC 2023 talk

And some examples on how to pick

And how it connects to everyday life. sorta.

An unexpected downside

On Nov 02, 1:30pm

Including from the JWST and Hubble!

There's a lot of stuff going on

Data products, all products

Maybe it's a thing

And difficult, and nothing new

Gamedev, Counting, and Ethics, oh my!

Took me a while to learn this one

And that's what makes it interesting

But it's optional

And your product folk might not know that

Because thinking about LLMs

And arithmetic!

Because we're not selling products

And other duplicated row situations

For the fun of it

Despite it being critical to our job

A just what the heck is it anyway?

Like touching set theoretical grass?

Gotta learn to handle it

For the education!

Because its different

However much I hate them, you can't avoid them

At least, the bits I regularly see

Or a month? Averaged?

It's a scoring function!

But it's not just me

It can have it. Please.

Before things spin out of control

Talking about using our tools instead of the tools themselves

For nerds who need to see the whole process

I want MORE damnit!

Involves forming an often bad opinions

But you can easily become one if you're not careful

Thankfully I've got them, but plenty of people don't

Why are we even effective at anything?

Because we learn a lot that way

Maybe? Not really? It's messy

It's about as weird as it sounds

We can't help but sell this one

A rant

This is SO HARD!!!!

It's all vibes

What'll it look like then?

If you like execution-type work like me

Because of course it'll happen some day

Data nerd bait

So much fun chaos

And then catching flak for it

Just like code reviews

But for how long?

Most of us probably don't even work at it

And they look nothing like the ones I make...

And slightly infuriating

HARD. WARES.

The corollary to this, of course, is that you know something that they don't!

Or just for yourself

That we need to balance out

Going with the flow

Because it is/was perf season

Yes, you too

Compared to many other data fields

The Newsletter Version!

It's hard to describe and very branchy

What has happened?

For 100 years anyways...

It all starts looking the same

We're lucky to be the former

Can take a lot of unexpected paths

Are they just wishful thinking?

oh dear gods...

In case "Data Twitter" becomes just "Data"

Warp speeeeeeeeed

🚩🚩🚩

A rant about failure and magical thinking

Some thoughts about the intersection that keeps me up at night a surprising amount

Some musings on designing spaces and software

We can do that?

An update for an evolving job title.

Sometimes product teams constantly hear a feature request and then think that everyone will adopt it upon launch. They are then very very disappointed.

Big shoe brand crashes in value due to a big strategy shift that relies heavily on data... so maybe we should pay attention to it a bit.

Industry tends to not care about theory, but we should talk about it a bit more because many aren't adverse to it when we talk about it right.

Pulling, aggregating, categorizing, and interpreting public data is always a hairy task with lots of details. Here, Howe shows an example of pulling startup funding data w/ the help of AI classifiers.

Frameworks for doing things like setting metrics or identifying issues abound. They're useful tools but it's worth knowing now to take them too far.

Consumer products are fairly straightforward and interesting, but I guess I like the weirder stuff.

More venting out product ideas, but finding some live examples and course correcting

Nor vice-versa. We need the right tools for the job.

Who would've thought scheduling random meetings would be hard?

In ways that I want

A PyData NYC 2023 talk

Data products, all products

And your product folk might not know that

However much I hate them, you can't avoid them

Or a month? Averaged?

It can have it. Please.

Because we learn a lot that way

It's about as weird as it sounds

Most of us probably don't even work at it

And they look nothing like the ones I make...

A career of chasing ghosts

And it's everywhere...

A bit of dogfood reporting

It's a lot

There's soooo much overlap

In all sorts of ways. Except the butter kind.

Learning from self inflicted pain

During a pandemic?!?!?

Knowing how you got into a situation can help you get out

Generating value upwards

But only in my brain

Shocker: Stretching a painful thing out doesn't make it less painful!?

Not how we deal with it ourselves

Everyone tends to complain about their work

I'm new to this part too

If you're not good at Viz, like me, there's hope for us

Don't expect to land the title easily, heck, don't even go for the title

Not exactly Data Scientist, but close enough you can barely tell the difference. It’s a fairly obscure position. This is, in broad strokes…

It's gotten easier to screw up someone else's survey!

Goodbye BLS as we knew you

An update for an evolving job title.

Three years since I wrote about this last time, it's still happening, meaning a negative leap second continues to be a possibility.

Data work tends to mean getting inconclusive results, which are really inconvenient in industry because they're... nothing... and it's quite inconvenient

Real world measurement means making compromises in the name of work speed. Because precision is time is money.

Meteorologists have a curious little metric called "accumulated cyclone energy" that they use to compare hurricanes and entire hurricane seasons. This is a dive into what that's about.

Subscriber post with thoughts about some of the dangers of giving people data and having them feel like they "know" how things work when there's plenty they don't know

Domain knowledge is critical to dat awork, but we really do get thrown into the deep end at times. Why does that even work at all?

The most instinctive reaction to wanting data is to collect and store as much of it as you can, figure it out later. For many reasons, that's probably a bad idea.

There once was a time where working with lots of data was so awkward, it became a giant hype cycle. Now, almost nothing is "big" any more.

There was a brief period in the early 2010s where data scientists were interested in bandit models... and then you barely hear about them any more.

And then catching flak for it

Are they just wishful thinking?

The insignificance of significance

It's a lot

As a data nerd, pretty interesting stuff

Wait, what?

A reminder to us all

With tons of math going over my head

It's just bad news for everyone involved

Modeling for incorrectness

Almost bizarrely so

Cleaning data is important!

At what point are you completely useless?

The dream...

Doing things the incorrect way, for science(?)

It's always been important. It's just a bit more important these days.

With sketchy charts!

Fun is hard

Some people are learning data science quickly without realizing it

What you can do when you actually collect email data

It’s not every day we can watch an expert explaining how hard counting is

Somewhere, someone’s having this conversation. Right. Now.

Stories of how a simple thing can get really messy.

A useful tool that's mostly unused because people usually don't like what it has to say.

TL;DR is, your job as a reviewer is to help the reviewee come up with better code than they initially did.

It's a really annoying turn of events but here we are...

Dysfunction is still dysfunction, regardless of how it got there.

🚩🚩🚩

A rant about failure and magical thinking

I always told my wife this newsletter was my backup plan...

It does go on and on and on ...

Every few years I forget and have to remind myself

It could be better, it could be worse

It's easy to not realize we manage a DAG surrounding... ourselves

Another episode in the ongoing logs analysis primer series

Some musings on designing spaces and software

The fact that it CAN work, but only with a ton of aggravation is the worst part of the whole thing...

Please, let's celebrate doing things the ugly way. For our peers that do things justly as ugly.

The audience for messy, one-off examples seems larger than it initially appears. If we are only willing to listen

In every one of us, there's a little gremlin that learned something cool, silly, weird, but oddly useful.

How cloudy is a sample of water or air? Well, someone's figured it out, right?

Thoughts on how people should teach and learn analytics based on the question at hand, and not the other way around.

Data scientists have got "one weird trick" and it's called optimization. Luckily there's a surprising amount of depth to the trick.

It'll be in person, and it's gonna be fun! I promise at least that much.

The bad habits of data over-collection run deep.

I tried to run some ML stuff for the first time in ages! And I failed! r-o-f-l

The surprising depth needed to analyze bits of looped string

Planning is hard ot do, especially at fast-paced startups. So how about you spend less time planning and more time breaking things down into useful chunks?

Because I don't know what else to do...

Says guy who writes more than most of the population... but hear me out.

There's more to date time formats than YYYY-MM-DD HH:MM:SS. Most of it confusing.

Just some thoughts about how refactoring software is nowhere near as complex as refactoring physical spaces.

I admire good dataviz, I also know that I'm hopeless at those. So instead I've found a place where it's good enough for the work I do.

It's never been easier, and cheaper, to measure things in our world, just to actually do it and learn how annoying it can be.

There are no shortcuts to writing something worth reading

If you use it all the time at work, you're probably already in the upper percentiles in skill.

Learning new programing languages is always tricky. But using tutorials from a language you already know to learn another one makes things surprisingly easier.

Industry tends to not care about theory, but we should talk about it a bit more because many aren't adverse to it when we talk about it right.

For this Thursday's Subscriber post, some thoughts bouncing around my head that might eventually turn into a sane post but needs some thinking-out-loud to see where it actually goes.

Consumer products are fairly straightforward and interesting, but I guess I like the weirder stuff.

"How hard can it be?" – these are the famous last words of many a data science project. We try not to underestimate the difficulty, but can go too far. Sometimes we just need to know the simple version.

Bi-weekly Thursday posts are primarily for subscribers and feature more in-progress type work and thoughts from Randy. This week is about experiments.

A full summary of all the talks from Data Mishaps Night 2024, lightly anonymized for privacy.

Making transitions smoother

Just enough networking knowledge to do data work and debug your way out of tight spots.

Normcore self-hosting a newsletter

Working with Data Quality

FINALLY

During slow stretches

A PyData NYC 2023 talk

And some examples on how to pick

Data products, all products

Maybe it's a thing

Because thinking about LLMs

And other duplicated row situations

Despite it being critical to our job

A just what the heck is it anyway?

Gotta learn to handle it

For the education!

But it's not just me

Before things spin out of control

But you can easily become one if you're not careful

Why are we even effective at anything?

It's about as weird as it sounds

Just like code reviews

Going with the flow

Because it is/was perf season

The Newsletter Version!

For 100 years anyways...

They're wonky creatures

It's either a bit of work, or a lot of work

Everywhere! EVERYWHERE!

Would it be so bad?

And then we'll have to work harder =O

Because knowledge gaps are hard to spot

And it's everywhere...

So it's ok to crack things open to see

Instead of mastering a thing

For something more general

One day you're going to have to

We are destroyers and preservers

It's a mess because everything's undefined

I love hobbies. This is no surprise to any of you.

It's like a storybook

It's complicated

and it's no one's and everyone's fault

Flashbacks of past pain

Not all of us sits at the big table

It's not all about big, complicated systems

Taken to extremes!

And it's a giant mess

And stop obsessing on surface technical skills

It can be done... with effort

Because the possibility just got slightly more plausible

And that's a good thing

Every so often I just have to indulge in this topic

While we usually want to avoid touching daylight savings time changes in our datasets, when you're working with people, local time actually does matter.

Three years since I wrote about this last time, it's still happening, meaning a negative leap second continues to be a possibility.

There's more to date time formats than YYYY-MM-DD HH:MM:SS. Most of it confusing.

A brief update on what's been going on at the newsletter. Nothing crazy.

I'll do it. You should do it too!

During slow stretches

Going with the flow

For 100 years anyways...

It's like a storybook

Os different from a lot of other work

Because the possibility just got slightly more plausible

And that's a good thing

Because you'll be buried otherwise.

Doing things the incorrect way, for science(?)

For (your) sanity's sake

Shocker: Stretching a painful thing out doesn't make it less painful!?

What else can you do =\

Some survival tips for all the crazy out there

Sometimes, people can actually tell when vanity metrics aren't useful

It really shows when people don't do this

Every so often I just have to indulge in this topic

There's probably more upsides for us than SWEs

Through the magic of RADIO WAVES~~~~

It does go on and on and on ...

It could be better, it could be worse

Another episode in the ongoing logs analysis primer series

The fact that it CAN work, but only with a ton of aggravation is the worst part of the whole thing...

As analysts, we push data systems until they creak and groan. Then we modify them to not do that. Then repeat. Here I'm just thinking through the process.

Every tool I can remember owning, and what I use it for.

An update for a year of changes. In hopes anyone wanting to write more can get something out of it

I tried to run some ML stuff for the first time in ages! And I failed! r-o-f-l

It's never been easier, and cheaper, to measure things in our world, just to actually do it and learn how annoying it can be.

Learning new programing languages is always tricky. But using tutorials from a language you already know to learn another one makes things surprisingly easier.

Post for subscribers -- I think the way we still want our many data tools to interop via common file formats is a sign of dynamism and optimism of future innovation

Using tools to figure out where to go in all the chaos

Just enough networking knowledge to do data work and debug your way out of tight spots.

Normcore self-hosting a newsletter

Working on it!

It's dangerous to go alone...

Where are the limits? Edge cases?

And is it "enough"?

Because we're not selling products

Talking about using our tools instead of the tools themselves

This is SO HARD!!!!

That we need to balance out

They're wonky creatures

Would it be so bad?

Because what choice do we have?

Instead of mastering a thing

It's like a storybook

Psst... run a survey

And it's a giant mess

Shooting slightly less of your foot off

Practice practicing!

Because, sometimes there's no choice

And learn about learning about data sets

tl;dr CSV (and friends)

Because we're in a post-MR world

Not everyone's on the bleeding edge

Mic mic revolution

Off by one errors are common in programming, right?

This is supposed to save us all, it’s merely the best we’ve got so far

Some survival tips for all the crazy out there

TL;DR: Disk is Frickin’ Slow. Network is Worse.

Because they’re awesome in a bunch of messy data science contexts.

Sometimes, index or no index, you’re forced to make some really ugly queries out of necessity given unruly tables or bizarre business…

Really, please, OMG, stop

TL;DR is, your job as a reviewer is to help the reviewee come up with better code than they initially did.

And there's no escaping it

Just about a decade of conference organizing smushed into one post

It all starts from something most of us has. Just powered up and tooled up.

The hot stuff right now is labeling with LLMs, but y'know, maybe consider not?

Big shoe brand crashes in value due to a big strategy shift that relies heavily on data... so maybe we should pay attention to it a bit.

A text transcript of my Quant UX Con 2024 talk for those who prefer text over video.

Industry tends to not care about theory, but we should talk about it a bit more because many aren't adverse to it when we talk about it right.

There once was a time where working with lots of data was so awkward, it became a giant hype cycle. Now, almost nothing is "big" any more.

Many people joining the data science field harbor the belief that they're gonna help change how things work, make stuff magically better. Sadly, we all learn that it's much harder than that.

Nor vice-versa. We need the right tools for the job.

It's something we "deliver" but what is it anyway?

TL;DR: Spread love, joy, or at least helpfulness

A PyData NYC 2023 talk

Because thinking about LLMs

For the education!

However much I hate them, you can't avoid them

Because of course it'll happen some day

And they look nothing like the ones I make...

It's hard to describe and very branchy

Before GPS or even lasers

Beep Beep~~~

We are destroyers and preservers

It's like a storybook

Not a "What happened?" post no one needs to read

You need to. Seriously.

Because I doubt most people, myself included, is

Also, most data cleaning articles suck

Shocker: Stretching a painful thing out doesn't make it less painful!?

I still can't find it at times

The History of a little cup

Black boxes are hot again

It needs to be said regularly —  it’s OK to be nothing like them

It’s surprising to think we rely on this as much as we do

Some survival tips for all the crazy out there

TL;DR: Disk is Frickin’ Slow. Network is Worse.

It’ll save your butt. Lots.

99% chance you’re not ready

Really, please, OMG, stop

Know your data, where it comes from, what’s in it, what it means. It all starts from there.

It’s nothing like at a big mature company.