Watching a data team get their credibility nuked in national news

Thanks to all the people who submitted talks to the DataBS Conference! The volunteers have started submitting their reviews already and I'm preparing to figure out what the program will look like. I think the final decision will be made next week and emails sent out.

Last week, August 1, the Bureau of Labor Statistics (BLS) issued their regular report on employment. The highlight was that the unemployment was about steady at 4.2%, and the economy added about +73k non-farm jobs. More importantly, the data from the previous two months were also significantly revised down almost 90% to show that there had been practically no jobs growth the past three months. Within hours of it hitting the news, the revisions were (without evidence) declared politically motivated by the President and the head of the Bureau was fired. As you would expect, a LOT of experts have opinions about this, mostly different shades of negative.

Given the current administration's propensity to lie about all sorts of things, but especially things that make them look bad, many people who paid even the slightest attention to were already surprised that economic data reports appeared to have been relatively untouched over the past year. If anything, some people (myself included) were a bit perplexed that despite all the self-inflicted economic headwinds since the start of the year, economic reports about inflation and employment seemed oddly strong. But given that there wasn't data to the contrary, everyone went with what was available. The fact that downward revisions even happened showed how relatively untouched those systems were.

The reason why most economists and practitioners trusted the BLS numbers , aside from decades of being an upstanding source of statistics, is because it is a well documented methodology. The data actually comes from two surveys the Current Population Survey (CPS; household survey) and the Current Employment Statistics survey (CES; establishment survey). The first survey is sent to a large sample of Census households and is actually administered on the BLS's behalf through the Census department. The second survey is a series of industry-specific forms sent to a large sample of employers and is collected by BLS. While there can be all sorts of biases written into any given survey, the important part is that these surveys haven't changed significantly so the whole reporting apparatus is largely a very labor intensive mechanical process. Presumably there is a paper trail of survey responses that could be used to reproduce the whole analysis, leaving little room for politically motivated manipulation. Similarly, revisions to older values are simply because initial reports use the survey responses that are available at the time, and late responses to the same survey period update the data. It's all methodologically sound stuff done by civil servants fulfilling their mandate

But now, with the blatant interference of the administration, all that trust, deserved or not, is out the window. I don't think anyone who works with this data (and there are so, so many) will view future reports without a very healthy chunk of skepticism. People might still find uses for the data, but I highly suspect that they're going to try to independently triangulate. As a data organization, I can't imagine the amount of frustration that every single employee there is experiencing because having institutional trust literally vaporized in an instant through no apparent wrongdoing on their part really sucks.

As a practitioner, the only real currency I have to trade in at work is whether people trust me and my work or not. Being trusted means I can say and show things and be taken seriously without much comment. That skips huge amounts of hours spent justifying and spelling out how conclusions are drawn. It means not having to deal with antagonistic questions about the work. You don't get people going out of their way to do their own analyses to attempt to disprove you. It's infinitely more efficient for everyone involved. Burning a relationship that typically takes weeks, months, even years to build by providing falsified findings is tantamount to organizational suicide.

If anything this is a good learning opportunity for us practitioners. We're all extremely aware of the potential risks of what happens when trust is lost, so it's not something we're going to actively try to do. We get to watch this one play out more or less in public.

So what happens now

Considering that the entire financial sector and every economist looking at the US economy relies on BLS data, there is a LOT of demand for this kind of information. In the coming days, I think a lot of people are going to come to their own conclusions about to what extent they're going to trust the data coming out of BLS. I guarantee you that next month econ nerds will be abuzz with looking at BLS data and seeing if they can spot any issues.

At the low end of disaster where the financial industry exerts enough pressure that the BLS can continue to operate largely as it has, I think we're mostly going to see a lot more scrutiny and attempts at triangulation. If you search around for variations of "alternatives to BLS data", there's not many options available and all have significant flaws. Private companies like ADP, Linkedin, and Indeed put out reports on their view of the job market. Most economists I've seen all seem to agree that the data quality is significantly worse than BLS. But given that everyone is interested in finding alternatives now, we might start seeing people attempt to create alternatives over time.

The main problem is that while the methodology is published and can be replicated by anyone sufficiently motivated, it is a giant pain in the butt to administer massive surveys at scale every single month. More than that, it is very expensive for a private enterprise to do. It's why we had the government do it as a public good instead. I'd be willing to bet that financial sector data brokers, especially the big ones like Bloomberg and Reuters, are hard at work at this very moment trying to come up with alternative data products that will help their clients corroborate future BLS data releases.

On the far end of disaster, where the whole US economic data infrastructure crumbles into a facsimile of the likes of countries that are known to put out false data, things are going to be ... exciting? It's going to take time for clever folks and researchers figure out what proxy data can be used to answer the fundamental questions about employment across the country. But given how important the data is to the entire financial industry, I have to assume there's a large market incentive to do so quickly.

Until people figure out how to get answers out of what is available, we're going to be in a situation that's very much like what I imagine when the data team of an organization gets completely discredited. Everyone scrambles for their own metrics to get a view of the world. There'll be hot debates over who's metric are better. There'll be no source of truth to compare against so we'll have to wait for proposed metrics to fail before we realize what doesn't work. It'd take a drastic reset to even begin the process of rebuilding trust.

Keep an eye on this

This story is a (relatively) slow moving one in that we get to wait until the next couple of data releases and see if anyone can catch them in a lie. Markets will probably react rather chaotically until consensus forms. I'll be interesting times seeing the dust settle.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

• randyau.com — homepage, contact info, etc.

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
• Send a one time tip (feel free to change the amount)
• Share posts you like with other people!
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!