Fixing consistency errors helps you present better

Recently I had the chance to meet with an intern at work whop was preparing for their final presentation of their internship UX research project. While they were mostly presenting the results of their qualitative and survey study, their team lead suggested that there may be a quantitative angle that could be added and suggested we get together to discuss.

We of course chatted about the expected things like how to set up and run a regression model, as well as things like picking potential factors, interpretation, and other technical details. We also talked about how to present any findings that may fall out of such an analysis. But in the end I think the most interesting part of the conversation came about from the innocent question of, are there any things that one should be careful of when presenting quantitative data to a stakeholder audience?

Obviously there's no single right answer, but in the heat of the moment, the one I settled on was something pretty simple – make sure your numbers are self-consistent.

We're all familiar with many of the bad things that can possibly happen during a presentation, from random tech failures (annoying), to making a point that rests on a single assumption and literally have someone raise their hand and tell you that the assumption is false (really hurts), to everything in between like accidentally saying the wrong thing and sparking an argument, getting ripped to shreds by pointed comments, or getting outright dismissed by an unfriendly room. In my view, many of those things are largely outside your full control – there's only so much time you can devote to being thorough and checking assumptions, there's only so much practice you can cram in before the live presentation. Mistakes are inevitable in work life, and you learn to gracefully handle the issues as you go along.

But of the few things that are entirely within your locus of control, not having your own presentation slides contradict each other is probably my number one "take super extra pains to not do this!" advice for people who are new to using data skills to persuade a room of people about a point.

Inconsistency blows everything up

For people whose jobs are in "data science for product development" or "user experience research", our primary job is to convince people who have the actual decision-making power that a certain course of action is better than some alternative. Trust and credibility are the currencies we have to accomplish our work, and we earn them in drops and can lose them by the bucket. So when we present work where one slide says "A", and a few slides later says "Not A", people who are paying attention are going to notice.

I have not encountered another issue that can derail a findings presentation as much as when someone spots an inconsistency and questions it. If you don't have some kind of explanation as to why the "same number" is somehow different in two places, AND you can't come up with a way to politely "look into the issue" ater, trust gets thrown out the window. Once one set of numbers suspect, then literally every single other number becomes suspect. People are generally upset when numbers that are supposed to point to "a single truth" don't line up and they want to get at the bottom of it. Were the numbers from different systems? Were the definitions of the numbers slightly different? If there's a big discrepancy is there a third "neutral" measurement that can help triangulate the real value? Should we call in someone from another team who may know the answer?

Depending on how bad the discrepancy is, you can forget about telling the persuasive narrative you've carefully built. Essentially, by showing contradictory evidence, you have completely blown up your primary goal, which is persuading people that they should lend an ear to your work and recommendations.

Consistency is not correctness

It's important to also note that being consistent does not necessarily mean the numbers themselves are actually correct. Fundamental errors of analysis can happen at any time for all sorts of reasons. There are plenty of times when I'd finish a presentation and a little bit after have to send out an apologetic clarification to a finding because a mistake had been made that materially changes the conclusion in some way. While those mistakes are also damaging to your trust and credibility, I've found it to be oddly less of a problem? Maybe it's because it's extremely rare that I'd find an error that fundamentally changes how everything needs to be interpreted.

Finding and fixing consistency errors does have a tendency to make you poke holes in your figures and reasoning, so in a way going through the process of reviewing your work carefully makes it less likely to encounter big "oops, everything has changed" errors.

But how does it even happen?

Even if you work from one single data set, or even one single Excel file from the entire start to finish, I find that it's entirely possible to have weird and inconsistent numbers in your presentation.

The main reason is because as you build out the story and narrative arc of any presentation, you have to go back to your data and make little side analyses to say what you want even after you've finalized the "this is the money chart I want to show everyone". You'll want to do a quick descriptive summary for the intro. You'll want detailed table crosstabs for the appendix. You'll play with various visualizations and charts to get specific points across. As you tell the story you'll realize you need to back up some extra statements with hard data, and that drives another mini analysis. All throughout, you'll have plenty of opportunity to discover errors and other issues with your analysis. Some of those errors and issues actually require you to change your analysis because, oh I forgot to account for those weird test users, or we ran that ad campaign in April and need to add corrections, etc..

Fixing all those details makes your overall final analysis stronger, which is great, but it also means you need to remember to update your old slides. If this happens only once or twice in an analysis, it would merely be an annoyance. But with any complex project, this could happen a bunch of times. If your fixes get particularly deep, you may wind up rewriting the entire narrative because you learned something important. At some point you stop updating every figure and table and mention of number until things settle down because going back is slowing you down... and then after a few weeks have gone by you completely forget to go back and update those loose ends. And then you stand up and show off your hard work and ten minutes in someone asks why does the total customer count on slide 3 disagree with the one on your big final chart.

How to check for consistency

The basic notion of internal consistency of numbers is that given all the numbers that are exposed in a presentation, the numbers can be involved in calculations and transformations that logically lead to the other numbers. For example, if you segment your user counts into categories, at the end they need to sum back up to the original total. Ratios should be using consistent and logical numerators and denominators, and if you do a lot of multiplying and dividing of different ratios, you can sanely convert from one unit to another and get roughly the same answer.

At the surface, this can seem like an impossible task. First, it's a giant pain to notice if there are mathematical connections between the different numbers you're showing, and the number of comparisons grows exponentially as you add numbers. On top of that, not only do your stakeholders look at the numbers you present and see if they are consistent with one another, but they bring their own domain knowledge to the table. The head of sales very likely has a very accurate idea of what the overall sales numbers for the quarter is to within a couple of sig figs, they might even know finer breakdowns. They will totally compare your numbers against their own understanding and call out things that look unreasonably far off. How are you supposed to dodge all those unknowable bullets?

The honest answer is that you can't find and defend against everything – errors do sometimes find their way through. It's important to have a strong process for making corrections and updates as they are needed so that you maintain your trust and credibility with everyone. That said, it's not hard to check the most obvious things – do totals and crosstabs sum consistently, do numbers with identical labels actually show the same number, do the orders of magnitude on every number make sense, do the reported ratios make sense? Those are all things that people in the audience can quickly check over and thus the most likely issues to pop up. You'd be surprised how many of those stupid bugs happen in your own work if you choose to pay attention to look for them.

Standing offer: If you created something and would like me to review or share it w/ the data community — just email me by replying to the newsletter emails.

Guest posts: If you’re interested in writing something, a data-related post to either show off work, share an experience, or want help coming up with a topic, please contact me. You don’t need any special credentials or credibility to do so.

"Data People Writing Stuff" webring: Welcomes anyone with a personal site/blog/newsletter/book/etc that is relevant to the data community.

Counting Stuff Official Forums: Discuss posts, or other data topics with the community.

About this newsletter

I’m Randy Au, Quantitative UX researcher, former data analyst, and general-purpose data and tech nerd. Counting Stuff is a weekly newsletter about the less-than-sexy aspects of data science, UX research and tech. With some excursions into other fun topics.

All photos/drawings used are taken/created by Randy unless otherwise credited.

Supporting the newsletter

All Tuesday posts to Counting Stuff are always free. The newsletter is self hosted. Support from subscribers is what makes everything possible. If you love the content, consider doing any of the following ways to support the newsletter:

• Consider a paid subscription – the self-hosted server/email infra is 100% funded via subscriptions, get access to the subscriber's area in the top nav of the site too
• Send a one time tip (feel free to change the amount)
• Join the Approaching Significance Discord — where data folk hang out and can talk a bit about data, and a bit about everything else. Randy moderates the discord. We keep a chill vibe.
• Get merch! If shirts and stickers are more your style — There’s a survivorship bias shirt!