There’s this clip of comedian Bill Burr on Conan talking about how he won’t listen to his friends list a source unless it comes from the library because at least the library separates fiction from nonfiction. As most successful comedians do, Bill Burr made me sit back and laugh but also ponder his words long after the clip ended. It’s true, and as stated by Conan O’Brien in the clip, if you’re looking for a “source” to support your claim, you’ll be able to find it. Whether that source is reputable or not is the real question.
I imagine that this idea of terrible sourcing for an untrue claim reminds you of someone or even something you saw. For me, I typically see it from some people on Facebook, for example. Before I jump further into this, I need to acknowledge that it is difficult to have a civil, open discussion about politics on social media. You are limited by your character count, and you can’t use tone of voice and things of that nature like you can in in-person conversations. If you took Facebook comments into account, you’d think that there would be numerous studies about how “masks are harmful to us,” or how “COVID-19 is a government hoax.” For the record, those two things are wrong: masks aren’t harmful, and COVID-19 is most definitely not a government hoax.
Those are two of the more extreme examples, but sometimes there are “stats/facts/studies” that are more difficult to tell what is real and what isn’t.
So, with that in mind, let’s review some ways that you can identify a quality source compared to a lower quality source, how to dive into data, and what to do once you have this information.
Source of the Claim(s)
If you are getting your sources from a poorly built website called “truth-news.com,” which also has no citations for the claims, that is an immediate red flag. If it’s from an accredited university (.edu) or government site (.gov), you can be sure that this is a reliable source. You can also count on sources like the New York Times, Washington Post, Los Angeles Times, and big-time newspapers.
Along with that, dig into other parts of the website to see how reputable this source is. Look at the company’s mission statement and values to see if those seem reliable (and if they even exist). Look at this website’s past content to see their track record. If it’s just rambling blogs with no citations or references, then you can immediately eliminate them from legitimate consideration.
But sometimes sites with misleading or misguided information are more difficult to determine, and for that you’ll have to put some more work in to determine the legitimacy of the primary source.
Dive into the Data
Ask anyone who works with numbers, and they’ll tell you how easy it is to manipulate statistics to appear in your favor. So sometimes you’ll see a headline that says “90% of people who do this are more likely to…” or “87% of people located in this area are…” You get the idea. You might even find this accidental (or intentional) misuse of data in larger, more reputable publications. The best way to get to the bottom of the claim is to dive into the data and ask some of these questions:
- When was the data collected?
- How many people were a part of the study?
- Where were the participants located?
- What is the demographic breakdown of the participants?
- How was the data collected? (Survey, door-to-door, etc.)
There are many other questions you could ask when looking into the data collected, but these are the ones that I typically find myself asking when looking into the validity of data with a claim.
After all of this, some might think that they’ll be able to immediately tell if information is “good” or “bad,” but it’s a lot more nuanced than that.
When you answer the questions about how the data was collected, who it was collected from, and more, you’ll have a greater idea of how much this data should weigh. It might be good early data but with room for growth. Or it might be a small sample size with a very specific demographic, which in that case, you might decide is not worth serious consideration.
What I’m saying here is that the more particulars you have about the information being shown, the better you can determine how strong or weak the data and the claim is.
Here’s another useful resource for determining how to identify manipulated data.
Keep in mind…
The reality is that you won’t be able to dive into the data for every single claim you see on every single news site – it’s impossible to keep up with. So, keep in mind that there is always more to the story than the headline, and if it’s a claim you’re really interested in check the website and dive into the data to ensure that your primary source is reputable. Or of course you can find reputable sources at a library, too (library takeout, anyone?)!
