Off the Grid

Despite sounding like an entertaining activity, my initial plan to measure the staying power of various news content by pop-quizzing my friends doesn't really hold up to much scrutiny. Never short of scrutiny, I've decided to divide the flaws of my planned experiment into two groups:

 

1. Confounding Variables — External factors that reduce the precision of measurement

 

2. Framework Errors — False assumptions that void my results regardless of confounding variables

 

Essentially, the former involves logistical execution, and the latter concerns theory. After meeting with my professor and discussing the confounding variables of my ‘news quiz’ experiment, I had originally planned to include these at the end of my analysis in a ‘limitations’ section. I remember doing this in high school science projects, following the conlcusion. In other words, it's a way to say "here are results of my experiment, and this is what it means, but keep in mind it's not perfect because ______." But after mapping our the erorrs conceptually, I realized that the confounding variables alone reduce the interpretative power of my planned ‘news experiment’ to little more than a buzzfeed video. Here are the three most revelant confouding variables:

 

 

Confounding Varible 1: Non-homogenous Consumption

 

One issue with measuring knowledge retention after consumption of news material stems from the fact that different people don't consume news the same way. My planned activity involved  a whopping 8 people, which means there was really no way to control for differences in consumption. After chatting with my roommates, I realized this manifests itself in two ways:

 

Coerced vs. Voluntary: Consider two of my roommates: James and Clay. James loves Last Week Tonight, and watches it almost religiously. Clay, on the other hand, isn't really into political humor and doesn't care for it much. If I tell both roommates to watch an episode, and surprise them with a pop quiz the next week to measure retention, it's likely James will score higher assuming he doesn't have the memory of a goldfish. Because James actually cares about the content, even if I pick an obscure episode he hasn't seen. He will watch intently. Clay, however, is watching because he's my friend. At the end of the day, he doesn't really care about Last Week Tonight, and probably won't watch intently. This difference in voluntary and coerced consumption distorts my ability to precisely measure how an 'average' person would retain the information, because I simply don't have the man-power to control for this variable. 

 

Casual vs. Serious: The second difference stems from how people consume content. In other words, not everyone watches TV shows with the same purpose. This applies more to satirical news than network news, because satiritical news are usually both funny and informative, and this can divide the shows demographic. To illustrate this point, take my friend Akshay as an example. Akshay is an analytical mastermind, and probably one of the smartest people I know. He scored in the 99.8th percentile on his dental school admission exam. But politics and history are not his strong suits. In 12th grade, after covering The Cold War for about 8 weeks, my history teacher caught him chatting during a lecture. In response, she bluntly called him out, and asked him "Can you even name the two Cold War powers?" Akshay thought pensively for a bit, and then responded with "I don't want to guess wrong , so I've decided I don't wish to answer the question."  This anecdote isn't meant to demean Akshay in any way, but simply demonstrate that some people really don't care much for politics. However, Akshay loves Last Week Tonight, a show that very frequently discusses domestic and foreign politics. He watches it because he finds the show hilarious.

 

Similar to Akshay, my roommmate James also loves Last Week Tonight. And he also finds it hilarious. The difference is that James has actively trying to become more politically aware. He's recently watched every presidential debate, every episode of The Daily Show, and even watches CNN occassionally like a real-life, grown-up human being. James' motivation for watching Last Week Tonight is not superior to Akshay's; they are simply different. Akshay focuses more on the jokes, and less on the policy implications of the content. James focuses less on the jokes and more on the policy implications. I recently had both of them watch this segment on Televangelism, and asked them what they learned. Here are their paraphrased responses:

 

Akshay: I didn't realize how crazy those televangists were man. They're snakes man. Also, what's with the dude getting sent an outline of the preacher's foot with a request for more money? That's hilarious. [see below]

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

James: That was absurd. Why are presidential candidates not talking about this? How is it not illegal to tell somebody on TV that their cancer will be cured if they send you money? How is that not regulated? How is that tax free? I basically learned that public bastardizations of our commercial and tax laws exist beccause somebody found a loophole in the intentionally broad IRS laws on churches.

 

These two people watch the show in completely different ways. Akshay enjoys the outlandish humor tied to real-life events, while James likes critically analyzing the issues presented to build his political repartee. Both watch for entertainment, but focus on different things. Plopping these two people into an experiment to measure information retention represents an uncontrolled variable, one that I wasn't really sure how to control. The existence of this bimodal method of consumption makes it incredibly difficult to measure 'informative power', unless you decide to only utilize serious or casual viewers in your experiment. But even still, those groups aren't concrete, monolithic blocks. If you wanted to measure the 'informative power' of a show using this type of experiment, you'd have to perfectly sample people to accurately represent differnt styles of consumption. I'm not even sure this is possible, let alone something I was capable doing.

 

 

Confounding Varible 2: Past Inquiry / Decoupled Information from Source

 

The second uncontrolled variable involves the impreciseness of reverse engineering information. Suppose for a second that we discovered an accurate quantitative measure of knowledge. Unfortunately, knowledge can't be measured like grams or milliliters, but suppose it could. Could we measure the imformative power of two shows by comparing their knowledge contribution? In other words, if a team of researchers conducted diligent interviews with 2,000 viewers that all only watched the same four shows, could I create the following chart like for an issue like the presidential election?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The answer is obviously a firm no, because reports on current events are not mutually exclusive in regards to knowledge. The vast majority of content overlaps, because... well... why would it not? If you're looking for the most recent contreversial quote by Donald Trump, you don't have to flip through ten different news channels to find it. Even if the genius investigative reporters at ABC uncover an amazing insight regarding a current event, it's not like they can protect this fact with a copyright. At the end of the day, for most issues, the overlap between content on various networks prevents you from reverse engineering the system to find the "source" of knowledge. And even if a certain fact is only presented by one network, the consumer of that fact likely won't remember where it came from. The only exception occurs when a fact is presented in an incredibly unique or engaging way, like through a metaphor involving a dingo and a baby. This phenomenom is relatively easy to test in the wild. Regarding a current event, ask a friend "what do you know about _______; can you list some facts?" After listing a few facts, ask your friend which specific news outlet he heard those facts from. Unless your friend only consumes one news outlet, you will probably get an answer like "all of them" or "I don't remember".

 

Therefore, because it's almost impossible to comprehensively review someone's previous consumption of news, and because knowledge travels from each source into an unretrievable brain dump, how are you supposed to measure 'informative' power without controlling for prior intake of news on the subject? Sure, I could choose topics that my subjects likely have no prior knowledge on, but the majority of mainstream news outlets don't cover niche topics like that. And thus, the idea of decoupled knowledge prevents you from accurately measuring 'informative power' through the experiment I planned. How was I supposed to know whether the retained information truly derived from the target source unless I controlled for people with no prior knowledge? And if I control for that, what's the point of the experiment when news is usually consumed with some sort of prior knowledge. It's just very hard to measure. 

 

 

Confounding Varible 3: Future Inquiry

 

The third issue with measuring the 'informative power' or retention rates of different shows involves the chosen time horizon. If Show A provides a detailed overview of the facts in a way that discourages future research, it might score highly in my quiz experiment. Alternatively, if Show B provides a cursory overview of the facts, but inspires 6 months of continuous research on the topic, then it might score relatively poorly in my news quiz, which was supposed to take place two weeks after consumption. But this doesn't mean Show A is more informative than Show B, because the results of my experiment depend entirely on the time horizon chosen. If I chose to spring the pop quiz 4 months after consumption, the Show A viewers may have forgotten almost all the content, while the Show B viewers would still be actively researching it. How do you measure future inquiry? How do you choose a time horizon to measure? Do we just test all reasonable time horizons? Unlike a financial cash flow, we can't just use interest rates to discount the value of knowledge over time. This remains another slippery variable preventing you from accurately measuring the 'informative power' of various shows.