Writer’s note: You're receiving this email because you signed up for our newsletter The Prediction Game. We’ll be sending out an update on the game next week, but in between updates, we’re exploring different methods of predicting the future, from the methodical to the mystical. 

Chance or Skill? 

I’m not one to brag, but I consider myself to be particularly good at rock, paper, scissors. 

That may seem like a ridiculous statement to those who consider rock, paper, scissors to be a game purely based on chance. After the countdown of “rock, paper, scissors, shoot!” you randomly throw down one of those three options, and your opponent does the same. But is your choice truly random? Is theirs? After enough rounds, can you begin to detect a pattern in the choices your opponent makes? For example, after each round where you both throw down the same choice (paper meets paper) are they more likely to stick to that same choice again in the next round or are they more likely to switch to the choice that beats it (scissors beat paper)? When you start to understand the people playing the game—how they think, how the past affects their decisions—etc. you can find patterns that turn what looks like a game of chance into a game of skill.

But maybe this isn't just limited to rock, paper, scissors. Maybe, if we get really good at analyzing the past and present we can get really good at predicting the future in lots of situations. What previously looked like random happenstance, suddenly becomes explainable—even predictable.

Take The Prediction Game for example. Is it a game of skill or a game of chance? To answer that question, let’s look at the purest example of a game of chance: playing the lottery. The lottery is an exercise in predicting the future that entirely comes down to chance. There is nothing you can do, no data to analyze or research to complete, that would increase your chances of selecting the right combination of random numbers. The Prediction Game, on the other hand, is a fairly different proposition. Or at least that’s the hypothesis that Wharton Professor Dr. Philip Tetlock has been researching for the better part of 3 decades. (Note: Everything we’re sharing here comes from Tetlock’s excellent book Superforecasting: The Art and Science of Prediction. This article is certainly no replacement for reading the book, and we highly recommend that you do. We don’t receive any affiliate revenue from this, we just think it’s a really great book and that you should support your local bookstores these days.)

Think like a Superforecaster.

Through his research, Tetlock has found that some people are uniquely better at predicting the future than others. They’re so good at it, in fact, that they are capable of beating betting markets. Tetlock calls these folks superforecasters. What’s most interesting is that these superforecasters are not made up of subject-matter experts for the questions they’re answering. They’re not statisticians or card carrying members of Mensa. What makes them unique is not some form of superhuman intelligence or paranormal clairvoyance (we’ll talk about those folks in a future update). 

At its core, superforecasting is a way of thinking and problem-solving. It’s a process that can be applied to pretty much any question regarding the future. It’s a skill that can be learned, practiced, and improved upon. And that’s exactly what we’re going to teach you how to do throughout the rest of this article.

We’re going to break down the superforecasting process to 5 parts, and then we’re going to apply each of the parts to one of the questions from The Prediction Game. 

Part 1: Break the question down

According to Tetock, in order to make an accurate prediction, we have to make sure we’re actually answering the right question. His research has found that our brains take all sorts of shortcuts when we read questions in order to arrive at the fastest possible answer. But in many cases, the fastest answer is not always the most accurate one. Consider the example Tetlock uses in Superforecasting:

“A bat and a ball together cost $1.10. The bat costs a dollar more than the ball. How much does the ball cost?” If you’re like most people, the answer that immediately comes to mind is 10 cents, but is that the right answer? Read the question again. Take the time to actually do the math. You’ll find that 10 cents is, in fact, not the right answer (it’s 5 cents). Our minds take shortcuts all of the time, and we end up answering the question we want to answer instead of the question that is being asked of us. So take your time, break the question down into smaller parts, and make sure to check your shortcuts and biases at the door.  

Part 2: Define the knowns and unknowns

Once you've broken down a question you can get to step two: separating the known and unknown components. Tetlock calls this "Fermi-izing" a question, a nod to nuclear physicist Enrico Fermi. Decades before Google used a question like “how many piano tuners are in the city of Chicago?” during product management interviews, it was Fermi himself who would ask this question of his students. The reason he would ask what first appears to be an unanswerable question, at least without the use of the Yellow Pages or a search engine, was not so much about getting to a correct answer, but rather seeing if his students were able to break down a question into knowns and unknowns, and finding close enough knowable proxies for the unknowns that they could narrow in on a plausible answer. An “unknowable” question can quickly become much more “knowable” if we break it down and look for the parts that are easier to answer. 

In Superforecasters, Tetlock tries to answer the piano tuner question by breaking it down into 4 smaller, more knowable questions: 

• How many pianos are in Chicago? 
• How often are pianos tuned each year? 
• How long does it take to tune a piano? 
• How many hours per year does a piano tuner expect to work?

Some of those questions are still pretty hard to answer, but they can also be broken down into easier, more knowable questions. We can break the number of pianos down by considering privately owned pianos versus pianos owned by schools or music venues. We can use the average 40 hour work week to figure out how many hours per year a piano tuner would expect to work, and then divide that by the total number of hours we believe will be spent tuning pianos in the city of Chicago. All of this will still only get us to an estimate, but it will be a much more informed estimate than simply picking a number out of the sky. 

Part 3: Adopt the outside view

Adopting the outside view is all about, as Tetlock puts it, recognizing that “nothing is 100% unique.” The situation you’re trying to predict, be it a pandemic, a natural disaster, or a presidential election, might feel entirely new and unforeseen. But each of these situations have to come from or are impacted by something that already exists. We can assume that several elements of future events have happened before, and therefore they can be studied. 

Let’s look at the 2016 US Presidential election for a moment (I know, I know, trust me, I’d also rather not). The result of that election shocked the world, but if you took an exclusively outside view of the race, a political party has only been able to retain the Presidency after a 2-term administration once in a 76 year period. That’s a 12.5% success rate. Not the best. Obviously that shouldn’t be the only factor you should consider, but it’s a useful outside perspective that may ultimately impact your prediction, or at the very least how confident you feel about its accuracy.  

Part 4: Adopt the inside view

If adopting the outside view is the practice of looking at a future event as not that unique, or simply the result or combination of previous knowable events, then taking the inside view is doing exactly the opposite. Taking the inside view is all about looking at all of the truly unique aspects of the situation. 

Let’s go back to the 2016 Presidential election (last time, I promise). There were several aspects of that election that were totally unique. It was the first time that a woman led the top of the ticket for a major political party. Several race-changing scandals broke out just a few weeks before election day. Each one of these unique details should help you “tune in” your prediction to a higher level of certainty. As these events took place, pundits and analysts all dynamically changed their forecasts to reflect the inside view.  

Part 5: Synthesize

Synthesis is its own superpower. Synthesis is the ability to take multiple points of view and pieces of data and merge them into “a single vision as acute as that of a dragonfly” as Tetlock puts it in Superforecasting. When you put the inside view and the outside view together with differing opinions and counterpoints, you should be able to come up with a well informed, thoughtfully calculated prediction.

It’s at this point that you’ll want to consider how much weight to place on different pieces of information. How important was that 12.5% success rate for the 2016 election? Turns out, it was pretty important. How much weight should we have put on the Access Hollywood tape vs. the Comey memo to swing the election results in one direction or the other? Synthesis isn’t just about putting together all of your information, it’s also about gauging how that information will impact your final prediction. 

Predicting 100 Million Shots

Now that we’ve gone through those five steps, let’s try to apply to an actual question from The Prediction Game: when will the US have administered 100 million doses of COVID-19 vaccine? Of course, we’ve already gotten there, but we’ll use the information we had available at the time we asked the question, and see how close it gets us to the actual answer. 

Step 1: Break the question down

We’re not asking when 100 million people will be vaccinated. We’re also not asking when 100 million doses will be available. Read the question carefully. When will the US have administered 100 million doses of COVID-19 vaccine? Answering the wrong question is an extremely common mistake. Our minds fixate on the “100 million” because it’s the only quantifiable number in the entire question, and then proceeds to ignore all of the other components. But the other components matter a great deal. 

Step 2: Define the knowns and unknowns

Our goal is to understand the pace of how each COVID-19 vaccine in the United States will be approved, manufactured, distributed, and administered, and to consider all of the factors that may speed up or slow down that process.

Some of those variables can be known. When we originally asked this question, back in mid-December of 2020, both the Pfizer and Moderna vaccines had been approved. The Johnson & Johnson vaccine had just begun its Phase 3 trial, with expected approval in February. We also knew that a new administration would be in place by January 20, with a stated goal of 100 million shots in their first 100 days. 

Other variables that could impact our prediction are unknown. Manufacturing could hit a roadblock, delaying the distribution of vaccines. Weather events like storms or floods could impact delivery and administration of vaccines. Just because these variables are unknown doesn’t mean we shouldn’t factor them into our prediction. If we expect the administration of 100 million shots to take over 4 to 5 months, it’s fair to expect that some major weather event will take place over that period of time that would impact that timeline (which, of course, it did). 

This gets us to the part of the process where we should adopt the outside and inside view of our question.  

Step 3: Adopt the Outside View

It’s tempting to feel like, when dealing with predicting the future, we’re always working with the unknown and unprecedented. Certainly when it comes to the question of administering hundreds of millions of doses of a brand new vaccine in a matter of months, all of it seems completely unique. It’s never been done before. 

Adopting the outside view of a problem means putting that kind of thinking aside. While the specific question we’re dealing with might be unique, its components are things we’ve done before, and therefore can be used to establish a sort of baseline that we can build our final prediction on. We can safely say that it’s impossible for 100 million vaccine doses to be administered in a week. We can assume that it’s unlikely for the administration of 100 million vaccine doses for such a widespread pandemic would take more than 2 years. Let’s try to narrow that down.

According to the CDC, 63.8% of Americans (roughly 209 million people) received the standard flu vaccine during the 2019-2020 season. That number has steadily increased over the last 3 years. That gives us a pretty strong baseline that over 200 million vaccine doses can be developed and administered over a 12 month period. Since the question is asking about 100 million doses, the time frame might be closer to 6 months. 

The outside view won’t get us to our final answer, but it will help us limit the range of possibilities. Just that one CDC data point has allowed us to narrow our range from 2 years to 6-12 months (a 50-75% improvement). More than a week, less than 12 months. That’s still a pretty wide range, but continuing to take the outside view will help us narrow that down. We just used our broad brush. Now it’s time for the inside view to give us the finer details.

Step 4: Adopt the Inside View

Of course, what we’re dealing with here is not just the flu or the flu vaccine. There are many factors that we need to consider that are totally unique to the situation we find ourselves in. 

Unlike other vaccine development and approval cycles, which typically take around 5 years, COVID-19 vaccines have been developed and approved in about 11 months due to the sheer scale of the pandemic. That also means a lot more resources, private and public, are being dedicated to the effort of administering this particular vaccine. All of these signs point to a speedier deployment. 

On the other hand, two of these vaccines are using an entirely new type of inoculation method: mRNA. They require specialized manufacturing, transportation, and storage. Adapting our existing infrastructure for these new types of vaccines will take some time, slowing down their deployment. 

Taking this inside view allows us to dial our outside view range in or out. When we put both together, through synthesis, we can finally arrive at an informed prediction.

Step 5: Synthesize

The outside view tells us that administering 100 million doses of the COVID-19 vaccine in the United States will take anywhere from 6 to 12 months. Let’s take a look at the inside view. We know that the COVID-19 vaccines were developed 5 times more quickly than the average time it takes to develop and approve a vaccine due to a higher level of urgency and increased resources. Let’s apply that knowledge to our original range, changing it to 3 to 5 months. We know that some new infrastructure will need to be built in order to mass deploy these new vaccines, so let’s take the average and say around 4 months. 

Given that vaccinations started in late December, our synthesis puts the 100 million dose milestone between late March to early April 2021.

In reality, the US hit the 100 million dose milestone in the 2nd week of March. That’s a little earlier than we would’ve originally predicted. So what didn’t we see coming that would’ve helped the accuracy of our prediction? For one thing, we may have put too much emphasis on the need for new infrastructure to deploy these vaccines. We also didn’t predict the ramp up in production that came from the early days of the Biden administration. If we had factored those two variables in, it may have tilted our prediction closer to the right answer. Either way, we were only about 2 weeks off. 

Thinking like a Superforecaster isn’t only helpful when playing The Prediction Game, though. This is a problem-solving process that you could and should apply whenever you’re trying to forecast what will take place in the future, whether that’s for your business, investments, or just trying to impress your friends. 

Stay tuned for next week for a quick recap of everything that’s happened in the last 3 months, and where the leader board stands in The Prediction Game.