The 10 Commandments of Superforecasting

Triage.Prioritize your efforts on questions where accurate predictions have the most significant impact or consequences, or in the realm of trading, where you’ll find the most edge. Focus on areas where your expertise and available information give you a reasonable chance of success. Some problems are too hard to predict (who will be elected president in 12 years) while others are “clocklike” questions (the sun will rise and fall each day), where you won’t find much edge.Concentrate on questions in the Goldilocks zone of difficulty, where your efforts will give you the most edge.

Break down seemingly intractable problems into tractable sub-problems.Complex problems can be overwhelming and prone to errors. Breaking problems down them down into smaller, more manageable sub-problems, allows for focused analysis, enables better assessment of each component, and increases the accuracy of predictions.To illustrate this, they give an example of Enrico Fermi who had a unique interest in brain teasers aimed at estimating various things.Enrico Fermi asks the question “How many piano tuners are there in Chicago?.” Rather than putting out a number that “seems about right” as most would, Fermi would break the problem down into subcomponents. Here’s an abridged excerpt from Superforecasters breaking down the subproblem…1. How many pianos are there in Chicago? To answer that…a. How many people are there in Chicago? Let’s guess 2.5 million people.b. What percentage of people own a piano? Pianos are too expensive for most families—and most who can afford one don’t really want one. So I’ll put it at one in one hundred.c. How many institutions—schools, concert halls, bars—own pianos? I’ll....say that it’s enough to double the per person number of pianos to roughly two in one hundred.d. With those guesses, I can do some simple math and conclude that there are fifty thousand pianos in Chicago.2. How often are pianos tuned? Maybe once a year.3. How long does it take to tune a piano? I’ll say two hours.4. How many hours a year does the average piano tuner work?a. The...American workweek is 40 hours, minus two weeks of vacation...multiply 40 hours by 50 weeks to come up with 2,000 hours a year.b. But piano tuners have to [travel] between pianos.. I’ll guess 20% of their work hours. So...the average piano tuner works 1,600 hours a year. If 50,000 pianos need tuning once a year, and it takes 2 hours to tune one piano, that’s 100,000 total piano-tuning hours. Divide that by the annual number of hours worked by one piano tuner and you get 62.5 piano tuners in Chicago.….[after looking in the yellow pages]....eighty-three listings for piano tuners in the Chicago yellow pages, but many were duplicates, such as businesses with more than one phone number…. But my estimate…looks surprisingly close to the mark. Fermi was renowned for his estimates. With little or no information at his disposal, he would often do back-of-the-envelope calculations like this to come up with a number that subsequent measurement revealed to be impressively accurate.”

Strike the Right Balance between outside views and inside views.To illustrate the difference between outside views and inside views let’s say you were tasked with answering the question: How likely are the Renzettis, a family that lives in an American suburb, to have a pet?The authors describe the “outside view” as the statistical base rate of a particular situation occurring. So you might start by asking “What percentage of Americans have a pet?” Google says 66%, so that’s a good place to start.Then you might try to understand the “inside view”, which is the specifics of the particular case. You might ask whether they live in the suburbs, what their income is, whether anyone is allergic to pets, etc., to get a number that better reflects the situation at hand.Tetlock and Gardner talk about the importance of starting with the outside view, then adjusting your prediction further by incorporating the inside view. Starting with the outside view is crucial because of a psychological concept called anchoring. When making estimates, we tend to start with a number (anchor) and adjust from there. Since we often under-adjust from that anchor, a bad anchor can lead to inaccurate estimates. Classic experiments by Daniel Kahneman and Amos Tversky demonstrate how easily people's judgment can be influenced by any number, even a meaningless one. By starting with the outside view, a forecaster avoids being swayed by an irrelevant anchor and benefits from a more meaningful starting point for analysis.

Find the right balance between underreacting and overreacting to new information.Throughout the book, Tetlock and Gardner highlight how Superforecasters are constantly updating their assessed probabilities as new information comes in. The more often the updates, and the more granular the updates, the better the performance on average. Update your forecasts as new evidence emerges, but avoid making drastic changes without sufficient justification. Continuously evaluate the weight of new evidence against existing knowledge.

Look for the clashing causal forces at work in each problem.The importance of understanding the counterargument to your thesis cannot be understated. As Charlie Munger, one of the best investors of all time has said “I never allow myself to hold an opinion on anything that I don't know the other side's argument better than they do.”All sides of an argument need to be considered and synthesized into a coherent and nuanced understanding. Tetlock and Gardner mention that Superforecasters tend to use words like “however,” “but,” “although,” and “on the other hand,” quite regularly when describing their views. “They talked about possibilities and probabilities, not certainties.”

Strive to distinguish as many degrees of doubt as the problem permits but no more.Amos Tversky, renowned cognitive psychologist known for his influential research on decision-making and cognitive biases, once said that most people have three settings: “gonna happen,” “not gonna happen,” and “maybe.” As in poker, those who can distinguish 60/40 bets from 40/60 bets-or 55/45 from 45/55, are going to have dramatically different outcomes over the long term. Superforecastes tend to go beyond the three settings and get quite granular in their assessed probabilities.At times it can be difficult to get extremely granular probabilities. At that point, a Superforecaster who is trading will either choose not to participate in that market, or find places where the odds are clearly far off from reality.

Strike the right balance between under- and over confidence, between prudence and decisiveness.Superforecasters tend to be a humble group - they’re constantly testing their assumptions, updating their priors, getting feedback. But ultimately, a decision needs to be made. What’s most important is the feedback mechanism to track how you’ve done and update the process over time.

Look for errors behind your mistakes but beware of rearview-mirror hindsight biases.Tetlock and Gardner spend a lot of time talking about hindsight bias. Hindsight bias refers to the tendency for people to perceive past events as more predictable or obvious than they actually were before they occurred. “When forecasts span months or years, the wait for a result allows the flaws of memory to creep in. You know how you feel now about the future. But as events unfold, will you be able to recall your forecast accurately? There is a good chance you won’t. Not only will you have to contend with ordinary forgetfulness, you are likely to be afflicted by what psychologists call hindsight bias.”To minimize hindsight bias, the authors recommend writing down in detail the facts and reasoning you’re using when making the initial prediction. And then conduct rigorous post-mortems, even when things go well. “Not all successes imply that your reasoning was right. You may have just lucked out by making offsetting errors. And if you keep confidently reasoning along the same lines, you are setting yourself up for a nasty surprise.” Poker champion Annie Duke refers to this as resulting, and the consequences of it can be fatal over time.

Bring out the best in others and let others bring out the best in you.In the first year of the Superforecasting competition, teams were 23% more accurate than individuals. Some of the elements that contributed to effective team performance were diverse perspectives, open-mindedness, intellectual humility, effective communication, collaborative synergy, and a commitment to continuous improvement through feedback and learning.

Master the error-balancing bicycle “Just as you can’t learn to ride a bicycle by reading a physics textbook, you can’t become a Superforecaster by reading training manuals.” “Also remember that practice is not just going through the motions of making forecasts, or casually reading the news and tossing out probabilities. Like all other known forms of expertise, Superforecasting is the product of deep, deliberative practice.”