Read or use AI analysis.

  • Use AI if you are looking for a step by step / how to guide for becoming a superforecaster.
  • Read if you are looking to get the background on the largest results based prediction study with some process and tips thrown in.

Project Background

In 2005 Philip Tetlock published a book called “Expert Political Judgement” where his main thesis can be boiled down to “Most expert predictions turn out to be no more accurate than a chimpanzee throwing a dart at a dart board.” It gets even more damning when his data showed that there is a direct correlation between how famous an expert was and how often they were wrong. Said differently, the more famous an expert was, the more likely they were to be wrong.

Why Track Prediction Accuracy?

We use predictions to help decide which actions we should take. What is the weather forecast for today? Is it supposed to rain? or snow? How likely is it that Afghanistan has weapons of mass destruction? Should we launch an all out war to prevent the harm that they could do with these weapons?

We don’t want to get caught out in the rain without an umbrella and we don’t want our home to get destroyed by a bomb. At the same time we don’t want to carry our winter coat around in the summer and launching a military invasion will cause it’s own batch of casualties. If a particular weather report is consistently wrong, you are going to want to switch to a new provider.

Tracking accuracy allows the predictor to improve over time by understanding what went wrong where and make corrections for next time. It also allows the general population to know who is the most trust worthy.

Tips for Making Predictions

Now for the good stuff, how to make your own predictions better.

The Good Judgement Project will pose questions that it asks for predictions for rather than asking for open ended predictions. The first step is to break down the problem into several manageable sub-problems. An example of this could be “How likely is X candidate to when the presidential election?”. You could turn this into 50 sub-problems asking how likely it is that the candidate when each state. This makes the large problem a bit easier to solve.

From there you need to take the “outside view”. Take a sub-problem and determine what the base rate would be. The outside view strips the specificity out of the problem, so an example would be how likely is the Republican candidate to win Texas? This allows us to anchor any of our specific or “inside view” variance from the general case. This is needed because as humans, we tend to get hung up on our gut instinct. Taking the general view as our first estimate helps us to not over emphasis the specifics of this situation.

Once you have the outside view, you can add the inside view adjustment to your prediction. Now roll up all of your sub-problems and you will have a prediction for you large problem.

Now that you have your initial prediction you can go from good to great by continually refining your prediction as new information comes to light. Did a candidate X do really well in a debate, or have a disastrous town hall? Adjust your prediction. Making adjustments is a balancing act between the new information and all of your past research. If a republican candidate has a poor debate performance, they are still likely to win Texas and many other red heavy states so don’t make to harsh of corrections.

Another way to refine your prediction is to pool your prediction with other predictors who did their own analysis. Tetlock found a sizable accuracy increase when he pooled many predictions together, especially when he weighted their impact based on how accurate they have been in the past. This is effective because it can smooth out the noise that each predictor has. Maybe you favor one political candidate or another? Maybe you live in an area that is highly polarized and even being conscious of that bias is not enough to fully remove it.

The final step is to review your predictions after the answer is known. Go back and look through your research and identify mistakes that were made or areas where you didn’t gather enough information. This will help you correct for the inside view in the future.

“Commitment to self-improvement is crucial for superforecasting success.” - Philip Tetlock