Dec 7, 2022Liked by Michael Story

A good reality check. My perspective is that we can use these observations to decide how the best organisations could, if they choose, do better forecasting / prediction markets than that which has come before. https://twitter.com/thatMikeBishop/status/1600568351151083520?s=20&t=Ymthl5tlKxzVAn_AAaYmmA

Expand full comment

Thanks for writing this! I was planning to start a forecasting tournament in my org at work with around 100 people. But from talking to other people, I couldn’t identify if we could get any signal before running but.

I think you’re spot on that it’s just a minority of people who enjoy logging int Metaculus et al. That’s a good reminder why trying to get buy in from a majority about participating in a forecasting tournament is a hard sell.

Expand full comment

There is another downside to forecasting tournaments, or any kind of prediction contest which has a significant winner-take-all component. Namely, basing your predictions on the actual probabilities of genuinely uncertain events is not a winning strategy if rewards are mostly for top finishers and if most players have the same information.

Essentially, gambling usually beats accurate modeling unless accuracy is genuinely hard / quite different from a general consensus estimate. The only way to fix this is to make losing proportionally painful (prediction markets handle this well).

Consider a game with a hundred players that involves predicting the probability of outcome for twenty coin flips, using Brier scoring to assign player scores. Rewards for top three finishers or what have you.

An accurate player will assign 50% to each flip and always achieve a score of 5, which is the lowest average score anyone can expect.

Assuming players use game theory, this is very unlikely to win. The Nash equilibrium is complicated but in general most players will end up making incorrect predictions for a significant number of flips with a minority doing the same for a smaller number of flips, leaving only a very small chance (less than 1/P) that the correct prediction will take first place.

The same applies even if players are predicting outcomes rather than probabilities, as it may make sense to predict unlikely-but-possible events just to increase variance at the cost of accuracy, and thereby maximize the chance of being first.

Expand full comment

Is “there there” a typo? Sorry, it seems Leo and me are grammar police today. I loved the article.

Expand full comment

Minor typo ICMB rather than ICBM

Expand full comment