8 Comments
Dec 7, 2022Liked by Michael Story

A good reality check. My perspective is that we can use these observations to decide how the best organisations could, if they choose, do better forecasting / prediction markets than that which has come before. https://twitter.com/thatMikeBishop/status/1600568351151083520?s=20&t=Ymthl5tlKxzVAn_AAaYmmA

Expand full comment

Thanks for writing this! I was planning to start a forecasting tournament in my org at work with around 100 people. But from talking to other people, I couldn’t identify if we could get any signal before running but.

I think you’re spot on that it’s just a minority of people who enjoy logging int Metaculus et al. That’s a good reminder why trying to get buy in from a majority about participating in a forecasting tournament is a hard sell.

Expand full comment

There is another downside to forecasting tournaments, or any kind of prediction contest which has a significant winner-take-all component. Namely, basing your predictions on the actual probabilities of genuinely uncertain events is not a winning strategy if rewards are mostly for top finishers and if most players have the same information.

Essentially, gambling usually beats accurate modeling unless accuracy is genuinely hard / quite different from a general consensus estimate. The only way to fix this is to make losing proportionally painful (prediction markets handle this well).

Consider a game with a hundred players that involves predicting the probability of outcome for twenty coin flips, using Brier scoring to assign player scores. Rewards for top three finishers or what have you.

An accurate player will assign 50% to each flip and always achieve a score of 5, which is the lowest average score anyone can expect.

Assuming players use game theory, this is very unlikely to win. The Nash equilibrium is complicated but in general most players will end up making incorrect predictions for a significant number of flips with a minority doing the same for a smaller number of flips, leaving only a very small chance (less than 1/P) that the correct prediction will take first place.

The same applies even if players are predicting outcomes rather than probabilities, as it may make sense to predict unlikely-but-possible events just to increase variance at the cost of accuracy, and thereby maximize the chance of being first.

Expand full comment

Is “there there” a typo? Sorry, it seems Leo and me are grammar police today. I loved the article.

Expand full comment

Should be: It seems like Leo and I are….

Who polices the police?

Expand full comment

I AM FIRE. I AM DEATH. I AM wrong here. Thank you for pointing that out. As a fellow fan, I love that your name is Bard the Bowman. :)

Yeah, I realize now that for the subject of the sentence, the correct option is “Leo and I.” I wasn't aware of this exception for the first-person pronoun. Of course, I find out humans do this out of politeness. I am a selfish, greedy dragon, so I am used to talking about myself.

I appreciate it again. (Also, rumours about me having a bare patch on my belly of armour are fake news, so don't even think of shooting me down with a black arrow). Sincerely, Smaug.

Expand full comment

Minor typo ICMB rather than ICBM

Expand full comment
author

got it thanks!

Expand full comment