Excerpt
2020, Paper, "There is evidence that prediction markets are useful tools to aggregate information about researchers’ beliefs about scientific results including forecasting the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We will set up prediction markets for hypotheses tested in DARPA’s Next Generation Social Science (NGS2) program. We will invite researchers to bet on whether 22 hypotheses will be supported or not. We define support as a test result in the same direction as the hypothesized, with a Bayes Factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared to the null hypothesis). In addition to betting on this binary outcome, we will ask participants to bet on the expected effect size (in Cohen’s d) for each hypothesis. We recruit at least 50 participants that sign up to participate in these markets. Participants will also complete a survey on both the binary result and the effect size. Our goals are to elicit peer beliefs about the outcomes of the hypotheses tested in NGS2, and to test if these peer beliefs can predict the outcomes of novel experimental designs rather than replications. This study will increase our knowledge about the predictability of scientific results and the dynamics of hypothesis testing."
Non-HKS Author Website - Yiling Chen