Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID events

Alexander Gruen; Karl R Mattingly; Ellen Morwitch; Frederik Bossaerts; Manning Clifford; Chad Nash; John P A Ioannidis; Anne-Louise Ponsonby

doi:10.1016/j.ebiom.2023.104783

Machine learning augmentation reduces prediction error in collective forecasting: development and validation across prediction markets with application to COVID events

EBioMedicine. 2023 Oct:96:104783. doi: 10.1016/j.ebiom.2023.104783. Epub 2023 Sep 12.

Authors

Alexander Gruen¹, Karl R Mattingly², Ellen Morwitch¹, Frederik Bossaerts², Manning Clifford³, Chad Nash², John P A Ioannidis⁴, Anne-Louise Ponsonby⁵

Affiliations

¹ The Florey Institute of Neuroscience and Mental Health, Melbourne, Australia.
² Dysrupt Labs by SlowVoice, Melbourne, Australia.
³ Boston Consulting Group, Melbourne, Australia.
⁴ Stanford Prevention Research Center, Department of Medicine, and Departments of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Meta-Research Innovation Center at Stanford, Stanford, CA, USA.
⁵ The Florey Institute of Neuroscience and Mental Health, Melbourne, Australia; Murdoch Children's Research Institute, Royal Children's Hospital, Melbourne, Australia; Centre of Epidemiology and Biostatistics, School of Population and Global Health, University of Melbourne, Australia. Electronic address: annelouise.ponsonby@florey.edu.au.

Abstract

Background: The recent COVID-19 pandemic highlighted the challenges for traditional forecasting. Prediction markets are a promising way to generate collective forecasts and could potentially be enhanced if high-quality crowdsourced inputs were identified and preferentially weighted for likely accuracy in real-time with machine learning.

Methods: We aim to leverage human prediction markets with real-time machine weighting of likely higher accuracy trades to improve performance. The crowd sourced Almanis prediction market longitudinal platform (n = 1822) and Next Generation Social Science (NGS2) platform (n = 103) were utilised.

Findings: A 43-feature model predicted accurate forecasters, those with top quintile relative Brier accuracy, with subsequent replication in two out-of-sample datasets (p_both <1 × 10^-9). Trades graded by this model as having higher accuracy scores than others produced a greater AUC temporal gain in the overall market after vs before trade. Accuracy score-weighted forecasts had higher accuracy than market forecasts alone, particularly when the two systems disagreed by 5% or more for binary event prediction: the hybrid system demonstrating substantial % AUC gains of 13.2%, p = 1.35 × 10^-14 and 13.8%, p = 0.003 in two out-of-sample datasets. When discordant, the hybrid model was correct for COVID-19 event occurrence 72.7% of the time vs 27.3% for market models, p = 0.007. This net classification benefit was replicated in the separate Almanis B dataset, p = 2.4 × 10^-7.

Interpretation: Real-time machine classification followed by weighting human trades according to likely accuracy improves collective forecasting performance. This could provide improved anticipation of and thus response to emerging risks.

Funding: This work was supported by an AusIndustry R and D tax incentive program from the Department of Industry, Science, Energy and Resources, Australia, to SlowVoice Pty Ltd. (IR 2101990) and Fellowship (GNT 1110200) and Investigator grant (GNT 1197234) to A-L Ponsonby by the National Health and Medical Research Council of Australia.

Keywords: COVID; Collective intelligence; Forecast accuracy; Human-machine forecast; Prediction market.

MeSH terms

Australia
COVID-19* / epidemiology
Forecasting
Humans
Machine Learning
Pandemics*