ACES: Crowd-based Forecasting of World Events

ACEing Predictions

Challenge

Decision makers (e.g. Congress) need forward looking opinions from subject matter experts to make policy decisions (e.g. the “National Intelligence Estimates”). They don’t know whom to ask, what to ask, how to combine their opinions, and how to interpret them. The main objective of this project was to provide accurate, timely, and continuous probabilistic forecasts and early warning of global events, by aggregating the judgments of many widely-dispersed analysts. A secondary objective was to enable decision makers to understand forecast results.

Solution

Funded by the Intelligence Advanced Research Projects Activity (IARPA) Aggregative Contingent Estimation (ACE) program, ARA developed methods for eliciting and aggregating many individual judgments to improve forecast accuracy beyond what any one person or small group could provide. These methods move beyond the state-of-the-art in crowd sourcing and prediction markets by assigning appropriate weights and/or transformations to individual opinions versus a simple averaging of those opinions. The methods developed on this project were incorporated in ARA's Aggregative Contingent Estimation System (ACES), a virtual collaboration platform that allows each person to identify topics of interest, engage with other forecasters and compete to make the best predictions. To test the accuracy of the methods, ARA invited volunteers from the general public to use the ACES platform to forecast events in areas such as politics, military, economics, science and technology, and social affairs. In return, ACES provided participants with feedback on their contributions and predictive successes, including tips and training on how to improve individual forecast accuracy. 

Impact

The technology developed on the ACES project was very successful in forecasting world events, including topics in politics, business, and economy, picking the right outcome in 82% of all cases.

Applied Cognition

Cognitive Solutions

Our cognitive capabilities combined with our expertise in the physical sciences allow us to develop solutions that enhance the performance of... more

Modeling and Simulation

Modeling and Simulation

Tools for decision support and training for defense applications, the transportation community, medical and life sciences, and emergency... more