Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Prediction markets

39 views

Published on

The aim of this project was to develop and evaluate a prediction market tool that higher education institutions can use to rank outputs for potential REF submissions as part of their internal REF (Research Excellence Framework) planning. A prediction market is a bit like the stock market, except instead of investing in companies, participants invest in the outcomes of future events (in this case, ratings of research outputs). This presentation will give some background to the project and details of the prediction markets that have been tested with Units of Assessment at the University of Bristol. It will include a demo of the tool used and the lessons learned from the first round of markets.  

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Prediction markets

  1. 1. New methods of REF assessment: Prediction Markets Jisc Research Analytics Webinar 1 July 2019 Jackie Thompson University of Bristol
  2. 2. The Problem
  3. 3. Prediction markets: replication studies
  4. 4. The market interface https://www.refpredictionmarkets.com/screencast.html
  5. 5. Prediction Market specs • 21 participants – psychology department • Postdocs and teaching/research academics • Financial incentives • 11 days • 29 papers
  6. 6. Rationale • Condenses crowd wisdom • Dynamically weighted • Incentives • Fine-grained rankings • Burden
  7. 7. Assessment methods • Close reading 1 (Internal panel) • Close reading 2 (external rater) • Prediction market • Machine learning
  8. 8. Results: Correlations Spearman's rho Internal CR External CR Prediction Market Machine Learning Internal CR 1 .534** .732*** .655*** External CR 1 .601*** .739*** Prediction Market 1 .643*** Machine Learning 1
  9. 9. 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 0.4 0.6 0.8 1 5 7 9 11 13 5 7 9 11 13 External CR Internal CR Prediction Market External CR Prediction Market Machine Learning 0.4 0.6 0.8 1 1 2 3 4 5 7 9 11 13 1 2 3 4 5 7 9 11 13 1 2 3 4
  10. 10. Further plans • User feedback mostly positive • Rolling out beyond Bristol soon (contact jackie.thompson@bristol.ac.uk)
  11. 11. Thank you! Research Contributors: Marcus Munafò (U. Bristol) Ian Penton-Voak (U. Bristol) Anna Dreber Almenberg (Stockholm School of Economics) Felix Holzmeister (U. Innsbruck)

×