1.
New methods of REF
assessment: Prediction
Markets
Jisc Research Analytics Webinar
1 July 2019
Jackie Thompson
University of Bristol

2.
The Problem

3.
Prediction markets: replication studies

4.
The market interface
https://www.refpredictionmarkets.com/screencast.html

5.
Prediction Market specs
• 21 participants – psychology department
• Postdocs and teaching/research academics
• Financial incentives
• 11 days
• 29 papers

6.
Rationale
• Condenses crowd wisdom
• Dynamically weighted
• Incentives
• Fine-grained rankings
• Burden

7.
Assessment methods
• Close reading 1 (Internal panel)
• Close reading 2 (external rater)
• Prediction market
• Machine learning

8.
Results: Correlations
Spearman's rho
Internal CR External CR
Prediction
Market
Machine
Learning
Internal CR 1 .534** .732*** .655***
External CR 1 .601*** .739***
Prediction Market 1 .643***
Machine Learning 1

9.
5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
5 7 9 11 13
External CR
Internal
CR
Prediction Market
External
CR
Prediction
Market
Machine Learning
0.4
0.6
0.8
1
1 2 3 4
5
7
9
11
13
1 2 3 4
5
7
9
11
13
1 2 3 4

10.
Further plans
• User feedback mostly positive
• Rolling out beyond Bristol soon
(contact jackie.thompson@bristol.ac.uk)

11.
Thank you!
Research Contributors:
Marcus Munafò (U. Bristol)
Ian Penton-Voak (U. Bristol)
Anna Dreber Almenberg
(Stockholm School of Economics)
Felix Holzmeister (U. Innsbruck)