The document discusses a webinar on new methods for research assessment using prediction markets, conducted by Jackie Thompson from the University of Bristol on July 1, 2019. It details the structure of a prediction market involving 21 participants from psychology, the results of correlations with machine learning, and plans for future implementation based on positive user feedback. The aim is to enhance research evaluation by condensing crowd wisdom and providing fine-grained rankings.

1. New methods of REF
assessment: Prediction
Markets
Jisc Research Analytics Webinar
1 July 2019
Jackie Thompson
University of Bristol

2. The Problem

3. Prediction markets: replication studies

4. The market interface
https://www.refpredictionmarkets.com/screencast.html

5. Prediction Market specs
• 21 participants – psychology department
• Postdocs and teaching/research academics
• Financial incentives
• 11 days
• 29 papers

6. Rationale
• Condenses crowd wisdom
• Dynamically weighted
• Incentives
• Fine-grained rankings
• Burden

7. Assessment methods
• Close reading 1 (Internal panel)
• Close reading 2 (external rater)
• Prediction market
• Machine learning

8. Results: Correlations
Spearman's rho
Internal CR External CR
Prediction
Market
Machine
Learning
Internal CR 1 .534** .732*** .655***
External CR 1 .601*** .739***
Prediction Market 1 .643***
Machine Learning 1

9. 5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
0.4 0.6 0.8 1
5
7
9
11
13
5 7 9 11 13
External CR
Internal
CR
Prediction Market
External
CR
Prediction
Market
Machine Learning
0.4
0.6
0.8
1
1 2 3 4
5
7
9
11
13
1 2 3 4
5
7
9
11
13
1 2 3 4

10. Further plans
• User feedback mostly positive
• Rolling out beyond Bristol soon
(contact jackie.thompson@bristol.ac.uk)

11. Thank you!
Research Contributors:
Marcus Munafò (U. Bristol)
Ian Penton-Voak (U. Bristol)
Anna Dreber Almenberg
(Stockholm School of Economics)
Felix Holzmeister (U. Innsbruck)