The document discusses peer review reform and proposes a new system called IOTA (I Owe the Academy Review). IOTA would allow academics to donate review tokens that represent a pledge to conduct peer reviews. These tokens could then be granted to various peer review initiatives in exchange for conducting experiments and sharing results openly. The goal is to better match reviewers with projects promoting high-quality scholarship, while generating more empirical evidence on effective peer review methods. Several example scenarios for how IOTA could work with different types of journals or research pools are provided.

1. philipncohen.com
philipncohen@gmail.com
@familyunequal
micahaltman.com
micah.altman@gmail.com
@drmaltman
socarxiv.org
Philip N. Cohen & Micah Altman
Academy owned peer review
ARL / SSRC meeting, December 11, 2018
Penny University, by Ricardo Levins Morales

2. Areas of concern for peer review
Review Process
•Time
•Incentives
•Quality
•Bias/Inclusion
Impact on Science and Scholarship
•Accountability
•Costs
•Societal Benefit
•Equity

3. Peer review initiatives
•Post-publication review
(ex. Advances in Combinatorics)
• Badging
(ex. Association for Computing Machinery)
• Open review
(ex. F1000)
• Portable Peer Review
(ex. Rubriq, Axios )
•Including replication
(ex. American Journal of Political Science, American
Economics Review)
Targeted Features
• Organizational decoupling
• Temporal resequencing
• Review criteria
(pre-registration, validity)
• Review target
(experimental design, data)
• Access
(license, sequencing of release)
• Accountability and attribution
(anonymity, APT criteria, citability, metrics)
Exemplar Initiatives

4. Supply: Academics are willing to do peer review --
when it contributes to a better system of scholarship
Demand: New peer review projects need (good) reviewers to launch
& community needs (good) empirical research on peer review
Challenge:
Match reviewers with initiatives that
promote better scholarship
& better evidence of what works
Matching (pent-up) supply and demand

6. 1. Academics donate review tokens
2. Profiles of reviewers are published
3. Tokens are granted to reviewing initiatives,
which pledge research results
4. Initiatives administered by universities,
societies, or journals
5. Research results are openly shared
I Owe the Academy Review (IOTA)
-- How it Works

7. IOTA Design
Design PrinciplesToken
Metadata : person (ORCID); contact method; creation time; review fields
Branding: for reviewer, projects
Service
Provides: create, grant, redeem of tokens; trade
Supports: evaluation of token system, review experiments
Social Contract
Reviewer is absolved from/for doing a closed review/submission now,
pledges to be responsive future request for review.
IOTA pledges to vet recipients with a principled & transparent process,
with flexibility at redemption to defer or trade reviews.

8. Can work within narrow niches or broad fields
Raises awareness of available resources
Facilitates experimentation
Aligns incentives with academy needs
IOTA -- Where to use it.

9. Scenario 1: Overlay journal OA initiative
● E.g., Open Journal of Family Demography announces
formation, calls for submissions and reviewers
● Family demographer receives review request from
paywall journal, declines review and submits IOTA token
● Journal publishes list of available reviewers, family
demographers submit papers
● Journal matches reviewers and papers on open paper
server, publishes decisions and reviews.

10. Scenario 2: Open society journal
● Reviewers volunteer tokens, with metadata
● Society submits proposal to use IOTA tokens
● Journal matches papers and reviewers using token
metadata
● Society journal facilitates review as they do now

11. Scenario 3: Open reviewed research pool
● Reviewers volunteer tokens, with metadata
● Token list is public
● Researchers self-nominate papers, reviewers volunteer or
admins match on metadata
● Open reviews and research available for publication

12. IOTA Review
TIME
INCENTIVES
QUALITYBIAS
ACCOUNTABILITY
Open review pool
Public reviewer lists
Allows experimentationAllows experimentation
Academy / society
collaborations