The document discusses issues related to the replication crisis in psychology and potential solutions. It notes that questionable research practices like p-hacking and HARKing are common but unintentional. Solutions proposed include transparency through open science, pre-registration of studies including pre-reviews, direct replication of studies, and higher evidentiary standards. Institutional changes are also needed to incentivize practices like pre-registration and increasing acceptance of replication studies. While rigorous methods may initially lower productivity, they can increase it long-term by allowing easier reuse of materials and data and identifying reliable findings sooner.

1. FROM
REPLICATION CRISIS
TO
CREDIBILITY REVOLUTION
KOKI IKEDA, CHUKYO UNIVERSITY
2018 NOV. 11

2. A CRISIS?
Psychology is f**ked

3. Diederik Stapel
The Lord of Data

4. English translation is now available!!
https://errorstatistics.com/2014/12/21/derailment
-faking-science-a-true-story-of-academic-fraud-
by-diederik-stapel-translated-into-english/

6. QRP
• Questionable research practices,
such as 𝑝 hacking, HARKing etc.
• Most of them are probably
unconscious without bad
intentions.
• Cannot be treated in the same as
frauds, such as data fabrication
etc.

8. QRPs 自己告白率 (%) 弁明可能性
1. 従属変数を選択的に報告 66.5 1.84
2. 結果が有意かどうかを見てからさらにデータを採る 58.0 1.79
3. 実験条件の選択的報告 27.4 1.77
4. 望む結果が出たので予定より早くデータ取得を終了 22.5 1.76
5. p 値を「切り捨て」て報告（例：5.4%を5%に） 23.3 1.68
6. うまくいった研究だけを選択的に報告 50.0 1.66
7. データ除外の影響を見てから除外するかどうかを決定 43.4 1.61
8. 予想していなかった結果を予測していたかのように報告 35.0 1.50
9. 実際は不確かなのに、性別など人口統計学的変数に
よって結果は影響されないと報告
4.5 1.32
10. データの改竄 1.7 0.16
(0 = no, 1 = possibly, and 2 = yes)

10. A REVOLUTION
Because we already found
solutions

12. SOLUTIONS
• Transparency or Open Science
• Pre-registration including ones
with pre-review
• Direct replication
• Higher standards of evidence

13. INSTITUTIONAL CHANGES
• They do not only depend on one’s
morality, but control one’s
behavior by changing incentives.
• In-principle acceptance for prereg’d
reports.
• Increased acceptance rate for replications.
• Formal requirements from journals.

14. IMPLICATIONS
• Productivity
• Creativity
• Progress (skipped in this talk)

15. PRODUCTIVITY
• Slower science?
• Rigor requires more time, effort,
and resources (e.g. larger N)

18. BUT IN A LONG RUN…
• Transparency allows easy re-use
of materials and data.
• Registered reports format allow
us to publish papers regardless of
the results
• Replications allow us to pre-select
reliable findings.

19. SOCIAL DILEMMA
• Personal incentives may
contradict with collective benefits.
• Institutional support is necessary.

20. CREATIVITY
• Lessens riskiness?
• Narrows diversity in data
collection? (e.g. MTurknization)
• Less freedom to choose methods
and standards?

25. Trustworthiness Dominance
http://tlab.princeton.edu/demonstrations/

26. “We will address this question by
replicating Oosterhof and Todorov’s
(2008) methodology across all world
regions (Africa, Asia, Central America
and Mexico, Eastern Europe, Middle
East, USA and Canada, Australia and
New Zealand, Scandinavia, South
America, UK, Western Europe, total N ≥
9525) and using a diverse set of face
stimuli.”

27. If we uncover systematic regional
differences in social judgments, this will
fundamentally change how social
perception research is done and
interpreted. If we find consistency
across regions, this will ground future
theory in an appropriately powered
empirical test of an underlying
assumption.