The document discusses the lack of political diversity in psychology. It notes that psychology has become increasingly liberal over the last 50 years. Some arguments made include that a lack of political diversity could undermine the validity of social psychology research findings and that increasing political diversity may improve the field. The document reviews surveys of social psychologists finding a small minority identify as conservative and most as liberal. It also discusses whether this lack of diversity could lead to biases in what types of research findings are published or supported.

1. @jayvanbavel
New York University
psyarxiv.com/6k3j5
Is there ideological bias
in psychology research?

2. @jayvanbavel
New York University
psyarxiv.com/6k3j5
Is there ideological bias
in psychology research?
Diego Reinero, Julian Wills,
Billy Brady, Peter Mende-
Siedlecki, Jarrett Crawford

5. “I consider the rapid loss of political diversity, over
the last 20 years, to be the second-greatest existential
threat to the field of social psychology, after the
‘replication crisis’ “
– Jon Haidt, 2016

6. Royal Netherlands Academy of Arts &
Sciences

8. 1. Lack of political diversity in psychology, mostly in last 50 years
2. Lack of political diversity undermines validity of social psychological
science
3. More political diversity would improve social psychological science
4. Lack of non-liberals in the field due to self-selection, hostile
climate, and discrimination
Duarte et al (2015) BBS article

9. 1. Lack of political diversity in psychology, mostly in last 50 years
2. Lack of political diversity undermines validity of social psychological
science
3. More political diversity would improve social psychological science
4. Lack of non-liberals in the field due to self-selection, hostile
climate, and discrimination
Duarte et al (2015) BBS article

10. Fig. 1. Percentage of academics
supporting the Conservatives and
major left-wing parties over time.
College Faculty Political
Leanings (1989 - 2014)
Faculty Voter Registration in
Economics, History, Journalism,
Law, and Psychology (2016)
A lack of political diversity
Langbert et al (2016)

11. Fig. 1. Percentage of academics
supporting the Conservatives and
major left-wing parties over time.
College Faculty Political
Leanings (1989 - 2014)
Faculty Voter Registration in
Economics, History, Journalism,
Law, and Psychology (2016)
Langbert et al (2016)
A lack of political diversity

12. Within Psychology

13. Within Social Psychology
• SPSP listserv survey in 2012
• Survey 1: N = 508
• 3.9% - 17.9% conservative
• 89.6% - 63.2% liberal
• Survey 2: N = 292
• 6% conservative
• 85% liberal
• SESP survey in 2015
• N = 335
• 2.5% right-of-center
• 89.3% left-of-center
Inbar & Lammers (2012); von Hippel & Buss (2015)

14. • SPSP listserv survey in 2012
• Survey 1: N = 508
• 3.9% - 17.9% conservative
• 89.6% - 63.2% liberal
• Survey 2: N = 292
• 6% conservative
• 85% liberal
• SESP survey in 2015
• N = 335
• 2.5% right-of-center
• 89.3% left-of-center
Inbar & Lammers (2012); von Hippel & Buss (2015)
Within Social Psychology

15. 1. Lack of political diversity in psychology, mostly in last 50 years
2. Lack of political diversity undermines validity of social
psychological science
3. More political diversity would improve social psychological science
4. Lack of non-liberals in the field due to self-selection, hostile
climate, and discrimination
Duarte et al (2015) BBS article

16. “…we suggest that one largely overlooked cause of failure is
a lack of political diversity. We review evidence suggesting
that political diversity and dissent would improve the
reliability and validity of social psychological science.” pg.1
Duarte et al (2015) BBS article

17. Liberal bias in standards of evidence?
• Asked research psychologists to rate the suitability of a manuscript
for publication.
• The methods and analyses were held identical for all reviewers;
however finding was either:
• Leftist political activists
• Mentally healthier than comparison group of non activists
• Mentally less healthy than comparison group of non activists
• When leftist activists were mentally healthier, the liberal reviewers
rated the manuscript as more publishable (Fint = 5.66, p = .025)
• Less liberal reviewers showed no such bias
Abramowitz (1975)
…findings that are at odds with liberal values are at risk of
being judged more harshly than they deserve; findings that
support liberal values are at risk of being waived through
without sufficiently critical review.” – Duarte et al (2015)
pg. 12

18. Eiten et al., (2018)

19. r = .16 r = .10
Eiten et al., (2018)

20. “Expecting trustworthy results on politically charged topics from
an ‘ideologically incestuous community,’ he explained, is
‘downright delusional.’"
“…even honest researchers are affected by the unconscious bias
that creeps in when everyone thinks the same way. Certain
results — especially when they reinforce commonly held ideas —
tend to receive a lower standard of scrutiny. This might help
explain why, when the Open Science Collaboration’s
Reproducibility Project recently sought to retest 100 social
science studies, the group was unable to confirm the original
findings more than half the time.”

21. Does this political imbalance
mean our science is less
robust/replicable?

22. The Partisan Brain
Van Bavel & Pereira (2018)

23. The Partisan Brain
Van Bavel & Pereira (2018)

24. The Partisan Brain
Van Bavel & Pereira (2018)

25. The Partisan Brain
Van Bavel & Pereira (2018)

26. - - - - - - - - >
Political lean of research Did it replicate?
Current research
Reinero, Wills, Brady, Mende-Siedlecki, Crawford, & Van Bavel, under review

27. - - - - - - - - >
Political lean of research Did it replicate?
Current research
WWBND: What would Brian Nosek do?
Reinero, Wills, Brady, Mende-Siedlecki, Crawford, & Van Bavel, under review

28. - - - - - - - - >
Political lean of research Did it replicate?
Current research
Reinero, Wills, Brady, Mende-Siedlecki, Crawford, & Van Bavel, under review

29. - - - - - - - - >
Political lean of research Did it replicate?
Current research
Reinero, Wills, Brady, Mende-Siedlecki, Crawford, & Van Bavel, under review

30. Databases of Replications
• Reproducibility Project: Psychology
• Attempted to replicate 100 psychology studies
• Many Labs 1
• Attempted to replicate 13 psychology effects (36 samples per effect)
• Many Labs 2
• Attempted to replicate 28 psychology effects (58 unique samples for 13 of the
effects, and 57 unique samples for the other 15 effects)
• Many Labs 3
• Attempted to replicate 10 psychology effects (20 samples per effect)
• APS Registered Replication Reports
• Social Psychology: Special Issue
• Pre-Publication Independent Replication
• Curate Science
• 195 unique articles
• 218 unique replication attempts

31. Study 1: Procedure
• Recruited 6 social psych doctoral raters from
across political spectrum via SPSP listserv;
matched on various characteristics
Political
Orientation
Very Lib. Very Lib. Moderate Moderate Very
Con.
Very
Con.
Sex Male Female Male Female Male Female
Age 25 26 28 28 25 23
Ethnicity White White White Multicultu
ral
White White
Religiosity† 0 0 10 25 100 90
Years lived in
US
25 26 28 28 25 23
Pre-Doctoral
Status
2nd year 2nd year 2nd year 3rd year 2nd year 2nd year
Sub-discipline Social Social Social Social Social Social
†Religiosity scale: 0 = not at all religious, 100 = very religious

32. • Given sample definition of liberalism and conservatism
+ flattering/unflattering portraits of liberals and
conservatives (Tetlock & Mitchell, 1993)
• Two practice rounds to build reliability
• Given obvious example abstracts representing each
endpoint of our “political lean” scale (raters unaware of
this aspect before rating)
• Raters given feedback after each practice round
• Then, rated political lean of original abstracts from
actual database
• Raters likely unaware of the replication attempt
results; blind to other features of the article (e.g.,
journal, year)
Study 1: Procedure

33. Study 1: Procedure

34. • Pre-registered rule: if > 4 raters said “does not
apply”, abstract deemed not politically relevant
• 52% of abstracts rated politically relevant
• If > 3 raters said it fell on 1-5 scale, deemed
politically relevant and averaged ratings ! used
for key analyses
Study 1: Procedure

35. • “Adversarial” collaboration
• Similar study, using MTurkers as raters
• Allows us to test if our effects replicate with a
different (and larger) sample of raters
• 511 Mturkers (47% male; Mavg = 37 years)
• Each coder rated random sample of 10 abstracts out
of possible 195
• Each abstract rated by between 18 and 31
participants (M = 26.15, SD = 2.33)
• Additional items
• Political knowledge
• Political engagement
• Political ideology
• Demographics
Jarret
Crawford
Study 2: Procedure

36. Ideological Distribution of Literature

37. Ideological Distribution of Literature

38. Ideological Distribution of Literature
Very left leaning Very right leaning
Psychologists are publishing many findings that
go against their political ideology

39. Does ideology predict replicability?
• Do you think liberal or
conservative findings are
less likely to replicate?
• Liberal less likely?
• Conservative less likely?
• Both less likely?
• Neither less likely?

40. Does ideology predict replicability?
• Do you think liberal or
conservative findings are
less likely to replicate?
• Liberal less likely?
• Conservative less likely?
• Both less likely?
• Neither less likely?
N = 699!

41. p = .781, OR = 1.03
Ideology does NOT predict replicability
p = .432, OR = 0.98

42. p = .781, OR = 1.03
Ideology does NOT predict replicability
p = .432, OR = 0.98
There is no evidence that liberal or conservative
findings are less replicable—across coder ideology

43. p = .781, OR = 1.03
Ideology does NOT predict replicability
p = .432, OR = 0.98
There is no evidence that liberal or conservative
findings are less robust (power, effect size)

44. Findings
• Our field has many more liberals than conservatives
(or non-liberals)
• Yet, there is much less ideological imbalance in the
results that we publish
• Most research is either not politically relevant or
shows no strong political lean
• Research that is politically relevant is not associated
with replicability (0.4% variance explained)
• This political imbalance does not seem to reduce
the replicability or robustness of research
• Lack of association holds across rater ideology
• Lack of association of replicability holds after adjusting
for covariates (sample size, effect size)

45. Duarte et al (2015) BBS article
1. Lack of political diversity in psychology, mostly in last 50 years
2. Lack of political diversity undermines validity of social psychological
science
3. More political diversity would improve social psychological science
4. Lack of non-liberals in the field due to self-selection, hostile
climate, and discrimination

46. • We need a wide latitude for political discourse—
this is especially critical for social & political
psychologists
• I have many colleagues who have been
maligned, threatened, harassed for their
research (I have had all three myself)
• But…we need rigorous analyses of our work
• Cherry picking examples is easy, and can be done
to fit almost any ideological narrative
• Claims of bias among scientists deserve the same
scrutiny we would give to other claims of bias
Discussion

47. Discussion
• Our intuitions are not
great: People overestimate
liberal bias in science

48. • Our intuitions are not
great: People overestimate
liberal bias in science
Discussion
Eiten et al., (2018)

49. Discussion
• Our intuitions are not
great: People overestimate
liberal bias in science

50. • If anything, there seems to be a political
extremism effect (exploratory)
Discussion

51. • Duarte et al admit that
liberal bias is likely limited
to a narrow range of studies
on taboo topics
• It seems that we (and the
media) vastly overgeneralize
Discussion

52. • Many allegations of liberal bias do not withstand
scrutiny
Discussion

53. • Many allegations of liberal bias do not withstand
scrutiny
Discussion

54. • Many allegations of liberal bias do not withstand
scrutiny
Discussion

55. • Many allegations of liberal bias do not withstand
scrutiny
Discussion

56. • Many allegations of liberal bias do not withstand
scrutiny
• Abramowitz, 1975: Assigned reviewer ideology
based on contributors to journals & societies
• More liberal: Fellows of APA Division 9, SPSSI, editors/
contributors to Journal of Social Issues
• Less liberal: Fellows of APA Division 8, editors/contributors to
JPSP, Journal of Consulting and Clinical Psychology, Journal of
Personality, Sociometry
• Many effects were non-significant or weak
Discussion
“…the amount of bias detected might be so slight as to
be meaningless in the real world of publish or perish”

57. • Many allegations of liberal bias do not withstand
scrutiny
• This issue might itself be a Rorschach test for our
own ideology or identity
Discussion

58. • Our system is almost
perfectly designed to avoid
groupthink
• We made independent
judgments
• Judgments are anonymous
• We have an identity and
norms focused on accuracy
Discussion

59. • Scientific identity and
norms value accuracy
Discussion

60. • Scientific identity and
norms value accuracy
Discussion

61. • Scientific identity and
norms value accuracy
Discussion

62. • What predicts replication: Effect size, sample size,
and other indices of robust science
• Shift focus from politics of scientists, to their
practices
Discussion

63. “In disputes upon moral or scientific points, ever
let your aim be to come at truth, not to conquer
your opponent. So you never shall be at a loss in
losing the argument, and gaining a new discovery.”
-Arthur Martine, 1860

64. Thank you
Diego
Reinero
Julian
Wills
Peter Mende-
Siedlecki
Jarret
Crawford
Billy Brady
Jonathan Haidt
Read more here:
psyarxiv.com/6k3j5