The study on whether author's gender affects academic article evaluation

1. Author's gender aects rating of academic article:
Author's gender aects rating of academic article:
evidence from an incentivized, deception-free laboratory experiment
(published in European Economic Review)
Magdalena Smyk
Michaª Krawczyk
Findings from the GENDEQU Project
University of Warsaw
June 22, 2016

2. Author's gender aects rating of academic article:
Table of contents
1 Introduction: an idea
2 Our experiment
3 Results
4 Conclusions

3. Author's gender aects rating of academic article:
Introduction: an idea
Introduction: an idea

4. Author's gender aects rating of academic article:
Introduction: an idea
Gender equality at the Univeristy
Gender-stereotypes - do they exist in academia?
Does gender of the scientist aect evaluation of his/her
work?

5. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration
How to check whether individuals take into account gender of a scientist
when they evaluate her/his work?

6. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration
How to check whether individuals take into account gender of a scientist
when they evaluate her/his work?
in the real life: confounding eects (like dierences in quality etc.)
ideally: comparing two researchers who dier only by gender

7. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration
How to check whether individuals take into account gender of a scientist
when they evaluate her/his work?
in the real life: confounding eects (like dierences in quality etc.)
ideally: comparing two researchers who dier only by gender
SOLUTION: create such circumstances in an EXPERIMENT

8. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration - other experiments
Borsuk et al. (2009): reviewing article with four dierent author
designations (female, male, initials, and no name at all)

9. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration - other experiments
Borsuk et al. (2009): reviewing article with four dierent author
designations (female, male, initials, and no name at all)
result: no gender bias was observed

10. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration - other experiments
Borsuk et al. (2009): reviewing article with four dierent author
designations (female, male, initials, and no name at all)
result: no gender bias was observed
Knobloch-Westerwick et al. (2013): reviewing conference abstracts
ostensibly signed with female or male names

11. Author's gender aects rating of academic article:
Introduction: an idea
Inspiration - other experiments
Borsuk et al. (2009): reviewing article with four dierent author
designations (female, male, initials, and no name at all)
result: no gender bias was observed
Knobloch-Westerwick et al. (2013): reviewing conference abstracts
ostensibly signed with female or male names
result: `male-authored' texts were rated higher on average, especially
when the topic was male-typed

12. Author's gender aects rating of academic article:
Our experiment
Our experiment

13. Author's gender aects rating of academic article:
Our experiment
Treatments
Articles
5 pairs written by mixed-gender couples
in each pair - one paper published in top economic journal (e.g.The
Quarterly Journal of Economics) and one working paper
no names
two by two design: a female economist, a male economist, a
young female economist, a young male economist

14. Author's gender aects rating of academic article:
Our experiment
What was the experiment about?
Task:
Each subject had to read one article from the pool and answer on a
scale 0-7 how she evaluate:
author's competence
language quality
methodology
literature review
scientic signicance
scientic quality
intelligibility

15. Author's gender aects rating of academic article:
Our experiment
What was the experiment about?
Task:
At the end, we asked if they believe the paper was published in one
of the top economic journals or not at all.
Reward: 80 zlotys (about 20EUR 25USD)
+ 10 zlotys - if the guess on publication was correct.

16. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)

17. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)
conditions preventing consultation with external sources

18. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)
conditions preventing consultation with external sources
deception free experiment

19. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)
conditions preventing consultation with external sources
deception free experiment
additional dimension: young

20. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)
conditions preventing consultation with external sources
deception free experiment
additional dimension: young
novel way of introducing a treatment

21. Author's gender aects rating of academic article:
Our experiment
What's dierent/new?
entire paper and ample time to evaluate a scientist (instead of just
an abstract or short paper)
conditions preventing consultation with external sources
deception free experiment
additional dimension: young
novel way of introducing a treatment
several additional questions

22. Author's gender aects rating of academic article:
Our experiment
Design
10 sessions (+ 2 additional blind treatment sessions)
190 subjects ( + 34 subjects in blind treatment sessions)

23. Author's gender aects rating of academic article:
Our experiment
Design
10 sessions (+ 2 additional blind treatment sessions)
190 subjects ( + 34 subjects in blind treatment sessions)
Subjects
Subjects/ Young Female Young Male Blind
Treatment female regardless male regardless
of age of age
Males 19 18 23 13 18
Females 26 31 25 35 16
Economists 20 28 31 28 13
Other majors 14 11 9 12 18
Non-students 6 5 3 3 3
Mean age 22.7 22.6 22.4 22.7 24.8

24. Author's gender aects rating of academic article:
Our experiment
Hypotheses
Hypothesis 1
Female authors will receive lower ratings than authors of unspecied
gender and, especially, males, and their papers will less often be judged
as published.
Hypothesis 2
Likewise, young female economists will be judged as inferior with regard
to female economists of unspecied age.
Intuition
The gender eect may have been greater for the incentivized and indirect
question (whether the paper was published)

25. Author's gender aects rating of academic article:
Results
Results

26. Author's gender aects rating of academic article:
Results
Fewer subjects believed that female-authored papers had
been published.
However, authors' young age was insignicant. And there was no
dierences between male and female subjects.

27. Author's gender aects rating of academic article:
Results
Dierences in ratings were mostly insignicant
Mean Mean Mean Mean Mean
young females young males blind
females regardless males regardless
of age of age
Judged published 0.53 0.41 0.60 0.60 0.56
0.46 0.60
Competence 5.22 4.9 5.25 5.06 0.32
4.99 5.14
Language 5.02 4.92 5.08 4.8 5
4.92 4.96
Methodology 5.18 4.47 5.15 4.56 4.85
4.76 4.82
Literature 5.04 4.35 4.81 4.63 4.59
4.66 4.77
Signicance 4.41 4.04 5.21 3.96 4.38
4.18 4.06
Overall 4.73 4.47 4.75 4.63 4.53
4.56 4.67

28. Author's gender aects rating of academic article:
Results
Was the paper judged published?
probit model ( 1 = YES) (1) (2) (3) (4)
Female author -0.34* -0.50* -0.54** -0.56**
(0.09) (0.27) (0.25) (0.24)
Young author 0.16 0 -0.07 -0.07
(0.23) (0.23) (0.22) (0.22)
Young female author 0.32 0.36 0.35
(0.48) (0.47) (0.47)
Female subject -0.41* -0.38
(0.23) (0.30)
Student 0.26 0.27
(0.30) (0.30)
Economics student -0.17 -0.17
(0.17) (0.18)
Female author*male subject 0.06
(0.32)
Paper eects included YES YES YES YES
Observations 190 190 190 190

29. Author's gender aects rating of academic article:
Results
Controversy in the interpretation of the results
Interpretation of results is very important to draw proper conclusions:
are female authors rated less likely to have published because they are
(wrongly) judged less competent than male authors or because subjects
believe there exists a gender bias in publication (due to gender
discrimination)?
Lack of the dierences in ratings, BUT high correlation with judged
published.

30. Author's gender aects rating of academic article:
Results
Controversy in the interpretation of the results
Interpretation of results is very important to draw proper conclusions:
are female authors rated less likely to have published because they are
(wrongly) judged less competent than male authors or because subjects
believe there exists a gender bias in publication (due to gender
discrimination)?
Lack of the dierences in ratings, BUT high correlation with judged
published.
Reviewer and some of the commenters - this explanation is too weak.

31. Author's gender aects rating of academic article:
Results
Controversy in the interpretation of the results
Interpretation of results is very important to draw proper conclusions:
are female authors rated less likely to have published because they are
(wrongly) judged less competent than male authors or because subjects
believe there exists a gender bias in publication (due to gender
discrimination)?
Lack of the dierences in ratings, BUT high correlation with judged
published.
Reviewer and some of the commenters - this explanation is too weak.
Additional sessions:
After rst predictions, we told subjects that the paper was either
male or female-authored
Do you want to change your answer?
If they choose to change - we asked them about the reason

32. Author's gender aects rating of academic article:
Results
Controversy in the interpretation of the results
Predictions:
If subjects were aware of gender bias in publication, they would
sometimes change decision from published to unpublished for female
authors...
and from unpublished to published for males.
Due to the fact the there is a gender bias in publication.

33. Author's gender aects rating of academic article:
Results
Controversy in the interpretation of the results
Predictions:
If subjects were aware of gender bias in publication, they would
sometimes change decision from published to unpublished for female
authors...
and from unpublished to published for males.
Due to the fact the there is a gender bias in publication.
No one decided to change their answer.
Rather unconscious beliefs than knowledge on gender discrimination.

34. Author's gender aects rating of academic article:
Conclusions
Conclusions

35. Author's gender aects rating of academic article:
Conclusions
Conclusions
The gender of the author, but not his or her age, aect evaluation of
a paper (at least in the eld of economics).

36. Author's gender aects rating of academic article:
Conclusions
Conclusions
The gender of the author, but not his or her age, aect evaluation of
a paper (at least in the eld of economics).
Why those results are important?
The chances that our subjects' views of the role of women in
academia will change are minor.
Our subjects were approaching the age at which important early
career decisions are made and when inexperienced researchers may
easily be discouraged.
Young women fail to see successful female economists that could
serve as role models.

37. Author's gender aects rating of academic article:
Conclusions
Thank you!