This work presents program evaluation findings affiliated with gender-based selection bias in an international high performance computing (HPC) student program. Our research demonstrates application reviewers from all countries are unconsciously biased against female applicants resulting in significantly lower applicant scores. Additional data that contradicts reviewer bias and supports gender parity among participants in HPC knowledge level and academic achievement are presented. Suggestions for reducing selection bias in light of our findings are discussed.

1. Debugging a
biased student
selection system
PEARC17| July 2017
Lorna Rivera |Research Faculty | Georgia Institute of Technology
NCSA Center Affiliate | University of Illinois at Urbana-Champaign

2. 2015 International Summer School on HPC Challenges in
Computational Sciences | Toronto, Canada

3. 3.75
4.26
3.63
1
5
e. the applicants struck me
as qualified regardless of
gender.
g. I feel confident about the
decisions I made.
h. On average, people in
advanced computing treat
women equally.
Gender specific statements were amongst
the lowest rated survey items.
Scale 1 (strongly disagree) to 5 (strongly agree)

4. “Unfortunately the pool of female applicants was a small
…and did not appear to come up to the level of the
maximally qualified males. I worry if the difference in
preparation between males and females will be visible to the
students at the event, and if so whether it will have a negative
(reinforcing) effect on the minority position of the women in HPC.”
3.75
4.26
3.63
1
5
e. the applicants struck me
as qualified regardless of
gender.
g. I feel confident about the
decisions I made.
h. On average, people in
advanced computing treat
women equally.

5. Selected male applicants were rated significantly
higher than selected female applicants.
Scale 1 (Poor) – 7 (Excellent)
4.73
4.25***
1
2
3
4
5
6
7
Men (N=20) Women (N=10)
***p < .001

6. 60%
35%
49%
32%
40%
65%
51%
68%
0% 50% 100%
Selected Women (N=10)
Selected Men (N=20)
All Women Applicants (N=39)
All Male Applicants (N=177)
Published Unpublished
Selected 2015 US females published nearly twice as much as
males.

7. Scale 0 (incorrect) – 1 (correct)
0.71 0.68
0
1
Men (N=60) Women (N=16)
No significant gender differences in knowledge of general
parallelism concepts exist.

8. Self report knowledge &
experience scale (novice – expert)
items rated significantly higher by
men than women.
55%
0% Self report frequency scale (almost
never – almost always) items with
significant gender differences.
N=22
N=7

9. 2015 Application Form
Mostly free text boxes describing experience
Research abstract
Discuss the research you are conducting and how HPC is being used or will be used to advance
your research. Please limit your text to 400 words.
Experience with computational science and HPC
Describe your use of HPC, including familiarity with MPI, OpenMP, multicore, accelerators, and
coprocessors, ad how you participate in software development. Please, limit your text to 400
words.
Motivation for your participation

10. 2016 Application Form
Mostly closed-ended ratings of usage frequency
0-3 mos. 4-6 mos. 7-12 mos. 1 – 2 yrs. More than 2 yrs.
MPI ⃝ ⃝ ⃝ ⃝ ⃝
OpenMP ⃝ ⃝ ⃝ ⃝ ⃝
CUDA/OpenCL ⃝ ⃝ ⃝ ⃝ ⃝
Never Occasionally
/ Sometimes
Monthly Weekly Daily
Linux/Unix ⃝ ⃝ ⃝ ⃝ ⃝
Note that responding highly (i.e. Weekly – Daily) will not necessarily improve your
likeliness of acceptance.
Note that responding highly (i.e. 1 – 2+ years) will not necessarily improve your likeliness
of acceptance.
How frequently do you use each of the following:
How long have you used each of the following:

11. 2016
No significant
gender
differences in
applicant
rankings
2015
Male applicants
rated
significantly
higher than
women

12. • 75% (195/258) of all applicants fell within the
5 -7 range (scale 0 – 9) using rubric
• 99% (79/80) of selected applicants scored 6 –
7 on rubric (scale 0-9)
• No gender differences were found in rubric
scores
Additional Outcomes:
Rubric/Tool for Scaling Review Process & reducing selection bias

13. "I think there's a lot of institutional bias against women in HPC;
I think there is often an assumption that women
are less skilled, and then anecdotal data is
sought to back up that assumption.”
- 2016 Reviewer

14. • Harvard’s Project Implicit: https://implicit.harvard.edu/implicit/education.html
• NCWIT’s Critical Listening Guide: Just because you always hear it, doesn’t
mean it’s true. https://www.ncwit.org/resources/critical-listening-guide
• Male Advocates and Allies: Promoting Gender Diversity In Technology
Workplaces https://www.ncwit.org/resources/male-advocates-and-allies-
promoting-gender-diversity-technology-workplaces
Now
what?
Identify Your Biases

15. • NSF – National Science Foundation: Science and Engineering
Doctorate Awards, www.nsf.gov/statistics/doctorates
• WebCASPAR and the Survey of Earned Doctorates (SED)
Tabulation Engine, https://webcaspar.nsf.gov and
https://ncses.norc.org/NSFTabEngine
• Women in Science and Engineering Leadership Institute
(WISELI), http://wiseli.engr.wisc.edu/pubtype.php
• Women in HPC (WHPC), www.womeninhpc.org
Even More Resources:
Evaluation tools, training materials, data sets, research, etc.

16. Q & A
Lorna Rivera |lorna.rivera@gatech.edu
Georgia Institute of Technology