This document presents methods for detecting emergent intersectional biases in contextualized word embeddings. It introduces the Contextualized Embedding Association Test (CEAT), which generates contextualized embeddings from a language model and calculates effect sizes of biases based on the Word Embedding Association Test. CEAT was found to accurately detect intersectional biases that do not overlap with constituent minority identities. Evaluation showed that contextualized embeddings contained widely shared biases around gender, race, social groups, and intersections of these attributes. Moreover, the magnitude of bias generally decreased as models became more contextualized.

1. Detecting Emergent Intersectional Biases:
Contextualized Word Embeddings Contain a Distribution of
Human-like Biases
Wei Guo Aylin Caliskan
August 1, 2021
George Washington University
1

2. Bias in NLP is everywhere!
• Bias in NLP perpetuates bias in society
• Incomprehensive measurement
• Cannot automatically identify bias
2

3. Intersectional bias is concerning!
3

4. Intersectional bias is concerning!
• Incomplete measurement
of social biases
• Unique experiences of
discrimination in ML system
3

5. Methods: implicit cognition → natural language → computer vision
Implicit Association Test (IAT)
• Tests for differential association of
two concepts
• Easier to categorize
stereotype-congruent pairs
• Harder to categorize
stereotype-incongruent pairs
• Effect d = difference in reaction time Weapon IAT (implicit.harvard.edu)
4

6. Methods: implicit cognition → natural language → computer vision
Implicit Association Test (IAT)
• Tests for differential association of
two concepts
• Easier to categorize
stereotype-congruent pairs
• Harder to categorize
stereotype-incongruent pairs
• Effect d = difference in reaction time
Weapon IAT (implicit.harvard.edu)
4

7. Methods: implicit cognition → natural language → computer vision
Implicit Association Test (IAT)
• Tests for differential association of
two concepts
• Easier to categorize
stereotype-congruent pairs
• Harder to categorize
stereotype-incongruent pairs
• Effect d = difference in reaction time
Weapon IAT (implicit.harvard.edu)
4

8. Methods: implicit cognition → natural language → computer vision
Implicit Association Test (IAT)
• Tests for differential association of
two concepts
• Easier to categorize
stereotype-congruent pairs
• Harder to categorize
stereotype-incongruent pairs
• Effect d = difference in reaction time
4

9. Methods: implicit cognition → static embeddings → contextualized embeddings
Word Embedding Association Test Implicit Association Test
5

10. Methods: implicit cognition → static embeddings → contextualized embeddings
Word Embedding Association Test
5

11. Methods: implicit cognition → static embeddings → contextualized embeddings
man
[
feature1 feature2 . . . featured
]
father
[
feature1 feature2 . . . featured
]
.
.
.
woman
[
feature1 feature2 . . . featured
]
mother
[
feature1 feature2 . . . featured
]
.
.
.
science
[
feature1 feature2 . . . featured
]
math
[
feature1 feature2 . . . featured
]
.
.
.
liberal arts
[
feature1 feature2 . . . featured
]
music
[
feature1 feature2 . . . featured
]
.
.
.
5

12. Methods: implicit cognition → static embeddings → contextualized embeddings
man
[
feature1 feature2 . . . featured
]
father
[
feature1 feature2 . . . featured
]
.
.
.
woman
[
feature1 feature2 . . . featured
]
mother
[
feature1 feature2 . . . featured
]
.
.
.
science
[
feature1 feature2 . . . featured
]
math
[
feature1 feature2 . . . featured
]
.
.
.
liberal arts
[
feature1 feature2 . . . featured
]
music
[
feature1 feature2 . . . featured
]
.
.
.
Word Embedding Association Test
(WEAT)
s(w, A, B) = meana∈A cos(w, a)−meanb∈B cos(w, b)
s(X, Y, A, B) =
∑
x∈X
s(x, A, B) −
∑
y∈Y
s(y, A, B)
5

13. Methods: implicit cognition → static embeddings → contextualized embeddings
Implicit Association Test
Word Embedding Factual Association
Test (WEFAT)
s(w, A, B) =
meana∈As(⃗
w,⃗
a) − meanb∈Bs(⃗
w,⃗
b)
stdx∈A∪Bs(⃗
w, x)
6

14. Intersectional Bias Detection (IBD)
s(w, A, B) =
meana∈As(⃗
w,⃗
a) − meanb∈Bs(⃗
w,⃗
b)
stdx∈A∪Bs(⃗
w, x)
7

15. Intersectional Bias Detection (IBD)
s(w, A, B) =
meana∈As(⃗
w,⃗
a) − meanb∈Bs(⃗
w,⃗
b)
stdx∈A∪Bs(⃗
w, x)
Detecting intersectional biases
associated with members of multiple minority groups.
7

16. Emergent Intersectional Bias Detection (EIBD)
8

17. Emergent Intersectional Bias Detection (EIBD)
Intersectional biases - Attributes highly associated with single social category =
Remaining set is the emergent intersectional biases
Detecting unique emergent intersectional biases that do not overlap with the
biases of their constituent minority identities.
8

18. Evaluation of IBD
Detection accuracy > 80% accuracy, where random chance < 15%
Validation set for intersectional biases from Ghavami Peplau, 2013
9

19. Methods: implicit cognition → static embeddings → contextualized embeddings
Extract the sentence containing the words X, Y, A, B
Contextualized Embedding Assoiciation Test (CEAT)
10

20. Methods: implicit cognition → static embeddings → contextualized embeddings
Generates the contextualized embeddings
10

21. Methods: implicit cognition → static embeddings → contextualized embeddings
Calculate the effect size of bias based on WEAT
10

22. Contextualized Embedding Association Test (CEAT)
Generates the distribution of effect magnitudes of biases
Calculate Combined Effect Size
CES(X, Y, A, B) =
∑N
i=1 viESi
∑N
i=1 vi
10

23. Evaluation of CEAT
Contextualized
embeddings from Corpus of
• Widely shared biases
• Flowers/insects
• Musical instru-
ments/weapons
• Social group biases
• Gender
• Race
• Intersectionality
• ...
11

24. Evaluation of CEAT
• Intersectional biases have
high magnitude.
• Biased: ELMo > BERT >
GPT > GPT-2
• The overall magnitude of
bias negatively correlates
with the level of
contextualization in the
language model.
11

25. Questions?
weiguo@gwu.edu
github.com/weiguowilliam/CEAT
paper code
Acknowledgements
my co-author Aylin Caliskan & many reviewers
11