SlideShare a Scribd company logo
Human Interface Laboratory
Towards Cross-Lingual Generalization of
Translation Gender Bias (ACM FAccT 2021)
2021. 4. 7 @SSU
Won Ik Cho
Contents
• Translation gender bias
 What’s the problem and why this matters?
 Significant in which language pairs? - Struggles so far
• Our approach
 Language pairs and template
 Dataset construction
 Measurement of fluency and biasedness
• Discussion
 Results and analysis
 Takeaways
1
Bias
• Bias in machine learning?
 Bias and variance
• Overfitting and underfitting
 Bias in view of fairness machine learning?
• Problem of individuality and context rather than of
statistics and system (Binns, 2017)
 Machine learning에서의 bias와 fairness machine learning에서의 bias, 그리
고 실제 social bias는 어떠한 관련이 있을까?
• e.g., image semantic role labeling
– Zhao et al., Men Also Like Shopping:
Reducing Gender Bias Amplification
using Corpus-level Constraints,
in Proc. EMNLP, 2017.
• Amazon recruiting issue
• Machines showing social/cultural bias
• This also happens in translation!
2
Bias
• What is shown (social) bias in AI and NLP?
 Sun et al., Mitigating Gender Bias in Natural Language Processing:
Literature Review, in Proc. ACL, 2019.
3
Overview: Gender bias in translation?
• Formulation #1
 Gender-neutral pronouns
• Target problem?
 Translation of gender-neutral pronouns to gender-specific ones
• Gender-neutral pronoun
 Pronouns with no biological
gender displayed
• `single they’ 같은 표현들
 Frequently appears in languages
like Korean, Japanese, Turkish, ...
 Prates et al., Assessing Gender
Bias in Machine Translation:
A Case Study with Google Trans
late, Neural Computing and
Applications, 2018.
4
Overview: Gender bias in translation?
• Formulation #2
 Gendered languages
• Target problem?
 Translation of expressions without
gender representation to gendered items
• Gendered languages
 관사, 명사, 형용사 등의 grammatical gender
 대부분의 Indo-European languages
 Differs from the biological gender
 Vanmassenhove et al.,
Getting Gender Right in
Neural Machine Translation,
in Proc. EMNLP, 2018.
5
Overview: Gender bias in translation?
• Why do they matter?
 The result can be offensive to end users
• When they do matter?
 사용자가 target/source language에 익숙하든 아니든 상관없이
• Who will potentially feel offensive?
 오역이 특정 집단에 대한 social stereotypes을 포괄하고 있다면?
• Research questions
 How can the evaluation incorporate various aspects of translation gender
bias?
 How will grammatical properties and resource condition influence the bias
issue?
6
Template-based attacks
• 걔(s/he)는 [##]이야!
 Cho et al., On measuring gender bias in translation of gender-neutral
pronouns," in Proc. GeBNLP, ACL Workshop, 2019.
• Why Korean?
 Displays various sentence styles
 Translation service
popular among the users
7
Semantic cues
• WinoMT
 Stanovsky et al., Evaluating Gender Bias in Machine Translation,
in Proc. ACL, 2019.
8
- Performance can differ among
a similar language family
- High accuracy in stereotypical
case does not guarantee the
accuracy in the counterpart
Our approach
• Combined approach – Cross-lingual evaluation?
 Two different types of translation gender bias
• Different types of gender bias can be observed in a single translation system
9
?
Our approach
• Language pairs
 Different types of `language pairs’ should be investigated
• 1) Languages with gender-neutral pronouns
• 2) Languages without (colloquially used) GNPs but not grammatically gendered
• 3) Languages with grammatical gender
10
Our approach
• Language pairs
 Linguistic criteria: On GNPs and genderedness
• Korean/Tagalog -> English -> German/Portuguese
 Resource criteria
• Korean > Tagalog
• German > Portuguese
11
Our approach
• Template
 How can the pairs be simultaneously evaluated?
12
Korean 그 여자에 대해 하나 말하자면, 걔는 초등학교 선생님이야.
Tagalog Isang bagay tungkol sa babae, siya ay isang guro sa elementarya.
English One thing about the woman, she is an elementary school teacher.
German Eine Sache über die Frau, sie ist eine Grundschullehrerin.
Portuguese Um facto sobre a mulher, ela é professora do ensino primário.
Our approach
• Template
 How can the pairs be simultaneously evaluated?
13
Our approach
• Evaluation
 Template-based evaluation set construction
 Inference with public MT modules
 Human evaluation (gender-related) and automatic metrics (fluency)
14
Our approach
• Measurement
 Biasedness
• Accuracy on biological gender
– Korean, Tagalog > English
• Accuracy on grammatical gender
– English > German, Portuguese
• Disparate impact
– Accuracy on female case divided by accuracy on male case
 Fluency
• BLEU
– Lexical한 측면의 번역 성능
– EN, DE, PT
• BERTScore
– Semantics 측면의 번역 성능
– Multilingual BERT
15
Results and analysis
• Results
 More bias-related errors in EN > DE/PT than in KO/TL > EN
• She is a game programmer > Sie ist ein professioneller Spieler
• aviador, soldado, monge (airman, soldier, monk)
• Exceptional cases for Bing KO-EN (경찰 (cop), 경비원 (guard), 배관공 (plumber))
16
Results and analysis
• Analysis
 Unbiasedness/Disparate impact
• Higher among type 1 languages
– DE, PT < KO, TL (overall)
• In the same type, resource seems
to matter
– DE < PT, KO < TL
 Fluency measurement
• Lexical and semantic approach have different results
– BLEU (lexical): DE > PT > KO, TL
– BERTScore (semantic): DE < PT, KO < TL
 Observations
• The amount of available language resource, though here assumed for public
MT modules, does not guarantee unbiased translation, albeit fluency measure
may be higher in some sense
• There is a difference regarding the evaluation on gender-related inference per
fluency measures
17
Takeaways
• Translation gender bias is problematic since wrong results can be
offensive to end users
• Translation gender bias matters regardless of the user proficiency
of the language, and especially offensive if the mistranslation
engages social stereotypes
• Our approach, including template and measurement, can combine
the translation gender bias evaluation regarding various language
pairs
• Our evaluation results suggest that the inductive bias as a social
stereotype is a major factor causing the errors and augmenting
training corpora may not be a solution
18
Reference (order of appearance)
• Binns, Reuben. "Fairness in Machine Learning: Lessons from Political Philosophy." arXiv preprint
arXiv:1712.03586 (2017).
• Zhao, Jieyu, et al. "Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus-
level Constraints." arXiv preprint arXiv:1707.09457 (2017).
• Sun, Tony, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza,
Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. "Mitigating Gender Bias in Natural
Language Processing: Literature Review." In Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics, pp. 1630-1640. 2019.
• Prates, Marcelo OR, Pedro H. Avelar, and Luís C. Lamb. "Assessing Gender Bias in Machine
Translation: A Case Study with Google Translate." Neural Computing and Applications (2018): 1-
19.
• Vanmassenhove, Eva, Christian Hardmeier, and Andy Way. "Getting Gender Right in Neural
Machine Translation." In Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pp. 3003-3008. 2018.
• Cho, Won Ik, et al. "On Measuring Gender Bias in Translation of Gender-neutral Pronouns."
GeBNLP 2019 (2019): 173.
• Stanovsky, Gabriel, Noah A. Smith, and Luke Zettlemoyer. "Evaluating Gender Bias in Machine
Translation." arXiv preprint arXiv:1906.00591 (2019).
19
Thank you!
EndOfPresentation

More Related Content

What's hot

Natural language processing
Natural language processingNatural language processing
Natural language processing
Hansi Thenuwara
 
Natural language procssing
Natural language procssing Natural language procssing
Natural language procssing
Rajnish Raj
 
Natural Language Processing in Alternative and Augmentative Communication
Natural Language Processing in Alternative and Augmentative CommunicationNatural Language Processing in Alternative and Augmentative Communication
Natural Language Processing in Alternative and Augmentative Communication
Divya Sugumar
 
Introduction to natural language processing
Introduction to natural language processingIntroduction to natural language processing
Introduction to natural language processing
Minh Pham
 
NLP & Machine Learning - An Introductory Talk
NLP & Machine Learning - An Introductory Talk NLP & Machine Learning - An Introductory Talk
NLP & Machine Learning - An Introductory Talk
Vijay Ganti
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
Yasir Khan
 
Natural language processing PPT presentation
Natural language processing PPT presentationNatural language processing PPT presentation
Natural language processing PPT presentation
Sai Mohith
 
Natural Language Processing: Parsing
Natural Language Processing: ParsingNatural Language Processing: Parsing
Natural Language Processing: Parsing
Rushdi Shams
 
Natural Language Processing for Games Research
Natural Language Processing for Games ResearchNatural Language Processing for Games Research
Natural Language Processing for Games Research
Jose Zagal
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language Processing
Mercy Rani
 
UCU NLP Summer Workshops 2017 - Part 2
UCU NLP Summer Workshops 2017 - Part 2UCU NLP Summer Workshops 2017 - Part 2
UCU NLP Summer Workshops 2017 - Part 2
Yuriy Guts
 
Frontiers of Natural Language Processing
Frontiers of Natural Language ProcessingFrontiers of Natural Language Processing
Frontiers of Natural Language Processing
Sebastian Ruder
 
A Light Introduction to Transfer Learning for NLP
A Light Introduction to Transfer Learning for NLPA Light Introduction to Transfer Learning for NLP
A Light Introduction to Transfer Learning for NLP
Lahore Garrison University
 
Natural language processing
Natural language processingNatural language processing
Natural language processing
Saurav Aryal
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
Mariana Soffer
 
natural language processing help at myassignmenthelp.net
natural language processing  help at myassignmenthelp.netnatural language processing  help at myassignmenthelp.net
natural language processing help at myassignmenthelp.net
www.myassignmenthelp.net
 
Natural Language Processing (NLP)
Natural Language Processing (NLP)Natural Language Processing (NLP)
Natural Language Processing (NLP)
Yuriy Guts
 
Natural language processing
Natural language processingNatural language processing
Natural language processing
National Institute of Technology Durgapur
 
NLP Project Presentation
NLP Project PresentationNLP Project Presentation
NLP Project Presentation
Aryak Sengupta
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language Processing
David Rostcheck
 

What's hot (20)

Natural language processing
Natural language processingNatural language processing
Natural language processing
 
Natural language procssing
Natural language procssing Natural language procssing
Natural language procssing
 
Natural Language Processing in Alternative and Augmentative Communication
Natural Language Processing in Alternative and Augmentative CommunicationNatural Language Processing in Alternative and Augmentative Communication
Natural Language Processing in Alternative and Augmentative Communication
 
Introduction to natural language processing
Introduction to natural language processingIntroduction to natural language processing
Introduction to natural language processing
 
NLP & Machine Learning - An Introductory Talk
NLP & Machine Learning - An Introductory Talk NLP & Machine Learning - An Introductory Talk
NLP & Machine Learning - An Introductory Talk
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
 
Natural language processing PPT presentation
Natural language processing PPT presentationNatural language processing PPT presentation
Natural language processing PPT presentation
 
Natural Language Processing: Parsing
Natural Language Processing: ParsingNatural Language Processing: Parsing
Natural Language Processing: Parsing
 
Natural Language Processing for Games Research
Natural Language Processing for Games ResearchNatural Language Processing for Games Research
Natural Language Processing for Games Research
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language Processing
 
UCU NLP Summer Workshops 2017 - Part 2
UCU NLP Summer Workshops 2017 - Part 2UCU NLP Summer Workshops 2017 - Part 2
UCU NLP Summer Workshops 2017 - Part 2
 
Frontiers of Natural Language Processing
Frontiers of Natural Language ProcessingFrontiers of Natural Language Processing
Frontiers of Natural Language Processing
 
A Light Introduction to Transfer Learning for NLP
A Light Introduction to Transfer Learning for NLPA Light Introduction to Transfer Learning for NLP
A Light Introduction to Transfer Learning for NLP
 
Natural language processing
Natural language processingNatural language processing
Natural language processing
 
Natural Language Processing
Natural Language ProcessingNatural Language Processing
Natural Language Processing
 
natural language processing help at myassignmenthelp.net
natural language processing  help at myassignmenthelp.netnatural language processing  help at myassignmenthelp.net
natural language processing help at myassignmenthelp.net
 
Natural Language Processing (NLP)
Natural Language Processing (NLP)Natural Language Processing (NLP)
Natural Language Processing (NLP)
 
Natural language processing
Natural language processingNatural language processing
Natural language processing
 
NLP Project Presentation
NLP Project PresentationNLP Project Presentation
NLP Project Presentation
 
Introduction to Natural Language Processing
Introduction to Natural Language ProcessingIntroduction to Natural Language Processing
Introduction to Natural Language Processing
 

Similar to 2104 Talk @SSU

2103 ACM FAccT
2103 ACM FAccT2103 ACM FAccT
2103 ACM FAccT
WarNik Chow
 
190802 GeBNLP
190802 GeBNLP190802 GeBNLP
190802 GeBNLP
WarNik Chow
 
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
NUS Institute of Applied Learning Sciences and Educational Technology
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health Programs
MEASURE Evaluation
 
Lepor: augmented automatic MT evaluation metric
Lepor: augmented automatic MT evaluation metricLepor: augmented automatic MT evaluation metric
Lepor: augmented automatic MT evaluation metric
Lifeng (Aaron) Han
 
LEPOR: an augmented machine translation evaluation metric - Thesis PPT
LEPOR: an augmented machine translation evaluation metric - Thesis PPT LEPOR: an augmented machine translation evaluation metric - Thesis PPT
LEPOR: an augmented machine translation evaluation metric - Thesis PPT
Lifeng (Aaron) Han
 
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
Antonio Toral
 
Automated Language Assessment Scoring and impact on instruction
Automated Language Assessment Scoring and impact on instructionAutomated Language Assessment Scoring and impact on instruction
Automated Language Assessment Scoring and impact on instruction
tfarny
 
A Survey of ‘Bias’ in Natural Language Processing Systems
A Survey of ‘Bias’ in Natural Language Processing SystemsA Survey of ‘Bias’ in Natural Language Processing Systems
A Survey of ‘Bias’ in Natural Language Processing Systems
subarna89
 
Psychological test adaptation
Psychological test adaptationPsychological test adaptation
Psychological test adaptation
Carlo Magno
 
Machine translator Introduction
Machine translator IntroductionMachine translator Introduction
Machine translator Introduction
Hamid Shahrivari Joghan
 
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
Jennifer Fabrizi
 
Aac October 2008
Aac   October 2008Aac   October 2008
Aac October 2008
Jantioco1
 
A Level English Language Exam Prep from AQA 2011
A Level English Language Exam Prep from AQA 2011A Level English Language Exam Prep from AQA 2011
A Level English Language Exam Prep from AQA 2011
ENSFCEnglish
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
PrithaVashisht1
 
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
Lifeng (Aaron) Han
 
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
Bilinguistics
 
Attitudes bolouri
Attitudes bolouriAttitudes bolouri
Attitudes bolouri
Allame Tabatabaei
 
Rapid Qaulitative Inquiry, extended presentation
Rapid Qaulitative Inquiry, extended presentationRapid Qaulitative Inquiry, extended presentation
Rapid Qaulitative Inquiry, extended presentation
James Beebe
 
Social Media Sentiments Analysis
Social Media Sentiments AnalysisSocial Media Sentiments Analysis
Social Media Sentiments Analysis
PratisthaSingh5
 

Similar to 2104 Talk @SSU (20)

2103 ACM FAccT
2103 ACM FAccT2103 ACM FAccT
2103 ACM FAccT
 
190802 GeBNLP
190802 GeBNLP190802 GeBNLP
190802 GeBNLP
 
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
Carolyn Rosé - WESST - From Data to Design of Dynamic Support for Collaborati...
 
Seven Steps to EnGendering Evaluations of Public Health Programs
 Seven Steps to EnGendering Evaluations of Public Health Programs Seven Steps to EnGendering Evaluations of Public Health Programs
Seven Steps to EnGendering Evaluations of Public Health Programs
 
Lepor: augmented automatic MT evaluation metric
Lepor: augmented automatic MT evaluation metricLepor: augmented automatic MT evaluation metric
Lepor: augmented automatic MT evaluation metric
 
LEPOR: an augmented machine translation evaluation metric - Thesis PPT
LEPOR: an augmented machine translation evaluation metric - Thesis PPT LEPOR: an augmented machine translation evaluation metric - Thesis PPT
LEPOR: an augmented machine translation evaluation metric - Thesis PPT
 
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Mach...
 
Automated Language Assessment Scoring and impact on instruction
Automated Language Assessment Scoring and impact on instructionAutomated Language Assessment Scoring and impact on instruction
Automated Language Assessment Scoring and impact on instruction
 
A Survey of ‘Bias’ in Natural Language Processing Systems
A Survey of ‘Bias’ in Natural Language Processing SystemsA Survey of ‘Bias’ in Natural Language Processing Systems
A Survey of ‘Bias’ in Natural Language Processing Systems
 
Psychological test adaptation
Psychological test adaptationPsychological test adaptation
Psychological test adaptation
 
Machine translator Introduction
Machine translator IntroductionMachine translator Introduction
Machine translator Introduction
 
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
Boston Mini Upa2011: Localization Research Presentation by Jennifer Fabrizi a...
 
Aac October 2008
Aac   October 2008Aac   October 2008
Aac October 2008
 
A Level English Language Exam Prep from AQA 2011
A Level English Language Exam Prep from AQA 2011A Level English Language Exam Prep from AQA 2011
A Level English Language Exam Prep from AQA 2011
 
Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
TSD2013 PPT.AUTOMATIC MACHINE TRANSLATION EVALUATION WITH PART-OF-SPEECH INFO...
 
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
Using Dynamic Assessment in Differential Diagnoses of Culturally and Linguist...
 
Attitudes bolouri
Attitudes bolouriAttitudes bolouri
Attitudes bolouri
 
Rapid Qaulitative Inquiry, extended presentation
Rapid Qaulitative Inquiry, extended presentationRapid Qaulitative Inquiry, extended presentation
Rapid Qaulitative Inquiry, extended presentation
 
Social Media Sentiments Analysis
Social Media Sentiments AnalysisSocial Media Sentiments Analysis
Social Media Sentiments Analysis
 

More from WarNik Chow

2312 PACLIC
2312 PACLIC2312 PACLIC
2312 PACLIC
WarNik Chow
 
2311 EAAMO
2311 EAAMO2311 EAAMO
2311 EAAMO
WarNik Chow
 
2211 HCOMP
2211 HCOMP2211 HCOMP
2211 HCOMP
WarNik Chow
 
2211 APSIPA
2211 APSIPA2211 APSIPA
2211 APSIPA
WarNik Chow
 
2211 AACL
2211 AACL2211 AACL
2211 AACL
WarNik Chow
 
2210 CODI
2210 CODI2210 CODI
2210 CODI
WarNik Chow
 
2206 FAccT_inperson
2206 FAccT_inperson2206 FAccT_inperson
2206 FAccT_inperson
WarNik Chow
 
2206 Modupop!
2206 Modupop!2206 Modupop!
2206 Modupop!
WarNik Chow
 
2204 Kakao talk on Hate speech dataset
2204 Kakao talk on Hate speech dataset2204 Kakao talk on Hate speech dataset
2204 Kakao talk on Hate speech dataset
WarNik Chow
 
2108 [LangCon2021] kosp2e
2108 [LangCon2021] kosp2e2108 [LangCon2021] kosp2e
2108 [LangCon2021] kosp2e
WarNik Chow
 
2106 PRSLLS
2106 PRSLLS2106 PRSLLS
2106 PRSLLS
WarNik Chow
 
2106 JWLLP
2106 JWLLP2106 JWLLP
2106 JWLLP
WarNik Chow
 
2102 Redone seminar
2102 Redone seminar2102 Redone seminar
2102 Redone seminar
WarNik Chow
 
2011 NLP-OSS
2011 NLP-OSS2011 NLP-OSS
2011 NLP-OSS
WarNik Chow
 
2010 INTERSPEECH
2010 INTERSPEECH 2010 INTERSPEECH
2010 INTERSPEECH
WarNik Chow
 
2010 PACLIC - pay attention to categories
2010 PACLIC - pay attention to categories2010 PACLIC - pay attention to categories
2010 PACLIC - pay attention to categories
WarNik Chow
 
2010 HCLT Hate Speech
2010 HCLT Hate Speech2010 HCLT Hate Speech
2010 HCLT Hate Speech
WarNik Chow
 
2009 DevC Seongnam - NLP
2009 DevC Seongnam - NLP2009 DevC Seongnam - NLP
2009 DevC Seongnam - NLP
WarNik Chow
 
2008 [lang con2020] act!
2008 [lang con2020] act!2008 [lang con2020] act!
2008 [lang con2020] act!
WarNik Chow
 
2007 CogSci 2020 poster
2007 CogSci 2020 poster2007 CogSci 2020 poster
2007 CogSci 2020 poster
WarNik Chow
 

More from WarNik Chow (20)

2312 PACLIC
2312 PACLIC2312 PACLIC
2312 PACLIC
 
2311 EAAMO
2311 EAAMO2311 EAAMO
2311 EAAMO
 
2211 HCOMP
2211 HCOMP2211 HCOMP
2211 HCOMP
 
2211 APSIPA
2211 APSIPA2211 APSIPA
2211 APSIPA
 
2211 AACL
2211 AACL2211 AACL
2211 AACL
 
2210 CODI
2210 CODI2210 CODI
2210 CODI
 
2206 FAccT_inperson
2206 FAccT_inperson2206 FAccT_inperson
2206 FAccT_inperson
 
2206 Modupop!
2206 Modupop!2206 Modupop!
2206 Modupop!
 
2204 Kakao talk on Hate speech dataset
2204 Kakao talk on Hate speech dataset2204 Kakao talk on Hate speech dataset
2204 Kakao talk on Hate speech dataset
 
2108 [LangCon2021] kosp2e
2108 [LangCon2021] kosp2e2108 [LangCon2021] kosp2e
2108 [LangCon2021] kosp2e
 
2106 PRSLLS
2106 PRSLLS2106 PRSLLS
2106 PRSLLS
 
2106 JWLLP
2106 JWLLP2106 JWLLP
2106 JWLLP
 
2102 Redone seminar
2102 Redone seminar2102 Redone seminar
2102 Redone seminar
 
2011 NLP-OSS
2011 NLP-OSS2011 NLP-OSS
2011 NLP-OSS
 
2010 INTERSPEECH
2010 INTERSPEECH 2010 INTERSPEECH
2010 INTERSPEECH
 
2010 PACLIC - pay attention to categories
2010 PACLIC - pay attention to categories2010 PACLIC - pay attention to categories
2010 PACLIC - pay attention to categories
 
2010 HCLT Hate Speech
2010 HCLT Hate Speech2010 HCLT Hate Speech
2010 HCLT Hate Speech
 
2009 DevC Seongnam - NLP
2009 DevC Seongnam - NLP2009 DevC Seongnam - NLP
2009 DevC Seongnam - NLP
 
2008 [lang con2020] act!
2008 [lang con2020] act!2008 [lang con2020] act!
2008 [lang con2020] act!
 
2007 CogSci 2020 poster
2007 CogSci 2020 poster2007 CogSci 2020 poster
2007 CogSci 2020 poster
 

Recently uploaded

CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
girewiy968
 
all about the data science process, covering the steps present in almost ever...
all about the data science process, covering the steps present in almost ever...all about the data science process, covering the steps present in almost ever...
all about the data science process, covering the steps present in almost ever...
palaniappancse
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
kevig
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
saadkhan1485265
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
vrvipin164
 
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
tanupasswan6
 
Introduction to the Red Hat Portfolio.pdf
Introduction to the Red Hat Portfolio.pdfIntroduction to the Red Hat Portfolio.pdf
Introduction to the Red Hat Portfolio.pdf
kihus38
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
huseindihon
 
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
ginni singh$A17
 
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
revolutionary575
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
palanisamyiiiier
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
norina2645
 
Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
TARIKU ENDALE
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
erynsouthern
 
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
gargnatasha985
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
LINAT
 

Recently uploaded (20)

CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
 
all about the data science process, covering the steps present in almost ever...
all about the data science process, covering the steps present in almost ever...all about the data science process, covering the steps present in almost ever...
all about the data science process, covering the steps present in almost ever...
 
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy DsouzaOpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
OpenMetadata Spotlight - OpenMetadata @ Aspire by Vinol Joy Dsouza
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
 
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured DataFine-Tuning of Small/Medium LLMs for Business QA on Structured Data
Fine-Tuning of Small/Medium LLMs for Business QA on Structured Data
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
 
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
Busty Girls Call Delhi 🎈🔥9711199171 🔥💋🎈 Provide Best And Top Girl Service And...
 
Introduction to the Red Hat Portfolio.pdf
Introduction to the Red Hat Portfolio.pdfIntroduction to the Red Hat Portfolio.pdf
Introduction to the Red Hat Portfolio.pdf
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
 
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
Celebrity Girls Call Noida 9873940964 Unlimited Short Providing Girls Service...
 
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
Verified Girls Call Andheri 9930245274 Unlimited Short Providing Girls Servic...
 
transgenders community data in india by govt
transgenders community data in india by govttransgenders community data in india by govt
transgenders community data in india by govt
 
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
Mumbai Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service And ...
 
Supervised Learning (Data Science).pptx
Supervised Learning  (Data Science).pptxSupervised Learning  (Data Science).pptx
Supervised Learning (Data Science).pptx
 
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDeliveryBDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
BDSM Girls Call Mumbai 👀 9820252231 👀 Cash Payment With Room DeliveryDelivery
 
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Vadodara 000XX00000 Provide Best And Top Girl Service And No1 in City
 
Willis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdfWillis Tower //Sears Tower- Supertall Building .pdf
Willis Tower //Sears Tower- Supertall Building .pdf
 

2104 Talk @SSU

  • 1. Human Interface Laboratory Towards Cross-Lingual Generalization of Translation Gender Bias (ACM FAccT 2021) 2021. 4. 7 @SSU Won Ik Cho
  • 2. Contents • Translation gender bias  What’s the problem and why this matters?  Significant in which language pairs? - Struggles so far • Our approach  Language pairs and template  Dataset construction  Measurement of fluency and biasedness • Discussion  Results and analysis  Takeaways 1
  • 3. Bias • Bias in machine learning?  Bias and variance • Overfitting and underfitting  Bias in view of fairness machine learning? • Problem of individuality and context rather than of statistics and system (Binns, 2017)  Machine learning에서의 bias와 fairness machine learning에서의 bias, 그리 고 실제 social bias는 어떠한 관련이 있을까? • e.g., image semantic role labeling – Zhao et al., Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints, in Proc. EMNLP, 2017. • Amazon recruiting issue • Machines showing social/cultural bias • This also happens in translation! 2
  • 4. Bias • What is shown (social) bias in AI and NLP?  Sun et al., Mitigating Gender Bias in Natural Language Processing: Literature Review, in Proc. ACL, 2019. 3
  • 5. Overview: Gender bias in translation? • Formulation #1  Gender-neutral pronouns • Target problem?  Translation of gender-neutral pronouns to gender-specific ones • Gender-neutral pronoun  Pronouns with no biological gender displayed • `single they’ 같은 표현들  Frequently appears in languages like Korean, Japanese, Turkish, ...  Prates et al., Assessing Gender Bias in Machine Translation: A Case Study with Google Trans late, Neural Computing and Applications, 2018. 4
  • 6. Overview: Gender bias in translation? • Formulation #2  Gendered languages • Target problem?  Translation of expressions without gender representation to gendered items • Gendered languages  관사, 명사, 형용사 등의 grammatical gender  대부분의 Indo-European languages  Differs from the biological gender  Vanmassenhove et al., Getting Gender Right in Neural Machine Translation, in Proc. EMNLP, 2018. 5
  • 7. Overview: Gender bias in translation? • Why do they matter?  The result can be offensive to end users • When they do matter?  사용자가 target/source language에 익숙하든 아니든 상관없이 • Who will potentially feel offensive?  오역이 특정 집단에 대한 social stereotypes을 포괄하고 있다면? • Research questions  How can the evaluation incorporate various aspects of translation gender bias?  How will grammatical properties and resource condition influence the bias issue? 6
  • 8. Template-based attacks • 걔(s/he)는 [##]이야!  Cho et al., On measuring gender bias in translation of gender-neutral pronouns," in Proc. GeBNLP, ACL Workshop, 2019. • Why Korean?  Displays various sentence styles  Translation service popular among the users 7
  • 9. Semantic cues • WinoMT  Stanovsky et al., Evaluating Gender Bias in Machine Translation, in Proc. ACL, 2019. 8 - Performance can differ among a similar language family - High accuracy in stereotypical case does not guarantee the accuracy in the counterpart
  • 10. Our approach • Combined approach – Cross-lingual evaluation?  Two different types of translation gender bias • Different types of gender bias can be observed in a single translation system 9 ?
  • 11. Our approach • Language pairs  Different types of `language pairs’ should be investigated • 1) Languages with gender-neutral pronouns • 2) Languages without (colloquially used) GNPs but not grammatically gendered • 3) Languages with grammatical gender 10
  • 12. Our approach • Language pairs  Linguistic criteria: On GNPs and genderedness • Korean/Tagalog -> English -> German/Portuguese  Resource criteria • Korean > Tagalog • German > Portuguese 11
  • 13. Our approach • Template  How can the pairs be simultaneously evaluated? 12 Korean 그 여자에 대해 하나 말하자면, 걔는 초등학교 선생님이야. Tagalog Isang bagay tungkol sa babae, siya ay isang guro sa elementarya. English One thing about the woman, she is an elementary school teacher. German Eine Sache über die Frau, sie ist eine Grundschullehrerin. Portuguese Um facto sobre a mulher, ela é professora do ensino primário.
  • 14. Our approach • Template  How can the pairs be simultaneously evaluated? 13
  • 15. Our approach • Evaluation  Template-based evaluation set construction  Inference with public MT modules  Human evaluation (gender-related) and automatic metrics (fluency) 14
  • 16. Our approach • Measurement  Biasedness • Accuracy on biological gender – Korean, Tagalog > English • Accuracy on grammatical gender – English > German, Portuguese • Disparate impact – Accuracy on female case divided by accuracy on male case  Fluency • BLEU – Lexical한 측면의 번역 성능 – EN, DE, PT • BERTScore – Semantics 측면의 번역 성능 – Multilingual BERT 15
  • 17. Results and analysis • Results  More bias-related errors in EN > DE/PT than in KO/TL > EN • She is a game programmer > Sie ist ein professioneller Spieler • aviador, soldado, monge (airman, soldier, monk) • Exceptional cases for Bing KO-EN (경찰 (cop), 경비원 (guard), 배관공 (plumber)) 16
  • 18. Results and analysis • Analysis  Unbiasedness/Disparate impact • Higher among type 1 languages – DE, PT < KO, TL (overall) • In the same type, resource seems to matter – DE < PT, KO < TL  Fluency measurement • Lexical and semantic approach have different results – BLEU (lexical): DE > PT > KO, TL – BERTScore (semantic): DE < PT, KO < TL  Observations • The amount of available language resource, though here assumed for public MT modules, does not guarantee unbiased translation, albeit fluency measure may be higher in some sense • There is a difference regarding the evaluation on gender-related inference per fluency measures 17
  • 19. Takeaways • Translation gender bias is problematic since wrong results can be offensive to end users • Translation gender bias matters regardless of the user proficiency of the language, and especially offensive if the mistranslation engages social stereotypes • Our approach, including template and measurement, can combine the translation gender bias evaluation regarding various language pairs • Our evaluation results suggest that the inductive bias as a social stereotype is a major factor causing the errors and augmenting training corpora may not be a solution 18
  • 20. Reference (order of appearance) • Binns, Reuben. "Fairness in Machine Learning: Lessons from Political Philosophy." arXiv preprint arXiv:1712.03586 (2017). • Zhao, Jieyu, et al. "Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus- level Constraints." arXiv preprint arXiv:1707.09457 (2017). • Sun, Tony, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. "Mitigating Gender Bias in Natural Language Processing: Literature Review." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1630-1640. 2019. • Prates, Marcelo OR, Pedro H. Avelar, and Luís C. Lamb. "Assessing Gender Bias in Machine Translation: A Case Study with Google Translate." Neural Computing and Applications (2018): 1- 19. • Vanmassenhove, Eva, Christian Hardmeier, and Andy Way. "Getting Gender Right in Neural Machine Translation." In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3003-3008. 2018. • Cho, Won Ik, et al. "On Measuring Gender Bias in Translation of Gender-neutral Pronouns." GeBNLP 2019 (2019): 173. • Stanovsky, Gabriel, Noah A. Smith, and Luke Zettlemoyer. "Evaluating Gender Bias in Machine Translation." arXiv preprint arXiv:1906.00591 (2019). 19

Editor's Notes

  1. .