Margaret Mitchell, Senior Research Scientist, Google, at MLonf Seattle 2017
Margaret Mitchell, Senior Research Scientist, Google, at MLonf Seattle 2017
Margaret Mitchell is the Senior Research Scientist in Google’s Research & Machine Intelligence group, working on advancing artificial intelligence towards positive goals. She works on vision-language and grounded language generation, focusing on how to help computers communicate based on what they can process. Her work combines computer vision, natural language processing, social media, many statistical methods, and insights from cognitive science. Before Google, She was the Researcher at Microsoft Research, and a founding member of their “Cognition” group. Her work focused on advancing artificial intelligence, with specific focus on generating language from visual inputs. Before MSR, she was a postdoctoral researcher at The Johns Hopkins University Center of Excellence, where she mainly focused on semantic role labeling and sentiment analysis using graphical models, working under Benjamin Van Durme. Margaret received her PhD in Computer Science from the University of Aberdeen, with supervision from Kees van Deemter and Ehud Reiter, and external supervision from the University of Edinburgh (with Ellen Bard) and Oregon Health & Science University (OHSU) (with Brian Roark and Richard Sproat). She worked on referring expression generation from statistical and mathematical perspectives as well as a cognitive science perspective, and also worked on prototyping assistive/augmentative technology that people with language generation difficulties can use. Her thesis work was on generating natural, human-like reference to visible, everyday objects. She spent a good chunk of 2008 getting a Master’s in Computational Linguistics at the University of Washington, studying under Emily Bender and Fei Xia. Simultaneously (2005 – 2012), She worked on and off at the Center for Spoken Language Understanding, part of OHSU, in Portland, Oregon. Her title changed with time (research assistant/associate/visiting scholar), but throughout, she worked under Brian Roark on technology that leverages syntactic and phonetic characteristics to aid those with neurological disorders. She continues to balance my time between language generation, applications for clinical domains, and core AI research. Abstract summary The Seen and Unseen Factors Influencing Machine Perception of Images and Language: The success of machine learning has recently surged, with similar algorithmic approaches effectively solving a variety of human-defined tasks. Tasks testing how well machines can perceive images and communicate about them have begun to show a lot of promise, and at the same time, have exposed strong effects of different types of bias, such as overgeneralization. In this talk, I will detail how machines are learning about the visual world, and discuss how machine learning and different kinds of bias interact.
Recommended
More Related Content
Viewers also liked
Viewers also liked(19)
More from MLconf
More from MLconf(20)
Recently uploaded
Recently uploaded(20)
Margaret Mitchell, Senior Research Scientist, Google, at MLonf Seattle 2017
- 1. Ishan Misra Larry Zitnick Ross Girshick Josh Lovejoy Hartwig Adam Blaise Agüera y Arcas CMU FAIR FAIR Google Google Google The Seen and Unseen Factors Influencing Machine Perception of Images and Language Margaret Mitchell Google
- 2. What do you see?
- 3. What do you see? ● Bananas
- 4. What do you see? ● Bananas ● Bananas at a store
- 5. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves
- 6. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas
- 7. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them
- 8. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them ● Bunches of bananas with stickers on them on shelves in a store
- 9. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them ● Bunches of bananas with stickers on them on shelves in a store Yellow Bananas
- 10. What do you see? Green Bananas Unripe Bananas
- 11. What do you see? Ripe Bananas Bananas with spots
- 12. What do you see? Ripe Bananas Bananas with spots Bananas good for banana bread
- 13. What do you see? Yellow Bananas Yellow is prototypical for bananas
- 14. A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
- 15. A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
- 16. “Female doctor” A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
- 18. The majority of test subjects overlooked the possibility that the doctor is a she - including men, women, and self-described feminists. Wapman & Belle, Boston University
- 19. Word Frequency in corpus “spoke” 11,577,917 “laughed” 3,904,519 “murdered” 2,834,529 “inhaled” 984,613 “breathed” 725,034 “hugged” 610,040 “blinked” 390,692 “exhale” 168,985 World learning from text Gordon and Van Durme, 2013
- 20. Word Frequency in corpus “spoke” 11,577,917 “laughed” 3,904,519 “murdered” 2,834,529 “inhaled” 984,613 “breathed” 725,034 “hugged” 610,040 “blinked” 390,692 “exhale” 168,985 World learning from text Gordon and Van Durme, 2013
- 21. Human Reporting Bias The frequency with which people write about actions, outcomes, or properties is not a reflection of real-world frequencies or the degree to which a property is characteristic of a class of individuals. (Gordon and Van Durme, 2013)
- 22. Training data are collected and annotated
- 23. Training data are collected and annotated Model is trained
- 24. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated
- 25. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output
- 26. Human Biases in Data Reporting bias Selection bias Overgeneralization Out-group homogeneity bias Stereotypical bias Historical unfairness Implicit associations Implicit stereotypes Prejudice Group attribution error Halo effect Training data are collected and annotated
- 27. Human Biases in Data Reporting bias Selection bias Overgeneralization Out-group homogeneity bias Stereotypical bias Historical unfairness Implicit associations Implicit stereotypes Prejudice Group attribution error Halo effect Training data are collected and annotated Human Biases in Collection and Annotation Sampling error Non-sampling error Insensitivity to sample size Correspondence bias In-group bias Bias blind spot Confirmation bias Subjective validation Experimenter’s bias Choice-supportive bias Neglect of probability Anecdotal fallacy Illusion of validity Automation bias
- 28. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output
- 29. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias
- 30. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias Human Bias Human Bias Human Bias
- 31. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias Biased data created from process becomes new training data Human Bias Human Bias Human Bias
- 32. Bias: A systematic deviation from full ground truth.
- 33. Bias: A systematic deviation from full ground truth.
- 34. Can be used as a signal.
- 35. Reporting Bias in Vision and Language
- 36. Data data everywhere … But not many labels to train Exhaustively annotated data is expensive dog, chair, pizza, donut dog, chihuahua, brown, chair, table, wall, space heater, pizza, greasy, donut 1, donut 2, pizza slice 1, pizza slice 2...
- 37. Simple Image Classification Linear Sigmoid Classifier CNN Banana Yellow Output Input Image Ground Truth “Gold standard” Annotation: Human-biased label yw ∈ {0, 1} Prediction hw (yw |I) w ∈ {banana, yellow} yw For each w
- 38. Factoring in Reporting Bias: Idea ● A human-biased prediction h can be factored into two terms
- 39. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? w ∈ {banana, yellow} Factoring in Reporting Bias: Idea
- 40. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? w ∈ {banana, yellow} Factoring in Reporting Bias: Idea
- 41. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? Factoring in Reporting Bias: Idea
- 42. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? Label Prediction Visually correct ground truth (Unknown) Available ground truth (human-biased) Is concept present? Given visual presence, is concept relevant? Factoring in Reporting Bias: Idea
- 43. End-to-End Approach SOURCE Misra, Ishan; Girshick, Ross; Mitchell, Margaret; Zitnick, Larry (2016). “Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels”. Proceedings of CVPR.
- 45. Ishan Misra Larry Zitnick Ross Girshick Josh Lovejoy Hartwig Adam Blaise Agüera y Arcas CMU FAIR FAIR Google Google Google Thanks! margarmitchell@gmail.com Margaret Mitchell Google