Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Margaret Mitchell, Senior Research Scientist, Google, at MLonf Seattle 2017

655 views

Published on

Margaret Mitchell is the Senior Research Scientist in Google’s Research & Machine Intelligence group, working on advancing artificial intelligence towards positive goals. She works on vision-language and grounded language generation, focusing on how to help computers communicate based on what they can process. Her work combines computer vision, natural language processing, social media, many statistical methods, and insights from cognitive science. Before Google, She was the Researcher at Microsoft Research, and a founding member of their “Cognition” group. Her work focused on advancing artificial intelligence, with specific focus on generating language from visual inputs. Before MSR, she was a postdoctoral researcher at The Johns Hopkins University Center of Excellence, where she mainly focused on semantic role labeling and sentiment analysis using graphical models, working under Benjamin Van Durme.

Margaret received her PhD in Computer Science from the University of Aberdeen, with supervision from Kees van Deemter and Ehud Reiter, and external supervision from the University of Edinburgh (with Ellen Bard) and Oregon Health & Science University (OHSU) (with Brian Roark and Richard Sproat). She worked on referring expression generation from statistical and mathematical perspectives as well as a cognitive science perspective, and also worked on prototyping assistive/augmentative technology that people with language generation difficulties can use. Her thesis work was on generating natural, human-like reference to visible, everyday objects. She spent a good chunk of 2008 getting a Master’s in Computational Linguistics at the University of Washington, studying under Emily Bender and Fei Xia. Simultaneously (2005 – 2012), She worked on and off at the Center for Spoken Language Understanding, part of OHSU, in Portland, Oregon. Her title changed with time (research assistant/associate/visiting scholar), but throughout, she worked under Brian Roark on technology that leverages syntactic and phonetic characteristics to aid those with neurological disorders.

She continues to balance my time between language generation, applications for clinical domains, and core AI research.

Abstract summary

The Seen and Unseen Factors Influencing Machine Perception of Images and Language:
The success of machine learning has recently surged, with similar algorithmic approaches effectively solving a variety of human-defined tasks. Tasks testing how well machines can perceive images and communicate about them have begun to show a lot of promise, and at the same time, have exposed strong effects of different types of bias, such as overgeneralization. In this talk, I will detail how machines are learning about the visual world, and discuss how machine learning and different kinds of bias interact.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Margaret Mitchell, Senior Research Scientist, Google, at MLonf Seattle 2017

  1. 1. Ishan Misra Larry Zitnick Ross Girshick Josh Lovejoy Hartwig Adam Blaise Agüera y Arcas CMU FAIR FAIR Google Google Google The Seen and Unseen Factors Influencing Machine Perception of Images and Language Margaret Mitchell Google
  2. 2. What do you see?
  3. 3. What do you see? ● Bananas
  4. 4. What do you see? ● Bananas ● Bananas at a store
  5. 5. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves
  6. 6. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas
  7. 7. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them
  8. 8. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them ● Bunches of bananas with stickers on them on shelves in a store
  9. 9. What do you see? ● Bananas ● Bananas at a store ● Bananas on shelves ● Bunches of bananas ● Bananas with stickers on them ● Bunches of bananas with stickers on them on shelves in a store Yellow Bananas
  10. 10. What do you see? Green Bananas Unripe Bananas
  11. 11. What do you see? Ripe Bananas Bananas with spots
  12. 12. What do you see? Ripe Bananas Bananas with spots Bananas good for banana bread
  13. 13. What do you see? Yellow Bananas Yellow is prototypical for bananas
  14. 14. A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
  15. 15. A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
  16. 16. “Female doctor” A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?
  17. 17. “Female doctor”“Doctor”
  18. 18. The majority of test subjects overlooked the possibility that the doctor is a she - including men, women, and self-described feminists. Wapman & Belle, Boston University
  19. 19. Word Frequency in corpus “spoke” 11,577,917 “laughed” 3,904,519 “murdered” 2,834,529 “inhaled” 984,613 “breathed” 725,034 “hugged” 610,040 “blinked” 390,692 “exhale” 168,985 World learning from text Gordon and Van Durme, 2013
  20. 20. Word Frequency in corpus “spoke” 11,577,917 “laughed” 3,904,519 “murdered” 2,834,529 “inhaled” 984,613 “breathed” 725,034 “hugged” 610,040 “blinked” 390,692 “exhale” 168,985 World learning from text Gordon and Van Durme, 2013
  21. 21. Human Reporting Bias The frequency with which people write about actions, outcomes, or properties is not a reflection of real-world frequencies or the degree to which a property is characteristic of a class of individuals. (Gordon and Van Durme, 2013)
  22. 22. Training data are collected and annotated
  23. 23. Training data are collected and annotated Model is trained
  24. 24. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated
  25. 25. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output
  26. 26. Human Biases in Data Reporting bias Selection bias Overgeneralization Out-group homogeneity bias Stereotypical bias Historical unfairness Implicit associations Implicit stereotypes Prejudice Group attribution error Halo effect Training data are collected and annotated
  27. 27. Human Biases in Data Reporting bias Selection bias Overgeneralization Out-group homogeneity bias Stereotypical bias Historical unfairness Implicit associations Implicit stereotypes Prejudice Group attribution error Halo effect Training data are collected and annotated Human Biases in Collection and Annotation Sampling error Non-sampling error Insensitivity to sample size Correspondence bias In-group bias Bias blind spot Confirmation bias Subjective validation Experimenter’s bias Choice-supportive bias Neglect of probability Anecdotal fallacy Illusion of validity Automation bias
  28. 28. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output
  29. 29. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias
  30. 30. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias Human Bias Human Bias Human Bias
  31. 31. Training data are collected and annotated Model is trained Media are filtered, ranked, aggregated, or generated People see output Human Bias Biased data created from process becomes new training data Human Bias Human Bias Human Bias
  32. 32. Bias: A systematic deviation from full ground truth.
  33. 33. Bias: A systematic deviation from full ground truth.
  34. 34. Can be used as a signal.
  35. 35. Reporting Bias in Vision and Language
  36. 36. Data data everywhere … But not many labels to train Exhaustively annotated data is expensive dog, chair, pizza, donut dog, chihuahua, brown, chair, table, wall, space heater, pizza, greasy, donut 1, donut 2, pizza slice 1, pizza slice 2...
  37. 37. Simple Image Classification Linear Sigmoid Classifier CNN Banana Yellow Output Input Image Ground Truth “Gold standard” Annotation: Human-biased label yw ∈ {0, 1} Prediction hw (yw |I) w ∈ {banana, yellow} yw For each w
  38. 38. Factoring in Reporting Bias: Idea ● A human-biased prediction h can be factored into two terms
  39. 39. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? w ∈ {banana, yellow} Factoring in Reporting Bias: Idea
  40. 40. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? w ∈ {banana, yellow} Factoring in Reporting Bias: Idea
  41. 41. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? Factoring in Reporting Bias: Idea
  42. 42. ● A human-biased prediction h can be factored into two terms ○ Visual presence v – Is the concept visually present? ○ Relevance r – Is the concept relevant for a human? Label Prediction Visually correct ground truth (Unknown) Available ground truth (human-biased) Is concept present? Given visual presence, is concept relevant? Factoring in Reporting Bias: Idea
  43. 43. End-to-End Approach SOURCE Misra, Ishan; Girshick, Ross; Mitchell, Margaret; Zitnick, Larry (2016). “Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels”. Proceedings of CVPR.
  44. 44. “Female doctor”“Male doctor”
  45. 45. Ishan Misra Larry Zitnick Ross Girshick Josh Lovejoy Hartwig Adam Blaise Agüera y Arcas CMU FAIR FAIR Google Google Google Thanks! margarmitchell@gmail.com Margaret Mitchell Google

×