Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understanding Blind People’s Experiences with Computer-Generated Captions of Social Media Images

103 views

Published on

Presented at CHI 2017

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Understanding Blind People’s Experiences with Computer-Generated Captions of Social Media Images

  1. 1. Understanding Blind People’s Experiences with Captions of Social Media Images Haley MacLeod CynthiaL. Bennett Meredith Ringel Morris Edward Cutrell Indiana University University of Washington Microsoft Research Microsoft Research @haley_macleod ww.haleymacleod.com
  2. 2. CaptionBot
  3. 3. “I think it's a young man jumping in the air on a skateboard.”
  4. 4. “I think it's a man in a suit and tie and he seems 😠😠”
  5. 5. “I am not really confident, but I think it's a group of men standing next to a vase and he seems 😐😐.”
  6. 6. These tools can empower blind and visually impaired people to know more about these images.
  7. 7. Typically, these systems are evaluated by sighted people.
  8. 8. Standardized Metrics Sighted Human Output Machine Output
  9. 9. User Studies • Rate the quality of the caption for a given image • Choose the best from a series of caption
  10. 10. The evaluationcriteria for caption quality are different than they would be if designed with accessibility as the primary scenario.
  11. 11. We explore how blind people experience automatically generated captions in the context of social media.
  12. 12. Contextual Inquiry Experiences with image captions in the Twitter feed (n=6) Online Experiment The impact of caption phrasing on trust and skepticism (n=100)
  13. 13. Study One: Contextual Inquiry
  14. 14. Observations
  15. 15. Observations Interviews
  16. 16. Observations Interviews Image Probes
  17. 17. Observations Interviews Image Probes More Interviews
  18. 18. • Taken from trending hashtags, popular accounts • Range of topics (news, memes, photography, cats, selfies, etc.) • Computer generated captions as alt text Tweet/Image Probes:
  19. 19. @WholeFoodsMarketReady for #summer entertaining?Plan w/ special diets in mind. #glutenfree #paleo #vegan
  20. 20. “I thinkit’s a plate of pasta and broccoli” @WholeFoodsMarketReady for #summer entertaining?Plan w/ special diets in mind. #glutenfree #paleo #vegan
  21. 21. @HillaryClinton Some on the other side may say our best days are behind us. Let's provethem wrong.
  22. 22. “I’m not really confident,but I think it’s a man doing a trick on a skateboardat night.” @HillaryClinton Some on the other side may say our best days are behind us. Let's provethem wrong.
  23. 23. “I felt pretty confident about them being accurate” (P2)
  24. 24. “I have to trust them because I don’t have any form of reference […] Unless somebody sighted were saying, ‘I don’t know why they said that, that’s not what it is at all…’” (P1)
  25. 25. “I always double check. Cause if I’m going to retweet something, I want to know that I’m retweeting what I think it is… I definitely wouldn’t retweet that without somebody looking at it first.” (P3)
  26. 26. “I’m not really confident,but I think it’s a man doing a trick on a skateboardat night.” @HillaryClinton Some on the other side may say our best days are behind us. Let's provethem wrong.
  27. 27. “I probably would have just retweeted it thinking it was a photo of a skateboarder” (P3)
  28. 28. “If you say, there’s an older man on a skateboard at night that would make more sense. That’s the way I take ‘our best days are behind us’ because I’m an old lady…” (P3)
  29. 29. @NewYorkTimes Why isn’t Italy kinder to gays? I'm not really confident,but I thinkit's a person holdinga tennis racquetand she seems neutral face.
  30. 30. “Okay, so it’s a tennis player. Who, if I recognized tennis players, maybe she’s gay.” (P2)
  31. 31. “Why is a tennis racket involved in this story? The way pictures are eye catching to people, it would engage a different side of my perception. Saying, oh tennis racket. I don’t play tennis, but maybe that’s even more interesting than I thought.” (P5)
  32. 32. Contextual Inquiry Experiences with image captions in the Twitter feed (n=100) Online Experiment The impact of caption phrasing on trust and skepticism (n=100)
  33. 33. Study Two: Online Experiment
  34. 34. Framing Effects X% chance of dying vs. Y% chance of surviving
  35. 35. Framing Effects X% chance of dying vs. a very small chance of dying
  36. 36. PositiveFraming Negative Framing Natural Language Framing “I'm really confident that's a cat sitting on a counter next to a window.” “There's a small chance I could be wrong, but I think that's a cat sitting on a counter next to a window.” Numeric Framing “There's an 80% chance that's a cat sitting on a counter next to a window.” “There's a 20% chance I'm wrong, but I think that's a cat sitting on a counter next to a window”
  37. 37. Background
  38. 38. Background Understanding
  39. 39. Background Understanding Overall Trust
  40. 40. Background Understanding Overall Trust Further Information
  41. 41. Background Understanding Overall Trust Further Information Optional Follow Up
  42. 42. Congruent Captions vs. Incongruent Captions
  43. 43. @ALensOnAmerica Share your best wildlife photo from a national park and get feedback from @chamiltonjames. #YourShot. “There's a 25% chance that's a bear is swimmingin the water.”
  44. 44. @SeattlePoliceDepartment Bank robber arrested after abandoningpants. “There’s a 5% chance that’sa personon a surfboardin a skatepark.”
  45. 45. People found tweets with high congruence to be more trustworthyand more helpful.
  46. 46. People who received negatively framed captions trusted incongruent captions significantly less often.
  47. 47. Negatively phrased captions better encouraged a lower level of reliance on incongruent captions, and a higher level of reliance on congruent captions.
  48. 48. Thus, we recommend that automatically generated captions be phrased in a way that reinforces the possibility the caption might be wrong (negative framing).
  49. 49. Negatively framed captions help blind and visually impaired people rely more on their intuition about a caption, rather than unquestioningly trusting a caption and making decisions based off misinformation.
  50. 50. Contextual Inquiry Experiences with image captions in the Twitter feed (n=6) Online Experiment The impact of caption phrasing on trust and skepticism (n=100)
  51. 51. • We observed that blind people are very trusting of captions, even when they don’t make much sense. • If we phrase captions in such a way that reinforces the possibility the caption might be wrong, people rely more on their intuition and expectations of a caption. • They create stories to resolve these differences, rather than suspect that captions might be wrong. Haley MacLeod CynthiaL. Bennett Meredith Ringel Morris Edward Cutrell www.haleymacleod.com @haley_macleod

×