Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AI & Work, with Transparency & the Crowd


Published on

Invited talk at the 2019 AAAI Fall Symposium ( on Artificial Intelligence and Work (

Published in: Technology
  • Get Paid To Manage Facebook Fan Pages! Facebook Fan Page Workers Required - Start Immediately. ●●●
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

AI & Work, with Transparency & the Crowd

  1. 1. AI & Work, with Transparency & the Crowd Matt Lease School of Information @mattlease University of Texas at Austin Slides:
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at over 100 universities around the world What’s an Information School? 2
  3. 3. Part I: Designing Human-AI Partnerships 3Matt Lease (University of Texas at Austin)
  4. 4. Matt Lease (University of Texas at Austin) 4 UT Austin Grand Challenge: Good Systems Challenge: Ensure a future of “good” AI for society • What does “good” AI mean? How do we measure & build it?
  5. 5. Matt Lease (University of Texas at Austin) Designing for Human Interaction with AI 5
  6. 6. Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking Joint work with An Thanh Nguyen (UT), Byron Wallace (Northeastern), & more… ACM UIST 2018
  7. 7. Matt Lease (UT Austin) • Believe it or not: Designing a Human-AI Partnership for Mixed-Initiative Fact-Checking 7 Nguyen et al., UIST’18
  8. 8. Anubrata Das, Kunjan Mehta and Matthew Lease SIGIR 2019 Workshop on Fair, Accountable, Confidential, Transparent, and Safe Information Retrieval (FACTS-IR). July 25, 2019 CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking
  9. 9. 9 ACM IUI Workshop on Explainable Smart Systems (ExSS), 2019.
  10. 10. Part II: AI Meets the Crowd 10Matt Lease (University of Texas at Austin)
  11. 11. Going Beyond AI 11 “Software developers with innovative ideas for businesses and technologies are constrained by the limits of artificial intelligence… If software developers could programmatically access and incorporate human intelligence into their applications, a whole new class of innovative businesses and applications would be possible. This is the goal of Amazon Mechanical Turk… people are freer to innovate because they can now imbue software with real human intelligence.”
  12. 12. Davis et al. (2010) The HPU. HPU 12
  13. 13. Matt Lease <> 13
  14. 14. Matt Lease (University of Texas at Austin) WWW 2011 CrowdConf 2010 14 Fort, Adda, & Cohen (2011) – Is crowdsourcing a “gold mine or coal mine” for the workers?
  15. 15. • Silberman, Irani, and Ross (2010) – “How should we… conceptualize the role of these people who we ask to power our computing?” • “Irani and Silberman (2013) – “…AMT helps employers see themselves as builders of innovative technologies, rather than employers unconcerned with working conditions.” How can we design for “good” win-wins? 15 Paul Hyman. Communications of the ACM, Vol. 56 No. 8, Pages 19-21, August 2013.
  16. 16.
  17. 17. The Future of Crowd Work, CSCW’13 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 19Matt Lease <>
  18. 18. Crowdsourcing & regulation • Wolfson & Lease (ASIS&T 2011) • As usual, technology is ahead of the law – employment law – patent inventorship – data security and the Federal Trade Commission – copyright ownership – securities regulation of crowdfunding 20
  19. 19. Design Activism for Minimum Wage Crowd Work Mankar, Shah, & Lease, HCOMP 2017 Inspiration See also: Whiting et al. Fair Work: Crowd Work Minimum Wage with One Line of Code. HCOMP 2019. Honorable Mention.
  20. 20. Digital “Dirty Jobs” • The Googler who Looked at the Worst of the Internet • Facebook content moderation • The dirty job of keeping Facebook clean • Even linguistic annotators report stress & nightmares from reading news articles (Strauss et al., LREC 2000) 22
  21. 21. Litigation & research • Soto & Blauert vs. Microsoft Corporation (2018) • Two content moderators report post-traumatic stress disorder (Ghoshal 2017) from having to watch child pornography as content moderators • Growing research awareness & interest • Conferences and workshops, e.g., at UCLA, Santa Clara University, USC, and Alexander von Humboldt Institute for Internet and Society
  22. 22. BUT WHO PROTECTS THE MODERATORS? BRANDON DANG1, MARTIN J. RIEDL2, AND MATTHEW LEASE1 1School of Information, 2School of Journalism (both students contributed equally) The University of Texas at Austin AAAI HCOMP -&- ACM Collective Intelligence July 2018, Zurich, Switzerland
  23. 23. But Who Protects the Moderators? 25 See also: Karunakaran & Ramakrishnan. Testing Stylistic Interventions to Reduce Emotional Impact of Content Moderation Workers. HCOMP 2019.
  24. 24. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Donna Vakharia & Matthew Lease University of Texas at Austin 2015 iConference
  25. 25. Matt Lease <> 27/20
  26. 26. Many Crowd Work Platforms Today And More! JobBoy, microWorkers, MiniFreelance, MiniJobz, MinuteWorkers, MyEasyTask, OpTask, ShortTask, SimpleWorkers Matt Lease <> 28
  27. 27. Here: Qualitative Study of 7 Platforms We shared preliminary findings with personnel from all platforms, requesting & incorporating feedback 29
  28. 28. Problems vs. Platform Capabilities Matt Lease <> 30
  29. 29. AAAI HCOMP 2013 Industry Panel Anand Kulkarni: “How do we dramatically reduce the complexity of getting work done with the crowd?” Greg Little: How can we post a task and with 98% confidence know we’ll get a quality result? Matt Lease <> 31/20
  30. 30. Robert Sim, MSR Summit’12 32Matt Lease <>
  31. 31. Ethics Checking: The Next Frontier in FAT AI? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 33
  32. 32. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 34
  33. 33. Matt Lease (University of Texas at Austin) – Thank You! 35 Lab: @mattlease Slides: