Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor

24 views

Published on

Background on Lydia Chilton's crowd algorithms work, Maneesh Agrawala's algorithms for generating good designs from design principles, and past and future work on crowdsourcing humor

Published in: Engineering
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor

  1. 1. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor Lydia Chilton 13 July 2016 MUGS Lunch Talk, Stanford
  2. 2. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  3. 3. Summary
  4. 4. Simple Crowd Task: Label this image Complex Crowd Algorithm: Decipher this message Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller. TurKit: Human Computation Algorithms on Mechanical Turk. UIST 2010. Crowdsourcing: Decompose Problems so People Can Collaboratively Solve them
  5. 5. Decomposing and Automate Design: Model Design as a Search Problem To automate well-designed visual communication: 1. Collect design principles from experts 2. Define a search space 3. Define Constraints 4. Define an objective function to maximize Maneesh Agrawala, et al. Design Principles for Visual Communication. CACM 2011.
  6. 6. Humor is a Difficult AI Problem Model Humor Generation as Constraint-based Search Descriptive Theories of Humor: Constraint: “Violate an expectation”
  7. 7. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  8. 8. Microtask Crowdsourcing 8
  9. 9. Microtasks were Parallel 9
  10. 10. TurKit: Crowd Algorithms 10
  11. 11. Adaptive Mechanism: Improve-and-Vote 11 initial artifact output improve vote improved original
  12. 12. Problems too hard for one person 12 You (misspelled) (several) (words). Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they get lost amidst the (writing). (signature)
  13. 13. 13 initial artifact output Improve-and-Vote Builds on Contextual Clues from Other Workers You (?) (?) (?) (work). You (misspelled) (several) (words). improve vote improved original You (?) (?) (?) (work). (?) (?) (?) (?) (?).
  14. 14. 14 MTurk TurKit Seaweed Soylent VizWiz Adrenaline Legion PlateMate Crowdy Cascade Frenzy … … … Other Crowd Algorithms TurKontrol Deluge
  15. 15. Cascade Distributes big problems with complex interdependencies such as creating taxonomies 15 Lydia B. Chilton, et al. Cascade: Crowdsourcing Taxonomy Creation. CHI 2013.
  16. 16. Frenzy Combines the knowledge of many experts to meet a global constraint such as creating conference sessions Lydia B. Chilton, et al. Frenzy: collaborative data organization for creating conference sessions. CHI 2014. 16 Used at CSCW 2013 and CHI 2014 (136 papers, 431 papers)
  17. 17. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  18. 18. LineDrive: Using Design Principles to Automate Design Problems 18
  19. 19. LineDrive: Using Design Principles to Automate Design Problems Methodology • Expert patterns • Observations from examples of expert work • Grounded in psychological principles. 19
  20. 20. LineDrive: Using Design Principles to Automate Design Problems Methodology • Expert patterns • Observations from examples of expert work • Grounded in psychological principles. 20
  21. 21. Applying Design Principles • Generative Rules: – Define design spaces • Descriptive Rules: – Identify quantitative evaluation functions 21
  22. 22. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  23. 23. Creativity Intelligence 23 Humor is a Desirable Property of Communication
  24. 24. Humor evokes Mirth 24 • Surprise • Delight
  25. 25. Persuasion 25
  26. 26. Capturing Attention 26
  27. 27. Education 27
  28. 28. Humor is Hard for Computers 28
  29. 29. Descriptive Theories of Humor
  30. 30. Expectation Violation Expectation Violation 30
  31. 31. Associations Association: Fun = Good 31
  32. 32. Descriptive Theories of Humor are Not Generative “Write something that violates an expectation.”
  33. 33. Approach To Creating a Generative Model of Humor 33 Examples + Theories Model + System + Study
  34. 34. 34 “I can’t believe people would just walk into an Apple store and start breaking things like it’s a Best Buy.” American Voices Examples People Bending iPhones At Apple Stores Friday September 30, 2014 Input Output
  35. 35. 35 1. Associations 2. Expectation Violation Books on humor by philosophers, linguists, comedians Theory
  36. 36. Satisfy a Constraint by searching a space of associations 36 Headline Joke Expectation Violation Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Model
  37. 37. Sure, I guess UConn’s course is fine if you couldn’t get into Yale’s football clinic. UConn Holding Football 101 Clinic for Female Fans Association: Alternative Thing 37
  38. 38. Alright, how many ‘Summer Savannah’ Backyard Garden Lion Pedestals do I have to order to turn this thing around? Report: ‘SkyMall’ Magazine May End Print Edition Association: Details 38
  39. 39. Just tell me what to do and where to go to guarantee happiness forever. Poll: Elite Colleges Don’t Produce Happier Graduates Association: Personality Flaw 39
  40. 40. Satisfying a Constraint by searching a space of associations 40 Headline Joke Expectation Violation Sarcasm Unexpected Angle Bait-and- Switch Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Model
  41. 41. Study: First Born Children More Ambitious As a second-born girl, I’d just like to say wooohoo! Spring break! Bait-and-Switch Bait: The study is false. Switch: The study is true. 41 Expectation Violation #1
  42. 42. Unexpected Angle “It’s an exciting time to be a shark, that’s for sure!” Great White Shark Populations Surging Off East Coast Expected Focus: This is bad for people Unexpected Focus: This is good for sharks 42 Expectation Violation #2
  43. 43. Sarcasm “There is absolutely no other explanation.” Mick Jagger Blamed for Brazil’s World Cup Defeat 43 Expectation Violation #3 Literal Subtext: This is true Sarcastic Flip Subtext: This is false
  44. 44. How Not to Make a Joke Static Workflow Why not? Does not adapt to the input The same recipe won’t apply to every headline.
  45. 45. 45 Headline Joke Expectation Violation Sarcasm Unexpected Angle Bait-and- Switch Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Instead, use an Adaptive Workflow That searches the space of associations until a constraint can be satisfied
  46. 46. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  47. 47. HumorTools Adaptive Workflow • Apply microtasks to the headline • Until you can find a way of violating an expectation • Follow your train of thought, until you get stuck, then backtrack • Not yet crowdsourced • But it is decomposed into microtasks 47 Aspects Violation Mechanism Headline Prototype & Test Joke Expected Reactions Associations
  48. 48. HumorTools Interface 48 Justin Bieber was baptized this weekend in an attempt to wash away his sins following a scandal in which the singer appears in videos uses racial slurs
  49. 49. Justin Bieber Baptized In NYC BathtubHeadline Joke 49
  50. 50. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects 50
  51. 51. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. Expected Reactions 51
  52. 52. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. Expected Reactions Violation Mechanism 52
  53. 53. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. 53
  54. 54. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. 54
  55. 55. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism 55
  56. 56. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism Prototype Test Joke … …… … Never let it be said that Bieber’s PR people aren’t bringing new ideas to the table. 56
  57. 57. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism Prototype Test Joke … …… … Never let it be said that Bieber’s PR people aren’t bringing new ideas to the table. 57
  58. 58. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke … …… … 58
  59. 59. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross … …… … 59
  60. 60. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub … …… … 60
  61. 61. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub Prototype Test …… …… …… … 61
  62. 62. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub Prototype Test …… …… …… … Oh my God! Can I lick the tub? 62
  63. 63. HumorTools Study • 20 users: Stanford undergraduates • Humor novices • 60-80 minutes: – Tutorial on 20 microtasks – Humor Writing • Write jokes for 3 headlines • Rate jokes against The Onion (21 people) 63
  64. 64. HumorTools Study Headlines 64 PETA Seeks Copyright for Primate A lawsuit filed by PETA claims that “monkey selfies” snapped by a macaque who stole a photographer’s camera should be considered the legal property of the macaque himself. Liquid Water Found on Mars NASA revealed Monday that they have found evidence of liquid water on the Mars, pointing to the possibility of life on the red planet. Can users synthesize humor?
  65. 65. HumorTools Evaluation • 75% of participants were able to synthesize humor • 25% of the HumorTools jokes were rated funny 65
  66. 66. 66 PETA Seeks Copyright for Primate This is great news for animal rights. Now my neighbor can take my dog to court instead of me the next time he deposits some of his “legal property” in their front lawn. This is why you always get the macaque to sign a release.
  67. 67. 67 PETA Seeks Copyright for Primate This is great news for animal rights. Now my neighbor can take my dog to court instead of me the next time he deposits some of his “legal property” in their front lawn. This is why you always get the macaque to sign a release. HumorTools 47% Funny 62% Funny
  68. 68. 68 Liquid Water Found on Mars I think it’s more convenient for me to get it from the tap. In other news, Pluto has been called even less of a planet now. Sounds refreshing!
  69. 69. 69 Liquid Water Found on Mars HumorTools HumorTools I think it’s more convenient for me to get it from the tap. In other news, Pluto has been called even less of a planet now. Sounds refreshing! 15% Funny 45% Funny 45% Funny
  70. 70. User Comments 70 “[Associations] helped me think of jokes in a wider conceptual space than I previously had.” (p9) Before this, I never really knew where to start with joke writing I just kind of sat and thought until I came up with something. (p5)
  71. 71. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  72. 72. Problem: The Search Space is Big
  73. 73. Solution: Maximize expected utility by selecting microtasks that are correlated with successfully making jokes
  74. 74. Headline Types (8 total) • Science – Scientists Developing Heat-Resistant Chickens To Withstand Climate Change • Celebrity / Entertainment – Mattel Announced Barbie Movie • Absurd – Company Unveils 'Drinkable Sunscreen'
  75. 75. Same Related Insult Alternative Detail POV AVERAGE 0.876 1.170 0.104 0.456 0.328 0.452 American Politics 1.00 1.093 0.130 0.389 0.407 0.440 Business 0.821 1.321 0.154 0.436 0.295 0.530 Entertainment 0.753 1.358 0.049 0.420 0.444 0.421 Science 0.877 1.231 0.036 0.536 0.290 0.588 Celebrity 0.636 1.273 0.242 0.394 0.636 0.389 Sports 1.133 1.133 0.333 0.933 0.200 0.333 Tech 1.111 0.944 0.139 0.417 0.500 0.429 Absurd 0.884 0.977 0.109 0.403 0.194 0.487 Frequency per joke (588 jokes total) Red indicates “well above average” Association TypesHeadline Type P(joke contains an association type | Headline Type)
  76. 76. Headline Type Correlation with Joke Strategy
  77. 77. Joke Strategies for Entertainment Headlines: BE TOO SERIOUS
  78. 78. Joke Strategies for Absurd Headlines ACCEPT THE ABSURD PREMISE
  79. 79. Crowdsourcing Humor Associations Violation Mechanisms Shorten and Polish
  80. 80. Future Work: Automate Some Microtasks Train ML system with user collected data 81 • Entity extraction • Sentiment analysis • Semantic parsing
  81. 81. Grant: Be funnier than the Onion 82
  82. 82. Summary
  83. 83. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor
  84. 84. Crowdsourcing: Decompose Problems so People Can Collaboratively Solve them Simple Crowd Task: Label this image Complex Crowd Task: Decipher this message Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller. TurKit: Human Computation Algorithms on Mechanical Turk. UIST 2010.
  85. 85. To Automate Design, People Model Design as a Search Problem 1. Collect design principles from experts 2. Define a search space 3. Define Constraints 4. Define an objective function to maximize Maneesh Agrawala, et al. Design Principles for Visual Communication. CACM 2011.
  86. 86. Humor is a Difficult AI Problem Model Humor Generation as Constraint-Based Search Descriptive Theories of Humor: Constraint: “Violate an expectation”
  87. 87. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Decomposing Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor

×