Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
HumorTools:
An Adaptive Microtask Workflow for
Crowdsourcing Humor
Lydia Chilton
13 July 2016
MUGS Lunch Talk, Stanford
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
Summary
Simple Crowd Task:
Label this image
Complex Crowd Algorithm:
Decipher this message
Greg Little, Lydia B. Chilton, Max
Gold...
Decomposing and Automate Design:
Model Design as a Search Problem
To automate well-designed visual
communication:
1. Colle...
Humor is a Difficult AI Problem
Model Humor Generation as Constraint-based Search
Descriptive Theories of Humor:
Constrain...
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
Microtask Crowdsourcing
8
Microtasks were Parallel
9
TurKit: Crowd Algorithms
10
Adaptive Mechanism:
Improve-and-Vote
11
initial
artifact output
improve vote
improved
original
Problems too hard for one person
12
You (misspelled) (several) (words). Please spellcheck your work next time.
I also noti...
13
initial
artifact output
Improve-and-Vote Builds on
Contextual Clues from Other Workers
You (?) (?) (?) (work).
You (mis...
14
MTurk TurKit
Seaweed
Soylent
VizWiz
Adrenaline
Legion
PlateMate
Crowdy
Cascade
Frenzy
…
…
…
Other Crowd Algorithms
TurK...
Cascade
Distributes big problems with complex interdependencies such
as creating taxonomies
15
Lydia B. Chilton, et al.
Ca...
Frenzy
Combines the knowledge of many experts to meet a
global constraint such as creating conference sessions
Lydia B. Ch...
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
LineDrive: Using Design Principles to
Automate Design Problems
18
LineDrive: Using Design Principles to
Automate Design Problems
Methodology
• Expert patterns
• Observations from examples ...
LineDrive: Using Design Principles to
Automate Design Problems
Methodology
• Expert patterns
• Observations from examples ...
Applying Design Principles
• Generative Rules:
– Define design spaces
• Descriptive Rules:
– Identify quantitative
evaluat...
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
Creativity Intelligence
23
Humor is a Desirable Property of
Communication
Humor evokes Mirth
24
• Surprise
• Delight
Persuasion
25
Capturing Attention
26
Education
27
Humor is Hard for Computers
28
Descriptive Theories of Humor
Expectation Violation
Expectation
Violation 30
Associations
Association:
Fun = Good
31
Descriptive Theories of Humor
are Not Generative
“Write something that violates an expectation.”
Approach To Creating a
Generative Model of Humor
33
Examples + Theories
Model + System + Study
34
“I can’t believe people would just walk into
an Apple store and start breaking things
like it’s a Best Buy.”
American V...
35
1. Associations
2. Expectation Violation
Books on humor by
philosophers, linguists,
comedians
Theory
Satisfy a Constraint
by searching a space of
associations
36
Headline
Joke
Expectation
Violation
Apple
Broken DisplaysBest...
Sure, I guess UConn’s course is fine if you couldn’t
get into Yale’s football clinic.
UConn Holding Football 101 Clinic fo...
Alright, how many ‘Summer Savannah’ Backyard
Garden Lion Pedestals do I have to order to turn
this thing around?
Report: ‘...
Just tell me what to do and where to go to guarantee
happiness forever.
Poll: Elite Colleges Don’t Produce Happier
Graduat...
Satisfying a Constraint
by searching a space of
associations
40
Headline
Joke
Expectation
Violation
Sarcasm
Unexpected
Ang...
Study: First Born Children More
Ambitious
As a second-born girl, I’d just like
to say wooohoo! Spring break!
Bait-and-Swit...
Unexpected
Angle
“It’s an exciting time to be a
shark, that’s for sure!”
Great White Shark Populations Surging Off East Co...
Sarcasm
“There is absolutely no other
explanation.”
Mick Jagger Blamed for Brazil’s World Cup Defeat
43
Expectation Violat...
How Not to Make a Joke
Static Workflow
Why not?
Does not adapt to the input
The same recipe won’t apply to
every headline.
45
Headline
Joke
Expectation
Violation
Sarcasm
Unexpected
Angle
Bait-and-
Switch
Apple
Broken DisplaysBest Buy
Best Buy
Al...
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
HumorTools
Adaptive Workflow
• Apply microtasks to the headline
• Until you can find a way of violating
an expectation
• F...
HumorTools Interface
48
Justin Bieber was baptized this weekend in an attempt to wash
away his sins following a scandal in...
Justin Bieber Baptized In NYC BathtubHeadline
Joke 49
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
50
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bad.
His music sucks.
E...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bad.
His music sucks.
E...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bad.
His music sucks.
53
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR peopleAssoc...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR peopleAssoc...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR peopleAssoc...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR peopleAssoc...
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR people
Bad....
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR people
Bad....
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR people
Bad....
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR people
Bad....
Justin Bieber Baptized In NYC BathtubHeadline
Joke
Justin Bieber Baptized Bathtub
Headline
Aspects
Bieber’s PR people
Bad....
HumorTools Study
• 20 users: Stanford undergraduates
• Humor novices
• 60-80 minutes:
– Tutorial on 20 microtasks
– Humor ...
HumorTools Study Headlines
64
PETA Seeks Copyright for Primate
A lawsuit filed by PETA claims that “monkey selfies” snappe...
HumorTools Evaluation
• 75% of participants were able to synthesize
humor
• 25% of the HumorTools jokes were rated funny
65
66
PETA Seeks Copyright for Primate
This is great news for animal rights.
Now my neighbor can take my dog to court
instead...
67
PETA Seeks Copyright for Primate
This is great news for animal rights.
Now my neighbor can take my dog to court
instead...
68
Liquid Water Found on Mars
I think it’s more convenient for me to
get it from the tap.
In other news, Pluto has been ca...
69
Liquid Water Found on Mars
HumorTools
HumorTools
I think it’s more convenient for me to
get it from the tap.
In other n...
User Comments
70
“[Associations] helped me think of jokes in
a wider conceptual space than I
previously had.” (p9)
Before ...
Outline
• My Background: Crowd Algorithms
– TurKit
– Cascade & Frenzy
• Related Work: Automated Design
– LineDrive
• Curre...
Problem: The Search Space is Big
Solution:
Maximize expected utility
by selecting microtasks
that are correlated
with successfully making jokes
Headline Types (8 total)
• Science
– Scientists Developing Heat-Resistant
Chickens To Withstand Climate Change
• Celebrity...
Same Related Insult Alternative Detail POV
AVERAGE 0.876 1.170 0.104 0.456 0.328 0.452
American
Politics
1.00 1.093 0.130 ...
Headline Type
Correlation
with Joke Strategy
Joke Strategies
for Entertainment Headlines:
BE TOO SERIOUS
Joke Strategies for Absurd Headlines
ACCEPT THE ABSURD PREMISE
Crowdsourcing Humor
Associations
Violation
Mechanisms
Shorten and Polish
Future Work:
Automate Some Microtasks
Train ML system with user collected data
81
• Entity extraction
• Sentiment analysis...
Grant: Be funnier than the Onion
82
Summary
HumorTools:
An Adaptive Microtask Workflow for
Crowdsourcing Humor
Crowdsourcing: Decompose Problems so
People Can Collaboratively Solve them
Simple Crowd Task:
Label this image
Complex Cro...
To Automate Design, People Model
Design as a Search Problem
1. Collect design principles from
experts
2. Define a search s...
Humor is a Difficult AI Problem
Model Humor Generation as Constraint-Based Search
Descriptive Theories of Humor:
Constrain...
HumorTools: An Adaptive Microtask Workflow
for Crowdsourcing Humor
• My Background: Crowd Algorithms
– TurKit
– Cascade & ...
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
What to Upload to SlideShare
Next
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

2

Share

HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor

Download to read offline

Background on Lydia Chilton's crowd algorithms work, Maneesh Agrawala's algorithms for generating good designs from design principles, and past and future work on crowdsourcing humor

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor

  1. 1. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor Lydia Chilton 13 July 2016 MUGS Lunch Talk, Stanford
  2. 2. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  3. 3. Summary
  4. 4. Simple Crowd Task: Label this image Complex Crowd Algorithm: Decipher this message Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller. TurKit: Human Computation Algorithms on Mechanical Turk. UIST 2010. Crowdsourcing: Decompose Problems so People Can Collaboratively Solve them
  5. 5. Decomposing and Automate Design: Model Design as a Search Problem To automate well-designed visual communication: 1. Collect design principles from experts 2. Define a search space 3. Define Constraints 4. Define an objective function to maximize Maneesh Agrawala, et al. Design Principles for Visual Communication. CACM 2011.
  6. 6. Humor is a Difficult AI Problem Model Humor Generation as Constraint-based Search Descriptive Theories of Humor: Constraint: “Violate an expectation”
  7. 7. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  8. 8. Microtask Crowdsourcing 8
  9. 9. Microtasks were Parallel 9
  10. 10. TurKit: Crowd Algorithms 10
  11. 11. Adaptive Mechanism: Improve-and-Vote 11 initial artifact output improve vote improved original
  12. 12. Problems too hard for one person 12 You (misspelled) (several) (words). Please spellcheck your work next time. I also notice a few grammatical mistakes. Overall your writing style is a bit too phoney. You do make some good (points), but they get lost amidst the (writing). (signature)
  13. 13. 13 initial artifact output Improve-and-Vote Builds on Contextual Clues from Other Workers You (?) (?) (?) (work). You (misspelled) (several) (words). improve vote improved original You (?) (?) (?) (work). (?) (?) (?) (?) (?).
  14. 14. 14 MTurk TurKit Seaweed Soylent VizWiz Adrenaline Legion PlateMate Crowdy Cascade Frenzy … … … Other Crowd Algorithms TurKontrol Deluge
  15. 15. Cascade Distributes big problems with complex interdependencies such as creating taxonomies 15 Lydia B. Chilton, et al. Cascade: Crowdsourcing Taxonomy Creation. CHI 2013.
  16. 16. Frenzy Combines the knowledge of many experts to meet a global constraint such as creating conference sessions Lydia B. Chilton, et al. Frenzy: collaborative data organization for creating conference sessions. CHI 2014. 16 Used at CSCW 2013 and CHI 2014 (136 papers, 431 papers)
  17. 17. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools: – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks – Reducing the Humor Search Space – Crowdsourcing Humor
  18. 18. LineDrive: Using Design Principles to Automate Design Problems 18
  19. 19. LineDrive: Using Design Principles to Automate Design Problems Methodology • Expert patterns • Observations from examples of expert work • Grounded in psychological principles. 19
  20. 20. LineDrive: Using Design Principles to Automate Design Problems Methodology • Expert patterns • Observations from examples of expert work • Grounded in psychological principles. 20
  21. 21. Applying Design Principles • Generative Rules: – Define design spaces • Descriptive Rules: – Identify quantitative evaluation functions 21
  22. 22. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  23. 23. Creativity Intelligence 23 Humor is a Desirable Property of Communication
  24. 24. Humor evokes Mirth 24 • Surprise • Delight
  25. 25. Persuasion 25
  26. 26. Capturing Attention 26
  27. 27. Education 27
  28. 28. Humor is Hard for Computers 28
  29. 29. Descriptive Theories of Humor
  30. 30. Expectation Violation Expectation Violation 30
  31. 31. Associations Association: Fun = Good 31
  32. 32. Descriptive Theories of Humor are Not Generative “Write something that violates an expectation.”
  33. 33. Approach To Creating a Generative Model of Humor 33 Examples + Theories Model + System + Study
  34. 34. 34 “I can’t believe people would just walk into an Apple store and start breaking things like it’s a Best Buy.” American Voices Examples People Bending iPhones At Apple Stores Friday September 30, 2014 Input Output
  35. 35. 35 1. Associations 2. Expectation Violation Books on humor by philosophers, linguists, comedians Theory
  36. 36. Satisfy a Constraint by searching a space of associations 36 Headline Joke Expectation Violation Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Model
  37. 37. Sure, I guess UConn’s course is fine if you couldn’t get into Yale’s football clinic. UConn Holding Football 101 Clinic for Female Fans Association: Alternative Thing 37
  38. 38. Alright, how many ‘Summer Savannah’ Backyard Garden Lion Pedestals do I have to order to turn this thing around? Report: ‘SkyMall’ Magazine May End Print Edition Association: Details 38
  39. 39. Just tell me what to do and where to go to guarantee happiness forever. Poll: Elite Colleges Don’t Produce Happier Graduates Association: Personality Flaw 39
  40. 40. Satisfying a Constraint by searching a space of associations 40 Headline Joke Expectation Violation Sarcasm Unexpected Angle Bait-and- Switch Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Model
  41. 41. Study: First Born Children More Ambitious As a second-born girl, I’d just like to say wooohoo! Spring break! Bait-and-Switch Bait: The study is false. Switch: The study is true. 41 Expectation Violation #1
  42. 42. Unexpected Angle “It’s an exciting time to be a shark, that’s for sure!” Great White Shark Populations Surging Off East Coast Expected Focus: This is bad for people Unexpected Focus: This is good for sharks 42 Expectation Violation #2
  43. 43. Sarcasm “There is absolutely no other explanation.” Mick Jagger Blamed for Brazil’s World Cup Defeat 43 Expectation Violation #3 Literal Subtext: This is true Sarcastic Flip Subtext: This is false
  44. 44. How Not to Make a Joke Static Workflow Why not? Does not adapt to the input The same recipe won’t apply to every headline.
  45. 45. 45 Headline Joke Expectation Violation Sarcasm Unexpected Angle Bait-and- Switch Apple Broken DisplaysBest Buy Best Buy Alternative Insult Associations Instead, use an Adaptive Workflow That searches the space of associations until a constraint can be satisfied
  46. 46. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  47. 47. HumorTools Adaptive Workflow • Apply microtasks to the headline • Until you can find a way of violating an expectation • Follow your train of thought, until you get stuck, then backtrack • Not yet crowdsourced • But it is decomposed into microtasks 47 Aspects Violation Mechanism Headline Prototype & Test Joke Expected Reactions Associations
  48. 48. HumorTools Interface 48 Justin Bieber was baptized this weekend in an attempt to wash away his sins following a scandal in which the singer appears in videos uses racial slurs
  49. 49. Justin Bieber Baptized In NYC BathtubHeadline Joke 49
  50. 50. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects 50
  51. 51. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. Expected Reactions 51
  52. 52. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. Expected Reactions Violation Mechanism 52
  53. 53. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bad. His music sucks. 53
  54. 54. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. 54
  55. 55. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism 55
  56. 56. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism Prototype Test Joke … …… … Never let it be said that Bieber’s PR people aren’t bringing new ideas to the table. 56
  57. 57. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR peopleAssociations Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Violation Mechanism Prototype Test Joke … …… … Never let it be said that Bieber’s PR people aren’t bringing new ideas to the table. 57
  58. 58. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke … …… … 58
  59. 59. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross … …… … 59
  60. 60. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub … …… … 60
  61. 61. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub Prototype Test …… …… …… … 61
  62. 62. Justin Bieber Baptized In NYC BathtubHeadline Joke Justin Bieber Baptized Bathtub Headline Aspects Bieber’s PR people Bad. His music sucks. Unexpected Angle: Expected Reaction: Bad Angle: PR people are clever Angle Reaction: Good. Joke Expected Reactions Bad. Dirty, Gross Violation Mechanism Unexpected Angle: A Bieber fan who’d worship his dirty bathtub Prototype Test …… …… …… … Oh my God! Can I lick the tub? 62
  63. 63. HumorTools Study • 20 users: Stanford undergraduates • Humor novices • 60-80 minutes: – Tutorial on 20 microtasks – Humor Writing • Write jokes for 3 headlines • Rate jokes against The Onion (21 people) 63
  64. 64. HumorTools Study Headlines 64 PETA Seeks Copyright for Primate A lawsuit filed by PETA claims that “monkey selfies” snapped by a macaque who stole a photographer’s camera should be considered the legal property of the macaque himself. Liquid Water Found on Mars NASA revealed Monday that they have found evidence of liquid water on the Mars, pointing to the possibility of life on the red planet. Can users synthesize humor?
  65. 65. HumorTools Evaluation • 75% of participants were able to synthesize humor • 25% of the HumorTools jokes were rated funny 65
  66. 66. 66 PETA Seeks Copyright for Primate This is great news for animal rights. Now my neighbor can take my dog to court instead of me the next time he deposits some of his “legal property” in their front lawn. This is why you always get the macaque to sign a release.
  67. 67. 67 PETA Seeks Copyright for Primate This is great news for animal rights. Now my neighbor can take my dog to court instead of me the next time he deposits some of his “legal property” in their front lawn. This is why you always get the macaque to sign a release. HumorTools 47% Funny 62% Funny
  68. 68. 68 Liquid Water Found on Mars I think it’s more convenient for me to get it from the tap. In other news, Pluto has been called even less of a planet now. Sounds refreshing!
  69. 69. 69 Liquid Water Found on Mars HumorTools HumorTools I think it’s more convenient for me to get it from the tap. In other news, Pluto has been called even less of a planet now. Sounds refreshing! 15% Funny 45% Funny 45% Funny
  70. 70. User Comments 70 “[Associations] helped me think of jokes in a wider conceptual space than I previously had.” (p9) Before this, I never really knew where to start with joke writing I just kind of sat and thought until I came up with something. (p5)
  71. 71. Outline • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Breaking Down Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  72. 72. Problem: The Search Space is Big
  73. 73. Solution: Maximize expected utility by selecting microtasks that are correlated with successfully making jokes
  74. 74. Headline Types (8 total) • Science – Scientists Developing Heat-Resistant Chickens To Withstand Climate Change • Celebrity / Entertainment – Mattel Announced Barbie Movie • Absurd – Company Unveils 'Drinkable Sunscreen'
  75. 75. Same Related Insult Alternative Detail POV AVERAGE 0.876 1.170 0.104 0.456 0.328 0.452 American Politics 1.00 1.093 0.130 0.389 0.407 0.440 Business 0.821 1.321 0.154 0.436 0.295 0.530 Entertainment 0.753 1.358 0.049 0.420 0.444 0.421 Science 0.877 1.231 0.036 0.536 0.290 0.588 Celebrity 0.636 1.273 0.242 0.394 0.636 0.389 Sports 1.133 1.133 0.333 0.933 0.200 0.333 Tech 1.111 0.944 0.139 0.417 0.500 0.429 Absurd 0.884 0.977 0.109 0.403 0.194 0.487 Frequency per joke (588 jokes total) Red indicates “well above average” Association TypesHeadline Type P(joke contains an association type | Headline Type)
  76. 76. Headline Type Correlation with Joke Strategy
  77. 77. Joke Strategies for Entertainment Headlines: BE TOO SERIOUS
  78. 78. Joke Strategies for Absurd Headlines ACCEPT THE ABSURD PREMISE
  79. 79. Crowdsourcing Humor Associations Violation Mechanisms Shorten and Polish
  80. 80. Future Work: Automate Some Microtasks Train ML system with user collected data 81 • Entity extraction • Sentiment analysis • Semantic parsing
  81. 81. Grant: Be funnier than the Onion 82
  82. 82. Summary
  83. 83. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor
  84. 84. Crowdsourcing: Decompose Problems so People Can Collaboratively Solve them Simple Crowd Task: Label this image Complex Crowd Task: Decipher this message Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller. TurKit: Human Computation Algorithms on Mechanical Turk. UIST 2010.
  85. 85. To Automate Design, People Model Design as a Search Problem 1. Collect design principles from experts 2. Define a search space 3. Define Constraints 4. Define an objective function to maximize Maneesh Agrawala, et al. Design Principles for Visual Communication. CACM 2011.
  86. 86. Humor is a Difficult AI Problem Model Humor Generation as Constraint-Based Search Descriptive Theories of Humor: Constraint: “Violate an expectation”
  87. 87. HumorTools: An Adaptive Microtask Workflow for Crowdsourcing Humor • My Background: Crowd Algorithms – TurKit – Cascade & Frenzy • Related Work: Automated Design – LineDrive • Current Project: HumorTools – The Onion’s American Voices – Descriptive Models of Humor – Decomposing Humor into Microtasks for Individuals – Reducing the Humor Search Space – Crowdsourcing Humor
  • andreskralj

    Aug. 29, 2020
  • andreskralj

    Aug. 29, 2020

Background on Lydia Chilton's crowd algorithms work, Maneesh Agrawala's algorithms for generating good designs from design principles, and past and future work on crowdsourcing humor

Views

Total views

329

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

6

Shares

0

Comments

0

Likes

2

×