Presented at AAAI 2016 Spring Symposium: Intelligent systems for supporting distributed human teamwork. Position paper available at: http://juhokim.com/files/AAAISymposium2016-Organic-Crowdsourcing-Systems.pdf

1. Juho Kim
Organic
Crowdsourcing Systems
with Krzysztof Gajos and Rob Miller

2. Image courtesy of Christine Daniloff at MIT

3. Human computation
human intelligence + computational process

4. Humans as computational units?
Passive & disengaged crowd

5. Organic crowdsourcing
Individual & collective benefits to the crowd
Icons by Martha Ormiston from Noun Project

6. User: personally
meaningful activity
Human computation artifact:
immediately available in UI
for future users
System:
algorithmic
coordination
Organic crowdsourcing system

7. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing

8. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing

9. how-to videos online

10. Navigating how-to videos is hard
find
repeat
skip

11. Learning with solution structure helps

12. Learning with solution structure helps

13. Learning with solution structure helps

14. How to extract & present
solution structure at scale?
Subgoal labeling is a costly operation:
Requires domain experts and knowledge extraction
experts to work together. [Catrambone, 2011]
Insight:
Subgoal labeling can be a good learning exercise!
Solution: LEARNERSOURCING

17. Multi-stage learnersourcing workflow
Learnersourcing Subgoal Labels for How-to Videos.
Sarah Weir, Juho Kim, Krzysztof Z. Gajos, & Robert C. Miller.
CSCW 2015.

19. Sidebar w/ interactive
subgoals & steps

20. Crowdy evaluation
Does participating in learnersourcing
improve learning?
Does the learnersourcing workflow
produce good subgoal labels?

21. Crowdy evaluation
Does participating in learnersourcing
improve learning?
Does the learnersourcing workflow
produce good subgoal labels?
Retention test: baseline < Crowdy
Matching expert quality labels

22. Learnersourcing applications for
educational videos
ToolScape [CHI 2014]
Interaction Peaks [L@S 2014] LectureScape [UIST2014] RIMES [CHI 2015]
Mudslide [CHI 2015]Crowdy[CSCW 2015]

23. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing

24. 57
420 papers
16 parallel rooms
200 sessions in 15 venues
4 days and 3442 attendees

25. 1pm
3pm
good or bad?
101 102 103 104

26. Community-guided
conference planning
Attendees / Authors / Committee members
explore program & express own preferences.
Final schedule
with less conflicts
& more preferences
Scheduling tool
encodes community
input & recommends
schedule changes.
organizer

27. Committee members
Authors: Customized survey [UIST 2013]
Attendees: Confer
[CHI 2014, HCOMP 2013]
[HCOMP 2014]

29. Authorsourcing
8,651 preferences and constraints from
645 authors, covering 87% of accepted papers
my paper
fits with my paper does not fit
paper E
paper
D
paper B
paper C
paper A
paper Z

30. Mixed-initiative scheduling tool
Cobi: A Community-Informed Conference Scheduling Tool.
Juho Kim, Haoqi Zhang, Paul André, Lydia B. Chilton, Wendy Mackay, Michel Beaudouin-Lafon, Robert C. Miller, Steven P. Dow.
UIST 2013.

31. Cobi is helping actual conferences

32. • Learning via online videos
• Planning a large conference
• Understanding a government budget
Three case studies of
organic crowdsourcing

36. BudgetMap:
Issue-driven budget navigation
BudgetMap: Engaging Taxpayers in the Issue-Driven Classification of a Government Budget.
Nam Wook Kim, Jonghyuk Jung, Eun-Young Ko, Songyi Han, Chang Won Lee, Juho Kim, Jihee Kim.
CSCW 2016.

37. Budget tagging interface

38. Organic crowdsourcing for
civic engagement
increased awareness &
understanding of
government budget
dynamically generated
social issue – budget program links

39. Organic crowdsourcing
design principles
Paid crowdsourcing
simple and concrete task
quality control
microscopic, focused task
to workers
cost: money
Organic crowdsourcing
personally meaningful task
incentive design
overall goal & contribution
awareness
cost: crowd’s time & effort

40. Organic Crowdsourcing Systems
Juho Kim | juho@juhokim.com | @imjuhokim | juhokim.com
User: personally
meaningful activity
Human computation artifact:
immediately available in UI
for future users
System:
algorithmic
coordination