1001_SNU UX LAB Seminar

1. Effects of Simultaneous and
Sequential Work Structures
on Distributed Collaborative
Interdependent Tasks
+ SIGCHI 2014
-Paul Andre, 2014
/이현정
x 2015 Autumn
Simultaneous
Distributed Collaborative
Sequential

2. Effects of Simultaneous and
Sequential Work Structures
on Distributed Collaborative
Interdependent Tasks
+ SIGCHI 2014
-Paul Andre
/이현정
x 2015 Autumn

3. The Author

4. The Author

5. The Author

6. • This work was partially supported by Carnegie Mellon’s Center for the Future of Work, Bosch,
Google, and NSF
The Paper
• Faculty and students from a range of CMU departments — the School of Computer Science,
Carnegie Institute of Technology, the School of Design at the College of Fine Arts, Heinz
College, and the Tepper School of Business - See more at: http://fow.heinz.cmu.edu/
#sthash.PiaD6CiJ.dpuf

7. Distributed Online Groups + complex creative tasks
Coordinating multiple group members to work together effectively
while minimizing process losses
What
The
Challenge
Comparing the effectiveness
of two coordination strategies(simultaneous vs. sequential work) on a complex
creative task as the number of group members increased
Experiment
Results • sequential work structure > simultaneous work structure
• territoriality partially accounts for these results (mediation analysis)
• giving workers specific roles mitigate the detrimental effects of the
simultaneous work structure
• contribute to small group theory
• practical understanding of how to use crowdsourcing
for creative and complex tasks
Implications
Abstract
design product
develop software
write encyclopedia

8. Distributed Online Groups + complex creative tasks
Coordinating multiple group members to work together effectively
while minimizing process losses
What
The
Challenge
Comparing the effectiveness
of two coordination strategies(simultaneous vs. sequential work) on a complex
creative task as the number of group members increased
Experiment
Results • sequential work structure > simultaneous work structure
• territoriality partially accounts for these results (mediation analysis)
• giving workers specific roles mitigate the detrimental effects of the
simultaneous work structure
• contribute to small group theory
• practical understanding of how to use crowdsourcing
for creative and complex tasks
Implications
Abstract
design product
develop software
write encyclopedia
Process Loss
(social psychology)
Any aspect of group interactions that
inhibits good problem solving.
Raw material -> finisher products

9. Distributed Online Groups + complex creative tasks
Coordinating multiple group members to work together effectively
while minimizing process losses
What
The
Challenge
Comparing the effectiveness
of two coordination strategies(simultaneous vs. sequential work) on a complex
creative task as the number of group members increased
Experiment
Results • sequential work structure > simultaneous work structure
• territoriality partially accounts for these results (mediation analysis)
• giving workers specific roles mitigate the detrimental effects of the
simultaneous work structure
• contribute to small group theory
• practical understanding of how to use crowdsourcing
for creative and complex tasks
Implications
Abstract
design product
develop software
write encyclopedia

10. Introduction
number of people
way to organize their effort
Crucial
Questions
pros and cons of sequential vs. simultaneous work structure on
interdependent tasks as the number of workers increases
optimal output
best use of each worker
Structural
contingency
Theory
optimal structure or coordination method varies according to factors such as
group size and task uncertainty
creative task
assembly line
work process not rigid
unable to be defined in advance
task is interdependent
Aim of Paper
+ +
+
+
+

11. Related Work
More
is good
• can offer new knowledge
• cognitive stimulation
• error detection
task type + coordination mechanism
Task Type
Both in one
• [larger > smaller]
- tasks with objective answers
- "Eureka" tasks(everyone will recognize a correct answer once it is
offered), because larger group is more likely to contain individual with
answer
• [larger< smaller]
- problem solving tasks
(Groups preform better than the best individuals...Lauglin, 2006)
• trip planning : generation of ideas + consider constraints
• collaborative poetry translation: creative + negotiation with coworkers
• iterative chair design: creative + practical consideration

12. Related Work
Coordination
Styles
• A group of workers can be utilized in different ways
• and their interactions defined by their environment
pooled sequential teem
simply the
aggregation of
individually
preformed tasks
one worker build
on another's
output
require
interaction
among members
creative taskassembly line
• synchronous work the best?
• high coordination costs, adding workers may increase process losses > benefit
the extra worker brings
• This paper argues that sequential work may mitigate such process loss

13. Related Work
Coordination
Styles
• A group of workers can be utilized in different ways
• and their interactions defined by their environment
pooled sequential teem
simply the
aggregation of
individually
preformed tasks
one worker build
on another's
output
require
interaction
among members
creative taskassembly line
• synchronous work the best?
• high coordination costs, adding workers may increase process losses > benefit
the extra worker brings
• This paper argues that sequential work may mitigate such process loss

14. Experiment 1
group limerick(five-line poem) writing
using etherpad
There was an old man with a beard
Who said, 'It is just as I feared!'
Two Owls and a Hen,
Four Larks and a Wren,
Have all built their nests in my beard!
-Edward Lear
What
Why uncertain + interdependent
production + problem-solving
How
1 worker 2 worker
sequential
1 worker 2 worker
simultaneous
1 worker 3 worker
sequential
1 worker 3 worker
simultaneous
2 x 2 experiment
add 1 worker in each iteration
e,g. 1, 1+1, 1+1+1
Where Amazon Mechanical Turk (Online market place for work)
GROUP SIZE IN SIMULTANEOUS&SEQUENTIAL WORK

15. Procedure
Simultaneous
Sequential
1."Help Write a limerick(a short rhyme)" on MTurk
2.Waiting Room - crowdsourcing experiments
3.Page open in Background -> 2-3 people ready -> alerted
-> 10 minutes X -> 1 worker contribution
4.Taken to page with instructions and shown an etherpad
• no time limit
• Payment - $0.70 per limerick, MTurk hourly rates
1."Improve /edit a limerick(short rhyme)" on MTurk
2.Waiting Room - crowdsourcing experiments
3.Page open in Background -> 2-3 people ready -> alerted
-> 10 minutes X -> 1 worker contribution
4.Taken to page with instructions and shown an etherpad
• no time limit
• Payment - $0.50 per limerick,
Same collaborative interface, similar instructions, same task artifacts

16. Measures
Outcome
Measures
• overall quality
• technical quality: is the limerick the right number of lines, correct rhyme structure
initially split assessment into four criteria to capture both objective and subjective
aspects (technical, story coherence, creativity, and overall quality)
initial testing found all to be correlated and for the remainder of the experiment to
use the two clearest features
Raters • 3 raters
: 2 undergraduate or masters in creative writing + 1 of the paper's author
Amabile, Hak&Bernts, Hennessey&Amabile
Rating • 1-7 likert scale
Technical dimension: "Consider the number of lines, the rhyme scheme, and
meter"
Quality dimension: "A holistic rating, consider the story, the coherence, the
creativity or interestingness, more generally: do you like it?
Calculation • Cronbach's alpha for the two dimensions (independent or combine?)
combined to single rating (0.87)
Agreement between raters (0.84)

17. Measures
Outcome
Measures
• overall quality
• technical quality: is the limerick the right number of lines, correct rhyme structure
initially split assessment into four criteria to capture both objective and subjective
aspects (technical, story coherence, creativity, and overall quality)
initial testing found all to be correlated and for the remainder of the experiment to
use the two clearest features
Raters • 3 raters
: 2 undergraduate or masters in creative writing + 1 of the paper's author
Amabile, Hak&Bernts, Hennessey&Amabile
Rating • 1-7 likert scale
Technical dimension: "Consider the number of lines, the rhyme scheme, and
meter"
Quality dimension: "A holistic rating, consider the story, the coherence, the
creativity or interestingness, more generally: do you like it?
문항들의 내적일관성에 기초하여 추정되는 신뢰도 지수의 하나이다. 각 문항을 하나의 테스트로 간주하고 반응자들이 문항들에 대해 얼마나 일관성 있게 반응하는가를 계산하게 된다
Calculation • Cronbach's alpha for the two dimensions (independent or combine?)
combined to single rating (0.87)
Agreement between raters (0.84)
Cronbach's alpha
일관성을 나타내는 계수
Cronbach's alpha will generally
increase as the intercorrelations
among test items increase, and is
thus known as an internal
consistency estimate of reliability of
test scores.

18. Results
• 716 participants
• 431 limericks
• 1 person: 79
2 person group: 185
3 person group: 167
Overall

19. Results
• the improvement in work quality that resulted from working in groups of two or
three, rather than working individually was reliably greater when the work was
organized sequentially

20. Analysis
additional workers improved production quality when sequential, but why?
• mediation analysis (territoriality and social loafing)
Territoriality
Mediation
analysis
• people feeling uncomfortable editing another person's work (e.g. wikipedia)
• non-physical but acknowledgment of the presence of co-worker
• avoid editing or deleting others' work due to territoriality
• could result in missed opportunities to improve, and fix error
• When sequential no fear of 'stepping on someone else's toes', because
previous worker has departed
• measured the amount of
deletion of someone else's work vs. deletion of my work
Social
Loafing
• less effort because they believe other group member will 'pick up the slack' or
they feel their ideas are dispensable
• When sequential, working alone -> less loafing
difference in quality?
• measured the amount of edits per person increase in group member
reduce effort from worker?

21. Experiment 2
Idea
Procedure
• Results from Experiment 1 indicate that additional sequential workers
outperformed additional simultaneous workers
• Mediation analysis suggests that a territoriality phenomenon (whereby
workers are reluctant to edit another’s work in the presence of that person
contributed to this difference in quality)
IMPROVING SIMULTANEOUS WORK
• Control: same to Experiment 1
• Roles: writer and editor role
Results • Results suggest that assigning writer and editor roles in simultaneous work is
able to mitigate process losses

22. Implication
how might we get the most benefits from multiple workers working on a complex,
interdependent task, where it is not clear what the answer is beforehand.
• prior work said simultaneous! but we say sequential!
• recent crowdsourcing work examined iterative work -> we are different because
we compared the two methods
• Simultaneous work was not able to capture the benefits of additional
workers, while iteration was able to do so, and removed communication and
coordination overheads
• territoriality partially accounts for the difference in quality between
simultaneous and iterative work
• This research appears at a time(2014) when the very notion of a team and
group research paradigm is under discussion by organizational
psychologists, as workers and collaboration are increasingly solely virtually
mediated

23. Discussion
• delete and edit을 보았는데... delete을 해야 edit을 하는거 아닌가
• distributed work를 보는데 있어서, 변수가 너무 많다.
• output의 퀄러티는 어떻게 측정할 것인가? (creative work에 대한 evaluation 가능성 takeaway)
• 영향을 주는 요인으로 무엇을 꼽을 것인가?
(domain에 따라? coworker들 사이의 관계에 따라?)
• sequential=/ iterative work (software development)=lean UX?

24. Discussion
• delete and edit을 보았는데... delete을 해야 edit을 하는거 아닌가
• distributed work를 보는데 있어서, 변수가 너무 많다.
• output의 퀄러티는 어떻게 측정할 것인가? (creative work에 대한 evaluation 가능성 takeaway)
• 영향을 주는 요인으로 무엇을 꼽을 것인가?
(domain에 따라? coworker들 사이의 관계에 따라?)
• sequential=/ iterative work (software development)=lean UX?

25. Thank You.