This document describes a sociotechnical systems study of virtual research and development organizations. It discusses a session that will present case studies of different virtual R&D projects, including a large video game company. The central research question is how virtual communication influences the quality of deliberations at various stages of the R&D process. The session will include panel presentations on specific virtual R&D projects, summarizing across cases, and a discussant and Q&A.

1. A
Sociotechnical
Systems
Study
of
Virtual
R&D
Organiza;ons
Co-Chairs
Ronald E. Purser & Ramkrishnan V. Tenkasi
Panelists:
Doug Austrom
Betty Barrett
Bert Painter
Pamela Posey
Discussant:
William A. Pasmore
Sponsors: ODC, OCIS, TIM
Supported by NSF-VOSS award #0943237

2. Central
Research
Ques;on
How
do
virtual
modes
of
communica3on
influence
the
quality
of
delibera3ons
at
various
stages
of
the
R&D
process?
-­‐ STS
theore3cal
lens
-­‐ Compara3ve
case
study
-­‐ Grounded
theory
building

3. Session
Overview
• Introduc3on
and
Framing:
1:15-­‐1:30
Panel
Presenta3ons
• Large
Video
Game
Inc
1:30-­‐1:40
• Alzheimer’s
Disease
center
1:40-­‐1:50
• CALTECH
Micro
&
Nano-­‐photonics
1:50-­‐2:00
• Summarizing
across
the
cases
2:00-­‐2:15
• Discussant
comments
2:15-­‐2:30
• Q&
A
2:30-­‐2:45

4. Central
Research
Ques;on
How
do
virtual
modes
of
communica3on
influence
the
quality
of
delibera3ons
at
various
stages
of
the
R&D
process?
-­‐ STS
framework
-­‐ Delibera3ons
-­‐ Stages
of
the
R&D

5. Evolu3on
of
Socio-­‐Technical
Systems
STS
v1.0
Rou3ne
work
in
a
single
organiza3on
–
e.g.,
coal
mines,
factories,
oil
refineries
• Work
groups
with
pooled
iden3ty
• Unitary
conversion
process
• Linear
conversion
sequence
STS
v2.0
Non-­‐rou3ne
knowledge
work
in
single
organiza3ons
–
e.g.,
white
collar
office
work,
professional
services
firm,
NPD
and
R&D
• Individual
performers,
specialized
exper3se
• Mul3ple,
concurrent
conversion
processes
• Nonlinear
conversion
flow
STS
v3.0
Virtual,
non-­‐rou3ne
work
–
e.g.,
R&D
consor3a,
complex
supply
chains
• Individual
performers
and
work
groups
distributed
across
mul3ple
loca3ons
and/or
organiza3ons
• ICT
enabled
• Mul3ple,
concurrent
independent
and
interdependent
conversion
processes
• Nonlinear
conversion
flows
*

6. Delibera;ons:
Defini;on
and
Elements
Pava,
1983
• Delibera;ons
are
sense
making
exchanges
in
which
people
engage
with
themselves
or
others
to
reduce
the
uncertainty
of
a
problema9c
issue.
-­‐-­‐i.e.
an
issue
subject
to
two
or
more
interpreta9ons
• The
salient
elements
of
a
delibera3on
include
the
…
• Topics
or
problema3c
issues
facing
the
social
en3ty
about
which
people
reflect
and
communicate
• Forums
in
which
they
occur
which
may
structured,
semi-­‐structured,
or
unstructured
or
ad
hoc
• Par;cipants
both
those
who
are
currently
involved
and
those
who
ideally
should
be
involved
in
the
delibera3on.

7. A
Six
Stage
Con;nuum
of
the
R&D
Process

8. Sampling
of
Virtual
R&D
Projects
across
Sites
and
R&D
Stages
Type of Research Development Program/Project
Research Site R1
Pure
R2
Applied
D1
Exploratory
D2
Advanced
D3
Start-up
D4
Scale-
up
Large Video Game
Inc.
X X
NIAAlzheimer’s
Centers
X X X
Caltech Micro &
Nano-Photonics
Research Group
X X

9. Large
Video
Game,
Inc.
Case
Study
Major
support
and
funding
for
this
project
has
been
provided
by
the
Na;onal
Science
Founda;on.
NSF
Award
#0943237

10. LVG
Inc.
Core
Team
Florida
QA
Tes;ng
Southern
USA
Engineering
and
Web
Development
USA
LVG
Inc.
Canada
Artwork
(Philiphines,
India,
China)
Virtual
Organiza;on
for
LVG
Inc.
Game
2012
Product
Development
Systems
Engineering
China
NSF
Award
#0943237

11. Systems
View
of
Game
Development
NSF
Award
#0943237

12. Core
Game
Team
NSF
Award
#0943237
QA
Art
EngineeringProduction
Design

13. Org
Structure
NSF
Award
#0943237
Producer & Director
Art Director Lead DesignerTechnical Director
Engine/System Coders
Gameplay Coders
Network Coders
Hardware Specialists
Audio Specialists
Rendering Specialists
Gameplay Designers
SP Level Designers
Scripting Specialists
Cinematic Specialists
MP Level Designers
Concept Artists
Character Artists
Environment Artists
3D Modelers
Texture Artists
Technical Artists
Animators
Audio Director
Assistant Producer Writers
Sound Engineers
Music Composer
Recording Studio
Voice Actors
Localization Studio

14. AGILE
-­‐
SCRUM
• Rapid
prototyping
process
+
mul3ple
itera3ons
to
feature
comple3on
• Episodic
surveys
of
“POD”
members,
assessing
their
virtual
interac3ons
with
vendors
aler
every
SPRINT
cycle
(3
week
cycles,
13
total
SPRINT
cycles,
we
entered
at
SPRINT
6).
• AGILE
has
significant
implica3ons
for
virtual
organiza3ons
insofar
as
expecta3ons
may
change
during
the
life
of
a
contract,
during
various
“Sprints”
NSF
Award
#0943237

15. SCRUM BOARD
Feature
to
develop
Tasks
in
queue
Tasks
in
progress
Tasks
complete
Feature
complete
NSF
Award
#0943237

16. déjà
vu
• Souder’s
NSF
research
–
“Task-­‐dominant
approach
to
R&D”
(Souder,
1987)
• “Rugby
vs.
Relay-­‐race”
project
management
• How
and
to
what
extent
do
various
vendors
synchronize
and
entrain
with
itera3ve
nature
of
AGILE/SCRUM
at
the
host
organiza3on?
NSF
Award
#0943237

17. NSF
Award
#0943237

18. Data
Collec3on
to
Date
• Ini3al
Scoping
Phone
Interviews
with
Various
General
and
Project
Managers
(Spring/Summer
2010)
• On-­‐site
interviews
(14)
with
“Franchise”
team
(Sept.
2010)
– (General
Manager,
Project
&
Program
Managers,
Ar3sts,
Solware
Engineers,
Crea3ve
Director,
Audio
Engineers,
Computer
Graphics,
Outsourcing
Managers,
Quality
Assurance)
• Milestone
telephone
interviews
with
Project
Managers,
Ar3sts,
Engineers
and
Quality
Assurance
(Jan.
2011)
• Episodic
electronic
surveys
at
the
end
of
each
“Sprint”
cycle,
and
at
the
end
of
“Alpha”
(@
3
weeks,
10/10
–
05/11)
• Electronic
surveys
of
key
vendors
(May-­‐June
2011)
• Follow-­‐up
phone
interviews
with
Project
Managers,
Ar3sts,
Engineers
and
Quality
Assurance
(Aug.
2011)
NSF
Award
#0943237

19. Uniqueness
of
Producing
Art
in
Virtual
Organiza;ons
• Art
studios
produce
art
assets,
not
games
– Large
knowledge
gap
between
making
art,
and
making
art
that
works
for
games
– Oversea
vendors
–
gap
larger
• How
is
tacit
knowledge
obtained
in
virtual
exchanges?
– Some
vendors
more
sophis3cated
than
others
• Ar3s3c
and
aesthe3c
judgment
is
part
of
the
socio-­‐
technical
system
• Rich,
mul3-­‐media
documenta3on
is
key
NSF
Award
#0943237

20. Socio-­‐Technical
Systems
of
Virtual
Art
Produc;on:
Emergent
Themes
– Rou;ne
vs.
Nonrou;ne
Work.
Art
assets
vary
in
terms
of
“Likeness.”
The
more
authen3c
and
life-­‐like
the
art
asset,
the
less
rou3ne,
but
this
actuality
reduces
“ar3s3c
license.”
– Unit
Opera;ons:
How
to
assign
and
parcel
out
pieces
of
work?
Too
litle
turns
the
vendor
into
a
“monkey,”
too
large
is
overwhelming.
– Expecta;on
se`ng
is
key
to
joint
op3miza3on
– Variances
take
the
form
of
revisions/
reitera3ons
and
delays
in
delivery
– Managing
front-­‐end
delibera;ons
is
key
to
seung
flexible
and
realis3c
contracts
NSF
Award
#0943237

21. Scoping
Vendors
Vendor
Selection
Defining &
Estimating
Project Work
Pre-test &
Trial Run
Key
Delibera;ons
in
Virtual
Art
Produc;on
Detailed
Documentation
& Requirements
Art Assets
Deliveries
Initial Testing
of Assets
Debugging Re-negotiating
Project Work
Contract
Negotiation
Critiques
Critiques
NSF
Award
#0943237

22. Reducing
Equivocality
in
Art
Delibera;ons
• Contrary
to
tradi3onal
studies
on
virtual
teams
that
strive
for
3ght
coupling,
ar3s3c
produc3on
relies
more
on
reducing
equivocality
of
front-­‐end
delibera3ons
by
providing
well
defined
documenta3on,
visual
targets,
and
reference
materials
– “I
write
my
emails
out
properly
(addressing
the
person,
wri3ng
organized
paragraphs,
and
then
using
closing
remarks
with
a
signature).
I
started
no3cing
that
a
lot
of
that
properness
started
to
fade
as
the
sprint
carried
on.
We
are
star3ng
to
make
the
transi3on
into
alpha
and
the
workload
has
been
plenty,
so
I
thought
those
emails
were
a
great
reflec3on
of
it.
(just
something
funny
I
no3ced)”
2/23/11
• Key
delibera3ons
in
virtual
art
produc3on
involve
gaining
shared
agreement
on
“standardiza3on
of
outputs”
(Mintzberg,
1993;
Ramesh
&
Dennis,
2002)
• The
use
of
both
“seman3cally
and
aesthe3cally
rich
media”
(e-­‐
mails,
documenta3on,
visual
targets,
screen
shots)
NSF
Award
#0943237

23. Virtuality
Impacts
the
Nego;a;on
of
Art-­‐Based
Project
Work
• Ambiguity
in
defining
3me
per
asset
• Pricing
es3mates
are
olen
inaccurate
• Cost
effec3veness
is
hard
to
predict
– “Cost
nego3a3on.
There
was
a
task
that
required
modifying
heads.
It
took
us
30min
to
1hr
to
do
one
head
in-­‐house,
so
I
tasked
them
out
as
4hr
jobs.
The
outsourcers
wanted
anywhere
from
6-­‐16hrs
to
do,
and
one
agreed
to
setle
at
4hrs
per
head
aler
doing
20
of
them.
Some3mes
you
win
with
outsourcing,
some3mes
it
makes
no
sense
to
outsource.”
NSF
Award
#0943237

24. Management
of
Risk
Virtual
Engineering
• A
Key
Delibera3on
involves
managing
risk
in
providing
access
to
source
code
packages
– Key
Variances:
Delays
in
obtaining
approvals,
incomplete
documenta3on,
changing
expecta3ons/func3onality,
mis-­‐
matches
in
preferred
modes
of
communica3ons
• Our
biggest
challenge
with
Vendor
E
has
been
geung
them
set
up
with
VPN
and
the
appropriate
permissions.
There
is
a
lot
of
red
tape
that
we
are
having
to
go
through
wi3n
our
company
to
get
them
properly
set
up.
It
ul3mately
took
up
to
5
weeks
to
get
them
fully
ready
to
start
work
for
us.
Thus,
the
project
is
behind
schedule
and
we
are
looking
at
the
project
coming
in
3
weeks
late
from
the
original
final
date.
…we
have
had
a
ton
of
email
and
conference
alls
trying
to
figure
out
the
next
steps
in
geung
Egeniq
set
up.
NSF
Award
#0943237

25. Emerging
Theory
and
Prac;ce
Ques;ons
• Can
we
observe
and
theorize
how
to
design
“high
performance”
virtual
organiza3ons?
• Can
non-­‐rou3ne
STS
principles
be
applied
to
virtual
art
produc3on?
• Can
theories
of
organiza3onal
aesthe3cs
help
to
understand
arts-­‐based
virtual
delibera3ons?
[Aesthe3cs
=
sensory
knowledge
and
felt
meaning
of
objects
and
experiences.]
• How
does
aesthe3cs
factor
into
virtual
art
delibera3ons
to
create
value
and
performance?
NSF
Award
#0943237

26. These
Slides
are
Op3onal
NSF
Award
#0943237

27. NSF
Award
#0943237

28. NSF
Award
#0943237

29. NSF
Award
#0943237

30. LVG
Core
Team
NSF
Award
#0943237

31. Vendors:
Factors
Impeding
Virtual
Communica3ons
NSF
Award
#0943237

32. NSF
Award
#0943237

33. Research
Site
and
Project
• A
virtual
R&D
eco-­‐system
comprised
of
29
NIA-­‐funded
Alzheimers
Disease
Centers
(ADCs)
and
the
Na3onal
Alzheimers
Coordina3ng
Center
Center
(NACC)
• Uniform
Data
Set

34. Key
Peformance
Outcomes
of
of
this
Alzheimers
Research
Eco-­‐System
Number
of
enrollees*
23,577
subjects
Both
demented
and
non-­‐demented
Number
of
variables
725
variables
using
up
to
18
standardized
forms
at
each
visit
Longitudinal
data
is
collected
by
clinicians,
neuropsychologists,
and
other
ADC
research
personnel
each
visit
-­‐-­‐
ini3al,
annual
follow-­‐up
and
milestones
such
as
death
or
dropout
Number
of
research
projects**
157
research
projects
have
been
approved
to
use
the
NACC
data
• 104
by
researchers
from
the
ADCs
• 27
by
NACC
staff
• 26
by
external
researchers
Number
of
publica;ons
using
NACC
data***
Since
2001
367
publica3ons
• 180
NACC-­‐funded
projects
• 53
by
NACC
staff
• 53
NACC-­‐ADC
collabora3ve
studies
• 72
by
non-­‐NACC
researchers
• 9
from
Indirect
NACC
support
*
As
of
June
2011
**
As
of
August
2011
***
As
of
January
2011

35. Alzheimers
Research
Eco-­‐System
and
a
“Culture
of
Collabora3on”
• Culture
of
collabora3on
among
Alzheimers
researchers
has
extended
and
has
led
to
other
major
collabora3ve
research
ini3a3ves
• Alzheimers
Disease
Neuro-­‐Imaging
Ini3a3ve
(ADNI)
• Genome-­‐Wide
Associa3on
Study
(GWAS)
• Alzheimers
Disease
Coopera3ve
Study
(ADCS)
• Dominantly
Inherited
Alzheimer
Network
(DIAN)
• Interna3onal
collabora3ve
ini3a3ve
with
Worldwide
ADNI

36. Research
Process
to
Date
• Conducted
14
interviews
with
a
representa3ve
cross-­‐sec3on
of
key
par3cipants
in
the
UDS
project
including
ADC
Center
Directors,
Na3onal
Ins3tute
of
Aging
(NIA)
ADC
Program
Director,
NACC
Director,
Clinical
Core
Directors,
and
Data
Core
Directors
• Determined
that
the
UDS
project
is
D1
in
the
R&D
con3nuum,
Exploratory
Development
work
–
“They
know
what;
but
they
didn’t
know
how
to
achieve
it.”
• Iden3fied
key
delibera3ons
in
the
ini3al
development,
ongoing
data
collec3on/submission/quality
control,
and
revision
of
the
Uniform
Data
Set
including
…
• Iden3fying
and
choosing
the
“best
fit”
data
collec3on
methods,
diagnos3c
protocols
and
instruments,
variables
for
the
UDS
• Achieving
agreement
on
the
UDS
across
the
29
ADCs
• Ensuring
adherence
with
UDS
protocols,
data
quality,
and
3meliness
of
data
submission
at
all
29
centers
• Revising
assessment
instruments
and
protocols
in
the
UDS
and
achieving
agreement
on
the
new
instruments

37. Factors
that
Contributed
to
Delibera3on
Effec3veness
• Delibera3on
processes
itera3vely
evolved
within
the
informal
network
and
the
formal
structure,
the
NACC
• Face-­‐to-­‐face
forums
twice
a
year
have
enabled
the
ICT-­‐
enabled
(informa3on
and
communica3on
technologies)
forums
and
delibera3ons
to
be
more
effec3ve
and
efficient
• Relied
on
high
levels
of
input
and
par3cipa3on
–
directly
on
commitees
and
sub-­‐commitees
and
via
surveys,
bulle3n
boards,
direct
feedback
to
Commitee
Chair
and
Members
–
to
build
agreement
on
the
UDS
• Rou3ne
delibera3ons
(3mely
data
submission
and
data
quality
are
predominantly
ICT-­‐enabled

38. Contextual
Factors
that
Contributed
to
Delibera3on
Effec3veness
• Highly
effec3ve
and
highly
collabora3ve
virtual
socio-­‐technical
eco-­‐sytem
has
evolved
itera3vely
in
large
part
because
…
• Leaders
in
the
Alzheimers
research
community
(from
the
ADCs
and
NIA)
with
considerable
source
credibility
(trusted
and
respected,
necessary
exper3se)
who
were
commited
to
the
success
of
the
network
and
a
transcendent
purpose
–
i.e.,
addressing
Alzheimers
disease
• NACC
has
provided
the
infrastructure
for
effec3ve
and
efficient
delibera3ons
within
this
virtual
STS
and
serves
the
func3ons
of
a
referent
organiza3on
(Trist,
1983)
–
coordina3on,
apprecia3on
of
emergent
issues,
and
infrastructure
support
• Dense
network
of
rela3onships
among
par3cipants
in
the
network
has
also
strengthened
collabora3on

39. CALTECH
(California
Ins;tute
of
Technology)
&
D.A.R.P.A.
(Defense
Advanced
Research
Projects
Agency)
‘ORCHID’
PROGRAM
of
Fundamental
Research
-­‐
Op;cal
Radia;on
Cooling
&
Hea;ng
in
Integrated
Devices

40. Theore3cal
&
Experimental
“Opto-­‐Mechanics”
Research
within
a
Virtual
Organiza3on
-­‐2
project
phases,
June
2010-­‐12
&
2012-­‐14:
VOSS
study
of
Phase
One-­‐-­‐Exploratory
-­‐Tightly-­‐Linked
Collabora;on
for
Design
of
Experiments
&
Device
Fabrica;on
among
Laboratories
using
3
Technology
plajorms
in
USA
&
Europe
-­‐ At-­‐distance
Theore;cal
scien;fic
Study
&
Input
to
work
of
the
Experimental
groups
-­‐ Major
CHALLENGE:
Crea3ve
Research
Design
&
Knowledge
Genera3on
in
a
complex
Virtual
se`ng
-­‐2
project
phases,
June
2010-­‐12
&
2012-­‐14:
VOSS
study
of
Phase
One-­‐-­‐Exploratory
-­‐Tightly-­‐Linked
Collabora;on
for
Design
of
Experiments
&
Device
Fabrica;on
among
Laboratories
using
3
Technology
plajorms
in
USA
&
Europe
-­‐ At-­‐distance
Theore;cal
scien;fic
Study
&
Input
to
work
of
the
Experimental
groups
-­‐ Major
CHALLENGE:
Crea3ve
Research
Design
&
Knowledge
Genera3on
in
a
complex
Virtual
se`ng

41. Orchid
Project
Network
Core
Experimental
Groups
at
CALTECH
Physics
&
Applied
Physics
Depts.
Quantum
Op;cs
Group
Physics
-­‐
Caltech
Micro
&
Nano-­‐Photonics
Research
Group
Applied
Physics
-­‐
Caltech
Meso-­‐Op;cs
Research
Group
Applied
Physics
-­‐
Caltech

42. Quantum
Op;cs
Group
Physics
-­‐
Caltech
Micro
&
Nano-­‐Photonics
Research
Group
Applied
Physics
-­‐
Caltech
Meso-­‐Op;cs
Research
Group
Applied
Physics
-­‐
Caltech
Photonics
&
Quantum
Measurement
Laboratory
Federal
Inst.of
Technology
Lausanne,
Switzerland
Quantum
Op;cs
&
Nanophysics
Group
Univ.
of
Vienna
-­‐
Austria
Orchid
Project
Network
Data-­‐Sharing
&
Collabora3on
between
Caltech
&
EUROPEAN
Laboratories

43. Quantum
Op;cs
Group
Physics
-­‐
Caltech
Micro
&
Nano-­‐Photonics
Research
Group
Applied
Physics
-­‐
Caltech
Meso-­‐Op;cs
Research
Group
Applied
Physics
-­‐
Caltech
Photonics
&
Quantum
Measurement
Laboratory
Federal
Inst.of
Technology
Lausanne,
Switzerland
Quantum
Op;cs
&
Nanophysics
Group
Univ.
of
Vienna
-­‐
Austria
Orchid
Project
Network
THEORETICAL
Physicists
Study
&
Input
to
the
work
of
the
‘Experimentalists’
Theore;cal
Physics
-­‐
McGill
Univ.
Canada
Op;cal
Sciences
&
Physics
-­‐
Univ.Arizona

44. Quantum
Op;cs
Group
Physics
-­‐
Caltech
Micro
&
Nano-­‐Photonics
Research
Group
Applied
Physics
-­‐
Caltech
Meso-­‐Op;cs
Research
Group
Applied
Physics
-­‐
Caltech
Photonics
&
Quantum
Measurement
Laboratory
Federal
Inst.of
Technology
Lausanne,
Switzerland
Quantum
Op;cs
&
Nanophysics
Group
Univ.
of
Vienna
-­‐
Austria
Theore;cal
Physics
-­‐
McGill
Univ.
Canada
Op;cal
Sciences
&
Physics
-­‐
Univ.Arizona
Orchid
Project
Network
European
Graduate
Students
“Embedded”in
Caltech
Laboratories

45. VOSS
-­‐
Preliminary
Finding
-­‐
Orchid
Project
-­‐ ‘Embedded
Researcher(s)’
provide
temporary
F2F
‘presence’
+
‘transla;on’
assis;ng
at-­‐distance
collaborators
to
resolve
high
ambiguity
delibera;ons
-­‐NOW,
’Habit’
and
sufficient
Shared
Understandings
are
established
for
Skype-­‐based
collabora;on
across
groups
without
‘embedded’
researcher(s)
-­‐Another
coincidental
forum
for
F2F
delibera;ons
-­‐
frequent
Interna;onal
Conferences
(>5
per
year)
-­‐Result:
Dynamic
Complementary
F2F
+
Technology-­‐mediated
Interac3ons
in
‘hybrid’
Virtual
SeHng
(Note:
Dixon,
K.
&
Panteli,
N.
(2010).
From
Virtual
Teams
to
Virtuality
in
Teams,
Human
Rela3ons,
63,
1177-­‐1197.)
-­‐ ‘Embedded
Researcher(s)’
provide
temporary
F2F
‘presence’
+
‘transla;on’
assis;ng
at-­‐
distance
collaborators
to
resolve
high
ambiguity
delibera;ons
-­‐NOW,
’Habit’
and
sufficient
Shared
Understandings
are
established
for
Skype-­‐based
collabora;on
across
groups
without
‘embedded’
researcher(s)
-­‐Another
coincidental
forum
for
F2F
delibera;ons
-­‐
frequent
Interna;onal
Conferences
(>5
per
year)
-­‐Result:
Dynamic
Complementary
F2F
+
Technology-­‐mediated
Interac3ons
in
‘hybrid’
Virtual
SeHng
(Note:
Dixon,
K.
&
Panteli,
N.
(2010).
From
Virtual
Teams
to
Virtuality
in
Teams,
Human
Rela3ons,
63,
1177-­‐1197.)