Your SlideShare is downloading. ×
'Living Lab' for HCI - presentation made at HCI International 2009
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

'Living Lab' for HCI - presentation made at HCI International 2009

3,754
views

Published on

HCI have long moved beyond the evaluation setting of a single user …

HCI have long moved beyond the evaluation setting of a single user
sitting in front of a single desktop computer, yet many of our fundamentally
held viewpoints about evaluation continues to be ruled by outdated biases
derived from this legacy. We need to engage with real users in 'Living
Laboratories', in which researchers either adopt or create functioning systems
that are used in real settings. These new experimental platforms will greatly
enable researchers to conduct evaluations that span many users, places, time,
location, and social factors in ways that are unimaginable before.

Published in: Technology, Business

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,754
On Slideshare
0
From Embeds
0
Number of Embeds
34
Actions
Shares
0
Downloads
44
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Ed
H.
Chi
 Area
Manager
and
Sr.
Research
Scientist
 Palo
Alto
Research
Center
 2009
HCI
International
Conference,
San
Diego,
CA

  • 2.   As
a
field,
early
fundamental
contributions
from:
 –  Computer
scientists
interested
in
changes
in
ways
we
 interact
with
information
systems
 –  Psychologists
interested
in
the
implications
of
these
 changes
   Combustible,
because:
 –  Computer
scientists
want
to
create
great
tools,
but
didn’t
 know
how
to
measure
impact
 –  Psychologists
want
to
go
beyond
classical
research
of
the
 brain
and
human
cognition
   The
need
to
establish
HCI
as
a
science
 –  Adopt
methods
from
psychology
 –  Good
Examples:
Fitts’
Law,
Models
of
Human
Memory,
 Cognitive
and
Behavioral
Modeling,
Information
Foraging
 –  Dual
purpose:
understand
nature
of
human
behavior
and
 build
up
a
science
of
HCI
techniques.
 7/24/09 HCIC "Living Lab" 2
  • 3.   Many
problems
don’t
fit
the
laboratory
experimental
methods
 anymore
 –  Beyond
a
user
in
front
of
computer;
Yet
evaluation
methods
mostly
 stayed
the
same
 –  Controlled
lab
study
as
the
gold
standard
for
acceptance
   Changes
and
Trends
in
Social
Computing
and
UbiComp
 Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with no time boundries Controllable experimental conditions Uncontrollable experimental conditions 7/24/09 HCIC "Living Lab" 3
  • 4. Artificial
experimental
setups
are
only
capable
of
telling
us
behaviors
in
 constrained
situations
   Hard
to
generalize
to
new
task
contexts
(with
interruptions,
 other
tasks,
other
goals,
unfocused
attention,
more
displays)
   Hard
to
generalize
to
other
tools,
apps
   Ecological
considerations
   Adoption
of
mobile
technology
   iPhones
in
Japan,
single‐handed
input
[PARC]
   Best
selling
phones
in
Indonesia
comes
with
a
compass
[Bell]
   Impossible
to
answer
questions
about
aggregate
behaviors
 of
groups
   Aggregate
behavior
of
Wikipedia
or
Delicious
users
 7/24/09 HCIC "Living Lab" 4
  • 5.   Conduct
research
on
real
platforms
and
services
 –  Not
to
replace
controlled
lab
studies
 –  Expand
our
arsenal
to
cover
new
situations
   Some
principles:
 –  Embedded
in
the
real
world
 –  Ecologically
valid
situations
 –  Embrace
the
complexity
 –  Rely
on
big‐data‐science
to
extract
patterns
   Not
first
to
suggest
this:
 –  S.
Carter,
J.
Mankoff,
S.
Klemmer
and
T.
Matthews.
Exiting
the
cleanroom:
On
 ecological
validity
and
ubiquitous
computing.
HCI
Journal,
2008
 –  EClass
[Abowd],
PlaceLab
[Intille],
Plasma
Poster
[Churchill
and
Nelson],
Digital
 Family
Portrait
[Rowan,
Mynatt]
 7/24/09 HCIC "Living Lab" 5
  • 6. GroupLens / MovieLens [Riedl, Konstan, Univ. Minnesota]
  • 7. Games with a Purpose [von Ahn et al]
  • 8. 7/24/09 HCIC "Living Lab" 8
  • 9. World of Warcraft [Yee, Ducheneaut et al]
  • 10. Wikipedia History Flow [Viégas et al]
  • 11. A B Bucket Testing or A/B Testing [Kohavi et al]
  • 12. UbiFit [Consolvo et al]
  • 13. 7/24/09 HCIC "Living Lab" 13
  • 14.   Master
degree
was
in
computational
molecular
biology
   Analogy:
Just
as
biologists
work
on
model
plants
and
 genomes
in
the
lab,
this
tells
us
just
how
it
behaves
in
an
 isolated
environment
under
controlled
conditions,
but
 not
how
the
plant
will
behave
in
the
real
world.
   Biologists
don’t
just
study
models
in
the
lab,
but
in
the
 wild
also.
 7/24/09 HCIC "Living Lab" 14
  • 15.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 15
  • 16.   Traditional
Approach;
Numerous
examples
   Favored
by
HCI
field
reviewers
   Typical
situation
is
the
study
of
some
interaction
technique
 –  Pen
input,
gestures,
perception
of
some
visualized
data,
reading
tasks,
 mobile
text
input
   Typical
measures
are
quantitative
in
nature
 –  performance
in
time,
performance
in
accuracy,
eyetracking,
learning
 measures,
user
preferences
   Issues:
 –  Not
always
ecologically
valid
 –  Hard
to
take
all
interactions
into
account
 –  Often
time‐consuming;
even
though
we
thought
we
could
do
it
fast.
 7/24/09 HCIC "Living Lab" 16
  • 17.   Harder
to
find
in
the
literature
   Often
comparing
against
an
older
system
as
baseline
   Typical
case
is
comparison
of
two
systems

 –  (one
website
with
another,
one
word
processor
vs.
another)
 –  Which
highlighting
feature
works
better
 –  Two
text
input
technique
on
a
cell
phone
   Typical
measures
are
similar
to
(1)
   Issues:
 –  Some
similar
issues
to
(1)
because
it’s
in
lab
 –  System
feature
not
in
control,
so
not
able
to
compare
fairly,
or
 isolate
the
feature
 7/24/09 HCIC "Living Lab" 17
  • 18.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 18
  • 19.   Real
applications
in
ecological
valid
situations
   Real
findings
can
be
applied
to
a
running
system
   Impact
of
research
is
more
immediate,
since
system
is
already
 running
   Typical
case
is
log
analytics
with
large
subject
pools
 –  log
studies
of
web
sites,
real
mobile
calling
usages,
web
search
logs,
 studies
of
Wikipedia
edits.
   Typical
measures
are
stickiness,
amount
of
activity,
clustering
 analysis,
correlational
analysis
   Issues:
 –  Factors
not
in
control,
findings
not
comparable
 –  Factors
cannot
be
isolated
 –  Reasons
for
failure
is
often
just
guesswork
 7/24/09 HCIC "Living Lab" 19
  • 20.   Hypothesis:
Conflict
is
what
drives
Wikipedia
forward.
   How
to
study
this?
 –  John
Tukey
paradigm
 –  Get
a
large
paper,
and
plot
all
of
the
data!
 –  Downloaded
all
of
Wikipedia
and
all
of
the
revisions
 –  Hadoop/MapReduce,
MySQL,
etc.
 7/24/09 HCIC "Living Lab" 20
  • 21. 100% 95% Maintenance 90% Percentage of total edits Other 85% 80% User Talk 75% User 70% Article Talk 65% Article 60% 2001 2002 2003 2004 2005 2006 7/24/09 HCIC "Living Lab" 21
  • 22. Group D Group A Group B Group C Number of users in user group A B C Total Users with Korean point of view 10 6 0 16 Users with Japanese point of view 1 8 7 16 7/24/09 Neutral or Unidentified HCIC "Living Lab" 7 3 6 2217
  • 23. Anonymous (vandals/ spammers) Sympathetic to husband Mediators Sympathetic to parents 7/24/09 HCIC "Living Lab" 23
  • 24. 7/24/09 HCIC "Living Lab" 24
  • 25. 7/24/09 HCIC "Living Lab" 25
  • 26. 7/24/09 HCIC "Living Lab" 26
  • 27. 7/24/09 HCIC "Living Lab" 27
  • 28.   Hypothesis:
Social
Tagging
doesn’t
scale
over
time.
   How
to
study
this?
 –  Crawl
as
much
tagging
data
as
we
can.
 –  Study
the
noise
in
the
system.
 –  40
machines
for
3
months
 7/24/09 HCIC "Living Lab" 28
  • 29. Concepts
 Topics
 Users
 Documents
 Noise
 Tags
 Decoding
 Encoding
 T1…Tn
 7/24/09 HCIC "Living Lab" 29
  • 30. Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz) 7/24/09 HCIC "Living Lab" 30
  • 31. 7/24/09 HCIC "Living Lab" 31
  • 32. Semantic Similarity Graph Web Tools Reference Guide Howto Tutorial Tips Help Tip Tutorials Tricks 7/24/09 HCIC "Living Lab" 32
  • 33. 7/24/09 HCIC "Living Lab" 33
  • 34.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 34
  • 35.   Similar
to
(3),
practical
for
running
systems;
ecologically
 valid,
impact
can
be
immediate.
 –  Good
for
cases
in
which
economics
makes
sense
[Google]
 –  Changes
to
system
is
possible;
Factors
can
be
controlled.
   Typical
case
might
be
A/B
testing,
large
subject
pools
   Typical
measures
are
being
developed
 –  Impact
measures
.

Large
visit
#
and
interest
(measured
by
blog
 posts?)

New
Business
inquiries?
 –  Usability
measures
vs.
Usefulness
measures
   Issues:
 –  Effort
and
resource
requirement
is
dropping
but
still
significant
 –  Hard
for
a
research
lab
to
take
on
 7/24/09 HCIC "Living Lab" 35
  • 36. HCIC "Living Lab" 7/24/09 36
  • 37. HCIC "Living 7/24/09 37 Lab"
  • 38. HCIC "Living 7/24/09 38 Lab"
  • 39. Personal Computing [Xerox PARC]
  • 40.   Evaluation
methods
are
in‐separable
from
the
kinds
of
 science
and
models
that
can
be
build
in
a
field.

   Platform
advances
enable
real
technology
insertion
into
 real
world
situations
cheaper
and
more
manageable.
 Characteriza7on
 Models
 Evalua7ons
 Prototypes
 7/24/09 HCIC "Living Lab" 43
  • 41.   Research
Vision:
Understand
how
social
computing
systems
can
 enhance
the
ability
of
a
group
of
people
to
remember,
think,
and
 reason.
   Living
Laboratory:
Create
applications
that
harness
collective
 intelligence
to
improve
knowledge
capture,
transfer,
and
discovery.
 http://asc‐parc.blogspot.com
 http://www.edchi.net
 echi@parc.com
 WikiDashboard
 MrTaggy
 SparTag.us


×