SlideShare a Scribd company logo
1 of 18
Download to read offline
Ed
H.
Chi

Area
Manager
and
Sr.
Research
Scientist

Palo
Alto
Research
Center


2009
HCIC
Workshop
at
YMCA
Camp,
Colorado

As
a
field,
early
fundamental
contributions
from:

 

     –  Computer
scientists
interested
in
changes
in
ways
we
interact
        
with
information
systems

     –  Psychologists
interested
in
the
implications
of
these
changes

     Combustible,
because:

 

     –  Computer
scientists
want
to
create
great
tools,
but
didn’t
know
        
how
to
measure
impact

     –  Psychologists
want
to
go
beyond
classical
research
of
the
brain
        
and
human
cognition

     Example:


 

     –  an
enduring
interest
in
‘augmenting
human
intellect’:
V.
Bush,
        
Licklider,
Engelbart,
in
turn
inspiring
Stu
Card,
Alan
Newell,
        
Alan
Kay,
and
many
others.


2/13/09                        HCIC quot;Living Labquot;                          2
The
need
to
establish
HCI
as
a
science

 

     –  Adopt
methods
from
psychology

     –  Convenient
and
fits
well
with
problems
at
hand

            Issues
around
personal
computing
(WIMP)


     –  Dual
purpose:
understand
nature
of
human

        behavior
and
build
up
a
science
of
HCI

        techniques.

     –  Good
Examples:
Fitts’
Law,
Models
of
Human

        Memory,
Cognitive
and
Behavioral
Modeling,

        Information
Foraging


     –  Stuart
K.
Card,
William
K.
English,
and
Betty
J.
Burr
(1978).

        Evaluation
of
mouse,
rate‐controlled
isometric
joystick,

        step
keys,
and
text
keys
for
text
selection
on
a

        CRT.
Ergonomics,
21(8):601–613,
1978.

2/13/09                               HCIC quot;Living Labquot;                  3
Beyond
a
user
in
front
of
computer

 

      –  Yet
evaluation
methods
mostly
stayed
the
same

      –  Perceived
CHI
paper
template
for
acceptance

     Many
problems
don’t
fit
the
laboratory
experimental

 

     methods
anymore

      –  Yesterday’s
discussion
about
Large
Data
and
HCI
was
largely
driven
by

         how
HCI
evaluation
methods
need
to
change
to
fit
the
wild

     The
best
examples:

Trends
in
Social
Computing
and
UbiComp
force

 

     us
to
think
about
new
context
of
use





 2/13/09                          HCIC quot;Living Labquot;                               4
Old Assumptions                          New Considerations
Single display                           Multiple displays
Knowledge work                           Games, communication, social apps
Isolated worker                          Collaborative and social groups
Stationary location                      Mobile and stationary
Short task durations                     Short and long tasks, and tasks with
                                         no time boundries
Controllable experimental conditions     Uncontrollable experimental conditions




   2/13/09                        HCIC quot;Living Labquot;                             5
Think
about
research
that
has
been
done
on
UI
 

     
animations
or
flashing
icons.

     From
visual
perception,
we
know
motion
in
the
 
     
periphery
is
more
noticeable
than
in
the
foveal
region
     
[DaVinci].





2/13/09                    HCIC quot;Living Labquot;                   6
Evaluations
surrounding
many
HCI
systems
for
 

     
knowledge
work
focus
on
productivity
increase,
but
     
what
about
factors
for
adoption?

     –  Argument:
if
no
productivity
increase,
then
adoption
is
        
irrelevant

     –  But
the
opposite
argument
is
just
as
right:
if
no
adoption,
no
        
amount
of
productivity
increase
shown
is
relevant!

     Academic
research
often
focus
on
productivity
 

     
improvements,
increasing
the
perceived
gulf
between
     
the
ivory
tower
and
the
trenches

     An
Example:
Color
copier
studies

 




2/13/09                        HCIC quot;Living Labquot;                         7
Artificial
experimental
setups
are
only
capable
of
telling
us

 

     behaviors
in
constrained
situations

      –  Ecological
considerations

      –  Hard
to
generalize
to
new
task
contexts
(with
interruptions,
other

         tasks,
other
goals,
unfocused
attention,
more
displays)

      –  Hard
to
generalize
to
other
tools,
apps

      –  Impossible
to
answer
questions
about
aggregate
behaviors
of
groups

     Example
problems:


 

      –  Adoption
of
mobile
technology

            iPhones
in
Japan,
single‐handed
input
[PARC]


            Best
selling
phones
in
Indonesia
comes
with
a
compass
[Bell]


      –  Aggregate
behavior
of
Wikipedia
or
Delicious
users

            Big
data
analysis
of
edit
logs





 2/13/09                          HCIC quot;Living Labquot;                            8
Was
a
computational
molecular
biologist

 

     Analogy:
Just
as
biologists
work
on
model
plants
and
 
     
genomes
in
the
lab,
this
tells
us
just
how
it
behaves
in
     
an
isolated
environment
under
controlled
conditions,
     
but
not
how
the
plant
will
behave
in
the
real
world.

     Biologists
don’t
just
study
models
in
the
lab,
but
in
the
 
     
wild
also.





2/13/09                     HCIC quot;Living Labquot;                    9
Observational
Studies

 

     –    Ethnography

     –    Social
Technical
Design

     –    Iterative
Design

     –    Diary
Studies

     –    Longitudinal
Studies
with
single
outcome

     Problems:

 

     –  Sampling
from
non‐normal
distributions

     –  Treat
variations
in
social
contexts
as
‘noise’

     –  What
about
adoption?

     Mixed
Methods
is
now
popular
to
partially
ameliorate

 

     –  Convergent
measures
tells
us
we’re
getting
closer
to
the
truth


2/13/09                         HCIC quot;Living Labquot;                         10
–  similar
to
Venture
Business
mentioned
by
David
Millen

     Conduct
research
on
real
platforms
and
services

 

     –  Not
to
replace
controlled
lab
studies

     –  Expand
our
arsenal
to
cover
new
situations

     Some
principles:

 

     –    Embedded
in
the
real
world

     –    Ecologically
valid
situations

     –    Embrace
the
complexity

     –    Rely
on
big‐data‐science
to
extract
patterns

     Not
first
to
suggest
this:

 
     –  S.
Carter,
J.
Mankoff,
S.
Klemmer
and
T.
Matthews.
Exiting
the
cleanroom:
On

        ecological
validity
and
ubiquitous
computing.
HCI
Journal,
2008

     –  EClass
[Abowd],
PlaceLab
[Intille],
Plasma
Poster
[Churchill
and
Nelson],
Digital

        Family
Portrait
[Rowan,
Mynatt]


2/13/09                              HCIC quot;Living Labquot;                                   11
Two
dimensions

 

     –  1.
Whether
the
system
is
under
the
control
of
the
researcher

     –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild



                    System Control             System Not in
                                               Control

       Laboratory   (1) Build a system,        (2) Adopt a system,
                    study in the Lab           study in the Lab

       Wild (Real   (4) Build a system,        (3) Adopt a system,
       World)       release it, study in       study in the Wild
                    the Wild



2/13/09                        HCIC quot;Living Labquot;                        12
Traditional
Approach;
Numerous
examples

 

     Favored
by
CHI
reviewers

 

     Typical
situation
is
the
study
of
some
interaction
technique

 

      –  Pen
input,
gestures,
perception
of
some
visualized
data,
reading
tasks,

         mobile
text
input

     Typical
measures
are
quantitative
in
nature

 

      –  performance
in
time,
performance
in
accuracy,
eyetracking,
learning

         measures,
user
preferences

     Issues:

 

      –  Not
always
ecologically
valid

      –  Hard
to
take
all
interactions
into
account

      –  Often
time‐consuming;
even
though
we
thought
we
could
do
it
fast.





 2/13/09                          HCIC quot;Living Labquot;                             13
Harder
to
find
in
the
literature

 

     Often
comparing
against
an
older
system
as
baseline

 

     Typical
case
is
comparison
of
two
systems


 

     –  (one
website
with
another,
one
word
processor
vs.
another)

     –  Which
highlighting
feature
works
better

     –  Two
text
input
technique
on
a
cell
phone

     Typical
measures
are
similar
to
(1)

 

     Issues:

 

     –  Some
similar
issues
to
(1)
because
it’s
in
lab

     –  System
feature
not
in
control,
so
not
able
to
compare
fairly,
or

        isolate
the
feature



2/13/09                        HCIC quot;Living Labquot;                            14
http://www.flickr.com/photos/
                              crush_images/3245220458/
2/13/09   HCIC quot;Living Labquot;                             15
Real
applications
in
ecological
valid
situations

 

     Real
findings
can
be
applied
to
a
running
system

 

     Impact
of
research
is
more
immediate,
since
system
is
already

 

     running

     Typical
case
is
log
analytics
with
large
subject
pools

 

      –  log
studies
of
web
sites,
real
mobile
calling
usages,
web
search
logs,

         studies
of
Wikipedia
edits.

     Typical
measures
are
stickiness,
amount
of
activity,
clustering

 

     analysis,
correlational
analysis

     Issues:

 

      –  Factors
not
in
control,
findings
not
comparable

      –  Factors
cannot
be
isolated

      –  Reasons
for
failure
is
often
just
guesswork



 2/13/09                           HCIC quot;Living Labquot;                               16
Hypothesis:
Conflict
is
what
drives
Wikipedia
forward.

 

     How
to
study
this?

 

     –  Tukey
paradigm

     –  Get
a
large
paper,
and
plot
the
damn
data!


     –  Downloaded
all
of
Wikipedia
and
all
of
the
revisions

     –  Hadoop/MapReduce,
MySQL,
etc.





2/13/09                        HCIC quot;Living Labquot;                17
100%

                                                                                      Maintenance
                            95%

                            90%
Percentage of total edits

More Related Content

Viewers also liked

Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006Ed Chi
 
Models of Wikipedia Dynamics
Models of Wikipedia DynamicsModels of Wikipedia Dynamics
Models of Wikipedia DynamicsEd Chi
 
Model-Driven Research in Social Computing
Model-Driven Research in Social ComputingModel-Driven Research in Social Computing
Model-Driven Research in Social ComputingEd Chi
 
Wikipedia Slowing Growth and Models
Wikipedia Slowing Growth and ModelsWikipedia Slowing Growth and Models
Wikipedia Slowing Growth and ModelsEd Chi
 
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)Ed Chi
 
CSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearningCSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearningEd Chi
 

Viewers also liked (6)

Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
 
Models of Wikipedia Dynamics
Models of Wikipedia DynamicsModels of Wikipedia Dynamics
Models of Wikipedia Dynamics
 
Model-Driven Research in Social Computing
Model-Driven Research in Social ComputingModel-Driven Research in Social Computing
Model-Driven Research in Social Computing
 
Wikipedia Slowing Growth and Models
Wikipedia Slowing Growth and ModelsWikipedia Slowing Growth and Models
Wikipedia Slowing Growth and Models
 
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
 
CSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearningCSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearning
 

Similar to 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction

2009 05 01 How To Build A Lean Startup Step By Step
2009 05 01 How To Build A Lean Startup Step By Step2009 05 01 How To Build A Lean Startup Step By Step
2009 05 01 How To Build A Lean Startup Step By StepEric Ries
 
GIPA
GIPAGIPA
GIPAESUG
 
Text Mining and SEASR
Text Mining and SEASRText Mining and SEASR
Text Mining and SEASRLoretta Auvil
 
UW ADC - Course 3 - Class 1 - User Stories And Acceptance Testing
UW ADC - Course 3 - Class 1 - User Stories And Acceptance TestingUW ADC - Course 3 - Class 1 - User Stories And Acceptance Testing
UW ADC - Course 3 - Class 1 - User Stories And Acceptance TestingChris Sterling
 
Creating a Winning Life Sciences Innovation Strategy in an Era of Scarcity
Creating a Winning Life Sciences Innovation Strategy in an Era of ScarcityCreating a Winning Life Sciences Innovation Strategy in an Era of Scarcity
Creating a Winning Life Sciences Innovation Strategy in an Era of ScarcitySteve Brown
 
Transforming Contexts: UC DAAP talk, May 8, 2009
Transforming Contexts: UC DAAP talk, May 8, 2009Transforming Contexts: UC DAAP talk, May 8, 2009
Transforming Contexts: UC DAAP talk, May 8, 2009Peter Jones
 
Web 2.0 Marketing in Higher Education
Web 2.0 Marketing in Higher EducationWeb 2.0 Marketing in Higher Education
Web 2.0 Marketing in Higher Educationskornegay
 
Diving Into The Yahoo Open Stack
Diving Into The Yahoo Open StackDiving Into The Yahoo Open Stack
Diving Into The Yahoo Open StackDustin Whittle
 
Building


















 Terrier by
 Open
 Collaboration
Building


















 Terrier by
 Open
 CollaborationBuilding


















 Terrier by
 Open
 Collaboration
Building


















 Terrier by
 Open
 CollaborationCrai Macdonald
 
The Yahoo Open Stack
The Yahoo Open StackThe Yahoo Open Stack
The Yahoo Open StackMegan Eskey
 
Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics

Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics
Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics

Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics
zmanian
 
Fedora App Slide 2009 Hastac
Fedora App Slide 2009 HastacFedora App Slide 2009 Hastac
Fedora App Slide 2009 HastacLoretta Auvil
 
Roll-out of the NYU HSL Website and Drupal CMS
Roll-out of the NYU HSL Website and Drupal CMSRoll-out of the NYU HSL Website and Drupal CMS
Roll-out of the NYU HSL Website and Drupal CMSChris Evjy
 
Scalability without going nuts
Scalability without going nutsScalability without going nuts
Scalability without going nutsJames Cox
 

Similar to 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction (20)

2009 05 01 How To Build A Lean Startup Step By Step
2009 05 01 How To Build A Lean Startup Step By Step2009 05 01 How To Build A Lean Startup Step By Step
2009 05 01 How To Build A Lean Startup Step By Step
 
GIPA
GIPAGIPA
GIPA
 
Text Mining and SEASR
Text Mining and SEASRText Mining and SEASR
Text Mining and SEASR
 
UW ADC - Course 3 - Class 1 - User Stories And Acceptance Testing
UW ADC - Course 3 - Class 1 - User Stories And Acceptance TestingUW ADC - Course 3 - Class 1 - User Stories And Acceptance Testing
UW ADC - Course 3 - Class 1 - User Stories And Acceptance Testing
 
Creating a Winning Life Sciences Innovation Strategy in an Era of Scarcity
Creating a Winning Life Sciences Innovation Strategy in an Era of ScarcityCreating a Winning Life Sciences Innovation Strategy in an Era of Scarcity
Creating a Winning Life Sciences Innovation Strategy in an Era of Scarcity
 
Transforming Contexts: UC DAAP talk, May 8, 2009
Transforming Contexts: UC DAAP talk, May 8, 2009Transforming Contexts: UC DAAP talk, May 8, 2009
Transforming Contexts: UC DAAP talk, May 8, 2009
 
Web 2.0 Marketing in Higher Education
Web 2.0 Marketing in Higher EducationWeb 2.0 Marketing in Higher Education
Web 2.0 Marketing in Higher Education
 
Diving Into The Yahoo Open Stack
Diving Into The Yahoo Open StackDiving Into The Yahoo Open Stack
Diving Into The Yahoo Open Stack
 
Brazil Keynote 2008
Brazil Keynote 2008Brazil Keynote 2008
Brazil Keynote 2008
 
Building


















 Terrier by
 Open
 Collaboration
Building


















 Terrier by
 Open
 CollaborationBuilding


















 Terrier by
 Open
 Collaboration
Building


















 Terrier by
 Open
 Collaboration
 
The Yahoo Open Stack
The Yahoo Open StackThe Yahoo Open Stack
The Yahoo Open Stack
 
Society 3 0
Society 3 0Society 3 0
Society 3 0
 
Service Design
Service DesignService Design
Service Design
 
Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics

Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics
Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics

Towards 
Indigenously 
Developed,

 Locally 
Relevant,
Affordable 
Diagnosics

 
From Work To Word
From Work To WordFrom Work To Word
From Work To Word
 
Fedora App Slide 2009 Hastac
Fedora App Slide 2009 HastacFedora App Slide 2009 Hastac
Fedora App Slide 2009 Hastac
 
RoSE Framework
RoSE FrameworkRoSE Framework
RoSE Framework
 
Roll-out of the NYU HSL Website and Drupal CMS
Roll-out of the NYU HSL Website and Drupal CMSRoll-out of the NYU HSL Website and Drupal CMS
Roll-out of the NYU HSL Website and Drupal CMS
 
Scalability without going nuts
Scalability without going nutsScalability without going nuts
Scalability without going nuts
 
MIE2009 Keynote Address: Clinical Decision Support
MIE2009 Keynote Address: Clinical Decision SupportMIE2009 Keynote Address: Clinical Decision Support
MIE2009 Keynote Address: Clinical Decision Support
 

More from Ed Chi

2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...Ed Chi
 
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social ComputingHCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social ComputingEd Chi
 
Crowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI researchCrowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI researchEd Chi
 
CIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited TalkCIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited TalkEd Chi
 
WikiSym 2011 Closing Keynote
WikiSym 2011 Closing KeynoteWikiSym 2011 Closing Keynote
WikiSym 2011 Closing KeynoteEd Chi
 
Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...Ed Chi
 
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI ResearchTutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI ResearchEd Chi
 
Crowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical TurkCrowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical TurkEd Chi
 
Eddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter StreamsEddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter StreamsEd Chi
 
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...Ed Chi
 
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...Ed Chi
 
Zerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item RecommenderZerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item RecommenderEd Chi
 
ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007Ed Chi
 
Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0Ed Chi
 
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...Ed Chi
 
2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research Overview2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research OverviewEd Chi
 
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica SinicaEd Chi
 
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...Ed Chi
 
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...Ed Chi
 
'Living Lab' for HCI - presentation made at HCI International 2009
'Living Lab' for HCI - presentation made at HCI International 2009'Living Lab' for HCI - presentation made at HCI International 2009
'Living Lab' for HCI - presentation made at HCI International 2009Ed Chi
 

More from Ed Chi (20)

2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
 
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social ComputingHCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
 
Crowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI researchCrowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI research
 
CIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited TalkCIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited Talk
 
WikiSym 2011 Closing Keynote
WikiSym 2011 Closing KeynoteWikiSym 2011 Closing Keynote
WikiSym 2011 Closing Keynote
 
Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...
 
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI ResearchTutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
 
Crowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical TurkCrowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical Turk
 
Eddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter StreamsEddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter Streams
 
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
 
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
 
Zerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item RecommenderZerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item Recommender
 
ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007
 
Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0
 
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
 
2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research Overview2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research Overview
 
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica
2010-02-22 Wikipedia MTurk Research talk given in Taiwan's Academica Sinica
 
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...
Information Seeking with Social Signals: Anatomy of a Social Tag-based Explor...
 
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...
Slowing Growth of Wikipedia and Models of its Dynamic (Presented at Wikimedia...
 
'Living Lab' for HCI - presentation made at HCI International 2009
'Living Lab' for HCI - presentation made at HCI International 2009'Living Lab' for HCI - presentation made at HCI International 2009
'Living Lab' for HCI - presentation made at HCI International 2009
 

Recently uploaded

Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesThousandEyes
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructureitnewsafrica
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Nikki Chapple
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 

Recently uploaded (20)

Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyesAssure Ecommerce and Retail Operations Uptime with ThousandEyes
Assure Ecommerce and Retail Operations Uptime with ThousandEyes
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical InfrastructureVarsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
Varsha Sewlal- Cyber Attacks on Critical Critical Infrastructure
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
Microsoft 365 Copilot: How to boost your productivity with AI – Part two: Dat...
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 

'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction

  • 2. As
a
field,
early
fundamental
contributions
from:
   –  Computer
scientists
interested
in
changes
in
ways
we
interact 
with
information
systems
 –  Psychologists
interested
in
the
implications
of
these
changes
 Combustible,
because:
   –  Computer
scientists
want
to
create
great
tools,
but
didn’t
know 
how
to
measure
impact
 –  Psychologists
want
to
go
beyond
classical
research
of
the
brain 
and
human
cognition
 Example:

   –  an
enduring
interest
in
‘augmenting
human
intellect’:
V.
Bush, 
Licklider,
Engelbart,
in
turn
inspiring
Stu
Card,
Alan
Newell, 
Alan
Kay,
and
many
others.
 2/13/09 HCIC quot;Living Labquot; 2
  • 3. The
need
to
establish
HCI
as
a
science
   –  Adopt
methods
from
psychology
 –  Convenient
and
fits
well
with
problems
at
hand
   Issues
around
personal
computing
(WIMP)
 –  Dual
purpose:
understand
nature
of
human
 behavior
and
build
up
a
science
of
HCI
 techniques.
 –  Good
Examples:
Fitts’
Law,
Models
of
Human
 Memory,
Cognitive
and
Behavioral
Modeling,
 Information
Foraging
 –  Stuart
K.
Card,
William
K.
English,
and
Betty
J.
Burr
(1978).
 Evaluation
of
mouse,
rate‐controlled
isometric
joystick,
 step
keys,
and
text
keys
for
text
selection
on
a
 CRT.
Ergonomics,
21(8):601–613,
1978.
 2/13/09 HCIC quot;Living Labquot; 3
  • 4. Beyond
a
user
in
front
of
computer
   –  Yet
evaluation
methods
mostly
stayed
the
same
 –  Perceived
CHI
paper
template
for
acceptance
 Many
problems
don’t
fit
the
laboratory
experimental
   methods
anymore
 –  Yesterday’s
discussion
about
Large
Data
and
HCI
was
largely
driven
by
 how
HCI
evaluation
methods
need
to
change
to
fit
the
wild
 The
best
examples:

Trends
in
Social
Computing
and
UbiComp
force
   us
to
think
about
new
context
of
use
 2/13/09 HCIC quot;Living Labquot; 4
  • 5. Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with no time boundries Controllable experimental conditions Uncontrollable experimental conditions 2/13/09 HCIC quot;Living Labquot; 5
  • 6. Think
about
research
that
has
been
done
on
UI   
animations
or
flashing
icons.
 From
visual
perception,
we
know
motion
in
the   
periphery
is
more
noticeable
than
in
the
foveal
region 
[DaVinci].
 2/13/09 HCIC quot;Living Labquot; 6
  • 7. Evaluations
surrounding
many
HCI
systems
for   
knowledge
work
focus
on
productivity
increase,
but 
what
about
factors
for
adoption?
 –  Argument:
if
no
productivity
increase,
then
adoption
is 
irrelevant
 –  But
the
opposite
argument
is
just
as
right:
if
no
adoption,
no 
amount
of
productivity
increase
shown
is
relevant!
 Academic
research
often
focus
on
productivity   
improvements,
increasing
the
perceived
gulf
between 
the
ivory
tower
and
the
trenches
 An
Example:
Color
copier
studies
   2/13/09 HCIC quot;Living Labquot; 7
  • 8. Artificial
experimental
setups
are
only
capable
of
telling
us
   behaviors
in
constrained
situations
 –  Ecological
considerations
 –  Hard
to
generalize
to
new
task
contexts
(with
interruptions,
other
 tasks,
other
goals,
unfocused
attention,
more
displays)
 –  Hard
to
generalize
to
other
tools,
apps
 –  Impossible
to
answer
questions
about
aggregate
behaviors
of
groups
 Example
problems:

   –  Adoption
of
mobile
technology
   iPhones
in
Japan,
single‐handed
input
[PARC]
   Best
selling
phones
in
Indonesia
comes
with
a
compass
[Bell]
 –  Aggregate
behavior
of
Wikipedia
or
Delicious
users
   Big
data
analysis
of
edit
logs
 2/13/09 HCIC quot;Living Labquot; 8
  • 9. Was
a
computational
molecular
biologist
   Analogy:
Just
as
biologists
work
on
model
plants
and   
genomes
in
the
lab,
this
tells
us
just
how
it
behaves
in 
an
isolated
environment
under
controlled
conditions, 
but
not
how
the
plant
will
behave
in
the
real
world.
 Biologists
don’t
just
study
models
in
the
lab,
but
in
the   
wild
also.
 2/13/09 HCIC quot;Living Labquot; 9
  • 10. Observational
Studies
   –  Ethnography
 –  Social
Technical
Design
 –  Iterative
Design
 –  Diary
Studies
 –  Longitudinal
Studies
with
single
outcome
 Problems:
   –  Sampling
from
non‐normal
distributions
 –  Treat
variations
in
social
contexts
as
‘noise’
 –  What
about
adoption?
 Mixed
Methods
is
now
popular
to
partially
ameliorate
   –  Convergent
measures
tells
us
we’re
getting
closer
to
the
truth
 2/13/09 HCIC quot;Living Labquot; 10
  • 11. –  similar
to
Venture
Business
mentioned
by
David
Millen
 Conduct
research
on
real
platforms
and
services
   –  Not
to
replace
controlled
lab
studies
 –  Expand
our
arsenal
to
cover
new
situations
 Some
principles:
   –  Embedded
in
the
real
world
 –  Ecologically
valid
situations
 –  Embrace
the
complexity
 –  Rely
on
big‐data‐science
to
extract
patterns
 Not
first
to
suggest
this:
   –  S.
Carter,
J.
Mankoff,
S.
Klemmer
and
T.
Matthews.
Exiting
the
cleanroom:
On
 ecological
validity
and
ubiquitous
computing.
HCI
Journal,
2008
 –  EClass
[Abowd],
PlaceLab
[Intille],
Plasma
Poster
[Churchill
and
Nelson],
Digital
 Family
Portrait
[Rowan,
Mynatt]
 2/13/09 HCIC quot;Living Labquot; 11
  • 12. Two
dimensions
   –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 2/13/09 HCIC quot;Living Labquot; 12
  • 13. Traditional
Approach;
Numerous
examples
   Favored
by
CHI
reviewers
   Typical
situation
is
the
study
of
some
interaction
technique
   –  Pen
input,
gestures,
perception
of
some
visualized
data,
reading
tasks,
 mobile
text
input
 Typical
measures
are
quantitative
in
nature
   –  performance
in
time,
performance
in
accuracy,
eyetracking,
learning
 measures,
user
preferences
 Issues:
   –  Not
always
ecologically
valid
 –  Hard
to
take
all
interactions
into
account
 –  Often
time‐consuming;
even
though
we
thought
we
could
do
it
fast.
 2/13/09 HCIC quot;Living Labquot; 13
  • 14. Harder
to
find
in
the
literature
   Often
comparing
against
an
older
system
as
baseline
   Typical
case
is
comparison
of
two
systems

   –  (one
website
with
another,
one
word
processor
vs.
another)
 –  Which
highlighting
feature
works
better
 –  Two
text
input
technique
on
a
cell
phone
 Typical
measures
are
similar
to
(1)
   Issues:
   –  Some
similar
issues
to
(1)
because
it’s
in
lab
 –  System
feature
not
in
control,
so
not
able
to
compare
fairly,
or
 isolate
the
feature
 2/13/09 HCIC quot;Living Labquot; 14
  • 15. http://www.flickr.com/photos/ crush_images/3245220458/ 2/13/09 HCIC quot;Living Labquot; 15
  • 16. Real
applications
in
ecological
valid
situations
   Real
findings
can
be
applied
to
a
running
system
   Impact
of
research
is
more
immediate,
since
system
is
already
   running
 Typical
case
is
log
analytics
with
large
subject
pools
   –  log
studies
of
web
sites,
real
mobile
calling
usages,
web
search
logs,
 studies
of
Wikipedia
edits.
 Typical
measures
are
stickiness,
amount
of
activity,
clustering
   analysis,
correlational
analysis
 Issues:
   –  Factors
not
in
control,
findings
not
comparable
 –  Factors
cannot
be
isolated
 –  Reasons
for
failure
is
often
just
guesswork
 2/13/09 HCIC quot;Living Labquot; 16
  • 17. Hypothesis:
Conflict
is
what
drives
Wikipedia
forward.
   How
to
study
this?
   –  Tukey
paradigm
 –  Get
a
large
paper,
and
plot
the
damn
data!
 –  Downloaded
all
of
Wikipedia
and
all
of
the
revisions
 –  Hadoop/MapReduce,
MySQL,
etc.
 2/13/09 HCIC quot;Living Labquot; 17
  • 18. 100% Maintenance 95% 90% Percentage of total edits

Editor's Notes

  1. First, I’m humbled that so many luminaries are willing to listen to me speaking about HCI and HCI evaluation.HCI have long moved beyond the evaluation setting of a single user sitting in front of a single desktop computer, yet many of our fundamentally held viewpoints about evaluation continues to be ruled by outdated biases derived from this legacy. We need to engage with real users in 'Living Laboratories', in which researchers either adopt or create functioning systems that are used in real settings. These new experimental platforms will greatly enable researchers to conduct evaluations that span many users, places, time, location, and social factors in ways that are unimaginable before.<number>
  2. Looking back on the history of Human-Computer Interaction as a field, we see fundamental contributions mainly from two groups of researchers: (1) computing scientists interested in how technology would change the way we all interact with information, and (2) psychologists (especially cognitive psychologists) interested in the implications of those changes.
  3. With this aim, during the formation of the field, the need to establish HCI as a science had pushed us to adopt methods from psychology, both because it was convenient as well as the methods fit the needs.
  4. Of course, the world has changed.
  5. Bastardization of HCIRational / taskAbandonment of the notation of task/goalBeing measureable is not the same as being able to control the conditions / experimentOutdated Evaluative AssumptionsOf course, the world has changed.
  6. One might argue that if using an application results in no productivity increase then the fact there is adoption of the application is irrelevant.
  7. <number>
  8. Re-thinking EvaluationsBoth trends have required re-thinking our evaluation methodologies.
  9. A proposal for evaluations using 'Living Laboratories'The Augmented Social Cognition group have been a proponent of the idea of 'Living Labratory' within PARC.
  10. Looking at two different dimensions in which HCI researchers could conduct evaluations, one dimension is whether the system is under the control of the researcher or not.
  11. So as far as whether the glass is half-empty of half-full. I think we’re still filling the glass, as we have so far really just focused on (1) and (2). <number>
  12. <number>
  13. 19