SlideShare a Scribd company logo
1 of 44
Download to read offline
Ed
H.
Chi

Area
Manager
and
Sr.
Research
Scientist

Palo
Alto
Research
Center


2009
HCI
International
Conference,
San
Diego,
CA

    As
a
field,
early
fundamental
contributions
from:

      –  Computer
scientists
interested
in
changes
in
ways
we

         interact
with
information
systems

      –  Psychologists
interested
in
the
implications
of
these

         changes

    Combustible,
because:

      –  Computer
scientists
want
to
create
great
tools,
but
didn’t

         know
how
to
measure
impact

      –  Psychologists
want
to
go
beyond
classical
research
of
the

         brain
and
human
cognition

    The
need
to
establish
HCI
as
a
science

      –  Adopt
methods
from
psychology

      –  Good
Examples:
Fitts’
Law,
Models
of
Human
Memory,

         Cognitive
and
Behavioral
Modeling,
Information
Foraging

      –  Dual
purpose:
understand
nature
of
human
behavior
and

         build
up
a
science
of
HCI
techniques.


      7/24/09                          HCIC "Living Lab"               2
    Many
problems
don’t
fit
the
laboratory
experimental
methods

     anymore

      –  Beyond
a
user
in
front
of
computer;
Yet
evaluation
methods
mostly

         stayed
the
same

      –  Controlled
lab
study
as
the
gold
standard
for
acceptance

    Changes
and
Trends
in
Social
Computing
and
UbiComp

      Old Assumptions                           New Considerations
      Single display                            Multiple displays
      Knowledge work                            Games, communication, social apps
      Isolated worker                           Collaborative and social groups
      Stationary location                       Mobile and stationary
      Short task durations                      Short and long tasks, and tasks with no
                                                time boundries

      Controllable experimental conditions      Uncontrollable experimental conditions

 7/24/09                               HCIC "Living Lab"                                  3
Artificial
experimental
setups
are
only
capable
of
telling
us
behaviors
in

   constrained
situations

    Hard
to
generalize
to
new
task
contexts
(with
interruptions,

     other
tasks,
other
goals,
unfocused
attention,
more
displays)

    Hard
to
generalize
to
other
tools,
apps

    Ecological
considerations

               Adoption
of
mobile
technology

               iPhones
in
Japan,
single‐handed
input
[PARC]

               Best
selling
phones
in
Indonesia
comes
with
a
compass
[Bell]


    Impossible
to
answer
questions
about
aggregate
behaviors

     of
groups

               Aggregate
behavior
of
Wikipedia
or
Delicious
users





 7/24/09                                  HCIC "Living Lab"                     4
    Conduct
research
on
real
platforms
and
services

     –  Not
to
replace
controlled
lab
studies

     –  Expand
our
arsenal
to
cover
new
situations

    Some
principles:

     –    Embedded
in
the
real
world

     –    Ecologically
valid
situations

     –    Embrace
the
complexity

     –    Rely
on
big‐data‐science
to
extract
patterns

    Not
first
to
suggest
this:

     –  S.
Carter,
J.
Mankoff,
S.
Klemmer
and
T.
Matthews.
Exiting
the
cleanroom:
On

        ecological
validity
and
ubiquitous
computing.
HCI
Journal,
2008

     –  EClass
[Abowd],
PlaceLab
[Intille],
Plasma
Poster
[Churchill
and
Nelson],
Digital

        Family
Portrait
[Rowan,
Mynatt]




7/24/09                              HCIC "Living Lab"                                       5
GroupLens / MovieLens [Riedl,
Konstan, Univ. Minnesota]
Games with a Purpose [von Ahn et al]
7/24/09   HCIC "Living Lab"   8
World of Warcraft [Yee, Ducheneaut et al]
Wikipedia History Flow [Viégas et al]
A                              B

Bucket Testing or A/B Testing [Kohavi et al]
UbiFit [Consolvo et al]
7/24/09   HCIC "Living Lab"   13
    Master
degree
was
in
computational
molecular
biology

    Analogy:
Just
as
biologists
work
on
model
plants
and

     genomes
in
the
lab,
this
tells
us
just
how
it
behaves
in
an

     isolated
environment
under
controlled
conditions,
but

     not
how
the
plant
will
behave
in
the
real
world.

    Biologists
don’t
just
study
models
in
the
lab,
but
in
the

     wild
also.





7/24/09                     HCIC "Living Lab"                  14
    Two
dimensions

     –  1.
Whether
the
system
is
under
the
control
of
the
researcher

     –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild



                    System Control             System Not in
                                               Control

       Laboratory   (1) Build a system,        (2) Adopt a system,
                    study in the Lab           study in the Lab

       Wild (Real   (4) Build a system,        (3) Adopt a system,
       World)       release it, study in       study in the Wild
                    the Wild



7/24/09                        HCIC "Living Lab"                        15
    Traditional
Approach;
Numerous
examples

    Favored
by
HCI
field
reviewers

    Typical
situation
is
the
study
of
some
interaction
technique

      –  Pen
input,
gestures,
perception
of
some
visualized
data,
reading
tasks,

         mobile
text
input

    Typical
measures
are
quantitative
in
nature

      –  performance
in
time,
performance
in
accuracy,
eyetracking,
learning

         measures,
user
preferences

    Issues:

      –  Not
always
ecologically
valid

      –  Hard
to
take
all
interactions
into
account

      –  Often
time‐consuming;
even
though
we
thought
we
could
do
it
fast.





 7/24/09                          HCIC "Living Lab"                             16
    Harder
to
find
in
the
literature

    Often
comparing
against
an
older
system
as
baseline

    Typical
case
is
comparison
of
two
systems


     –  (one
website
with
another,
one
word
processor
vs.
another)

     –  Which
highlighting
feature
works
better

     –  Two
text
input
technique
on
a
cell
phone

    Typical
measures
are
similar
to
(1)

    Issues:

     –  Some
similar
issues
to
(1)
because
it’s
in
lab

     –  System
feature
not
in
control,
so
not
able
to
compare
fairly,
or

        isolate
the
feature



7/24/09                        HCIC "Living Lab"                            17
    Two
dimensions

     –  1.
Whether
the
system
is
under
the
control
of
the
researcher

     –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild



                    System Control             System Not in
                                               Control

       Laboratory   (1) Build a system,        (2) Adopt a system,
                    study in the Lab           study in the Lab

       Wild (Real   (4) Build a system,        (3) Adopt a system,
       World)       release it, study in       study in the Wild
                    the Wild



7/24/09                        HCIC "Living Lab"                        18
    Real
applications
in
ecological
valid
situations

    Real
findings
can
be
applied
to
a
running
system

    Impact
of
research
is
more
immediate,
since
system
is
already

     running

    Typical
case
is
log
analytics
with
large
subject
pools

      –  log
studies
of
web
sites,
real
mobile
calling
usages,
web
search
logs,

         studies
of
Wikipedia
edits.

    Typical
measures
are
stickiness,
amount
of
activity,
clustering

     analysis,
correlational
analysis

    Issues:

      –  Factors
not
in
control,
findings
not
comparable

      –  Factors
cannot
be
isolated

      –  Reasons
for
failure
is
often
just
guesswork



 7/24/09                           HCIC "Living Lab"                               19
    Hypothesis:
Conflict
is
what
drives
Wikipedia
forward.

    How
to
study
this?

     –  John
Tukey
paradigm

     –  Get
a
large
paper,
and
plot
all
of
the
data!


     –  Downloaded
all
of
Wikipedia
and
all
of
the
revisions

     –  Hadoop/MapReduce,
MySQL,
etc.





7/24/09                         HCIC "Living Lab"               20
100%

                            95%                                                       Maintenance


                            90%
Percentage of total edits




                                                                                      Other
                            85%

                            80%
                                                                                      User Talk
                            75%
                                                                                      User
                            70%
                                                                                      Article Talk
                            65%
                                                                                      Article
                            60%
                               2001   2002   2003        2004           2005   2006




            7/24/09                                 HCIC "Living Lab"                                21
Group D   

          Group A   


Group B   
                 Group C    
                                      Number of users in user group       A       B       C   Total
                                      Users with Korean point of view     10          6   0      16
                                      Users with Japanese point of view       1       8   7      16

7/24/09                           Neutral or Unidentified
                                  HCIC "Living Lab"                           7       3   6   2217
Anonymous (vandals/
                                          spammers)




          Sympathetic to husband




                                                Mediators




                            Sympathetic to parents
7/24/09                     HCIC "Living Lab"                   23
7/24/09   HCIC "Living Lab"   24
7/24/09   HCIC "Living Lab"   25
7/24/09   HCIC "Living Lab"   26
7/24/09   HCIC "Living Lab"   27
    Hypothesis:
Social
Tagging
doesn’t
scale
over
time.

    How
to
study
this?

     –  Crawl
as
much
tagging
data
as
we
can.

     –  Study
the
noise
in
the
system.


     –  40
machines
for
3
months





7/24/09                       HCIC "Living Lab"             28
Concepts
                                          Topics





Users
                                             Documents



                 Noise

                    Tags

     Decoding
                         Encoding

                   T1…Tn




 7/24/09           HCIC "Living Lab"                            29
Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)

7/24/09                  HCIC "Living Lab"                        30
7/24/09   HCIC "Living Lab"   31
Semantic Similarity Graph
                 Web
   Tools
                           Reference

                 Guide
 Howto

                         Tutorial
               Tips
 Help

         Tip             Tutorials

                Tricks




   7/24/09                           HCIC "Living Lab"   32
7/24/09   HCIC "Living Lab"   33
    Two
dimensions

     –  1.
Whether
the
system
is
under
the
control
of
the
researcher

     –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild



                    System Control             System Not in
                                               Control

       Laboratory   (1) Build a system,        (2) Adopt a system,
                    study in the Lab           study in the Lab

       Wild (Real   (4) Build a system,        (3) Adopt a system,
       World)       release it, study in       study in the Wild
                    the Wild



7/24/09                        HCIC "Living Lab"                        34
    Similar
to
(3),
practical
for
running
systems;
ecologically

     valid,
impact
can
be
immediate.

     –  Good
for
cases
in
which
economics
makes
sense
[Google]

     –  Changes
to
system
is
possible;
Factors
can
be
controlled.

    Typical
case
might
be
A/B
testing,
large
subject
pools

    Typical
measures
are
being
developed

     –  Impact
measures
.

Large
visit
#
and
interest
(measured
by
blog

        posts?)

New
Business
inquiries?

     –  Usability
measures
vs.
Usefulness
measures

    Issues:

     –  Effort
and
resource
requirement
is
dropping
but
still
significant

     –  Hard
for
a
research
lab
to
take
on


7/24/09                       HCIC "Living Lab"                        35
HCIC "Living Lab"

7/24/09   36
HCIC
               "Living
7/24/09   37     Lab"
HCIC
               "Living
7/24/09   38     Lab"
Personal Computing [Xerox PARC]
    Evaluation
methods
are
in‐separable
from
the
kinds
of

     science
and
models
that
can
be
build
in
a
field.


    Platform
advances
enable
real
technology
insertion
into

     real
world
situations
cheaper
and
more
manageable.



                     Characteriza7on
     Models





                     Evalua7ons
        Prototypes





 7/24/09                    HCIC "Living Lab"               43
    Research
Vision:
Understand
how
social
computing
systems
can

     enhance
the
ability
of
a
group
of
people
to
remember,
think,
and

     reason.

    Living
Laboratory:
Create
applications
that
harness
collective

     intelligence
to
improve
knowledge
capture,
transfer,
and
discovery.


http://asc‐parc.blogspot.com

http://www.edchi.net

echi@parc.com


WikiDashboard
               MrTaggy
                    SparTag.us


More Related Content

Viewers also liked

Next Generation Data Center Strategies
Next Generation Data Center StrategiesNext Generation Data Center Strategies
Next Generation Data Center Strategies
Venkat Nambiyur
 

Viewers also liked (17)

TOON Stephen Galsworthy
TOON Stephen GalsworthyTOON Stephen Galsworthy
TOON Stephen Galsworthy
 
Low-Latency Analytics with NoSQL – Introduction to Storm and Cassandra
Low-Latency Analytics with NoSQL – Introduction to Storm and CassandraLow-Latency Analytics with NoSQL – Introduction to Storm and Cassandra
Low-Latency Analytics with NoSQL – Introduction to Storm and Cassandra
 
Modernizing the Legacy - How Dish is Adapting its SOA Services for a Cloud Fi...
Modernizing the Legacy - How Dish is Adapting its SOA Services for a Cloud Fi...Modernizing the Legacy - How Dish is Adapting its SOA Services for a Cloud Fi...
Modernizing the Legacy - How Dish is Adapting its SOA Services for a Cloud Fi...
 
Pre-Con Ed: Learn What's New in CA Spectrum®
Pre-Con Ed: Learn What's New in CA Spectrum®Pre-Con Ed: Learn What's New in CA Spectrum®
Pre-Con Ed: Learn What's New in CA Spectrum®
 
Pesla
PeslaPesla
Pesla
 
Technical Radar (Chinese version) 2014-06
Technical Radar (Chinese version) 2014-06Technical Radar (Chinese version) 2014-06
Technical Radar (Chinese version) 2014-06
 
Rise of Container (RoC)
Rise of Container (RoC)Rise of Container (RoC)
Rise of Container (RoC)
 
Next Generation Data Center Strategies
Next Generation Data Center StrategiesNext Generation Data Center Strategies
Next Generation Data Center Strategies
 
A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...A modern, flexible approach to Hadoop implementation incorporating innovation...
A modern, flexible approach to Hadoop implementation incorporating innovation...
 
Sudan tanıtımı
Sudan tanıtımıSudan tanıtımı
Sudan tanıtımı
 
Deploy, Monitor and Manage in Style with WebSphere Liberty Admin Center
Deploy, Monitor and Manage in Style with WebSphere Liberty Admin CenterDeploy, Monitor and Manage in Style with WebSphere Liberty Admin Center
Deploy, Monitor and Manage in Style with WebSphere Liberty Admin Center
 
Agile Operations Keynote: Redefine the Role of IT Operations With Digital Tra...
Agile Operations Keynote: Redefine the Role of IT Operations With Digital Tra...Agile Operations Keynote: Redefine the Role of IT Operations With Digital Tra...
Agile Operations Keynote: Redefine the Role of IT Operations With Digital Tra...
 
Boston Devops Meetup June 22nd
Boston Devops Meetup June 22ndBoston Devops Meetup June 22nd
Boston Devops Meetup June 22nd
 
BVBA SOSIS van Jeroen Meus kent rustige start
BVBA SOSIS van Jeroen Meus kent rustige startBVBA SOSIS van Jeroen Meus kent rustige start
BVBA SOSIS van Jeroen Meus kent rustige start
 
Equipping IT to Deliver Faster, More Flexible Service Management
Equipping IT to Deliver Faster, More Flexible Service ManagementEquipping IT to Deliver Faster, More Flexible Service Management
Equipping IT to Deliver Faster, More Flexible Service Management
 
Primend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure InfrastruktuurPrimend Pilvekonverents - Azure Infrastruktuur
Primend Pilvekonverents - Azure Infrastruktuur
 
Oow2016 review--paas-microservices-
Oow2016 review--paas-microservices-Oow2016 review--paas-microservices-
Oow2016 review--paas-microservices-
 

Similar to 'Living Lab' for HCI - presentation made at HCI International 2009

Investigating cellular metabolism with the 3D Cell Explorer
Investigating cellular metabolism with the 3D Cell ExplorerInvestigating cellular metabolism with the 3D Cell Explorer
Investigating cellular metabolism with the 3D Cell Explorer
MathieuFRECHIN
 
10th e concertation-brussels-06march2013-v2
10th e concertation-brussels-06march2013-v210th e concertation-brussels-06march2013-v2
10th e concertation-brussels-06march2013-v2
Alex Hardisty
 
Using Visualizations in Remote Online Labs - Talk at CyTSE
Using Visualizations in Remote Online Labs - Talk at CyTSEUsing Visualizations in Remote Online Labs - Talk at CyTSE
Using Visualizations in Remote Online Labs - Talk at CyTSE
Megan Sauter
 
DeepLabCut AI Residency
DeepLabCut AI ResidencyDeepLabCut AI Residency
DeepLabCut AI Residency
Vic Shao-Chih Chiang
 
Rattani - Ph.D. Defense Slides
Rattani - Ph.D. Defense SlidesRattani - Ph.D. Defense Slides
Rattani - Ph.D. Defense Slides
Pluribus One
 

Similar to 'Living Lab' for HCI - presentation made at HCI International 2009 (20)

Structuring User Involvement Dimitri Schuurman et al summer school research day
Structuring User Involvement Dimitri Schuurman et al summer school research dayStructuring User Involvement Dimitri Schuurman et al summer school research day
Structuring User Involvement Dimitri Schuurman et al summer school research day
 
Model repositories and standard formats for model reusability
Model repositories and standard formats for model reusabilityModel repositories and standard formats for model reusability
Model repositories and standard formats for model reusability
 
Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017Being Reproducible: SSBSS Summer School 2017
Being Reproducible: SSBSS Summer School 2017
 
Detecting Food and Activities in Lifelogging Images
Detecting Food and Activities in Lifelogging ImagesDetecting Food and Activities in Lifelogging Images
Detecting Food and Activities in Lifelogging Images
 
Schuurman phd presentation 2015 02 27
Schuurman phd presentation 2015 02 27Schuurman phd presentation 2015 02 27
Schuurman phd presentation 2015 02 27
 
Neri conference Task 24
Neri conference Task 24Neri conference Task 24
Neri conference Task 24
 
Investigating cellular metabolism with the 3D Cell Explorer
Investigating cellular metabolism with the 3D Cell ExplorerInvestigating cellular metabolism with the 3D Cell Explorer
Investigating cellular metabolism with the 3D Cell Explorer
 
10th e concertation-brussels-06march2013-v2
10th e concertation-brussels-06march2013-v210th e concertation-brussels-06march2013-v2
10th e concertation-brussels-06march2013-v2
 
Living Lab research landscape
Living Lab research landscapeLiving Lab research landscape
Living Lab research landscape
 
Using Visualizations in Remote Online Labs - Talk at CyTSE
Using Visualizations in Remote Online Labs - Talk at CyTSEUsing Visualizations in Remote Online Labs - Talk at CyTSE
Using Visualizations in Remote Online Labs - Talk at CyTSE
 
Adding value to scientific results: COMBINE standards & guidelines for system...
Adding value to scientific results: COMBINE standards & guidelines for system...Adding value to scientific results: COMBINE standards & guidelines for system...
Adding value to scientific results: COMBINE standards & guidelines for system...
 
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
2019 Triangle Machine Learning Day - Biomedical Image Understanding and EHRs ...
 
Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology Open and Collaborative Software for Digital Pathology
Open and Collaborative Software for Digital Pathology
 
Systems Biology Systems
Systems Biology SystemsSystems Biology Systems
Systems Biology Systems
 
Open PHACTS April 2017 Science webinar Workflow tools
Open PHACTS April 2017 Science webinar Workflow toolsOpen PHACTS April 2017 Science webinar Workflow tools
Open PHACTS April 2017 Science webinar Workflow tools
 
Compound Management Focus 2010
Compound Management Focus 2010Compound Management Focus 2010
Compound Management Focus 2010
 
Elsevier‘s RDM Program: Habits of Effective Data and the Bourne Ulitmatum
Elsevier‘s RDM Program: Habits of Effective Data and the Bourne UlitmatumElsevier‘s RDM Program: Habits of Effective Data and the Bourne Ulitmatum
Elsevier‘s RDM Program: Habits of Effective Data and the Bourne Ulitmatum
 
Varieties of Self-Awareness and Their Uses in Natural and Artificial Systems ...
Varieties of Self-Awareness and Their Uses in Natural and Artificial Systems ...Varieties of Self-Awareness and Their Uses in Natural and Artificial Systems ...
Varieties of Self-Awareness and Their Uses in Natural and Artificial Systems ...
 
DeepLabCut AI Residency
DeepLabCut AI ResidencyDeepLabCut AI Residency
DeepLabCut AI Residency
 
Rattani - Ph.D. Defense Slides
Rattani - Ph.D. Defense SlidesRattani - Ph.D. Defense Slides
Rattani - Ph.D. Defense Slides
 

More from Ed Chi

2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
Ed Chi
 
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Ed Chi
 
Crowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI researchCrowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI research
Ed Chi
 
CIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited TalkCIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited Talk
Ed Chi
 
WikiSym 2011 Closing Keynote
WikiSym 2011 Closing KeynoteWikiSym 2011 Closing Keynote
WikiSym 2011 Closing Keynote
Ed Chi
 
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Ed Chi
 
Zerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item RecommenderZerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item Recommender
Ed Chi
 
ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007
Ed Chi
 

More from Ed Chi (20)

2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...2017 10-10 (netflix ml platform meetup) learning item and user representation...
2017 10-10 (netflix ml platform meetup) learning item and user representation...
 
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social ComputingHCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
HCI Korea 2012 Keynote Talk on Model-Driven Research in Social Computing
 
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)Location and Language in Social Media (Stanford Mobi Social Invited Talk)
Location and Language in Social Media (Stanford Mobi Social Invited Talk)
 
Crowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI researchCrowdsourcing using MTurk for HCI research
Crowdsourcing using MTurk for HCI research
 
CIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited TalkCIKM 2011 Social Computing Industry Invited Talk
CIKM 2011 Social Computing Industry Invited Talk
 
WikiSym 2011 Closing Keynote
WikiSym 2011 Closing KeynoteWikiSym 2011 Closing Keynote
WikiSym 2011 Closing Keynote
 
CSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearningCSCL 2011 Keynote on Social Computing and eLearning
CSCL 2011 Keynote on Social Computing and eLearning
 
Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...Replication is more than Duplication: Position slides for CHI2011 panel on re...
Replication is more than Duplication: Position slides for CHI2011 panel on re...
 
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI ResearchTutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
Tutorial on Using Amazon Mechanical Turk (MTurk) for HCI Research
 
Crowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical TurkCrowdsourcing for HCI Research with Amazon Mechanical Turk
Crowdsourcing for HCI Research with Amazon Mechanical Turk
 
Eddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter StreamsEddi: Topic Browsing of Twitter Streams
Eddi: Topic Browsing of Twitter Streams
 
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
Large Scale Social Analytics on Wikipedia, Delicious, and Twitter (presented ...
 
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
Model-based Research in Human-Computer Interaction (HCI): Keynote at Mensch u...
 
Zerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item RecommenderZerozero88 Twitter URL Item Recommender
Zerozero88 Twitter URL Item Recommender
 
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
Smart eBooks: ScentIndex and ScentHighlight research published at VAST2006
 
Model-Driven Research in Social Computing
Model-Driven Research in Social ComputingModel-Driven Research in Social Computing
Model-Driven Research in Social Computing
 
ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007ASC Disaster Response Proposal from Aug 2007
ASC Disaster Response Proposal from Aug 2007
 
Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0Using Information Scent to Model Users in Web1.0 and Web2.0
Using Information Scent to Model Users in Web1.0 and Web2.0
 
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
China HCI Symposium 2010 March: Augmented Social Cognition Research from PARC...
 
2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research Overview2010-03-10 PARC Augmented Social Cognition Research Overview
2010-03-10 PARC Augmented Social Cognition Research Overview
 

Recently uploaded

CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
Wonjun Hwang
 

Recently uploaded (20)

TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
2024 May Patch Tuesday
2024 May Patch Tuesday2024 May Patch Tuesday
2024 May Patch Tuesday
 
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptxCyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
Cyber Insurance - RalphGilot - Embry-Riddle Aeronautical University.pptx
 
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdfFrisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
Frisco Automating Purchase Orders with MuleSoft IDP- May 10th, 2024.pptx.pdf
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi Daparthi
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Top 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development CompaniesTop 10 CodeIgniter Development Companies
Top 10 CodeIgniter Development Companies
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 

'Living Lab' for HCI - presentation made at HCI International 2009

  • 2.   As
a
field,
early
fundamental
contributions
from:
 –  Computer
scientists
interested
in
changes
in
ways
we
 interact
with
information
systems
 –  Psychologists
interested
in
the
implications
of
these
 changes
   Combustible,
because:
 –  Computer
scientists
want
to
create
great
tools,
but
didn’t
 know
how
to
measure
impact
 –  Psychologists
want
to
go
beyond
classical
research
of
the
 brain
and
human
cognition
   The
need
to
establish
HCI
as
a
science
 –  Adopt
methods
from
psychology
 –  Good
Examples:
Fitts’
Law,
Models
of
Human
Memory,
 Cognitive
and
Behavioral
Modeling,
Information
Foraging
 –  Dual
purpose:
understand
nature
of
human
behavior
and
 build
up
a
science
of
HCI
techniques.
 7/24/09 HCIC "Living Lab" 2
  • 3.   Many
problems
don’t
fit
the
laboratory
experimental
methods
 anymore
 –  Beyond
a
user
in
front
of
computer;
Yet
evaluation
methods
mostly
 stayed
the
same
 –  Controlled
lab
study
as
the
gold
standard
for
acceptance
   Changes
and
Trends
in
Social
Computing
and
UbiComp
 Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with no time boundries Controllable experimental conditions Uncontrollable experimental conditions 7/24/09 HCIC "Living Lab" 3
  • 4. Artificial
experimental
setups
are
only
capable
of
telling
us
behaviors
in
 constrained
situations
   Hard
to
generalize
to
new
task
contexts
(with
interruptions,
 other
tasks,
other
goals,
unfocused
attention,
more
displays)
   Hard
to
generalize
to
other
tools,
apps
   Ecological
considerations
   Adoption
of
mobile
technology
   iPhones
in
Japan,
single‐handed
input
[PARC]
   Best
selling
phones
in
Indonesia
comes
with
a
compass
[Bell]
   Impossible
to
answer
questions
about
aggregate
behaviors
 of
groups
   Aggregate
behavior
of
Wikipedia
or
Delicious
users
 7/24/09 HCIC "Living Lab" 4
  • 5.   Conduct
research
on
real
platforms
and
services
 –  Not
to
replace
controlled
lab
studies
 –  Expand
our
arsenal
to
cover
new
situations
   Some
principles:
 –  Embedded
in
the
real
world
 –  Ecologically
valid
situations
 –  Embrace
the
complexity
 –  Rely
on
big‐data‐science
to
extract
patterns
   Not
first
to
suggest
this:
 –  S.
Carter,
J.
Mankoff,
S.
Klemmer
and
T.
Matthews.
Exiting
the
cleanroom:
On
 ecological
validity
and
ubiquitous
computing.
HCI
Journal,
2008
 –  EClass
[Abowd],
PlaceLab
[Intille],
Plasma
Poster
[Churchill
and
Nelson],
Digital
 Family
Portrait
[Rowan,
Mynatt]
 7/24/09 HCIC "Living Lab" 5
  • 6. GroupLens / MovieLens [Riedl, Konstan, Univ. Minnesota]
  • 7. Games with a Purpose [von Ahn et al]
  • 8. 7/24/09 HCIC "Living Lab" 8
  • 9. World of Warcraft [Yee, Ducheneaut et al]
  • 10. Wikipedia History Flow [Viégas et al]
  • 11. A B Bucket Testing or A/B Testing [Kohavi et al]
  • 13. 7/24/09 HCIC "Living Lab" 13
  • 14.   Master
degree
was
in
computational
molecular
biology
   Analogy:
Just
as
biologists
work
on
model
plants
and
 genomes
in
the
lab,
this
tells
us
just
how
it
behaves
in
an
 isolated
environment
under
controlled
conditions,
but
 not
how
the
plant
will
behave
in
the
real
world.
   Biologists
don’t
just
study
models
in
the
lab,
but
in
the
 wild
also.
 7/24/09 HCIC "Living Lab" 14
  • 15.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 15
  • 16.   Traditional
Approach;
Numerous
examples
   Favored
by
HCI
field
reviewers
   Typical
situation
is
the
study
of
some
interaction
technique
 –  Pen
input,
gestures,
perception
of
some
visualized
data,
reading
tasks,
 mobile
text
input
   Typical
measures
are
quantitative
in
nature
 –  performance
in
time,
performance
in
accuracy,
eyetracking,
learning
 measures,
user
preferences
   Issues:
 –  Not
always
ecologically
valid
 –  Hard
to
take
all
interactions
into
account
 –  Often
time‐consuming;
even
though
we
thought
we
could
do
it
fast.
 7/24/09 HCIC "Living Lab" 16
  • 17.   Harder
to
find
in
the
literature
   Often
comparing
against
an
older
system
as
baseline
   Typical
case
is
comparison
of
two
systems

 –  (one
website
with
another,
one
word
processor
vs.
another)
 –  Which
highlighting
feature
works
better
 –  Two
text
input
technique
on
a
cell
phone
   Typical
measures
are
similar
to
(1)
   Issues:
 –  Some
similar
issues
to
(1)
because
it’s
in
lab
 –  System
feature
not
in
control,
so
not
able
to
compare
fairly,
or
 isolate
the
feature
 7/24/09 HCIC "Living Lab" 17
  • 18.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 18
  • 19.   Real
applications
in
ecological
valid
situations
   Real
findings
can
be
applied
to
a
running
system
   Impact
of
research
is
more
immediate,
since
system
is
already
 running
   Typical
case
is
log
analytics
with
large
subject
pools
 –  log
studies
of
web
sites,
real
mobile
calling
usages,
web
search
logs,
 studies
of
Wikipedia
edits.
   Typical
measures
are
stickiness,
amount
of
activity,
clustering
 analysis,
correlational
analysis
   Issues:
 –  Factors
not
in
control,
findings
not
comparable
 –  Factors
cannot
be
isolated
 –  Reasons
for
failure
is
often
just
guesswork
 7/24/09 HCIC "Living Lab" 19
  • 20.   Hypothesis:
Conflict
is
what
drives
Wikipedia
forward.
   How
to
study
this?
 –  John
Tukey
paradigm
 –  Get
a
large
paper,
and
plot
all
of
the
data!
 –  Downloaded
all
of
Wikipedia
and
all
of
the
revisions
 –  Hadoop/MapReduce,
MySQL,
etc.
 7/24/09 HCIC "Living Lab" 20
  • 21. 100% 95% Maintenance 90% Percentage of total edits Other 85% 80% User Talk 75% User 70% Article Talk 65% Article 60% 2001 2002 2003 2004 2005 2006 7/24/09 HCIC "Living Lab" 21
  • 22. Group D Group A Group B Group C Number of users in user group A B C Total Users with Korean point of view 10 6 0 16 Users with Japanese point of view 1 8 7 16 7/24/09 Neutral or Unidentified HCIC "Living Lab" 7 3 6 2217
  • 23. Anonymous (vandals/ spammers) Sympathetic to husband Mediators Sympathetic to parents 7/24/09 HCIC "Living Lab" 23
  • 24. 7/24/09 HCIC "Living Lab" 24
  • 25. 7/24/09 HCIC "Living Lab" 25
  • 26. 7/24/09 HCIC "Living Lab" 26
  • 27. 7/24/09 HCIC "Living Lab" 27
  • 28.   Hypothesis:
Social
Tagging
doesn’t
scale
over
time.
   How
to
study
this?
 –  Crawl
as
much
tagging
data
as
we
can.
 –  Study
the
noise
in
the
system.
 –  40
machines
for
3
months
 7/24/09 HCIC "Living Lab" 28
  • 29. Concepts
 Topics
 Users
 Documents
 Noise
 Tags
 Decoding
 Encoding
 T1…Tn
 7/24/09 HCIC "Living Lab" 29
  • 30. Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz) 7/24/09 HCIC "Living Lab" 30
  • 31. 7/24/09 HCIC "Living Lab" 31
  • 32. Semantic Similarity Graph Web Tools Reference Guide Howto Tutorial Tips Help Tip Tutorials Tricks 7/24/09 HCIC "Living Lab" 32
  • 33. 7/24/09 HCIC "Living Lab" 33
  • 34.   Two
dimensions
 –  1.
Whether
the
system
is
under
the
control
of
the
researcher
 –  2.
Whether
the
study
is
conducted
in
the
lab
or
in
the
wild
 System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 34
  • 35.   Similar
to
(3),
practical
for
running
systems;
ecologically
 valid,
impact
can
be
immediate.
 –  Good
for
cases
in
which
economics
makes
sense
[Google]
 –  Changes
to
system
is
possible;
Factors
can
be
controlled.
   Typical
case
might
be
A/B
testing,
large
subject
pools
   Typical
measures
are
being
developed
 –  Impact
measures
.

Large
visit
#
and
interest
(measured
by
blog
 posts?)

New
Business
inquiries?
 –  Usability
measures
vs.
Usefulness
measures
   Issues:
 –  Effort
and
resource
requirement
is
dropping
but
still
significant
 –  Hard
for
a
research
lab
to
take
on
 7/24/09 HCIC "Living Lab" 35
  • 37. HCIC "Living 7/24/09 37 Lab"
  • 38. HCIC "Living 7/24/09 38 Lab"
  • 40.
  • 41.
  • 42.
  • 43.   Evaluation
methods
are
in‐separable
from
the
kinds
of
 science
and
models
that
can
be
build
in
a
field.

   Platform
advances
enable
real
technology
insertion
into
 real
world
situations
cheaper
and
more
manageable.
 Characteriza7on
 Models
 Evalua7ons
 Prototypes
 7/24/09 HCIC "Living Lab" 43
  • 44.   Research
Vision:
Understand
how
social
computing
systems
can
 enhance
the
ability
of
a
group
of
people
to
remember,
think,
and
 reason.
   Living
Laboratory:
Create
applications
that
harness
collective
 intelligence
to
improve
knowledge
capture,
transfer,
and
discovery.
 http://asc‐parc.blogspot.com
 http://www.edchi.net
 echi@parc.com
 WikiDashboard
 MrTaggy
 SparTag.us