'Living Lab' for HCI - presentation made at HCI International 2009

Ed H. Chi  Area Manager and Sr. Research Scientist  Palo Alto Research Center  2009 HCI International Conference, San Diego, CA 

  As a ﬁeld, early fundamental contributions from:  –  Computer scientists interested in changes in ways we  interact with information systems  –  Psychologists interested in the implications of these  changes    Combustible, because:  –  Computer scientists want to create great tools, but didn’t  know how to measure impact  –  Psychologists want to go beyond classical research of the  brain and human cognition    The need to establish HCI as a science  –  Adopt methods from psychology  –  Good Examples: Fitts’ Law, Models of Human Memory,  Cognitive and Behavioral Modeling, Information Foraging  –  Dual purpose: understand nature of human behavior and  build up a science of HCI techniques.  7/24/09 HCIC "Living Lab" 2

  Many problems don’t ﬁt the laboratory experimental methods  anymore  –  Beyond a user in front of computer; Yet evaluation methods mostly  stayed the same  –  Controlled lab study as the gold standard for acceptance    Changes and Trends in Social Computing and UbiComp  Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with no time boundries Controllable experimental conditions Uncontrollable experimental conditions 7/24/09 HCIC "Living Lab" 3

Artiﬁcial experimental setups are only capable of telling us behaviors in  constrained situations    Hard to generalize to new task contexts (with interruptions,  other tasks, other goals, unfocused attention, more displays)    Hard to generalize to other tools, apps    Ecological considerations    Adoption of mobile technology    iPhones in Japan, single‐handed input [PARC]    Best selling phones in Indonesia comes with a compass [Bell]    Impossible to answer questions about aggregate behaviors  of groups    Aggregate behavior of Wikipedia or Delicious users  7/24/09 HCIC "Living Lab" 4

  Conduct research on real platforms and services  –  Not to replace controlled lab studies  –  Expand our arsenal to cover new situations    Some principles:  –  Embedded in the real world  –  Ecologically valid situations  –  Embrace the complexity  –  Rely on big‐data‐science to extract patterns    Not ﬁrst to suggest this:  –  S. Carter, J. Mankoﬀ, S. Klemmer and T. Matthews. Exiting the cleanroom: On  ecological validity and ubiquitous computing. HCI Journal, 2008  –  EClass [Abowd], PlaceLab [Intille], Plasma Poster [Churchill and Nelson], Digital  Family Portrait [Rowan, Mynatt]  7/24/09 HCIC "Living Lab" 5

GroupLens / MovieLens [Riedl, Konstan, Univ. Minnesota]

Games with a Purpose [von Ahn et al]

7/24/09 HCIC "Living Lab" 8

World of Warcraft [Yee, Ducheneaut et al]

Wikipedia History Flow [Viégas et al]

A B Bucket Testing or A/B Testing [Kohavi et al]

UbiFit [Consolvo et al]

  Master degree was in computational molecular biology    Analogy: Just as biologists work on model plants and  genomes in the lab, this tells us just how it behaves in an  isolated environment under controlled conditions, but  not how the plant will behave in the real world.    Biologists don’t just study models in the lab, but in the  wild also.  7/24/09 HCIC "Living Lab" 14

  Two dimensions  –  1. Whether the system is under the control of the researcher  –  2. Whether the study is conducted in the lab or in the wild  System Control System Not in Control Laboratory (1) Build a system, (2) Adopt a system, study in the Lab study in the Lab Wild (Real (4) Build a system, (3) Adopt a system, World) release it, study in study in the Wild the Wild 7/24/09 HCIC "Living Lab" 15

  Traditional Approach; Numerous examples    Favored by HCI ﬁeld reviewers    Typical situation is the study of some interaction technique  –  Pen input, gestures, perception of some visualized data, reading tasks,  mobile text input    Typical measures are quantitative in nature  –  performance in time, performance in accuracy, eyetracking, learning  measures, user preferences    Issues:  –  Not always ecologically valid  –  Hard to take all interactions into account  –  Often time‐consuming; even though we thought we could do it fast.  7/24/09 HCIC "Living Lab" 16

  Harder to ﬁnd in the literature    Often comparing against an older system as baseline    Typical case is comparison of two systems   –  (one website with another, one word processor vs. another)  –  Which highlighting feature works better  –  Two text input technique on a cell phone    Typical measures are similar to (1)    Issues:  –  Some similar issues to (1) because it’s in lab  –  System feature not in control, so not able to compare fairly, or  isolate the feature  7/24/09 HCIC "Living Lab" 17

  Real applications in ecological valid situations    Real ﬁndings can be applied to a running system    Impact of research is more immediate, since system is already  running    Typical case is log analytics with large subject pools  –  log studies of web sites, real mobile calling usages, web search logs,  studies of Wikipedia edits.    Typical measures are stickiness, amount of activity, clustering  analysis, correlational analysis    Issues:  –  Factors not in control, ﬁndings not comparable  –  Factors cannot be isolated  –  Reasons for failure is often just guesswork  7/24/09 HCIC "Living Lab" 19

  Hypothesis: Conﬂict is what drives Wikipedia forward.    How to study this?  –  John Tukey paradigm  –  Get a large paper, and plot all of the data!  –  Downloaded all of Wikipedia and all of the revisions  –  Hadoop/MapReduce, MySQL, etc.  7/24/09 HCIC "Living Lab" 20

100% 95% Maintenance 90% Percentage of total edits Other 85% 80% User Talk 75% User 70% Article Talk 65% Article 60% 2001 2002 2003 2004 2005 2006 7/24/09 HCIC "Living Lab" 21

Group D Group A Group B Group C Number of users in user group A B C Total Users with Korean point of view 10 6 0 16 Users with Japanese point of view 1 8 7 16 7/24/09 Neutral or Unidentified HCIC "Living Lab" 7 3 6 2217

Anonymous (vandals/ spammers) Sympathetic to husband Mediators Sympathetic to parents 7/24/09 HCIC "Living Lab" 23

  Hypothesis: Social Tagging doesn’t scale over time.    How to study this?  –  Crawl as much tagging data as we can.  –  Study the noise in the system.  –  40 machines for 3 months  7/24/09 HCIC "Living Lab" 28

Concepts  Topics  Users  Documents  Noise  Tags  Decoding  Encoding  T1…Tn  7/24/09 HCIC "Living Lab" 29

Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz) 7/24/09 HCIC "Living Lab" 30

Semantic Similarity Graph Web Tools Reference Guide Howto Tutorial Tips Help Tip Tutorials Tricks 7/24/09 HCIC "Living Lab" 32

  Similar to (3), practical for running systems; ecologically  valid, impact can be immediate.  –  Good for cases in which economics makes sense [Google]  –  Changes to system is possible; Factors can be controlled.    Typical case might be A/B testing, large subject pools    Typical measures are being developed  –  Impact measures .  Large visit # and interest (measured by blog  posts?)  New Business inquiries?  –  Usability measures vs. Usefulness measures    Issues:  –  Eﬀort and resource requirement is dropping but still signiﬁcant  –  Hard for a research lab to take on  7/24/09 HCIC "Living Lab" 35

HCIC "Living Lab" 7/24/09 36

HCIC "Living 7/24/09 37 Lab"

HCIC "Living 7/24/09 38 Lab"

Personal Computing [Xerox PARC]

  Evaluation methods are in‐separable from the kinds of  science and models that can be build in a ﬁeld.     Platform advances enable real technology insertion into  real world situations cheaper and more manageable.  Characteriza7on  Models  Evalua7ons  Prototypes  7/24/09 HCIC "Living Lab" 43

  Research Vision: Understand how social computing systems can  enhance the ability of a group of people to remember, think, and  reason.    Living Laboratory: Create applications that harness collective  intelligence to improve knowledge capture, transfer, and discovery.  http://asc‐parc.blogspot.com  http://www.edchi.net  echi@parc.com  WikiDashboard  MrTaggy  SparTag.us 

http://edchi.blogspot.com	503
http://edchi.blogspot.fr	4
http://edchi.blogspot.com.au	3
http://edchi.blogspot.co.uk	2
http://edchi.blogspot.ca	2
http://edchi.blogspot.de	2
http://edchi.blogspot.jp	1
http://edchi.blogspot.in	1
https://courseware.ku.edu	1
http://edchi.blogspot.it	1
http://www.slideshare.net	1
http://edchi.blogspot.com.ar	1
http://edchi.blogspot.pt	1

'Living Lab' for HCI - presentation made at HCI International 2009

by Ed Chi on Jul 25, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

13 Embeds 523

Statistics

’Living Lab’ for HCI - presentation made at HCI International 2009 — Presentation Transcript