How to do MobileHCI Research inthe large?Niels HenzeUniversity of StuttgartVisualization and Interactive SystemsInstituteM...
… but lets start with a question:Who of you ever participated in a user study?
do you think that any of these guys     ever did?Photo by Robertobra,http://en.wikipedia.org/wiki/File:Guarani_Family.JPG ...
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
User studies at  MobileHCI 2010  20% acceptance rate  43 short+long papers
User studies at                             MobileHCI 2010                             20% acceptance rate                ...
User studies at                             MobileHCI 2010                             20% acceptance rate                ...
all with a university degree, recruited in        the Institute community students or employees at our university         ...
small samples
artificial context
artificial task
convenient samples
Some male students from the labtook part in our study...Small sample size isn’t necessarily an issue for a  studyNot every...
User studies at                             MobileHCI 2011                             22.8% acceptance rate              ...
Some motivationLarge numbers are expensive in the lab  – 1,000 subjects for an hour -> 10,000€  – 1,000 subjects for an ho...
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
Example of getting large…   Target selection on                               mobile phones                               ...
Target selection on                         mobile phones                         thirty right-handed                     ...
…same thing in the  large  game published on the  Android Market  we inform the player  about the study  just looks like a...
published on the  Android Market  100,000 installations in  three months  120 million touch events  more than hundred  dif...
[Park2008MobileHCI]
[Henze2011MobileHCI]
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
Types of workProof of concept   –   Showing that an idea/concept/product works   –   Lots of users, good ratings, positive...
Proof of concept
Smule’s iPhone                   Ocarina                   music instrument for the                   iPhone              ...
Shapewriter                  developed gesture-based                  keyboard + notepad                  qualitative feed...
App stores as research tool
Into the wild with                             Hungry Yoshi                             location based game for           ...
100%                                83.68% 81.31%80%60%                    54.76%                                         ...
Local vs. wild                        locale study with 11                        participants                        wild...
Investigatingapp-specificaspects
Ratings for Mobile                         Applications                         compare amount of                         ...
Compare off-screen                                              visualisations                                            ...
Observing general aspects
Falling Asleep with … appazaar                        [Böhmer2011MobileHCI]
A Study of Battery Life                          [Ferreira2011Pervasive]
app stores as a          investigating app-        investigatingproof of concept                         research tool    ...
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
but what is special about app storestudies?
App-based vs. other studiesCommon con- Mining existing App-basedtrolled studies data        studiesFew participants     Ma...
You have to “sell” your studyThe study has a goal  – Collect information about specific behaviour  – Performance for a spe...
Types of appsApplications    Games   Widgets
100,000 90,000 80,000                                            Participants 70,000                                      ...
US Android users     US population60%                                             Participants50%                         ...
Collecting informationObjective data  – As early as possible [Henze2011IJMHCI]  – More than just the task performance    •...
Collecting informationSubjective data  – App Store comments can provide information    •      but usually dont [Henze2011I...
Collecting informationYou have to measure what you intend to   measure!Case Study: Pocket Navigator [Pielot2012CHI]
motivation:                                                                     distraction                               ...
pocketnavigator  navigation system similar  to Google Maps  runs on OpenStreet Maps
pocketnavigator  navigation system similar  to Google Maps  runs on OpenStreet Maps  key innovation: convey  navigation in...
evaluated in a   field study   vibration patterns found to   be effective   they reduce level of   distraction
evaluated in   field study   vibration patterns found to   be effective   they reduce level of   distraction   but, users ...
evaluated in                                    field study                                      vibration patterns found ...
Collecting data,Feb – Dec 2011
quick facts   18,000 downloads   mostly US and Europe
quick facts   18,000 downloads   mostly US and Europe   Between Feb – Dec 2011   8,187 routes calculated   34,035,316 log ...
quick facts   18,000 downloads   mostly US and Europe   Between Feb – Dec 2011   8,187 routes calculated   34,035,316 log ...
pedestrian  navigation?
pedestrian  navigation?  we cannot prevent people  from using the app  anywhere, e.g. in cars
pedestrian  navigation?  we cannot prevent people  from using the app  anywhere, e.g. in cars  in fact, 87% of all log dat...
pedestrian  navigation?  we cannot prevent people  from using the app  anywhere, e.g. in cars  in fact, 87% of all log dat...
lessons learned                     double-check that you                     measure the intended use!                   ...
Collecting informationYou have to measure what you intend to   measure!Another Example: TypeIt
TypeIt                                                   compare approaches to                                            ...
TypeIt                  condition affects the                  number of played levels4 conditions
TypeIt                                   condition affects theAn ANOVA shows that the            number of played levelsfe...
TypeIt                                  condition affects theAnalysis of covariance            number of played levels(ANC...
Realy stupid            hope                  Stupid waste of time!!!                                                     ...
Ready for prime   time  Users don’t care if it’s a  research prototype  Low quality results in low  ratings
Ready for prime   time  users don’t care if it’s a  research prototype  low quality results in low  ratings  and few insta...
Ethical and legal issues“One should treat others as one would likeothers to treat oneself” [Flew1979Dictionary]           ...
6.96%       57.28%                           Informed consent                             Presentation highly affects     ...
Informed consent                      Presentation highly affects                      the conversion rate                ...
Regulations  Which rules to follow?
“any information relating to an identifiedor identifiable natural person”                 Regulations     • Transparency: ...
Outline1.   Limitations of common studies2.   Into the large3.   Types of studies4.   What is so special?5.   What works f...
… or what works for us
number of installations            400            350                                            Games vs. AppsThousands  ...
games         15.6%                                              Games vs. Apps                                           ...
Games vs. Apps  our games are more  successful  there are more apps than  games  players execute the  strangest tasks
Games vs. Apps  our games are more  successful  there are more apps than  games  players execute the  strangest tasks  wid...
Games vs. Apps  our games are more  successful  there are more apps than  games  players execute the  strangest tasks  wid...
Informing the user   provide information in the   Market
Informing the user   provide information in the   Market   show a modal dialog at the   first start
Informing the user   provide information in the   Market   show a modal dialog at the   first start   provide more informa...
Publishing   fancy screenshots and icon   (that’s the first thing   someone sees)   title & description contain   words us...
Playing with the   market   frequent updates
Playing with the   market   frequent updates   rate your app as soon as it   becomes available
Keep it simple   focused and specialized   studies
Keep it simple   focused and specialized   studies   learning by doing
Keep it simple   focused and specialized   studies   learning by doing   release early, often, and   try it again if it do...
Logging  use http and port 80  to transmit data
Logging                   use http and port 80                   to transmit data                   store unaggregated    ...
CSV files from ~400,000 users                                Logging                                  use http and port 80...
Compressed binary data fromless than 3,000 users         Logging                                use http and port 80      ...
200$ for AdMob over a couple of days                                          Advertisements                              ...
100$ for AppBrain on a single day                                           Advertisements                                ...
100$ for AppBrain on a single day                                           Advertisements                                ...
What do?No harm!                                 Release     Inform the user                          Keywords, descriptio...
small samples
large   small samples
artificial context
natural?   artificial context
artificial task
artificial task?
convenient samples
veryconvenient samples         but how         bad is it?
How to do Mobile                               HCI Research in   ethnography, controlled     the large?experiments, observ...
References[Morrison 2012CHI] Alistair Morrison, Donald McMillan, Stuart Reeves, Scott Sherwood, Matthew Chalmers: A Hybrid...
References[Pielot2011ELV] Martin Pielot, Niels Henze, Susanne Boll: Experiments in App Stores – How to Ask Users for their...
References[Hood2011IJTR] Jeffrey Hood, Elizabeth Sall, Billy Charlton: A GPS-based Bicycle Route Choice Model for San Fran...
My App is an Apparatus: How to do Mobile HCI Research in the large
Upcoming SlideShare
Loading in …5
×

My App is an Apparatus: How to do Mobile HCI Research in the large

407 views
353 views

Published on

Since the introduction of application stores for mobile devices there has been an increasing interest to use this distribution platform to collect user feedback. Mobile application stores can make research prototypes widely available and enable to conduct user studies “in the wild” with participants from all over the world. Using apps as an apparatus goes beyond just distributing research prototypes. Consider apps as a tool for research means distributing specifically designed prototypes in order to extend our understanding of mobile HCI. In this tutorial we will provide an overview about recent research in this domain. It will be shown that stringent tasks and users’ motivation are crucial aspects. We will discuss how to design app-based experiments, what kind of users one can expect, and how to avoid ethical and legal issues.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
407
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • http://en.wikipedia.org/wiki/File:Large_number_of_flamingos_at_Lake_Nakuru.jpg
  • http://henriklundqvisthockey.blogspot.com/2011/07/henrik-lundqvist-construction-worker.html
  • Award for a paperwith 8 participants
  • Evaluate mobile systems in the lab
  • In textentrypeopleareaskedtocopytext
  • Guys fromthe lab
  • ~5 participants: Tactics for homing in mobile life - a fieldwalk study of extremely mobile people
  • [Park2008MobileHCI] Y. S. Park, S. H. Han, J. Park, Y. Cho: Touch key design for target selection on a mobile phone. Proc. MobileHCI, 2008.
  • Park et al.
  • Park et al.
  • http://mangelnoah07.blogspot.com/2010/10/victory.html
  • [Wang2009NIME] Ge Wang: DesigningSmule’siPhoneOcarina. Proc. NIME, 2009.Image from: http://www.zdnet.com/blog/apple/shapewriter-must-try-iphone-app/4263
  • previously developed an innovative gesture-based keyboardpublished notepad with that keyboard to Apple's App Storedownload rate peaked at about 30,000 per day.Provide qualitative feedback from App Store comments[Zhai2009CHI] Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Peng, S., Liu, L. Dunnigan, A., Shapewriter on theiPhone: fromthelaboratorytothe real world. AdjunctProc. CHI, 2009.
  • http://www.mattcarlisle.com/webministry/church-ethnography/
  • ported their location-based game Hungry Yoshi to iPhoneparticipants 94,642 unique downloader 24,408 agreed to be part of the trial 8,676 active usersInvestigated how to get subjective feedback "task" with dynamically loaded questions (e.g age, gender, open questions) Integration with Facebook Interviewed 10 people over VoIP or telephone for $25Used user feedback for iterative design[McMillan2010Pervasive] Donald McMillan, Alistair Morrison, Owain Brown, Malcolm Hall & Matthew Chalmers: Further into the Wild: Running Worldwide Trials of Mobile Systems, Proc. Pervasive 2010.
  • [Henze2011IJMHCI] Niels Henze, Martin Pielot, Benjamin Poppinga, TorbenSchinke, Susanne Boll: My App is an Experiment: Experience from User Studies in Mobile App Stores, accepted by the International Journal of Mobile Human Computer Interaction (IJMHCI), 2011.
  • [Henze2011IJMHCI] Niels Henze, Martin Pielot, Benjamin Poppinga, TorbenSchinke, Susanne Boll: My App is an Experiment: Experience from User Studies in Mobile App Stores, accepted by the International Journal of Mobile Human Computer Interaction (IJMHCI), 2011.
  • http://www.redmondelptsa.org/enrichment/sciencefair.html
  • [Girardello2010DSZ] A. Girardello, F. Michahelles, Explicit and Implicit Ratings for Mobile Applications. In 3. Workshop “Digitale Soziale Netze” and der 40. Jahrestagung der Gesellshaft für Informatik, September 2010, Leipzig.
  • [Henze2010MobileHCI] Niels Henze, Susanne Boll: Push the Study to the App Store: Evaluating Off-Screen Visualizations for Maps in the Android Market, Adjunct. Proc. MobileHCI, 2010[Henze2010NordiCHI] Niels Henze, Benjamin Poppinga, Susanne Boll: Experiments in the Wild: Public Evaluation of Off-Screen Visualizations in theAndroid Market, Proc. NordiCHI, 2010.
  • http://www.mattcarlisle.com/webministry/church-ethnography/
  • [Böhmer2011MobileHCI] Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, Gernot Bauer: FallingAsleepwithAngry Birds, Facebook andKindle – A Large Scale Study on Mobile ApplicationUsage. Proc. MobileHCI, 2011.
  • [Ferreira2011Pervasive] Denzil Ferreira, Anind K. Dey, Vassilis Kostakos: Understanding Human-Smartphone Concerns: A Study ofBattery Life. Proc. Pervasive, 2011.
  • Using a serious applications can be as close to the task you want to investigate as you can ever get. E.g. with the PocketNavigator [Pielot2010MobileHCI] the developers try to investigate tactile feedback for navigation systems with an app that IS a navigation system. Unfortunately the competition is very strong for navigation systems – including Google Navigation that is preinstalled on Android devices.games attract a lot of players *HungryYoshi*, off-screen stuffarteficial tasksrepetative tasks are natural [*off-screen stuff*] great for experimentsWidgets and Wallpapers no interaction/tasks great for collecting longitudinal data
  • [Morrison2010RiL] askes “What is 'a user'?” and discusses the difference to controlled studies. They provide different perspectives on how the number of participants can be counted.In a study using the game HungryYoshidescribed in [McMillan2010Pervasive] the authors provided the following numbers: 94,642 unique downloader 24,408 agreed to be part of the trial 8,676 active users[Morrison2010RiL] Alistair Morrison, Stuart Reeves, Donald McMillan, Matthew Chalmers: Experiences of Mass Participation in Ubicomp Research, Proc. Research In The Large Workshop at Ubicomp, 2010.[McMillan2010Pervasive] Donald McMillan, Alistair Morrison, Owain Brown, Malcolm Hall & Matthew Chalmers: Further into the Wild: Running Worldwide Trials of Mobile Systems, Proc. Pervasive 2010.
  • The Nielsen Company looked at the number of smartphone users in the US for different platforms in the third quater of 2010 [Nielsen2011]. Comparing the demographics of, for example, Android users with the US population [USCensusBureau2008] shows a clear difference. Gender and origin are obviously also biased. Furthermore, you cannot expect to get the same distribution for a specific app.[Nielsen2011] http://blog.nielsen.com/nielsenwire/online_mobile/mobile-snapshot-smartphones-now-28-of-u-s-cellphone-market/[USCensusBureau2008] http://www.google.com/publicdata/explore?ds=kf7tgg1uo9ude_&ctype=c&strail=false&nselm=s&met_y=population&scale_y=lin&ind_y=false&idim=age_group:1:3:4:5:6:7:8:9:10:11:12:13:14:15:16:17:18:2&ifdim=age_group&tunit=M&pit=1216850400000&uniSize=0.035&iconSize=0.5&icfg
  • Image from: http://theagecases.blogspot.com/2010_10_01_archive.html
  • Image from: http://theagecases.blogspot.com/2010_10_01_archive.html
  • Image from: http://theagecases.blogspot.com/2010_10_01_archive.html
  • Image from: http://hellomynameisrichard.com/ethics-in-business-personal-writing-assignment/
  • Image from: http://theagecases.blogspot.com/2010_10_01_archive.html
  • Image from: http://theagecases.blogspot.com/2010_10_01_archive.html
  • Image from: http://insidenorthpoint.org/kids/2010/01/13/groups-directors-best-practices/best-practice-pic/
  • http://www.androlib.com/appstatstype.aspx
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Image from: http://www.nucleusnetworks.co.uk/3g-broadband-ports.htm
  • Award for a paperwith 8 participants
  • Award for a paperwith 8 participants
  • Evaluate mobile systems in the lab
  • Evaluate mobile systems in the lab
  • In textentrypeopleareaskedtocopytext
  • In textentrypeopleareaskedtocopytext
  • Guys fromthe lab
  • Guys fromthe lab
  • My App is an Apparatus: How to do Mobile HCI Research in the large

    1. 1. How to do MobileHCI Research inthe large?Niels HenzeUniversity of StuttgartVisualization and Interactive SystemsInstituteMartin PielotTelefónica I+DHCI and Mobile Computing Group
    2. 2. … but lets start with a question:Who of you ever participated in a user study?
    3. 3. do you think that any of these guys ever did?Photo by Robertobra,http://en.wikipedia.org/wiki/File:Guarani_Family.JPG (GFDL)
    4. 4. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    5. 5. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    6. 6. User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers
    7. 7. User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers subjects per paperhttp://nhenze.net/?p=810
    8. 8. User studies at MobileHCI 2010 20% acceptance rate 43 short+long papers subjects per paper subject’s genderhttp://nhenze.net/?p=810
    9. 9. all with a university degree, recruited in the Institute community students or employees at our university User studies at recruited through flyers, posters and various mailing lists at the university MobileHCI 2010 20% acceptance rate 10 university students and 2 participants 43 short+long papers are marketing professionals subjects per paper undergraduate or graduate students at subject’s gender the local university studying a variety of majors often a biased sample university studentsmost subjects were students with abackground in computer sciences most participants were studentsstudying or working in the University ofGlasgow members in a joint research projecthttp://nhenze.net/?p=810
    10. 10. small samples
    11. 11. artificial context
    12. 12. artificial task
    13. 13. convenient samples
    14. 14. Some male students from the labtook part in our study...Small sample size isn’t necessarily an issue for a studyNot every study needs a perfect sample of the populationFocussing on studies with few subjects prevents many findingsWe stew in our own juices if using our own students by default
    15. 15. User studies at MobileHCI 2011 22.8% acceptance rate 63 short+long papers subjects per paperhttp://nhenze.net/?p=865
    16. 16. Some motivationLarge numbers are expensive in the lab – 1,000 subjects for an hour -> 10,000€ – 1,000 subjects for an hour -> 6 month – 1,000 subjects from around the world -> impossibleDifferent contexts are hard to address – We have no airplane in our lab – Don’t want to train ticket for my participant – And what are the relevant contexts anyway?
    17. 17. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    18. 18. Example of getting large… Target selection on mobile phones thirty right-handed subjects different target locations and sizes[Park2008MobileHCI]
    19. 19. Target selection on mobile phones thirty right-handed subjects different target locations and sizes Taps are skewed fixed posture single device Korean students vague results[Park2008MobileHCI]
    20. 20. …same thing in the large game published on the Android Market we inform the player about the study just looks like an ordinary game participants get some introduction they tap the targets We vary targets’ size and position there is even a high score list
    21. 21. published on the Android Market 100,000 installations in three months 120 million touch events more than hundred different devices players from all over the world
    22. 22. [Park2008MobileHCI]
    23. 23. [Henze2011MobileHCI]
    24. 24. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    25. 25. Types of workProof of concept – Showing that an idea/concept/product works – Lots of users, good ratings, positive comments, ...App stores as research tool – Experience report – Ethical and legal issuesInvestigating app-specific aspects – How a specific app is used – Compare different visualizationsObserving general aspects – Learn about how people and devices behave – How are apps how, how people touch the screen, ...
    26. 26. Proof of concept
    27. 27. Smule’s iPhone Ocarina music instrument for the iPhone million installations[Wang2009NIME]
    28. 28. Shapewriter developed gesture-based keyboard + notepad qualitative feedback from App Store comments[Zhai2009CHI]
    29. 29. App stores as research tool
    30. 30. Into the wild with Hungry Yoshi location based game for the iPhone 94,642 unique downloader investigated how to get subjective feedback[McMillan2010Pervasive]
    31. 31. 100% 83.68% 81.31%80%60% 54.76% Experience from 5 Studies40% compare amount of collected data20% experience with collecting 7.32% qualitative data 0.46% 0% discuss internal and external validity[Henze2011IJMHCI]
    32. 32. Local vs. wild locale study with 11 participants wild study with over 10,000 users combine the findings of both approaches[Morrison 2012CHI]
    33. 33. Investigatingapp-specificaspects
    34. 34. Ratings for Mobile Applications compare amount of collected data experience with collecting qualitative data discuss internal and external validity[Girardello2010DSZ]
    35. 35. Compare off-screen visualisations using repeated measures using a tutorial for a map application and using a simple game[Henze2010MobileHCI] [Henze2010MobileHCI]
    36. 36. Observing general aspects
    37. 37. Falling Asleep with … appazaar [Böhmer2011MobileHCI]
    38. 38. A Study of Battery Life [Ferreira2011Pervasive]
    39. 39. app stores as a investigating app- investigatingproof of concept research tool specific aspects general aspects [Wang2009NIME] [McMillan2010RiL] [Girardello2010DSZ] [Hood2011IJTR] [McMillan2010Pervasive] [Zhai2009CHI] [Riccamboni2010IB] [Henze2011MobileHCIa] [Henze2011IJMHCI][Gilbertson2008CiE] [Kuhn2010MM] [Henze2011MobileHCIb] [Miluzzo2010RiL] [Watzdorf2010LocWeb] [Poppinga2010OMUE] [Yan2011MobiSys] [Ferreira2011Pervasive] [Oliver2010HotPlanet] [Budde2010IoT] [Morrison2010RiL] [Buddharaju2010CHI] [Karpischek2011RiL] [Sahami2011CHI] [Henze2010MobileHCI] [Verkasalo2010MB] [Pielot2011ELV] [Henze2010NordiCHI] [Böhmer2011MobileHCI] [Cramer2010UbiComp] [Morrison2011CHI] [Henderson2009HotPlanet] [Norcie2011ELV] Ethics and legal issues
    40. 40. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    41. 41. but what is special about app storestudies?
    42. 42. App-based vs. other studiesCommon con- Mining existing App-basedtrolled studies data studiesFew participants Many participants Many participantsArtificial context Natural context Natural context Defined tasksDefined task No tasks (if needed)Total control over Weak control over No controlparticipants participantsHeavily biased Biased to unbiased Unbiased samplesample sample
    43. 43. You have to “sell” your studyThe study has a goal – Collect information about specific behaviour – Performance for a specific taskUsers have to install the app on their own will – App needs a purpose – Good ratings, high rankingFind a compromise – Maintain the goals of the study – Attract sufficient participants
    44. 44. Types of appsApplications Games Widgets
    45. 45. 100,000 90,000 80,000 Participants 70,000 How do we count the 60,000 number of participant? 50,000 40,000 30,000 20,000 10,000 0 installations opt-in active users [McMillan2010Pervasive] [Morrison2010RiL]
    46. 46. US Android users US population60% Participants50% How do we count the40% number of participant? A good sample of the30% population?20%10%0% 18-34 35-44 45-54 55-64 65+[Nielsen2011] [USCensusBureau2008]
    47. 47. Collecting informationObjective data – As early as possible [Henze2011IJMHCI] – More than just the task performance • All aspects that affect the results • E.g. device type, local, time, screen size, resolution, ... • In particular: a version number – Compromise between permissions and data to collect
    48. 48. Collecting informationSubjective data – App Store comments can provide information • but usually dont [Henze2011IJMHCI] • Might help to claim an app is great (e.g. [Zhai2009CHI]) • Ratings without baseline are meaningless – Investigated how to get subjective feedback [McMillan2010Pervasive] • In-game “tasks” with dynamically loaded questions • Integration with Facebook • Interviewed 10 people over VoIP for $25
    49. 49. Collecting informationYou have to measure what you intend to measure!Case Study: Pocket Navigator [Pielot2012CHI]
    50. 50. motivation: distraction one in six (17%) cell-toting adults say they have been so distracted while talking or texting that they have physically bumped into another person or an objectMadden and Rainie, 2010,http://pewinternet.org/Reports/2010/Cell-Phone-Distractions.aspx
    51. 51. pocketnavigator navigation system similar to Google Maps runs on OpenStreet Maps
    52. 52. pocketnavigator navigation system similar to Google Maps runs on OpenStreet Maps key innovation: convey navigation information in vibration patterns
    53. 53. evaluated in a field study vibration patterns found to be effective they reduce level of distraction
    54. 54. evaluated in field study vibration patterns found to be effective they reduce level of distraction but, users were no experts and did not use navigation support out of a necessity
    55. 55. evaluated in field study vibration patterns found to be effective they reduce level of distraction but, users were no experts and did not use navigationInstead of bringing the user into the “lab” of a necessity support outwe bring the lab to the user’s daily life
    56. 56. Collecting data,Feb – Dec 2011
    57. 57. quick facts 18,000 downloads mostly US and Europe
    58. 58. quick facts 18,000 downloads mostly US and Europe Between Feb – Dec 2011 8,187 routes calculated 34,035,316 log entries 9,400 hours of usage
    59. 59. quick facts 18,000 downloads mostly US and Europe Between Feb – Dec 2011 8,187 routes calculated 34,035,316 log entries 9,400 hours of usage a lot of data! But …
    60. 60. pedestrian navigation?
    61. 61. pedestrian navigation? we cannot prevent people from using the app anywhere, e.g. in cars
    62. 62. pedestrian navigation? we cannot prevent people from using the app anywhere, e.g. in cars in fact, 87% of all log data are from indoor use 
    63. 63. pedestrian navigation? we cannot prevent people from using the app anywhere, e.g. in cars in fact, 87% of all log data are from indoor use  hence filtering (route length, travel time, movement speed) required
    64. 64. lessons learned double-check that you measure the intended use! filter data might be necessary acknowledge the fact that there is always uncertainty[Pielot2012CHI]
    65. 65. Collecting informationYou have to measure what you intend to measure!Another Example: TypeIt
    66. 66. TypeIt compare approaches to improve text entry people play as along as they want[Henze2012CHIa, Henze2012CHIb, Henze2012Text]
    67. 67. TypeIt condition affects the number of played levels4 conditions
    68. 68. TypeIt condition affects theAn ANOVA shows that the number of played levelsfeedback has a significanteffect on the total number oflevels played (p<.01).
    69. 69. TypeIt condition affects theAnalysis of covariance number of played levels(ANCOVA) is a general linear Factor the number of played levels out using anmodel which blends ANOVA ANCOVAand regression. (Wikipedia)
    70. 70. Realy stupid hope Stupid waste of time!!! cailan FC the rabbit.... uninstalled Godimus Prime Ready for primeIts ok Stupid waste of time. erika lance time boring and dumb. Users don’t care if it’s a Beba research prototype Stupid and offinciveto my pet rabbit bayleigh Logan 1 word...... dumb! josue 5 stars if there is a way to turn the music off. Doesnt go to well with slipknot Allen What the hell is this?? Boo! Luci Cullen Girl
    71. 71. Ready for prime time Users don’t care if it’s a research prototype Low quality results in low ratings
    72. 72. Ready for prime time users don’t care if it’s a research prototype low quality results in low ratings and few install installations
    73. 73. Ethical and legal issues“One should treat others as one would likeothers to treat oneself” [Flew1979Dictionary] “Primum non nocere”/”First, do no harm” (Thomas Sydenham)
    74. 74. 6.96% 57.28% Informed consent Presentation highly affects the conversion rate 67.42% 87.57%[Pielot2011ELV]
    75. 75. Informed consent Presentation highly affects the conversion rate Participants arent aware what data is collected[Morrison2011CHI]
    76. 76. Regulations Which rules to follow?
    77. 77. “any information relating to an identifiedor identifiable natural person” Regulations • Transparency: the persons whose data Which rules to follow? are being collected or accessed have the right to be informed when such data e.g. EU Data Protection processing is taking place. Directive • Legitimate purpose: data can only be collected for specific purposes • Proportionality: data should be processed in a fashion that is not excessive beyond the purposes for which they were collected [Henderson2009HotPlanet]
    78. 78. Outline1. Limitations of common studies2. Into the large3. Types of studies4. What is so special?5. What works for us6. Wrap up
    79. 79. … or what works for us
    80. 80. number of installations 400 350 Games vs. AppsThousands 300 250 our games are more successful 200 150 100 50 0
    81. 81. games 15.6% Games vs. Apps our games are more successful there are more apps than games apps 84.4% available in the Android Markethttp://www.androlib.com/appstatstype.aspx
    82. 82. Games vs. Apps our games are more successful there are more apps than games players execute the strangest tasks
    83. 83. Games vs. Apps our games are more successful there are more apps than games players execute the strangest tasks widgets and background services are perfect for longitudinal observations
    84. 84. Games vs. Apps our games are more successful there are more apps than games players execute the strangest tasks widgets and background services are perfect for longitudinal observations but sometimes an app is just the only option
    85. 85. Informing the user provide information in the Market
    86. 86. Informing the user provide information in the Market show a modal dialog at the first start
    87. 87. Informing the user provide information in the Market show a modal dialog at the first start provide more information and a link to an about page
    88. 88. Publishing fancy screenshots and icon (that’s the first thing someone sees) title & description contain words users search for of course I don’t want to miss a single user prepare a dedicated webpage for each app
    89. 89. Playing with the market frequent updates
    90. 90. Playing with the market frequent updates rate your app as soon as it becomes available
    91. 91. Keep it simple focused and specialized studies
    92. 92. Keep it simple focused and specialized studies learning by doing
    93. 93. Keep it simple focused and specialized studies learning by doing release early, often, and try it again if it doesn’t work
    94. 94. Logging use http and port 80 to transmit data
    95. 95. Logging use http and port 80 to transmit data store unaggregated measures[Henze2012CHI]
    96. 96. CSV files from ~400,000 users Logging use http and port 80 to transmit data store unaggregated measures consider limited resources in total: 392,401 files 27,331,383,646 bytes
    97. 97. Compressed binary data fromless than 3,000 users Logging use http and port 80 to transmit data store unaggregated measures consider limited resources seriously!
    98. 98. 200$ for AdMob over a couple of days Advertisements does not work!TapSnap: http://tiny.cc/tapsnap
    99. 99. 100$ for AppBrain on a single day Advertisements does not work! well sometimes it does!TypeIt II: http://tiny.cc/TypeIt2
    100. 100. 100$ for AppBrain on a single day Advertisements does not work! well sometimes it does! focus all your efforts on a very short time get additional users naturallyTypeIt II: http://tiny.cc/TypeIt2
    101. 101. What do?No harm! Release Inform the user Keywords, description, ... Dont store data you dont want Rate and comment Focus your advertisement effortsChoose a type of app Games worked for me Test it But if you have a great system Well I dont do that anyway... At least fix itSell you study Think about the data You compete with commercial apps Do you store everything interesting Graphics, design, ... Can you store data from 10,000 users? Can you analyse it?
    102. 102. small samples
    103. 103. large small samples
    104. 104. artificial context
    105. 105. natural? artificial context
    106. 106. artificial task
    107. 107. artificial task?
    108. 108. convenient samples
    109. 109. veryconvenient samples but how bad is it?
    110. 110. How to do Mobile HCI Research in ethnography, controlled the large?experiments, observations, … can all work in the large collect data Niels Henze University of Stuttgartearly, release often, be Visualization and Interactive Systems flexible Institute respect ethics, consider Martin Pielot Telefónica I+D regulations HCI and Mobile Computing Group
    111. 111. References[Morrison 2012CHI] Alistair Morrison, Donald McMillan, Stuart Reeves, Scott Sherwood, Matthew Chalmers: A Hybrid Mass Participation Approach to Mobile Software Trials. Proceedings of CHI, 2012.[Wang2009NIME] Ge Wang: Designing Smule’s iPhone Ocarina. Proc. NIME, 2009.[Zhai2009CHI] Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Peng, S., Liu, L. Dunnigan, A., Shapewriter on the iPhone: from the laboratory to the real world. Adjunct Proc. CHI, 2009.[Gilbertson2008CiE] Paul Gilbertson, Paul Coulton, Fadi Chehimi, Tamas Vajk: Using Tilt as an Interface to control No Button 3-D Mobile Games. ACM Computers in Entertainment, 2008.[Oliver2010HotPlanet] Earl Oliver. The Challenges in Large-Scale Smartphone User Studies. Invited talk @ HotPlanet, 2010.[McMillan2010RiL] Donald McMillan: iPhone Software Distribution for Mass Participation. Proc. Research in the Large Workshop @ UbiComp, 2010.[Miluzzo2010RiL] Emiliano Miluzzo, Nicholas D. Lane, Hong Lu, Andrew T. Campbell: Research in the App Store Era: Experiences from the CenceMe App Deployment on the iPhone. Proc. Research in the Large Workshop @ UbiComp, 2010.[Henze2011IJMHCI] Niels Henze, Martin Pielot, Benjamin Poppinga, Torben Schinke, Susanne Boll: My App is an Experiment: Experience from User Studies in Mobile App Stores, accepted by the International Journal of Mobile Human Computer Interaction (IJMHCI), 2011[McMillan2010Pervasive] Donald McMillan, Alistair Morrison, Owain Brown, Malcolm Hall & Matthew Chalmers: Further into the Wild: Running Worldwide Trials of Mobile Systems, Proc. Pervasive 2010.[Cramer2010UbiComp] Henriette Cramer, Mattias Rost, Nicolas Belloni, Didier Chincholle, Frank Bentley: Research in the Large. Using App Stores, Markets, and Other Wide Distribution Channels in Ubicomp Research. Adjunct Proc. Ubicomp, 2010.[Morrison2010RiL] Alistair Morrison, Stuart Reeves, Donald McMillan, Matthew Chalmers: Experiences of Mass Participation in Ubicomp Research, Proc. Research In The Large Workshop at Ubicomp, 2010.[Poppinga2010OMUE] Benjamin Poppinga, Martin Pielot, Niels Henze, Susanne Boll: Unsupervised User Observation in the App Store: Experiences with the Sensor-based Evaluation of a Mobile Pedestrian Navigation Application. Proc. OMUE in conjunction with NordiCHI, 2010.
    112. 112. References[Pielot2011ELV] Martin Pielot, Niels Henze, Susanne Boll: Experiments in App Stores – How to Ask Users for their Consent?, Proceedings of the CHI workshop on Ethics, logs & videotape, 2011.[Henderson2009HotPlanet] Tristan Henderson, Fehmi Ben Abdesslem: Scaling Measurement Experiments to Planet- Scale: Ethical, Regulatory and Cultural Considerations. Proc. HotPlanet, 2009.[Morrison2011CHI] Alistair Morrison, Owain Brown, Donald McMillan, Matthew Chalmers: Informed Consent and Users Attitudes to Logging in Large Scale Trials. Adjunct Proc. CHI, 2011.[Norcie2011ELV] Greg Norcie: Ethical and Practical Considerations For Compensation of Crowdsourced Research Participants, Proc. ETHICS, LOGS and VIDEOTAPE @ CHI, 2011.[Girardello2010DSZ] A. Girardello, F. Michahelles, Explicit and Implicit Ratings for Mobile Applications. In 3. Workshop “Digitale Soziale Netze” and der 40. Jahrestagung der Gesellshaft für Informatik, September 2010, Leipzig.[Riccamboni2010IB] Rodolfo Riccamboni, Alessio Mereu, Chiara Boscarol: Keys to Nature: A test on the iPhone market. Tools for Identifying Biodiversity: Progress and Problems, 2010.[Kuhn2010MM] Michael Kuhn, Roger Wattenhofer, Samuel Welten: Social Audio Features for Advanced Music Retrieval Interfaces. Proc. MM, 2010.[Yan2011MobiSys]Bo Yan, Guanling Chen: AppJoy: Personalized Mobile Application Discovery. Proc. MobiSys, 2011.[Budde2010IoT] Andreas Budde, Florian Michahelles: Product Empire - Serious play with barcodes. Proc. IoT, 2010.[Karpischek2011RiL] Stephan Karpischek, Geron Gilad, Florian Michahelles: Towards a Better Understanding of Mobile Shopping Assistants - A Large Scale Usage Analysis of a Mobile Bargain Finder Application. Workshop on Research in the Large @ UbiComp, 2011.[Henze2010MobileHCI] Niels Henze, Susanne Boll: Push the Study to the App Store: Evaluating Off-Screen Visualizations for Maps in the Android Market, Proc. MobileHCI, 2010[Henze2010NordiCHI] Niels Henze, Benjamin Poppinga, Susanne Boll: Experiments in the Wild: Public Evaluation of Off- Screen Visualizations in the Android Market, Proc. NordiCHI, 2010.
    113. 113. References[Hood2011IJTR] Jeffrey Hood, Elizabeth Sall, Billy Charlton: A GPS-based Bicycle Route Choice Model for San Francisco, California. Transportation Letters: The International Journal of Transportation Research, 2011[Henze2011MobileHCIa] Niels Henze, Enrico Rukzio, Susanne Boll: 100,000,000 Taps: Analysis and Improvement of Touch Performance in the Large, Proceedings of MobileHCI, 2011[Henze2011MobileHCIb ] Niels Henze, Susanne Boll: Release Your App on Sunday Eve: Finding the Best Time to Deploy Apps, Adjunct proceedings of MobileHCI, 2011[Henze2012CHIa] Niels Henze, Enrico Rukzio, Susanne Boll: Observational and Experimental Investigation of Typing Behaviour using Virtual Keyboards on Mobile Devices, Proceedings of CHI 2012.[Henze2012CHIb] Niels Henze: Hit it!: an apparatus for upscaling mobile HCI studies. Proceeding of CHI Extended Abstracts, 2012.[Henze2012Text] Niels Henze: Ten male colleagues took part in our lab-study about mobile texting, Proceedings of the Workshop on Designing and Evaluating Text Entry Methods in conjunction with CHI, 2012.[Watzdorf2010LocWeb] Stephan von Watzdorf, Florian Michahelles: Accuracy of Positioning Data on Smartphones. Proc. LocWeb, 2010.[Ferreira2011Pervasive] Denzil Ferreira, Anind K. Dey, Vassilis Kostakos: Understanding Human-Smartphone Concerns: A Study of Battery Life. Proc. Pervasive, 2011.[Buddharaju2010CHI] Pradeep Buddharaju, Yuichi Fujiki, Ioannis Pavlidis, Ergun Akleman: A Novel Way to Conduct Human Studies and Do Some Good. Adcunct Proc. CHI, 2010.[Sahami2011CHI] Alireza Sahami, Michael Rohs, Robert Schleicher, Sven Kratz, Alexander Müller, Albrecht Schmidt: Real-Time Nonverbal Opinion Sharing through Mobile Phones during Sports Events, Proc. CHI 2011.[Verkasalo2010MB] Hannu Verkasalo: Analysis of Smartphone User Behavior, Proc. Ninth International Conference on Mobile Business, 2010.[Böhmer2011MobileHCI] Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, Gernot Bauer: Falling Asleep with Angry Birds, Facebook and Kindle – A Large Scale Study on Mobile Application Usage. Proc. MobileHCI, 2011.[Agarwal2010HotNets] Sharad Agarwal, Ratul Mahajan, Alice Zheng, Victor Bahl: There’s an app for that, but it doesn’t work. Diagnosing Mobile Applications in the Wild. Proc. Hotnets, 2010.[Morrison2010RiL] Alistair Morrison, Matthew Chalmers: SGVis: Analysis of Mass Participation Trial Data. Proc. Research In The Large Workshop at Ubicomp, 2010.[Lane2010CM] Nicholas D. Lane, Emiliano Miluzzo, Hong Lu, Daniel Peebles, Tanzeem Choudhury, Andrew T. Campbell: A Survey of Mobile Phone Sensing. IEEE Communications Magazine, 2010.

    ×