Evaluation beyond the desktop

  • 1,949 views
Uploaded on

Evaluating non-desktop user interfaces focuses on the evaluation of pervasive, ubicomp, augmented reality, etc . appliances. Examples, Methods, and Tools are discussed

Evaluating non-desktop user interfaces focuses on the evaluation of pervasive, ubicomp, augmented reality, etc . appliances. Examples, Methods, and Tools are discussed

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,949
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Evaluation beyond the desktop ... Thomas Grill mailto:thomas.grill@sbg.ac.at 17.06.2009 Human Computer Interaction & Usability Unit Thursday, May 27, 2010 http://icts.sbg.ac.at
  • 2. What’s the difference? ... ● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5 Thursday, May 27, 2010 2 / 41
  • 3. ● Why is it different? • The range of computerized systems is far greater than just desktop application. • In particular, unconventional input channels such as Tangible user interfaces • Physical objects are representations and controls for digital objects  Gesture inputs, movement tracking  Eye- and gaze tracking  • and environmental profiles such as Virtual environments  Augmented reality  • are difficult or impossible to test using ordinary usability labs ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 3 / 45
  • 4. ● General problems related to the usability of unconventional user interaction • In UbiComp applications The feasibility rather than the usability is tested  No established software usability culture for mobile and UbiComp applications  • Lack of trained specialists • Screenbased testing methods are often not feasible • Direct porting between different platforms is not feasible nor does it cover the particular platform requirements • Fast-paced software market environment  No time for long-lasting user testing? ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 4 / 45
  • 5. What to evaluate? ... ● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5 Thursday, May 27, 2010 5 / 41
  • 6. 1 What to design - what to evaluate? GUIs Interaction with Interfaces Screen Print Interactions Objects Information Devices Architecture Non-GUIs ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 6 / 45
  • 7. 1 Usability of ... • desktop applications WIMP environment  Standard interaction devices (mouse/keyboard/...)  • mobile applications mobile environment  mobile devices (smartphones, handhelds, ubicomp appliances, ...)  • real world applications real world environment (mobile+static)  real world objects, ubicomp objects  Requirements Usability ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience 7 / 45
  • 8. 1 Evaluation in a UCD process Context of use User requirements - User tasks, Task characteristics - User groups, User characteristics - environment - Usability parameters - UI design requirements Product evaluations iterative formative evaluations of Lo/Fi prototypes - user experience analysis - usability evaluations ??? iterative formative evaluations of Hi/Fi Prototypes and products summative usability evaluation ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 8 / 45
  • 9. 1 ... identify and evaluate Requirements Usability Utility User Experience • desktop applications • mobile applications • ubicomp applications • real world applications ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 9 / 45
  • 10. 1 Requirements • Usability Utility User Experience User requirements Which are the targeted users?  Data for defining personas, user profiles  • Task requirements functionality, task  what to design?  • Design requirements which information do I need for this  is there an existing workflow? (reuse prior knowledge)  Usage environment  Type of information • mobile vs. static • outside vs. indoor, desk vs. standing, ... • public places vs. private places • ...  ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 10 / 45
  • 11. 1 Requirements • Usability Utility User Experience Evaluate according to usability heuristics  http://www.useit.com/papers/heuristic/heuristic_list.html  • Task performance Task completion  Delay  • Errors  • Error rate, error messages, error handling, ... Intuitiveness, Simplicity, Learnability, ...  Simple and natural dialogues Consistent interface and interaction • Feedback •  • No states where the user does not know what to do. Visibility of System State ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 11 / 45
  • 12. 1 Requirements • Utility  Usability Utility User Experience refers to the design's functionality: Does it do what users need? Usefulness  Is the functionality needed  Is the functionality appropriate  • Usability and utility are equally important! ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 12 / 45
  • 13. 1 Requirements Usability ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience http://sakshigupta.files.wordpress.com/2009/11/ 13 / 45
  • 14. 1 Requirements Usability What constitutes a good user experience…………. 1. 2. 3. 4. 5. 6. 7. 8. 9. useful functional intuitive reliable efficient effective usable innovative aesthetically pleasing beautiful 10. delightful, ‘aha’ moment wow-factor ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience How to design for a good user experience…………. 1. by understanding people needs, wants,behavior,constraints 2. based in social & cultural context 3. exploring opportunities 4. based on people’s past experiences 5. power to evoke emotions 6. forgiving to errors 7. simplicity 8. optimized for most frequent tasks 9. informative & timely feedback 10. story-telling 11. human touch 12. multiple iterations 13. prototyping http://sakshigupta.files.wordpress.com/2009/11/ 14 / 45
  • 15. Where to evaluate? the field lab ... ● Thursday, May 27, 2010 15 / 41
  • 16. 2 Evaluation environment video camera audio usability lab system video monitor tasks Usability laboratory one way mirror facilitator recording unit Field evaluation evaluator what to apply? ● - What - Where - How - Tools Thursday, May 27, 2010 16 / 45
  • 17. 2 Select the appropriate evaluation environment • What • In to evaluate? the lab (in-Vitro) “Realistic” usage scenario can be setup/simulated in a usability laboratory  Usually in-door usage  Usability lab, Smart Homes  Evaluating concrete tasks, performing user tests  lower costs  ● - What - Where - How - Tools Thursday, May 27, 2010 17 / 45
  • 18. 2 Select the appropriate evaluation environment • What • In to evaluate? the field (in-Situ) Mobile task scenario  Outdoor usage  Observing users in real world scenarios  higher costs  ● - What - Where - How - Tools Thursday, May 27, 2010 18 / 45
  • 19. 2 Select the appropriate evaluation environment • What • In to evaluate? the real world (in-Vivo) Long time usability observations  User experience analysis  Indoor and outdoor usage  Observing/Sensing/Logging of users in real world scenarios  higher costs / higher effort  ● - What - Where - How - Tools Thursday, May 27, 2010 19 / 45
  • 20. How to evaluate? ... ● Thursday, May 27, 2010 20 / 41
  • 21. 3 Evaluate a product http://www.ambientdevices.com/products/umbrella.html ● - What - Where - How - Tools Thursday, May 27, 2010 21 / 45
  • 22. 3 Define the goals of the evaluation ... • What do I want to find out? I have a design / idea. • Does it make sense? • Is it useful?  Usability  User Experience  Utility  General usage behaviour  Which user group to address  ...  ● - What - Where - How - Tools Thursday, May 27, 2010 22 / 45
  • 23. 3 Evaluate behaviour tool • Observation Audio/Video recording  note taking  diary methods (e.g. photo diary, post-it notes, DRM - day reconstruction method, ...)  • Interviews  structured vs. narrative • Focus  groups A group of people are asked about their perceptions, opinions, beliefs and attitudes towards a product, service, concept, advertisement, idea, or packaging ... ● - What - Where - How - Tools Thursday, May 27, 2010 23 / 45
  • 24. 3 Evaluate behaviour ● - What - Where - How - Tools Thursday, May 27, 2010 24 / 45
  • 25. 3 Evaluate usability • Evaluate behaviour • Measure task completion  error rate  critical incidents method  • Measuring Sensing  Logging  • Important Time synchronization of all measured data!!!  Analysis tools help here (ELAN, NVivo, MacEval, Morae, ...)  ● - What - Where - How - Tools Thursday, May 27, 2010 25 / 45
  • 26. 3 Evaluate user experience http://tpgblog.com Quick UX - Heuristics for User Experience The Quick-UX evaluates the degree to which a product successfully addresses the following 3 questions: Can I use it? (Usability) • Should I use it? (Usefulness) • Do I want to use it? (Desirability) customer ty abili g mar ke tin brand management n ce y e cr il i dib e sa tisf act ion expected function ality no u erro nexpecte d rs expected information ty d e si r abi lit co ty bili ifya ver r te fo opria appr se purpo ity rm nfo y alit naming f rd o wo uth o m as e ess en iqu un usef uln es s fer men ts dif es s user require c ac VALUE AG WC li comp conte separ color and c med use ia gra ele phic me nts typ og ra ph y on t ti nc fu d an c on te n se ness l ua nt me ts ce e n pla elem of ion igat nav intuitive str ucture naming categ and orizat ion co n sist enc y Thursday, May 27, 2010 u s er er ility ws tib ro pa b m co ard c nd sta mplian co http://userexperienceproject.blogspot.com/2007/04/user-experience-wheel.html to vo ne o ice f ● - What - Where - How - Tools f i n d a bi l i t y usability ic eg rat st launch on dati oun f ex p e ri ic provider Usability, Utility  Emotions  Social distinction  Stress  Trust  Privacy  Safety  ...  str a ph teg user experience factors e nt i at ion • Observe s ea stra rch en gine teg y r es po tim n e se • 26 / 45
  • 27. 3 Evaluate user experience • How? • Evaluation methods based on cognitive and social sciences Interviews  Questionnaires  long-term evaluations with diary method  Critical Incident Method (ICT)  ...  ● - What - Where - How - Tools Thursday, May 27, 2010 27 / 45
  • 28. 3 How to evaluate ... • Select, create, or define an appropriate Evaluation Method  Evaluation Environment  The participants of the evaluation scenario • experts • test users  Prepare • test scenarios • questionnaires • necessary forms (e.g. non-disclosure form, session checklists, ...)  tool • • • http://www.stcsig.org/usability/resources/toolkit/toolkit.html A set of document templates useful during usability evaluations NASA Usability toolkit (http://www.hq.nasa.gov/pao/portal/usability/resources/index.htm) • .... ● - What - Where - How - Tools Thursday, May 27, 2010 28 / 45
  • 29. 3 How to evaluate ... • Conduct the evaluation • Analyze the data depends on the type of data collected  video/audio analysis  text analysis  questionnaire evaluation  quantitative / qualitative methods  • Compile findings compile the identified issues  prepare a report  ● - What - Where - How - Tools Thursday, May 27, 2010 29 / 45
  • 30. 3 !"#$%&'()&*'#$%&*+',-$+,&%.&-,'/"0"12&345& Measuring usability ● - What - Where - How - Tools Thursday, May 27, 2010 30 / 45
  • 31. Evaluation tools in the field ... ... tools ● Thursday, May 27, 2010 31 / 41
  • 32. 4 Questionnaires • SUS  - System Usability Scale "SUS: a "quick and dirty" usability scale" http://www.usabilitynet.org/trump/documents/Suschapt.doc • AttrakDiff • SUMI Addresses quality of use of software  http://sumi.ucc.ie/  • QUIS • IsoMetrics Operationalising the design principles of ISO 9241 Part 10  http://www.isometrics.uni-osnabrueck.de/  • ... ● - What - Where - How - Tools Thursday, May 27, 2010 32 / 45
  • 33. 4 Traditional tools ... • Paper & Pencil - Wizard of Oz Evaluation • Storyboarding download video! ● - What - Where - How - Tools Thursday, May 27, 2010 33 / 45
  • 34. 4 Recording tools • Audio  • cap with mobile camera Mobile audio recorders Video maybe show cap/ Morae/Camtasia scenario from • Excellent tools for capturing video, logging issues, and creating highlight gerhard leitner tapes • http://www.techsmith.com/  Noldus Observer • http://www.noldus.com  Silverback • http://silverbackapp.com/  MacEval • http://www.tomgrill.info/maceval  OvoLogger • http://www.ovostudios.com  ● - What - Where - How - Tools Thursday, May 27, 2010 34 / 45
  • 35. 4 Recording tools • Text Notebooks  Instrumented tools that log data  Keylogger tools  • Transformed audio/video via annotation tools NVivo  Elan  Interact • http://www.mangoldinternational.com  ● - What - Where - How - Tools Thursday, May 27, 2010 35 / 45
  • 36. 4 Recording tools • UTE - Usability Testing Environment Comprehensive tool for capturing and analyzing usability data  Automatically calculates success rates, time-on-task, and many other metrics  Remote Usability testing  http://www.mindd.com  • Data Logger Free Excel program used to collect and analyze usability test data.  Records task success, time-on-task, survey questions, and automatically generates charts.  www.userfocus.co.uk/resources/datalogger .html  ● - What - Where - How - Tools Thursday, May 27, 2010 36 / 45
  • 37. 4 EyeTracking • Record eye movement • Gaze • Focus of attention • Systems eye tracking system on computer screen  eye tracking in real world  SMI  Tobii  ● - What - Where - How - Tools Thursday, May 27, 2010 37 / 45
  • 38. 4 Motion and position tracking • Tracking motion • Tracking position • Finding information about movement • Crowd behaviour • ... • Systems based on Optical: video, IR  Accoustic  RFID  GPS  ● - What - Where - How - Tools Thursday, May 27, 2010 Optical RFID 38 / 45
  • 39. 4 Recording tools • The Observer XT from Noldus Sophisticated software for usability data collection, analysis and presentation.  Ability to integrate multiple video feeds, analyze eye-tracking data, and other physiological measurements.  Allows to integrate eye-trackers  http://www.noldus.com  • FaceReader  FaceReader is a tool that is capable of automatically analyzing facial expressions, providing users with an objective assessment of a person’s emotion. ● - What - Where - How - Tools Thursday, May 27, 2010 39 / 45
  • 40. 4 Recording tools Noldus mobile camera ... ● - What - Where - How - Tools Thursday, May 27, 2010 40 / 45
  • 41. 4 Recording tools Video system using regular and IP based moveable cameras Recording & Notetaking Receiver for mobile camera Noldus observer Usability lab - ICTS Salzburg observation place ● - What - Where - How - Tools Thursday, May 27, 2010 41 / 45
  • 42. 4 Recording tools Note taking and evaluation control interface Video recorder - monitor Noldus observer - recording mode Usability lab - ICTS Salzburg ● - What - Where - How - Tools Thursday, May 27, 2010 42 / 45
  • 43. Thursday, May 27, 2010
  • 44. 4 Analysis tools • Video/Audio/Text annotation tools ELAN  NVivo  MacEval  • Statistical Analysis tools SPSS • http://www.spss.com R • http://www.r-project.org/  Microsoft Excel • statistics add-in available: “Analyze-it” • http://analyse-it.com/  Stata • http://www.stata.com  ● - What - Where - How - Tools Thursday, May 27, 2010 44 / 45
  • 45. * Contact HCI & Usability Unit ICT&S Center, University of Salzburg Sigmund-Haffner-Gasse 18 5020 Salzburg, Austria hci-unit@icts.sbg.ac.at Dr. Thomas Grill thomas.grill@sbg.ac.at Human Computer Interaction & Usability Unit Thursday, May 27, 2010 hci-unit@icts.sbg.ac.at