Evaluation beyond the desktop ...
Thomas Grill
mailto:thomas.grill@sbg.ac.at
17.06.2009

Human Computer Interaction & Usab...
What’s the difference?

...
● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5
Thursday, May 27, 2010

2 / 41
● Why is it different?
• The

range of computerized systems is far greater than just
desktop application.
• In particular,...
●

General problems related to the usability of
unconventional user interaction
• In

UbiComp applications
The feasibility...
What to evaluate?

...
● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5
Thursday, May 27, 2010

5 / 41
1

What to design - what to evaluate?
GUIs

Interaction
with
Interfaces
Screen

Print

Interactions

Objects

Information
...
1

Usability of ...
• desktop

applications

WIMP environment
 Standard interaction devices (mouse/keyboard/...)


• mob...
1

Evaluation in a UCD process
Context of use

User requirements

- User tasks, Task characteristics
- User groups, User c...
1

... identify and evaluate
Requirements

Usability

Utility

User
Experience

• desktop

applications
• mobile applicati...
1

Requirements

•

Usability

Utility

User
Experience

User requirements
Which are the targeted users?
 Data for defini...
1

Requirements

•

Usability

Utility

User
Experience

Evaluate
according to usability heuristics
 http://www.useit.com...
1

Requirements

• Utility


Usability

Utility

User
Experience

refers to the design's functionality:

Does it do what ...
1

Requirements

Usability

● - What - Where - How - Tools - Assignment
Thursday, May 27, 2010

Utility

User
Experience

...
1

Requirements

Usability

What constitutes a good user
experience………….
1.
2.
3.
4.
5.
6.
7.
8.
9.

useful
functional
int...
Where to evaluate?

the field lab ...
●
Thursday, May 27, 2010

15 / 41
2

Evaluation environment
video camera

audio

usability lab

system

video monitor

tasks

Usability laboratory

one way ...
2

Select the appropriate
evaluation environment
• What
• In

to evaluate?

the lab (in-Vitro)
“Realistic” usage scenario ...
2

Select the appropriate
evaluation environment
• What
• In

to evaluate?

the field (in-Situ)
Mobile task scenario
 Out...
2

Select the appropriate
evaluation environment
• What
• In

to evaluate?

the real world (in-Vivo)
Long time usability o...
How to evaluate?

...
●
Thursday, May 27, 2010

20 / 41
3

Evaluate a product

http://www.ambientdevices.com/products/umbrella.html

● - What - Where - How - Tools
Thursday, May ...
3

Define the goals of the evaluation ...
• What

do I want to find out?

I have a design / idea.
• Does it make sense?
• ...
3

Evaluate behaviour

tool

• Observation
Audio/Video recording
 note taking
 diary methods (e.g. photo diary, post-it ...
3

Evaluate behaviour

● - What - Where - How - Tools
Thursday, May 27, 2010

24 / 45
3

Evaluate usability
• Evaluate

behaviour

• Measure
task completion
 error rate
 critical incidents method


• Measu...
3

Evaluate user experience
http://tpgblog.com

Quick UX - Heuristics for User Experience

The Quick-UX evaluates the degr...
3

Evaluate user experience
• How?
• Evaluation

methods based on cognitive and social

sciences
Interviews
 Questionnair...
3

How to evaluate ...
• Select,

create, or define an appropriate

Evaluation Method
 Evaluation Environment
 The parti...
3

How to evaluate ...
• Conduct

the evaluation
• Analyze the data
depends on the type of data collected
 video/audio an...
3

!"#$%&'()&*'#$%&*+',-$+,&%.&-,'/"0"12&345&
Measuring usability

● - What - Where - How - Tools
Thursday, May 27, 2010

...
Evaluation tools in the field ...

... tools
●
Thursday, May 27, 2010

31 / 41
4

Questionnaires
• SUS


- System Usability Scale

"SUS: a "quick and dirty" usability scale"
http://www.usabilitynet.or...
4

Traditional tools ...
• Paper

& Pencil - Wizard of Oz Evaluation
• Storyboarding
download video!

● - What - Where - H...
4

Recording tools
•

Audio


•

cap with mobile camera

Mobile audio recorders

Video

maybe show cap/
Morae/Camtasia
sc...
4

Recording tools
• Text
Notebooks
 Instrumented tools that log data
 Keylogger tools


• Transformed

audio/video via...
4

Recording tools
• UTE

- Usability Testing Environment

Comprehensive tool for capturing and analyzing usability data
...
4

EyeTracking
• Record

eye movement

• Gaze
• Focus

of attention
• Systems
eye tracking system on computer screen
 eye...
4

Motion and position tracking
• Tracking

motion
• Tracking position
• Finding information about movement
• Crowd behavi...
4

Recording tools
• The

Observer XT from Noldus

Sophisticated software for usability data
collection, analysis and pres...
4

Recording tools

Noldus mobile camera ...
● - What - Where - How - Tools
Thursday, May 27, 2010

40 / 45
4

Recording tools

Video system using
regular and IP
based moveable
cameras
Recording & Notetaking

Receiver for mobile c...
4

Recording tools

Note taking and evaluation
control interface

Video recorder - monitor

Noldus observer - recording mo...
Thursday, May 27, 2010
4

Analysis tools
• Video/Audio/Text

annotation tools

ELAN
 NVivo
 MacEval


• Statistical

Analysis tools

SPSS
• ht...
* Contact
HCI & Usability Unit
ICT&S Center, University of Salzburg
Sigmund-Haffner-Gasse 18
5020 Salzburg, Austria
hci-un...
Upcoming SlideShare
Loading in...5
×

Evaluation beyond the desktop

2,068

Published on

Evaluating non-desktop user interfaces focuses on the evaluation of pervasive, ubicomp, augmented reality, etc . appliances. Examples, Methods, and Tools are discussed

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,068
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Evaluation beyond the desktop

  1. 1. Evaluation beyond the desktop ... Thomas Grill mailto:thomas.grill@sbg.ac.at 17.06.2009 Human Computer Interaction & Usability Unit Thursday, May 27, 2010 http://icts.sbg.ac.at
  2. 2. What’s the difference? ... ● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5 Thursday, May 27, 2010 2 / 41
  3. 3. ● Why is it different? • The range of computerized systems is far greater than just desktop application. • In particular, unconventional input channels such as Tangible user interfaces • Physical objects are representations and controls for digital objects  Gesture inputs, movement tracking  Eye- and gaze tracking  • and environmental profiles such as Virtual environments  Augmented reality  • are difficult or impossible to test using ordinary usability labs ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 3 / 45
  4. 4. ● General problems related to the usability of unconventional user interaction • In UbiComp applications The feasibility rather than the usability is tested  No established software usability culture for mobile and UbiComp applications  • Lack of trained specialists • Screenbased testing methods are often not feasible • Direct porting between different platforms is not feasible nor does it cover the particular platform requirements • Fast-paced software market environment  No time for long-lasting user testing? ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 4 / 45
  5. 5. What to evaluate? ... ● - HCI - Topic 2 - Topic 3 - Topic 4 - Topic 5 Thursday, May 27, 2010 5 / 41
  6. 6. 1 What to design - what to evaluate? GUIs Interaction with Interfaces Screen Print Interactions Objects Information Devices Architecture Non-GUIs ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 6 / 45
  7. 7. 1 Usability of ... • desktop applications WIMP environment  Standard interaction devices (mouse/keyboard/...)  • mobile applications mobile environment  mobile devices (smartphones, handhelds, ubicomp appliances, ...)  • real world applications real world environment (mobile+static)  real world objects, ubicomp objects  Requirements Usability ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience 7 / 45
  8. 8. 1 Evaluation in a UCD process Context of use User requirements - User tasks, Task characteristics - User groups, User characteristics - environment - Usability parameters - UI design requirements Product evaluations iterative formative evaluations of Lo/Fi prototypes - user experience analysis - usability evaluations ??? iterative formative evaluations of Hi/Fi Prototypes and products summative usability evaluation ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 8 / 45
  9. 9. 1 ... identify and evaluate Requirements Usability Utility User Experience • desktop applications • mobile applications • ubicomp applications • real world applications ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 9 / 45
  10. 10. 1 Requirements • Usability Utility User Experience User requirements Which are the targeted users?  Data for defining personas, user profiles  • Task requirements functionality, task  what to design?  • Design requirements which information do I need for this  is there an existing workflow? (reuse prior knowledge)  Usage environment  Type of information • mobile vs. static • outside vs. indoor, desk vs. standing, ... • public places vs. private places • ...  ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 10 / 45
  11. 11. 1 Requirements • Usability Utility User Experience Evaluate according to usability heuristics  http://www.useit.com/papers/heuristic/heuristic_list.html  • Task performance Task completion  Delay  • Errors  • Error rate, error messages, error handling, ... Intuitiveness, Simplicity, Learnability, ...  Simple and natural dialogues Consistent interface and interaction • Feedback •  • No states where the user does not know what to do. Visibility of System State ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 11 / 45
  12. 12. 1 Requirements • Utility  Usability Utility User Experience refers to the design's functionality: Does it do what users need? Usefulness  Is the functionality needed  Is the functionality appropriate  • Usability and utility are equally important! ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 12 / 45
  13. 13. 1 Requirements Usability ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience http://sakshigupta.files.wordpress.com/2009/11/ 13 / 45
  14. 14. 1 Requirements Usability What constitutes a good user experience…………. 1. 2. 3. 4. 5. 6. 7. 8. 9. useful functional intuitive reliable efficient effective usable innovative aesthetically pleasing beautiful 10. delightful, ‘aha’ moment wow-factor ● - What - Where - How - Tools - Assignment Thursday, May 27, 2010 Utility User Experience How to design for a good user experience…………. 1. by understanding people needs, wants,behavior,constraints 2. based in social & cultural context 3. exploring opportunities 4. based on people’s past experiences 5. power to evoke emotions 6. forgiving to errors 7. simplicity 8. optimized for most frequent tasks 9. informative & timely feedback 10. story-telling 11. human touch 12. multiple iterations 13. prototyping http://sakshigupta.files.wordpress.com/2009/11/ 14 / 45
  15. 15. Where to evaluate? the field lab ... ● Thursday, May 27, 2010 15 / 41
  16. 16. 2 Evaluation environment video camera audio usability lab system video monitor tasks Usability laboratory one way mirror facilitator recording unit Field evaluation evaluator what to apply? ● - What - Where - How - Tools Thursday, May 27, 2010 16 / 45
  17. 17. 2 Select the appropriate evaluation environment • What • In to evaluate? the lab (in-Vitro) “Realistic” usage scenario can be setup/simulated in a usability laboratory  Usually in-door usage  Usability lab, Smart Homes  Evaluating concrete tasks, performing user tests  lower costs  ● - What - Where - How - Tools Thursday, May 27, 2010 17 / 45
  18. 18. 2 Select the appropriate evaluation environment • What • In to evaluate? the field (in-Situ) Mobile task scenario  Outdoor usage  Observing users in real world scenarios  higher costs  ● - What - Where - How - Tools Thursday, May 27, 2010 18 / 45
  19. 19. 2 Select the appropriate evaluation environment • What • In to evaluate? the real world (in-Vivo) Long time usability observations  User experience analysis  Indoor and outdoor usage  Observing/Sensing/Logging of users in real world scenarios  higher costs / higher effort  ● - What - Where - How - Tools Thursday, May 27, 2010 19 / 45
  20. 20. How to evaluate? ... ● Thursday, May 27, 2010 20 / 41
  21. 21. 3 Evaluate a product http://www.ambientdevices.com/products/umbrella.html ● - What - Where - How - Tools Thursday, May 27, 2010 21 / 45
  22. 22. 3 Define the goals of the evaluation ... • What do I want to find out? I have a design / idea. • Does it make sense? • Is it useful?  Usability  User Experience  Utility  General usage behaviour  Which user group to address  ...  ● - What - Where - How - Tools Thursday, May 27, 2010 22 / 45
  23. 23. 3 Evaluate behaviour tool • Observation Audio/Video recording  note taking  diary methods (e.g. photo diary, post-it notes, DRM - day reconstruction method, ...)  • Interviews  structured vs. narrative • Focus  groups A group of people are asked about their perceptions, opinions, beliefs and attitudes towards a product, service, concept, advertisement, idea, or packaging ... ● - What - Where - How - Tools Thursday, May 27, 2010 23 / 45
  24. 24. 3 Evaluate behaviour ● - What - Where - How - Tools Thursday, May 27, 2010 24 / 45
  25. 25. 3 Evaluate usability • Evaluate behaviour • Measure task completion  error rate  critical incidents method  • Measuring Sensing  Logging  • Important Time synchronization of all measured data!!!  Analysis tools help here (ELAN, NVivo, MacEval, Morae, ...)  ● - What - Where - How - Tools Thursday, May 27, 2010 25 / 45
  26. 26. 3 Evaluate user experience http://tpgblog.com Quick UX - Heuristics for User Experience The Quick-UX evaluates the degree to which a product successfully addresses the following 3 questions: Can I use it? (Usability) • Should I use it? (Usefulness) • Do I want to use it? (Desirability) customer ty abili g mar ke tin brand management n ce y e cr il i dib e sa tisf act ion expected function ality no u erro nexpecte d rs expected information ty d e si r abi lit co ty bili ifya ver r te fo opria appr se purpo ity rm nfo y alit naming f rd o wo uth o m as e ess en iqu un usef uln es s fer men ts dif es s user require c ac VALUE AG WC li comp conte separ color and c med use ia gra ele phic me nts typ og ra ph y on t ti nc fu d an c on te n se ness l ua nt me ts ce e n pla elem of ion igat nav intuitive str ucture naming categ and orizat ion co n sist enc y Thursday, May 27, 2010 u s er er ility ws tib ro pa b m co ard c nd sta mplian co http://userexperienceproject.blogspot.com/2007/04/user-experience-wheel.html to vo ne o ice f ● - What - Where - How - Tools f i n d a bi l i t y usability ic eg rat st launch on dati oun f ex p e ri ic provider Usability, Utility  Emotions  Social distinction  Stress  Trust  Privacy  Safety  ...  str a ph teg user experience factors e nt i at ion • Observe s ea stra rch en gine teg y r es po tim n e se • 26 / 45
  27. 27. 3 Evaluate user experience • How? • Evaluation methods based on cognitive and social sciences Interviews  Questionnaires  long-term evaluations with diary method  Critical Incident Method (ICT)  ...  ● - What - Where - How - Tools Thursday, May 27, 2010 27 / 45
  28. 28. 3 How to evaluate ... • Select, create, or define an appropriate Evaluation Method  Evaluation Environment  The participants of the evaluation scenario • experts • test users  Prepare • test scenarios • questionnaires • necessary forms (e.g. non-disclosure form, session checklists, ...)  tool • • • http://www.stcsig.org/usability/resources/toolkit/toolkit.html A set of document templates useful during usability evaluations NASA Usability toolkit (http://www.hq.nasa.gov/pao/portal/usability/resources/index.htm) • .... ● - What - Where - How - Tools Thursday, May 27, 2010 28 / 45
  29. 29. 3 How to evaluate ... • Conduct the evaluation • Analyze the data depends on the type of data collected  video/audio analysis  text analysis  questionnaire evaluation  quantitative / qualitative methods  • Compile findings compile the identified issues  prepare a report  ● - What - Where - How - Tools Thursday, May 27, 2010 29 / 45
  30. 30. 3 !"#$%&'()&*'#$%&*+',-$+,&%.&-,'/"0"12&345& Measuring usability ● - What - Where - How - Tools Thursday, May 27, 2010 30 / 45
  31. 31. Evaluation tools in the field ... ... tools ● Thursday, May 27, 2010 31 / 41
  32. 32. 4 Questionnaires • SUS  - System Usability Scale "SUS: a "quick and dirty" usability scale" http://www.usabilitynet.org/trump/documents/Suschapt.doc • AttrakDiff • SUMI Addresses quality of use of software  http://sumi.ucc.ie/  • QUIS • IsoMetrics Operationalising the design principles of ISO 9241 Part 10  http://www.isometrics.uni-osnabrueck.de/  • ... ● - What - Where - How - Tools Thursday, May 27, 2010 32 / 45
  33. 33. 4 Traditional tools ... • Paper & Pencil - Wizard of Oz Evaluation • Storyboarding download video! ● - What - Where - How - Tools Thursday, May 27, 2010 33 / 45
  34. 34. 4 Recording tools • Audio  • cap with mobile camera Mobile audio recorders Video maybe show cap/ Morae/Camtasia scenario from • Excellent tools for capturing video, logging issues, and creating highlight gerhard leitner tapes • http://www.techsmith.com/  Noldus Observer • http://www.noldus.com  Silverback • http://silverbackapp.com/  MacEval • http://www.tomgrill.info/maceval  OvoLogger • http://www.ovostudios.com  ● - What - Where - How - Tools Thursday, May 27, 2010 34 / 45
  35. 35. 4 Recording tools • Text Notebooks  Instrumented tools that log data  Keylogger tools  • Transformed audio/video via annotation tools NVivo  Elan  Interact • http://www.mangoldinternational.com  ● - What - Where - How - Tools Thursday, May 27, 2010 35 / 45
  36. 36. 4 Recording tools • UTE - Usability Testing Environment Comprehensive tool for capturing and analyzing usability data  Automatically calculates success rates, time-on-task, and many other metrics  Remote Usability testing  http://www.mindd.com  • Data Logger Free Excel program used to collect and analyze usability test data.  Records task success, time-on-task, survey questions, and automatically generates charts.  www.userfocus.co.uk/resources/datalogger .html  ● - What - Where - How - Tools Thursday, May 27, 2010 36 / 45
  37. 37. 4 EyeTracking • Record eye movement • Gaze • Focus of attention • Systems eye tracking system on computer screen  eye tracking in real world  SMI  Tobii  ● - What - Where - How - Tools Thursday, May 27, 2010 37 / 45
  38. 38. 4 Motion and position tracking • Tracking motion • Tracking position • Finding information about movement • Crowd behaviour • ... • Systems based on Optical: video, IR  Accoustic  RFID  GPS  ● - What - Where - How - Tools Thursday, May 27, 2010 Optical RFID 38 / 45
  39. 39. 4 Recording tools • The Observer XT from Noldus Sophisticated software for usability data collection, analysis and presentation.  Ability to integrate multiple video feeds, analyze eye-tracking data, and other physiological measurements.  Allows to integrate eye-trackers  http://www.noldus.com  • FaceReader  FaceReader is a tool that is capable of automatically analyzing facial expressions, providing users with an objective assessment of a person’s emotion. ● - What - Where - How - Tools Thursday, May 27, 2010 39 / 45
  40. 40. 4 Recording tools Noldus mobile camera ... ● - What - Where - How - Tools Thursday, May 27, 2010 40 / 45
  41. 41. 4 Recording tools Video system using regular and IP based moveable cameras Recording & Notetaking Receiver for mobile camera Noldus observer Usability lab - ICTS Salzburg observation place ● - What - Where - How - Tools Thursday, May 27, 2010 41 / 45
  42. 42. 4 Recording tools Note taking and evaluation control interface Video recorder - monitor Noldus observer - recording mode Usability lab - ICTS Salzburg ● - What - Where - How - Tools Thursday, May 27, 2010 42 / 45
  43. 43. Thursday, May 27, 2010
  44. 44. 4 Analysis tools • Video/Audio/Text annotation tools ELAN  NVivo  MacEval  • Statistical Analysis tools SPSS • http://www.spss.com R • http://www.r-project.org/  Microsoft Excel • statistics add-in available: “Analyze-it” • http://analyse-it.com/  Stata • http://www.stata.com  ● - What - Where - How - Tools Thursday, May 27, 2010 44 / 45
  45. 45. * Contact HCI & Usability Unit ICT&S Center, University of Salzburg Sigmund-Haffner-Gasse 18 5020 Salzburg, Austria hci-unit@icts.sbg.ac.at Dr. Thomas Grill thomas.grill@sbg.ac.at Human Computer Interaction & Usability Unit Thursday, May 27, 2010 hci-unit@icts.sbg.ac.at
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×