Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
APPLICATION
0F THE
THINK ALOUD METHOD
IN
USE, USER AND USABILITY RESEARCH
IN
CARTOGRAPHY & GEOGRAPHIC
INFORMATION SCIENCE
...
2
THINKING ALOUD
ONE OF THE MANY METHODS OF USER RESEARCH
Method Similar or Related Methods
Expert-
based
heuristic evalua...
3
Important aspects of usability research
Requirements Analysis
Evaluate early design
Evaluate elaborate design
User feedb...
Analyse
Requirements
Produce Design
Solutions
Conceptual
Design
& Prototype
Final Design
& Prototype
Interim
Designs &
Pro...
 Distinction between qualitative and quantitative research
 Working with experts or with (representatives of) “real” end...
 Interviews / Focus Groups
 Surveys / Questionnaires
 Product analysis
 Card sorting
 Interaction study
 Observation...
asking subjects to execute a particular problem-solving
task and to speak out their thoughts while solving that
problem / ...
Some investigations into knowledge acquisition from existing (static) maps:
 Thorndyke & Stasz (1980)
 Kinnear & Wood a....
Investigations into the use of cartographic visualization systems for
geographical data exploration:
 McGuinness, van Wer...
 Form pairs
 Execute map reading activity:
Locate the Google StreetView image on the map fragment provided while
thinkin...
11
HANDS-ON EXPERIENCE
MAP 1: BAILEUX
12
HANDS-ON EXPERIENCE
MAP 2: FORGES
 What are your experiences with applying the think aloud method?
 What are the potential advantages?
 What are the disa...
 Leads to valid, unique, direct and the most complete data on cognitive
processes
 Least disturbance of the thought proc...
 Very time-consuming and tedious
 Protocol analysis difficult and subject to interpretation by researcher
 Protocols ma...
 Verbatim transcription of recordings: verbal protocols.
 Tools for qualitative data analysis
e.g. Atlas.ti or MacSHAPA
...
Keep your research questions in mind!
(but leave room to discover the unexpected ….)
PROCESSING & ANALYSIS OF THINK ALOUD ...
18
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
pr...
19
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
pr...
20
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
pr...
21
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
pr...
22
OVERVIEW OF PROTOCOL ANALYSIS
AFTER ERICSSON & SIMON (1993)
Theory
Coding
scheme
Task
analysis
Model of
geographical
pr...
INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION
Corné van Elzakker
The Use of Maps
in the
Explor...
24
Exploratory cartography
maps as tools
for discovery
25
Exploration of geographic data
This research project >
investigation into the use of maps:
the actual use of an existin...
26
User task: regional exploratory studies
gaining insight into the geography of an
unknown region
27
Example of a graphical model
constructed by a test person
28
Database of the Overijssel region
Available to the test persons:
 digital geographic (attribute and base map)
data, to...
29
Searching the Overijssel database
30
Model of regional exploratory problem-
solving: map selection matrix
31
Part C
32
Model of regional
exploratory
problem-solving:
map use matrix
33
Map use matrix (upper part enlarged)
34
Model of regional exploratory problem-
solving
35
Protocol analysis
36
Map use for model construction
37
Research methodology applied
 questionnaire
 thinking aloud
 video recording
 screen transaction logging
 thinking...
Thinking aloud is a very useful qualitative method to
discover use, user and usability issues ánd
requirements.
Most profi...
39
ADDED VALUE OF THINKING ALOUD IN EYE-TRACKING
FIND STREETVIEW IMAGE ON MAP CHIMAY
40
EFFECTS OF THINKING ALOUD ON ET RESULTS?
 Synchronization of recordings to facilitate analysis ...
Further practical hints will be provided by Kristien Ooms.
For ...
Thinking aloud is a very useful qualitative method to
discover use, user and usability issues ánd
requirements.
Most profi...
APPLICATION
0F THE
THINK ALOUD METHOD
IN
USE, USER AND USABILITY RESEARCH
IN
CARTOGRAPHY & GEOGRAPHIC
INFORMATION SCIENCE
...
Upcoming SlideShare
Loading in …5
×

2016 iccgis module4_thinking_aloud

306 views

Published on

Traning workshop on ‘Designing an conducting user studies”
Module 4 - Thinking Aloud (Corné van Elzakker)
@ ICC&GIS
June 15th, 2016
Albena, Bulgaria

Published in: Education
  • Be the first to comment

  • Be the first to like this

2016 iccgis module4_thinking_aloud

  1. 1. APPLICATION 0F THE THINK ALOUD METHOD IN USE, USER AND USABILITY RESEARCH IN CARTOGRAPHY & GEOGRAPHIC INFORMATION SCIENCE CORNÉ VAN ELZAKKER WORKSHOP DESIGNING & CONDUCTING USER STUDIES 6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016
  2. 2. 2 THINKING ALOUD ONE OF THE MANY METHODS OF USER RESEARCH Method Similar or Related Methods Expert- based heuristic evaluation rulesof thumb conformity inspection feature inspection, consistency inspection, standards inspection, guideline checklist cognitive walkthroughs pluralistic walkthroughs, prototyping, storyboarding, Wizard of Oz Theory- based scenario-based design personas, scenarios of use, use case, context of use, theatre secondary sources contentanalysis, competitive analysis automated evaluation automated interaction logs, unmoderated user-based methods User-based participant observation ethnographies, field observation, MILCs, journal/diary sessions, screenshot captures surveys questionnaires, entry/exit surveys, blind voting, cognitive workload assessment interviews structured interviews, semi-structured interviews, unstructured interviews, contextual inquiry focus groups supportive evaluation Delphi card sorting Q methodology, conceptmapping, aflnity diagramming, brainstorming talk/think aloud verbal protocol analysis, co-discovery study interaction study performance measurement,controlled experiments Source: Robert Roth
  3. 3. 3 Important aspects of usability research Requirements Analysis Evaluate early design Evaluate elaborate design User feedback Focus groups Brainstorm sessions Interviews Surveys & questionnaires Stakeholder meetings Literature review Card sorting ... Paper prototype Storyboarding Heuristic evaluation Automated evaluation Wizard of Oz Thinking aloud Eye tracking Card sorting Cognitive walkthrough .... Performance testing User satisfaction Heuristic evaluation Cognitive walkthrough Functional prototyping Critical incidents Logging user interactions .... Questionnaires User satisfaction Logging user interactions Observations Diary keeping ...
  4. 4. Analyse Requirements Produce Design Solutions Conceptual Design & Prototype Final Design & Prototype Interim Designs & Prototypes Identify the need for user-centred design Design satisfies requirements Evaluate Designs Testing with representative users input Business User StakeholderOrganisational Characteristics Preferences Use contexts Tasks 4 (OF GEOSPATIAL DATA PROCESSING AND DISSEMINATION SYSTEMS) Method = User-Centred Design Prototyping Requirement analysis Usability evaluation Techniques e.g.: Qualitative Focus groups Interviews Observation Thinking aloud Questionnaires Screen logging Eye tracking Performance an. Quantitative
  5. 5.  Distinction between qualitative and quantitative research  Working with experts or with (representatives of) “real” end-users  Doing research in the proper context (field-based vs. laboratory based research)  Distinction between methods & techniques of data collection and data analysis 5 IMPORTANT ASPECTS OF UUU RESEARCH
  6. 6.  Interviews / Focus Groups  Surveys / Questionnaires  Product analysis  Card sorting  Interaction study  Observation … COLLECTING USE, USER & USABILITY DATA ANOTHER LOOK AT CATEGORIES OF TOOLS
  7. 7. asking subjects to execute a particular problem-solving task and to speak out their thoughts while solving that problem / performing that task participants are asked to say out loud whatever they are looking at, thinking, doing and feeling at each moment thereafter: analyzing the resulting verbal and action protocols 7 THE THINK ALOUD METHOD
  8. 8. Some investigations into knowledge acquisition from existing (static) maps:  Thorndyke & Stasz (1980)  Kinnear & Wood a.o. (1987/1988)  Kulhavy, Pridemore & Stock (1992)  Crampton (1992)  Heidmann & Johann (1997)  Board (1997)  Ungar, Blades & Spencer (1997) EARLIER APPLICATIONS OF THE THINK ALOUD METHOD IN MAP USE RESEARCH
  9. 9. Investigations into the use of cartographic visualization systems for geographical data exploration:  McGuinness, van Wersch, Stringer (1993)  MacEachren, Boscoe, Haug, Pickle (1998)  See online bibliography ICA Commission on UUUI for more examples of case studies! EARLIER APPLICATIONS OF THE THINK ALOUD METHOD IN MAP USE RESEARCH
  10. 10.  Form pairs  Execute map reading activity: Locate the Google StreetView image on the map fragment provided while thinking aloud  Observer takes notes or records the thinking aloud  Switch roles with Map 2  After this, we will discuss your experiences 10 HANDS-ON EXPERIENCE MAP READING ACTIVITY IN PAIRS
  11. 11. 11 HANDS-ON EXPERIENCE MAP 1: BAILEUX
  12. 12. 12 HANDS-ON EXPERIENCE MAP 2: FORGES
  13. 13.  What are your experiences with applying the think aloud method?  What are the potential advantages?  What are the disadvantages? What could be the problems?  Questions? HANDS-ON EXPERIENCE WITH THINKING ALOUD MAP READING ACTIVITY IN PAIRS 13
  14. 14.  Leads to valid, unique, direct and the most complete data on cognitive processes  Least disturbance of the thought process  No memory errors  No interpretation, rationalization or explanation of thoughts by subjects  No need to bring thoughts in predefined form ADVANTAGES THINK ALOUD METHOD
  15. 15.  Very time-consuming and tedious  Protocol analysis difficult and subject to interpretation by researcher  Protocols may be incomplete  Verbalization problems  Situation may be unnatural for test participant DISADVANTAGES THINK ALOUD METHOD
  16. 16.  Verbatim transcription of recordings: verbal protocols.  Tools for qualitative data analysis e.g. Atlas.ti or MacSHAPA PROCESSING & ANALYSIS OF THINK ALOUD DATA TOOLS
  17. 17. Keep your research questions in mind! (but leave room to discover the unexpected ….) PROCESSING & ANALYSIS OF THINK ALOUD DATA ANALYSIS
  18. 18. 18 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
  19. 19. 19 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
  20. 20. 20 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
  21. 21. 21 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
  22. 22. 22 OVERVIEW OF PROTOCOL ANALYSIS AFTER ERICSSON & SIMON (1993) Theory Coding scheme Task analysis Model of geographical problem solving Predicted coded protocols Raw protocols Coded protocols Segmented protocols
  23. 23. INTERNATIONAL INSTITUTE FOR GEO-INFORMATION SCIENCE AND EARTH OBSERVATION Corné van Elzakker The Use of Maps in the Exploration of Geographic Data
  24. 24. 24 Exploratory cartography maps as tools for discovery
  25. 25. 25 Exploration of geographic data This research project > investigation into the use of maps: the actual use of an existing cartographic display plus the cognitive process of finding, selecting, retrieving, adjusting or generating 'private' map displays
  26. 26. 26 User task: regional exploratory studies gaining insight into the geography of an unknown region
  27. 27. 27 Example of a graphical model constructed by a test person
  28. 28. 28 Database of the Overijssel region Available to the test persons:  digital geographic (attribute and base map) data, to visualize within the ArcGIS environment  existing paper maps and atlases  existing digital maps (static, interactive or dynamic maps in different formats, ArcGIS ‘map documents’, maps on CD-ROM or on diskette and Web maps
  29. 29. 29 Searching the Overijssel database
  30. 30. 30 Model of regional exploratory problem- solving: map selection matrix
  31. 31. 31 Part C
  32. 32. 32 Model of regional exploratory problem-solving: map use matrix
  33. 33. 33 Map use matrix (upper part enlarged)
  34. 34. 34 Model of regional exploratory problem- solving
  35. 35. 35 Protocol analysis
  36. 36. 36 Map use for model construction
  37. 37. 37 Research methodology applied  questionnaire  thinking aloud  video recording  screen transaction logging  thinking aloud in retrospect  analysis of verbal protocols  analysis of action protocols  product analysis (graphical model)
  38. 38. Thinking aloud is a very useful qualitative method to discover use, user and usability issues ánd requirements. Most profitable use in a mixed methods approach (e.g. including video observation, survey to establish participants’ backgrounds, screen / action logging, etc.) Combining Eye-Tracking and Thinking Aloud: demonstration Kristien Ooms 38 CONCLUSION INCORPORATE THINK ALOUD IN MIXED METHODS APPROACH
  39. 39. 39 ADDED VALUE OF THINKING ALOUD IN EYE-TRACKING FIND STREETVIEW IMAGE ON MAP CHIMAY
  40. 40. 40 EFFECTS OF THINKING ALOUD ON ET RESULTS?
  41. 41.  Synchronization of recordings to facilitate analysis ... Further practical hints will be provided by Kristien Ooms. For now... 41 IMPLEMENTATION ISSUES THINKING ALOUD IN MIXED METHODS APPROACHES
  42. 42. Thinking aloud is a very useful qualitative method to discover use, user and usability issues ánd requirements. Most profitable use in a mixed methods approach. 42 CONCLUSION THINK ALOUD METHOD
  43. 43. APPLICATION 0F THE THINK ALOUD METHOD IN USE, USER AND USABILITY RESEARCH IN CARTOGRAPHY & GEOGRAPHIC INFORMATION SCIENCE c.vanelzakker@utwente.nl WORKSHOP DESIGNING & CONDUCTING USER STUDIES 6TH ICC & GIS, ALBENA, BULGARIA, 15 JUNE 2016

×