2. Roots in psychological research
Developed from introspection method
K. Anders Ericsson
and Herbert A. Simon
(1980)
Clayton Lewis , IBM, 1982
3. A qualitative methodology
An observational method/protocol analysis
Gathers rich, in-depth data from small sample
Ericcson and Simon’s definition “Participants are
asked to verbalize their
thoughts, opinions, decisions, likes and dislikes
as they work on a task.”
Measures not only what the participant is doing
but also what they are thinking and feeling
“Eavesdropping on someone's thinking"
Testing the process, not the user
5. Subject signs consent form
Individual session in a quiet setting
Entire session is audiotaped/videotaped then
transcribed
Subject instructed to think aloud constantly
If necessary, the investigator reminds subject
to “keep thinking aloud”
Investigator may take notes during session
A practice session
Brief follow up interview if necessary
Collect demographic data
8. Data transcribed by someone familiar with
clinical language and terminology
Fillers eliminated from transcript
Three steps of analysis
Referring phrase analysis
Assertional analysis
Script analysis
9. Underline nouns and noun phrases
Phrases are coded and defined
Another researcher analyzes random portions for
agreement, identify discrepancies
Experts in the area evaluate accuracy of codes
All verbal data is accounted for, nothing in
transcript is left uncoded
10. Identify set of assertions (confident and forceful
statements of facts)
Determine how relationships form between
concepts
Identify purpose of assertions
11. • What information subjects attended to
• How they structured the problem
• Rationale for choices and decisions
• Plan for problem resolution
13. In a physician’s office
Chess match
Wait staff
On a date
14. Quick way to obtain qualitative data in short time
In a usability study, helps identify problems with
a product and user’s misconceptions
Sequential method of use
In-depth, unbiased detail
Data from independent user’s point of view
Real time documentation
User does not transcribe data so can just think
Is recorded to be used for inter-observer results
and for further analysis
15. Slow, laborious process of analysis
Small sample
Communication
“Split brain” especially for children
Awkward, distracting
Difficult to simulate real life environment
Skilled participants
16. Disturbance of cognitive process
Memory errors
Interpretation by subject
Incomplete due to synchronization problems
Technical glitches
17.
18. Ericsson, K. A. , Simon, H. A. (1998) How to study thinking in everyday
life: Contrasting think-aloud protocols with descriptions and
explanations of thinking. Mind, Culture and Activity 5(3). 178-186.
Fonteyn, M.E . (1993) A description of think aloud method and
protocol analysis. Qualitative Health Research November 1993 3(4).
430-441.
Novotny, E. (2004) I don’t think I click: A protocol analysis study of use
of a library online catalog in a digital age. College & Research Libraries
November 2004 65(6). 525-537
Van Someren, M.W. et al (1994). The Think Aloud Method, University
of Amsterdam, Academic Press, London
YouTube The Think Aloud protocol,
http://www.youtube.com/watch?v=vgdZz49Y1kw
Make sure the sound is working and is turned up for the video clip.
The think aloud protocol traces its roots to psychological researchIt was developed from the introspection method – from observing events that take place in the consciousnessTwo pioneer s of think aloud protocol are K . Anders Ericcson from the University of Florida and Herbert A Simon from Carnegie Mellon University.In 1980, they published their first paper on the idea.In 198, Clayton Lewis from IBM used a think aloud protocol for usability studies.
What is the think aloud protocol?According to our textbook, the TAP isA qualitative research methodologyAn observational method/protocol or analysisIt gathers rich, in-depth information from a small sampleEriccson and Simon described the think aloud protocol as “participants are asked to verbalize their thought, opinions, decision, likes and dislikes as they work on a task.”The Think aloud protocol measures not only what the participant is doing but also what they are thinking and feelingIt has been referred to as “eavesdropping on someone's thinking.Remember, We are testing the process, not the user
There are two different types of thinking aloudThe first is concurrent – thinking aloud while performing the taskThe other is retrospective, sometimes called “think after” – the subject watches a video of themselves performing a task and then tries to recall what they were thinking and feeling at the time.We will focus tonight on the concurrent type of think aloud protocol.
How does this work?The collection process involves having the subject sign a consent form The collection process occurs as an individual session in a quiet settingThe entire session is audiotaped or videotaped and then transcribed.The subject is instructed to think aloud constantlyIf necessary, the investigator quietly reminds them to Keep thinking aloud” This should be their only interaction with the subject. The investigator may take notes during the session and indicate any areas that need clarificationA trial run or warm up exercise or practice session is used to get the subject used to talking aloud and comfortable with the level of detail required.There is a brief follow pup interview if needed to clarify thinking and identify reasoning strategiesFinally, demographic information is collected
I found a short video that demonstrates the process
After the data is collected, it is coded and analyzed.The data is transcribed by someone who is familiar with the clinical language and terminology used.Filler words, “ah, um, “ are eliminated from the transcriptThe remaining words undergo three steps of analysis – Referring analysisAssertional analysisScript analysos
In referring phrase analysisAll of the nouns and noun phrases are underlinedPhrases are defined and coded Another researcher analyzes random portions for agreement and identifies any discrepanciesExperts in the subject area evaluate the accuracy of the codesAll verbal data is accounted for, nothing in the transcript is left uncoded.
In assertional analysis, the confident and forceful statements of fact, called assertions are identifiedIt is determined how relationships form between conceptsThe purpose of assertions is identified.
Script analysis studies what information the subjects attended toHow they structured the problemThe rationale for their choices and decisionsAnd a plan for problem resolution, if there is one.
The Think aloud protocol can be used hereIn libraries to test new software or technology such as the online catalogue , library website, or databasesIn corporation s for usability studies for new products, methods, services, or databases. My daughter did a Think aloud protocol for Amazon.com in which she evaluated the ease of use of an online shoe website. She enjoyed it and got a $25 gift care from AmazonTeachers use the think aloud protocol especially for reading or math The Think aloud method can be used for health care workers in a controlled environment to determine problem solving as it relates to patient care– if this happened, what would you do?Knowledge engineers use it to integrate knowledge into their computer system.I read about a professional actor who used this method to understand a script on a far deeper level than would be provided by a detailed understanding of the story line. He was given a tape recorder and instructed to verbalize his thoughts as they came to him while working on a script. He first focused on structure, punctuation, and linguistic devices in the script to determine the character's internal states (e.g., intentions and motivations) and then focused on specific performance characteristics (e.g., speech and movement patterns) to turn it into a living conversation.
And these would be some places not to use the think aloud protocolIf while examining a patient a doctor voiced all of the possible diagnoses and treatments, the patient would be overwhelmed and maybe terrifiedDuring a chess match, if you think aloud you would give your opponent an advantageAt a restaurant where wait staff remember your order, you would probably not want to know the devices to do so – the large man in the stripes would like to have the pasta Alfred – that is the last thing he should be ordering, etc.On a date it is best not to verbalize everything that comes to mind.
Strengths of the Think aloud methodA quick way to obtain qualitative data in a short timeIn a usability study, it helps identify problems with a product and the user’s misconceptionsStep by step sequential method of use of the product by the userProvides in depth unbiased detailData is from an independent user’s point of viewReal time documentationThe user does not transcribe data so they can just be free to thinkIt is recorded and can easily be used for inter-observer results as well as used for further analysis by investigator
Weaknesses of the think aloud methodA slow, laborious process of collecting dataIt is usually done on a small sample, is this representative?It is difficult for children and non native language speakers to communicateIt requires “split brain” – the ability to talk while doing. This may be difficult especially for children who have trouble dong both things at once.It is awkward, not naturalIt is difficult to simulate a real life environmentDifficult to get participants esp. if a certain skill level is needed.
Problems that threaten validityCognitive disturbance – forget to think aloud, especially if it is one you do automatically; if you repeat it, you may do it differentlyMemory errors – incomplete or false recallInterpretation by subject may be different than plannedProcess may be incomplete due to synchronization – takes longer when thkning aloudTechnical glitches – failure
If you ‘d like, you can friend the think aloud protocol on Facebook
