This document discusses methods for collecting data from prototypes to test design features. It describes collecting learner responses to questions, outputs from design activities, and automated recording of learner actions. Observation and stimulated recall interviews can help interpret data. The document presents a study that automatically captured video, audio, screen actions, writing, and eye movements to analyze user response. Tools are recommended for coordinating and analyzing multiple data streams.

1. Testing the prototypes
Diana Laurillard for OLDSMOOC
Index
1. Testing and data collection from prototypes
2. Selected types of data collection and their features
3. Interpreting observation data
4. A study of a design that used automatic data capture
5. Data capture and analysis tools
6. Activity: Observation 1
7. Activity: Observation 2
Play slides or use these hyperlinks to navigate to sections.
Notes pages show any relevant bibliographic references
1

2. Testing and collecting data
• Some of the prototyping techniques we’ve looked at provide their own
means of collecting learner data on their response to a design – e.g.
responses to a ppt visualisation collected in the Notes page in answer to
specific questions as in the ‘Storyboarding functionality’ above.
• Ways of collecting data on learner responses depend on the technique
being used, and on what you want to find out.
• They can all be a basis for further follow-up with an interview to find out
more about learner perceptions and ways of conceptualising the task and
what they are trying to learn.
Menu 2

3. Selected types of data collection
Learners’ answers to questions on specific design issues
can collect from several learners at once; elicits their short text-
based reflections
Learners’ outputs on activities based on a design
can collect from several learners at once; elicits their perceptions of
the task; provides a detailed record for analysis
Program monitoring/recording of learner actions
automates data collection; may need observation as well to assist
interpretation; provides a detailed record for analysis
Notes/recording of pairs’ discussions
a researcher can only listen to and record one pair at a time; more
natural than one person thinking aloud; useful to record as well; link
to program monitoring of learner actions
Observation of learner actions
a researcher can only observe and record one person or group at a
time; may need program monitoring or video capture to assist
interpretation; provides a detailed record for analysis
Menu 3

4. Interpreting observation data
Interviews with the student after they have worked through a
prototype can be very useful to help with your interpretation of what
they were thinking about, or how they interpreted the task or
presentation.
Some researchers use a ‘think-aloud’ technique, asking the learner to
talk about their reactions as they go. This can be useful, except when
the task absorbs the learner’s attention, and then they stop talking.
A good alternative is to use ‘Stimulated recall’ (Bloom1953)
• the researcher notes the critical incidents or interesting moments in
the learner’s actions
• then returns to that part of the software in the interview and asks
‘what were you thinking at this point?’
• this can often elicit quite detailed accounts of how they were
interpreting the visuals, or the task – or, of course, why they might
have been confused or uncertain about what to do.
Menu 4

5. A study of a design: automatic data capture
• Data collection was carried out on 18 students with A-
level Maths or higher
• They were observed doing tasks involving interpretations
of prototype mathematical and graphical displays
• Each task was presented in either static, dynamic or
interactive forms
• Data capture was automatic:
– video and audio of the interaction
– real-time capture of work on a tablet
– recording of actions on the screen
– eye-tracking of eye movements across the screen.
Menu 5

6. These techniques generate copious data!
Video What the
actions, ges learner sees
tures, talk on screen
Fixation
Real-time
writing on Saccade
the Tablet PC
Allows detailed analysis of user response to interface and interactivity
Important to know what you’re looking for – what’s the hypothesis?
6

7. Relating talk to action
Participant’s talk: “This is going from minus two…”
When you are involved in different kinds of data capturing techniques, there is some
fine-grained observational data that’s not possible to record with just traditional
observation. In this figure and the next you can see the eye-movement following the
shapes in a particular area, with individual eye movements building up into a
coherent picture. This process is evident from the eye tracking data because you can
re-construct the saccades.
7

8. Analysing focus of attention
Interesting that the focus here
is much more on the numbers
than on the graphs
8

9. Comparing attention paid to
representations
Contrasting the data from
different students – it’s
important to know what you’re
looking for – are these similar
because they all focus on
numbers, or different because
they spend more time (larger
circles) on different elements?
9

10. Data capture and analysis tools
Try these if you want to do
this kind of data capture
and analysis
To coordinate and analyse
the multiple streams of data
To capture eye
movements, logged
interactions, and for eye-
tracking data analysis To capture screen activities
Menu 10

11. Typical Activity: Observation 1
Set Goal: To locate a specific document
Describe Task: Find and open the article X
Suggest Tool: A course Moodle site
Observation actions:
• take notes of all actions, timings and any talk
• refer back to an interesting moment and ask what
they were thinking about at that moment, or why
they took that action
• would structured notes help?
Menu 11

12. Typical Activity: Observation 2
Set Goal: Use an online tool to complete a task
Describe Task: Add a new page to the wiki, paste
in the link to an online resource, and add a short
comment
Suggest Tool: A course Moodle site
Observation actions:
• take notes of all actions, timings and any talk
• refer back to an interesting moment and ask what
they were thinking about at that moment, or why
they took that action
• would structured notes help?
Menu 12