OLDSMOOC Week5 part 2: Testing the prototypes. Diana Laurillard
Testing the prototypes Diana Laurillard for OLDSMOOCIndex1. Testing and data collection from prototypes2. Selected types of data collection and their features3. Interpreting observation data4. A study of a design that used automatic data capture5. Data capture and analysis tools6. Activity: Observation 17. Activity: Observation 2Play slides or use these hyperlinks to navigate to sections.Notes pages show any relevant bibliographic references 1
Testing and collecting data • Some of the prototyping techniques we’ve looked at provide their own means of collecting learner data on their response to a design – e.g. responses to a ppt visualisation collected in the Notes page in answer to specific questions as in the ‘Storyboarding functionality’ above. • Ways of collecting data on learner responses depend on the technique being used, and on what you want to find out. • They can all be a basis for further follow-up with an interview to find out more about learner perceptions and ways of conceptualising the task and what they are trying to learn.Menu 2
Selected types of data collection Learners’ answers to questions on specific design issues can collect from several learners at once; elicits their short text- based reflections Learners’ outputs on activities based on a design can collect from several learners at once; elicits their perceptions of the task; provides a detailed record for analysis Program monitoring/recording of learner actions automates data collection; may need observation as well to assist interpretation; provides a detailed record for analysis Notes/recording of pairs’ discussions a researcher can only listen to and record one pair at a time; more natural than one person thinking aloud; useful to record as well; link to program monitoring of learner actions Observation of learner actions a researcher can only observe and record one person or group at a time; may need program monitoring or video capture to assist interpretation; provides a detailed record for analysisMenu 3
Interpreting observation data Interviews with the student after they have worked through a prototype can be very useful to help with your interpretation of what they were thinking about, or how they interpreted the task or presentation. Some researchers use a ‘think-aloud’ technique, asking the learner to talk about their reactions as they go. This can be useful, except when the task absorbs the learner’s attention, and then they stop talking. A good alternative is to use ‘Stimulated recall’ (Bloom1953) • the researcher notes the critical incidents or interesting moments in the learner’s actions • then returns to that part of the software in the interview and asks ‘what were you thinking at this point?’ • this can often elicit quite detailed accounts of how they were interpreting the visuals, or the task – or, of course, why they might have been confused or uncertain about what to do.Menu 4
A study of a design: automatic data capture • Data collection was carried out on 18 students with A- level Maths or higher • They were observed doing tasks involving interpretations of prototype mathematical and graphical displays • Each task was presented in either static, dynamic or interactive forms • Data capture was automatic: – video and audio of the interaction – real-time capture of work on a tablet – recording of actions on the screen – eye-tracking of eye movements across the screen.Menu 5
These techniques generate copious data! Video What the actions, ges learner sees tures, talk on screen Fixation Real-time writing on Saccadethe Tablet PC Allows detailed analysis of user response to interface and interactivity Important to know what you’re looking for – what’s the hypothesis? 6
Relating talk to action Participant’s talk: “This is going from minus two…”When you are involved in different kinds of data capturing techniques, there is somefine-grained observational data that’s not possible to record with just traditionalobservation. In this figure and the next you can see the eye-movement following theshapes in a particular area, with individual eye movements building up into acoherent picture. This process is evident from the eye tracking data because you canre-construct the saccades. 7
Analysing focus of attention Interesting that the focus here is much more on the numbers than on the graphs 8
Comparing attention paid to representations Contrasting the data from different students – it’s important to know what you’re looking for – are these similar because they all focus on numbers, or different because they spend more time (larger circles) on different elements? 9
Data capture and analysis tools Try these if you want to do this kind of data capture and analysis To coordinate and analyse the multiple streams of data To capture eye movements, logged interactions, and for eye- tracking data analysis To capture screen activitiesMenu 10
Typical Activity: Observation 1 Set Goal: To locate a specific document Describe Task: Find and open the article X Suggest Tool: A course Moodle site Observation actions: • take notes of all actions, timings and any talk • refer back to an interesting moment and ask what they were thinking about at that moment, or why they took that action • would structured notes help?Menu 11
Typical Activity: Observation 2 Set Goal: Use an online tool to complete a task Describe Task: Add a new page to the wiki, paste in the link to an online resource, and add a short comment Suggest Tool: A course Moodle site Observation actions: • take notes of all actions, timings and any talk • refer back to an interesting moment and ask what they were thinking about at that moment, or why they took that action • would structured notes help?Menu 12
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.