Talk at EC-TEL 2014 presenting joint work from Manolis Mavrikis, Beate Grawemeyer, Alice Hansen, and Sergio Gutierrez-Santos. London Knowledge Lab.

1. Exploring the Potential of Speech
Recognition to Support Problem
Solving and Reflection
Wizards go to school in the elementary maths classroom
Manolis Mavrikis, Beate Grawemeyer, Alice
Hansen, and Sergio Gutierrez-Santos
London Knowledge Lab

2. Aim
• Enabling natural interaction with a learning
platform through multimodal interaction -
speech.
• Wizard-of-Oz study to
– Collect realistic data
– Inform the design of the system
– explore children’s inner speech
– reflection as a learning mechanism

3. Voice interaction for learning
• Educational research has shown benefits of verbalization of learning
(1,2,3).
• More natural and efficient form of communication will have
positive learning gains (4).
• Verbal communication in learning mathematics is important, as it is
similar to learning a foreign language.
• Self-explanation can be beneficial for learning (5) as it can be seen
as a tool to address student’s own misunderstandings.
• Speech can also provide cues for drawing inferences on students
emotions and attitude toward the learning situation

4. Wizard-of-oz study

5. (3) (4)
wizard
student
(1)
(2) (6)
(5)
(5)
WOZ Setup

6. Ecological validity

7. Ecological validity

8. Wizard-of-oz study

9. Wizard-of-oz study
• Overall 60 students aged between 9 – 10 took
part in a series of sessions with the iTalk2learn
platform.
• 12/10 Students were wizarded for around 30
minutes.
• The students that were wizarded answered a
questionnaire and took part in a focus group.

10. Fractions Lab – exploratory learning
environment

11. Fractions Lab – exploratory learning
environment

12. Wizard’s script for support

13. • Talk aloud
Feedback types
• “Remember to talk aloud, and tell us what are you thinking”
• “What is the task asking you to do?”
• “Please think aloud, what are your thoughts or feelings?”
• Talk maths
• “Can you explain that again using the terms denominator,
numerator?”
• Affect
• “It may be hard, but keep trying”
• “If you find this easy, check your work and change the task”
• Problem solving
• “You can’t add fractions with different denominators”
• Reflection
• “What did you learn from this task?”
• “What do you notice about the two fractions?”

14. Wizard of Oz results
• 170 messages were sent to 10 students
80
70
60
50
40
30
20
10
0
Affect Talk aloud Talk maths Problem
solving
Reflection

15. Wizard of Oz results
• Was the feedback immediately related to what
the student said?
80
70
60
50
40
30
20
10
0
Affect Talk maths Problem
solving
Reflection
No
Yes

16. Wizard of Oz results
• Student reacted?
80
70
60
50
40
30
20
10
0
Affect Talk aloud Talk maths Problem
solving
Reflection
No
Yes

17. Wizard of Oz results
• No significant correlation between the
provision of feedback immediately after what
the student said, and student’s reaction
(r=.18, p>.05).
• However, there was a significant correlation if
problem solving support was provided (r=.16,
p<.05).

18. Questionnaire
5
4
3
2
1
helpful repetitive understand like think aloud

19. Discussion
• “Now you are talking”
– An ecological valid setting is important.
• “Sarcasm doesn’t become you”
– Voice production is important.
• “Can the headsets look in my head?”
– Students attribute higher abilities to the system.
• “Hmm – let me rephrase that “
– Prompting students to use mathematic vocabulary might help to
think more carefully what they are saying.
• “Talking the talk”
– Individual differences.

20. Conclusion
• The results indicate that there is potential in
expecting young students to think-aloud while
interacting with educational technology.
• Even rudimentary ASD has the potential to
support basic reflection on the learning task and
of domain terminology.
• Future work includes identifying how interruptive
the messages are and carefully selecting timings
on reflective prompts.

21. References
1. Askeland, M.: Sound-based strategy training in multiplication. European
Journal of Special Needs Education 27(2), 201-217 (2012)
2. Borasi, R., Siegel, M., Fonzi, J., Smith, C.: Using transactional reading
strategies to support sense-making and discussion in mathematics
classrooms: An exploratory study. Journal of Research in Mathematics
Education 29, 275-305 (1998)
3. Mercer, N., Sams, C.: Teaching children how to use language to solve maths
problems. Language and Education 20(6), 507-528 (2007)
4. Mayer, R.E., Moreno, R.: Nine ways to reduce cognitive load in multimedia
learning. Educational Psychologist 38(1), 43-52 (2003)
5. Chi, M.: Self-explaining expository texts: The dual processes of generating
inferences and repairing mental models. In: Glaser, R., (ed.) Advances in
Instructional Psychology, 161-238 (2000)