This document discusses the implementation of principles of multimodal processing and learning for education. It outlines research on using animated tutors for vocabulary learning and describes studies evaluating the effectiveness of tutoring for hard of hearing children, autistic children, and English language learners. It also introduces a Lesson Creator application that allows teachers and parents to build personalized lessons.

1. Just in Time Learning
Implementing Principles
of Multimodal Processing
and Learning for
Education

2. Outline
 Introduction
 Animated Tutors
 Vocabulary Learning
 Direct Vocabulary Instruction
 Graded Word Information
 Need For Language Tutoring
 Effectiveness of Tutoring
 Hard Of Hearing Children
 Autistic Children
 English Language Learners
 Lesson Creator
 Errorless Learning
 Building Lessons with the Lesson Creator
 Example Lesson
 Evaluation

3. Introduction
 To provid both a theoretical and empirical framework
for addressing the important issue of the presentation of
multimodal displays to the user
 The major assumption is that multiple sensory
influences are continuously combined during perception,
categorization, and language processing
 Results from a variety of experiments on speech,
emotion, and gesture have supported this type of
framework for language processing

4. Animated Tutors
 Baldi, a 3-D computer-animated talking head
 The goal in this paper is to review applications in language
tutoring using this technology, propose a user-friendly
Lesson Creator that can be used to implement research and
tutoring applications, and suggest how it can be evaluated
more formally
 Several advantages of our technology and pedagogy
 the popularity of interactive agents
 instruction can be tailored exactly to the student’s need
 the ability to seamlessly meld spoken and written language
 provide a semblance of a game-playing experience while actually
learning

5. Vocabulary Learning
 vocabulary knowledge is positively correlated with both
listening and reading comprehension and predicts
overall success in school
 Direct Vocabulary
Instruction
 Graded Word
Information
 Need For Language
Tutoring

6. Direct Vocabulary Instruction
 some direct teaching of vocabulary is essential for
appropriate language development
 Contrary to a common belief that learning vocabulary is
a necessary outcome of reading in which new words are
experienced in a meaningful context, context seldom
disambiguates the meaning of a word completely

7. Graded Word Information
 Knowing a word is not an all-or-none proposition
 the dimension of vocabulary depth (as measured by
synonymy, polysemy, and collocation) is as important
as that of vocabulary size in predicting performance on
academic reading
 Thus, it is important to over-train vocabulary, and to
present the items in a variety of contexts in order to
develop rich representations

8. Need For Language Tutoring
 The need for language tutoring is pervasive in today’s world
 There are millions of elementary school children who face
language and speech challenges
 All of these groups have or are at high risk for language
learning disabilities and require additional instruction in
language learning.
 there are not enough skilled teachers and professionals to give them
the one-on- one attention that they need
 Other available resources are not easily personalized to the
students’ needs, lack the engaging capability of a teacher
 we developed a Vocabulary Tutor to provide direct
instruction in vocabulary learning (3)

9. Effectiveness of Tutoring
 Several evaluation experiments with the Vocabulary
Tutor have carried out with three different groups of
children with language challenges: hard of hearing,
autistic, and English learners
 Hard Of Hearing Children
 Autistic Children
 English Language Learners

10. Hard Of Hearing Children
 eight children with hearing loss, who needed help with
their vocabulary building skills as suggested by their
regular day teachers
 The experimenter developed a set of lessons with a
collection of vocabulary items that was individually
composed for each student.
 Each collection of items was comprised of 24 items,
broken down into 3 categories of 8 items each.
 Three lessons with 8 items each were made for each
child

11. Hard Of Hearing Children (cont.)
 the results from one student in this study, because of the
value of single-student analyses and the fact that this
student was representative of all of the students tested

12. Hard Of Hearing Children (cont.)

13. Autistic Children
 evaluating vocabulary acquisition, retention and
generalization in eight children diagnosed with autism,
ranging in age from 7-11 years
 The results indicated that the children learned many new
words, grammatical constructions and concepts, proving
that the application provided a valuable learning
environment for these children.
 In addition, a delayed test given more than 30 days after the
learning sessions took place showed that the children
retained over 85% of the words that they learned

14. Autistic Children (cont.)
 a second investigation used the single subject multiple
probe design
 Given training, all of the students attained our
criterion(accurately identify at least 5 out of 6
vocabulary items) for identification accuracy for each
word set and were also able to generalize accurate
identification to four instances of untrained images
 These results show that our tutoring program is effective
for autistic children, as well for hard of hearing children

15. English Language Learners
 Nine children ranging in age from 6-7 years were tested
in the summer before first grade
 Three different lessons were tested, corresponding to the
three sets of items used in the multiple baseline design
 Each child was pretested in order to find vocabulary that
was unknown to him or her
 The test session: two select one, No feedback
 A training session: error correct, Imitation

16. Lesson Creator
 A new application, the Lesson Creator, adds flexibility
and many new pedagogical features
 Teachers, parents, and even students with minimal
computer experience can build original lessons with
personalized vocabulary and pictures
 Errorless Learning
 Building Lessons with the Lesson Creator
 Example Lesson
 Evaluation

17. Errorless Learning
 A giraffe is an animal with a long neck
 An elephant is an animal with a long trunk
 With this type of supportive and corrective feedback, the
student learns about both animals, and is encouraged to
think about their differences

18. Building Lessons with the Lesson Creator
 We know that educational instructors are overworked, and it
is a challenge to ask teachers to create new lessons for their
students rather than simply using existing content
 Lesson Creator was developed to provide a highly flexible
framework to include a variety of content while
simultaneously providing a user-friendly interface to design,
create, and pilot test the lesson
 Each screen allows the coach to specify the greeting and
instructions, the questions, and the feedback, or the coach
can simply accept the general default dialog

19. Building Lessons with the Lesson Creator

20. Example Lesson

21. Evaluation
 First, although user feedback and testimonials have their
limitations, they provide an initial source of the
application’s effectiveness
 There appear to be two approaches to formal tests
The first would be to use a think-aloud protocol to monitor both
novice and experience users while they are composing lessons
The second would be to ask teachers to think of a lesson that
they would like to implement in an interactive tutoring situation

22. Think-Aloud Protocol
 Users are asked to say whatever they are looking at, thinking,
doing, and feeling, as they go about their task.
 This enables observers to see first-hand the process of task
completion (rather than only its final product).
 Observers at such a test are asked to objectively take notes of
everything that users say, without attempting to interpret their
actions and words.
 Test sessions are often audio and video taped so that developers
can go back and refer to what participants did, and how they
reacted.
 The purpose of this method is to make explicit what is implicitly
present in subjects who are able to perform a specific task