Literature Review

1. Multimedia Technology and Second Language Vocabulary Acquisition
Mia Howard
Hunter College
May 19, 2009

2. In the field of Second Language learning vocabulary acquisition is an essential
component in the student’s mastery of the language being studied. With the increased use
of multimedia programs in schools, many programmers are faced with the added
challenge of creating programs that are effective for a broad range of students with
different learning styles. One of the most fundamental questions being explored in
educational research is: Which presentation mode is most effective for vocabulary
acquisition? It is hypothesized that the use of multimedia programs that incorporates the
use of word annotations with different types of information such as pictures and videos
enhances vocabulary acquisition. An annotation in multimedia software can be in the
form of an image, video, audio recording, or text, which provides a description or
definition of the highlighted word or passage. The following literature reviewed for this
paper attempts to demonstrate and support this hypothesis with conflicting positions on
which presentation mode is more beneficial for the learner.
In a research article by Chun & Plass (1996), two specific questions were
addressed for guiding the study. First, how well is vocabulary learned “incidentally”
when the purpose of the task is reading comprehension? Second, what is the
effectiveness of annotations with different media types for vocabulary acquisition? The
focus of the investigation was second-year German student’s foreign language
vocabulary acquisition when viewing a multimedia German text with annotations for
words in the form of text, pictures and videos. Each student was subsequently given a
vocabulary test to assess the learning outcomes. In regard to incidental vocabulary
learning, previous research suggests that second language learners are not likely to
2

3. remember words that appear only once within the text. As for vocabulary acquisition
with the use of words coupled with images or videos, research suggests that words
associated with actual objects or imagery is learned more easily. “The results showed a
higher rate of incidental learning than expected (25% accuracy on production tests, 77%
on recognition tests), significantly higher scores for words that were annotated with
pictures + text than for those with video + text or text only, and a correlation between
looking up a certain annotation type and using this type as the retrieval cue for remember
the word.” (Chun & Plass, 1996, p.183) An explanation for the higher percentage of
vocabulary recall can be attributed to dual coding effect. “According to dual coding
theory, learning of a vocabulary item is best when both visual and verbal information are
presented.” (Chun & Plass, 1996, p. 189)
The research findings in this study support the proposed hypothesis that the use of
multimedia programs that incorporates the use of word annotations with different types of
information such as pictures and videos enhances vocabulary acquisition. One limitation
to the study is that individual cognitive learning styles were not investigated. “For
example, pictures may not be more useful than definitions for all learners but might help
visualizers, whereas definitions might improve the learning of verbalizers.” (Chun &
Plass, 1996, p. 195) Next, the topic of foreign language vocabulary acquisition will focus
on the benefit of video annotations versus still picture annotations.
In a published journal article by Al-Seghayer (2001), the research study was
guided by the question, which is more effective for facilitating vocabulary acquisition:
3

4. video mode or static picture? Al-Seghayer (2001) hypothesized that video is a more
effective tool to foster the acquisition of new words in a foreign language. Participants in
the study consisted of 30 ESL students (17 males, 13 females) who were enrolled in the
English Language Institute at the University of Pittsburgh. Al-Seghayer (2001) provided
a brief overview of previous research on second language vocabulary acquisition
enhanced by multimedia annotations. With a focus on three issues: the value and effect
of multimedia annotations, the effect of electronic glossing, and the efficacy of dynamic
videos and static pictures. The research reviewed on multimedia annotations focused on
research studies that focused on modes that presented imagery. Some findings suggested
that dual coding of vocabulary words better facilitates learning and different types of
media furthers richness of recall cues and boosts the likelihood of retention. “The mean
and percentage of correct answers for words with video and text annotations were 4.3
(87%), compared to 3.3 (67%) for words with picture and text, and 2.7 (53%) for text
alone” (Al-Seghayer, 2001). The overall results on both the recognition and production
vocabulary tests show that words presented with both the printed text and the video clips
produced the best results.
The results of this study support the researcher’s hypothesis that video is a more
effective tool to foster the acquisition of new words in a foreign language. Although the
participants recalled more words when the video clips were provided than when pictures
or just text was present, I still question what level of retention was achieved. The
researchers’ choice not to administer a delayed post test limits the findings to temporary
recall. There is a need to access whether different annotation modes facilitate a deeper
4

5. level of learning and retention that a student will need to become fluent in a second
language. Other limitations to this study are the small sample size which hinders the
validity of the results and does not allow us to generalize the results. Al-Seghayer (2001)
admits that alternate assessment techniques are needed to tap various aspects of
vocabulary knowledge (Al-Seghaver, 2001, p. 217). In addition, the use of multiple
choice questions as well and the questionnaire and survey are not ideal measurements of
vocabulary retention. The research thus far has not explored the use of auditory
annotations. The next study will incorporate research on the effectiveness of text plus
picture and sound as well as investigate the effects of different learning styles.
Researchers at the National Tsing Hua University, Yeh & Wang (2003) focused
their study on two topics; one goal was to investigate the effectiveness of three types of
vocabulary annotations on vocabulary learning for English Foreign Language students in
Taiwan as well as to determine whether learners with different perceptual learning styles
benefit more from a particular type of annotation. Perceptual learning styles refer to
students, “… different sensor preferences for processing information (Kinsella, 1995).”
(as cited by Yeh & Wang, 2003, p. 133). A student who tends to process information
through listening is categorized as an auditory learner and learners who prefer to process
information by reading printed text are labeled as visual learners. The research
participants were 82 freshmen from the Department of Material Science and Technology
and the Department of Chemistry at the Nation Tsing Hua University in Taiwan.
Students were given two questionnaires and two tests before and after using the
multimedia courseware. The results of the post-tests indicated that the students who
5

6. randomly received the courseware with text + picture annotations score higher than those
who did not, indicating that text + picture annotations was most effective for vocabulary
learning. To answer the second research question, that asked if perceptual learning styles
influence the effectiveness of annotations each student was given a self-report
questionnaire and categorized into groups according the their scored responses on the
questionnaire. The results suggest that there seems to be little correlation between three
versions of the courseware and the learning styles. With the exception of the
auditory/visual group scoring significantly higher than those who received the text-only
version, no other group showed any correlation between perceptual styles and the
different vocabulary annotations. These findings fall in line with the premise of previous
research that supports the hypothesis that text + visually annotations are most effective in
multimedia second language learning. As mentioned earlier, this study explored the
effect of dual annotations (text + picture) plus audio annotation. The students who
received the courseware with text + picture + auditory proved to be ineffective. The
researchers attributed this to the learning styles used in learning Chinese as opposed to
English, the speech rate of the recorded annotations and the amount of time students were
given to complete the tasks.
Although findings in this study supported the overall hypothesis that text +
picture annotations are most effective in the field of second language vocabulary
acquisition, there were several limitations to the study. The researchers attempted to
investigate the concept of auditory annotations which had not been widely researched
before but failed to execute the study properly. Failure to give participants adequate time
6

7. to complete the tasks as well as presenting recorded audio at a speech rate that exceeded
the users listening proficiency limited the study from providing sound research on the use
of audio annotations. Although the research did not further our understanding of auditory
annotations, the findings will help shape future studies. Thus far the research reviewed
has been conducted on college level participants. The next study explores the use of
different vocabulary annotations in multimedia programs designed for grade school
children studying a foreign language.
In a research study by Acha (2009, p.25), the following research question was
proposed, which presentation mode of a multimedia program will help in learning more
vocabulary in an unknown language: one with pictures, one with words or one combining
both representations? The research participants were clearly stated as such: “…135
Spanish children (67 female, 68 male), 66 in Grade 3 (age M = 8 years) and 69 in Grade
4 (age M = 9 years).”(Acha, 2009) These children were from Vizcaya and Guipuzcoa,
Spain attending a primary school that serviced middle to low socioeconomic populations
from urban zones. All of the children were assessed by their teacher based on an exam
given at the beginning of the school year as making normal progress in English language.
Previous research findings conducted on adults support the premise that presenting words
and pictures simultaneously in multimedia programs is more effective than words or
pictures alone when learning second language vocabulary. It was hypothesized that since
children’s cognitive ability is lower than adults, presenting words and pictures
simultaneously in multimedia programs would not yield the same results. Each child was
given access to a personal computer that presented an interactive multimedia short story.
7

8. The story consisted of 101 English words, 12 of which were inside a button: donkey,
drawer, penknife, hammer, bricks, ladybird, waistcoat, bonnet, mittens, jug, tray and
napkin. For each of these words different types of multimedia annotations were
available. In addition to the use of personal computers the children were given a paper
and pencil English vocabulary pre-test, post-test and a test of verbal and spatial ability.
The vocabulary pretest consisted of 60 English words and the post test consisted of only
the 12 annotated English words within the button. The children were instructed to write
the meaning of the words they knew. The groups of children were randomly selected and
assigned to three groups: “word only”, “picture only” and “word and picture”. Each
child was given instruction on how to access the annotations for the words, how to use
the mouse properly and they were also told that they needed to access the cues for each of
the 12 words within the button because the learning task was to remember the meanings
of these words. 20 minutes was allotted for the children to read the story twice and
another 10 minutes was allotted for the students to fill out the post-test.
The empirical evidence suggests that in a second language vocabulary program
for children, presenting a word only is more effective than presenting just a picture
representation of the word or a picture and the word simultaneously. These findings are
in line with the proposed hypothesis theorizing that school age children’s limited
cognitive ability affects second language vocabulary acquisition when a word is
presented along with a picture. The researchers attribute these findings based on previous
research on the effects of cognitive load on the working memory. Working memory
refers to the structure and processes used for temporarily storing and manipulating
8

9. information. Cognitive load is defined as the effort used with thinking and reasoning that
can interfere with other thought processes. “Cognitive load may occur when two types of
stimuli that supply the same information (e.g., a written word and a picture related to the
same concept) are perceived through the same information processing channel, which is a
visual channel in this case.”(Acha, 2009, p.25) The presentation of both the word and the
picture requires the child to exhaust more cognitive resources and is less effective in
learning the task at hand. One limitation to the study was the time each child was
allocated to reach each passage. Since the researchers were looking to assess vocabulary
retention each child should have been given a longer time to read the passages. Setting a
20 minute time limit to complete the task suggests that the children could not retain many
words in long term memory. Next, the topic of reading a foreign language text with
multimedia aids according to cognitive load theory will be addressed.
In a research study by Plass, Chun, Mayer & Leutner (2003), the authors of this
study are concerned with the question of what role cognitive load plays in multimedia
learning, and, in particular, how cognitive load affects the way learners with different
cognitive abilities process verbal and visual information. The focus of the investigation
was on English-speaking college students (N = 152), enrolled in a second-year German
course. It is hypothesized that low-ability learners would be less likely to learn the
translation of German words than high-ability learners when they were required to select
and process both the verbal information of a foreign-language reading passage coupled
with visual annotations for unknown vocabulary words. The researchers base this theory
on the fact that low-ability learners use more cognitive resources when processing visual
9

10. and verbal information simultaneously. As stated earlier, 152 college students
participated in the study, all of which were enrolled in a second-year German language
course. Each student was given access to a personal computer loaded with an interactive
multimedia version of a short story presented in German. Throughout the program
selected vocabulary words were annotated with verbal or video multimedia annotations.
A German vocabulary pre-test and post-test was administered to each student to assess
vocabulary acquisition. The research finding supported the proposed hypothesis, as
predicted when low-ability students processed verbal information from a reading text and
had to simultaneously process visual annotations for vocabulary words within the text,
they learned fewer vocabulary words in comparison to high-ability students.
In Summary, the results of each of these studies provide implications for
multimedia programmers who are developing new multimedia programs for second
language instruction. In particular, programmers should be conscious of these findings
and they should make an effort to include both visual and verbal annotations to
accommodate the different learning styles of students. Pedagogical implications of these
findings suggest that for some students multiple representations of information are not
beneficial for all learners. In fact, for some students it can hinder learning for low-ability
learners as well as grade school children. There are numerous instructional multimedia
software packages and online multimedia websites for adults as well as children geared to
foreign-language learning and we can assume many more are in the pre-production
phase. In respect to this growing market, additional research is required to fully
10

11. understand the process of multimedia learning so that educators and their students can
fully appreciate and benefit from this popular and growing field of technology.
11

12. References
Acha, J. (2009). The Effectiveness of Multimedia Programmes in Children's Vocabulary
Learning. British Journal of Educational Technology, 40(1), 23-31.
Al-Seghayer, K. (2001). The Effect of Multimedia Annotation Modes on L2 Vocabulary
Acquistition: A Comparitive Study. Language Learning & Technology, 5(1),
202-232.
Chun, D., & Plass, J. (1996). Effects of Multimedia Annotations on Vocabulary
Acquisition. The Modern Language Journal, 80(2), 183-198.
Mayer, R., & Moreno, R. (1998). A Split-Attention Effect in Multimedia Learning.
Journal of Educational Psychology, 90(2), 312-320.
Mayer, R., & Moreno, R. (2002). Aids to computer-based multimedia learning. Learning
and Instruction, 12(1), 107-119.
Plass, J. L., Chun, D. M., Mayer, R. E., & Leutner, D. (2003). Cognitive Load in Reading
a Foreign Language text with Multimedia Aids and the Influence of Verbal and
Spatial Abilities. Computers in Human Behavior, 19, 221 - 243.
Yeh, Y., & Wang, C. (2003). Effects of Multimedia Vocabulary Annotations and
Learning Styles on Vocabulary Learning. CALICO Journal, 21(1), 131 - 144.
12