Pusack Otto Lecture 2016 University of Iowa

1. Gillian Lord
University of Florida
glord@ufl.edu
The Promise of Online
Language Learning Programs:
Myth or Reality?

2. • Powerful, omnipresent marketing by
stand-alone online language programs
• Hyperbolic claims to be the
only / fastest / easiest / best/ most ___…
way to learn a language
• K-12 (and higher ed!) programs
feeling threatened…
Why this topic?

3. Why Rosetta Stone?
SOURCE:“CrowdsourcingDrivesLatestDisruptiveTechin
OnlineLanguageLearning.”Forbes.March24,2015.
“Rosetta Stone … spent $98.5
million on advertising in 2011,
up from $70.5 million in 2010,
according to Kantar Media”
www.nytimes.com/2012/06/20/business/media/rosetta
-stone-ads-emphasize-fun-not-efficiency.html

4. Why Rosetta Stone?

5. GOAL OF PRESENTATION:
• Provide linguistic evidence to be able to assess
these kinds of marketing claims
• Offer multiple data sources to consider the
advantages and disadvantages of this kind of
program
• Take as a case study example Rosetta Stone for
learning (Spanish as) a second/foreign language
or

6. 1. Professional assessments
2. (Previous) Empirical studies
3. Student attitudes (a priori)
4. Perceived quality of materials
5. Affective factors
6. Outcomes: (“Does it work?”)
a) Self-perceived communicative abilities
b) Quantitative measures of communicative abilities
c) Qualitative measures of communicative abilities
How to assess the program?

7. 1. Professional assessments
2. (Previous) Empirical studies
3. Student attitudes (a priori)
4. Perceived quality of materials
5. Affective factors
6. Outcomes: (“Does it work?”)
a) Self-perceived communicative abilities
b) Quantitative measures of communicative abilities
c) Qualitative measures of communicative abilities
How to assess the program?

8. University of Florida students enrolled in Beginning
Spanish 1 (avg. age = 20)
• L1 English
• No other L2 proficiency (beyond h.s. requirement)
• Course designed for those with NO prior Spanish
instruction
3 environments:
• Classroom (C): N=4
• Rosetta Stone (RS): N=4
• Classroom+Rosetta Stone (RS+C): N=4
My data – Participants
Original population had
20-25 participants in
each of the three groups.

9. Classroom
• In-tact section
• Followed regular
syllabus with
standard materials
My data – 3 environments
Rosetta Stone
• Self-selected (required by
IRB)
• Not required to attend any
regular class
• Used RS package
(“Conversational
Spanish”):
– 16-week course designed
to cover material
comparable to a face-to-
face beginning class
– 6 units of Rosetta Stone®
Version 4 TOTALe®
Spanish, each has 4
lessons [Level 1, half of
Level 2]
– Minimum of 6 Rosetta
StudioTM sessions
– Minimum of 8 hours in
Rosetta WorldTM
– Monitoring of program
access and time on task
RS + Class
• In-tact section of Beginning
Spanish class
• Same instructor as
Classroom group
• Used Rosetta Stone
materials as their textbook
– (including all features
described for RS
group)

10. Quantitative Data
• Portion of Spanish CLEP test (grammatical competence)
• Versant Proficiency Test in Spanish (oral proficiency)
• Attitude survey (Likert, style, pre- and post)
• Acoustic analysis of vowel production in Spanish
interviews
o 10 tokens of /e/ extracted from each participant’s interview.
 Tokens separated for male and female
 Each formant measured at midpoint using Praat
 F1 and F2 extracted using a script and compared to
standard native values
Data used here (1)

11. Qualitative Data
• Fluency analysis of Spanish interviews
o Total number of words spoken
o Number of Spanish words; Number of English words
o Number of dysfluencies
o Lexical density (number of unique Spanish words)
o Number of fillers/non-lexical items
• Content analysis of English interviews
o Affective factors
o Reactions to instructional materials
o Student perceived learning
o Language communication and use
o Other specific language problem or comment
Data used here (2)

12. PROFESSIONAL
ASSESSMENTS OF THE
ROSETTA STONE PROGRAM
Criterion 1:

13. • Godwin-Jones (2007, 2009)
o Traditional computer-based training programs often informed by
technology specialists, not SLA specialists
o Rosetta Stone provides visualization feedback for specific sounds
(pronunciation); mostly accurate
• Lafford & Sykes (2007)
o Evaluate if programs provide the tools necessary for effective language
learning, based on features that research has shown to be important
(interaction, relevant contextualization of language, etc.)
o “… these products do not incorporate a number of the [necessary]
research-based insights (e.g., the need for culturally authentic, task-
based activities) that informed SLA scholars might have given them.”
Professional Assessments (1)

14. • Santos (2011)
o Lack of context
o General inability to respond to spontaneous student speech
o What Rosetta Stone calls interaction is “a rather poor and limited
version of what one would encounter in a real-life conversation”
• DeWaard (2013)
o Based on personal experience, professional reactions
o “Not a viable replacement of current instruction at the
postsecondary level”
Professional Assessments (2)

15. EMPIRICAL STUDIES INTO
THE OUTCOMES OF THE
ROSETTA STONE PROGRAM
Criterion 2:

16. • Vesselinov (2009)
o Commissioned by Rosetta Stone; RS beginning users
demonstrated increased knowledge of the language after 55
hours of use.
• Nielson (2011)
o Self-study programs in workplace; some success among
committed users, but extreme attrition.
• Stevenson & Liu (2010)
o Use of social interaction in web-based language learning tools;
lack of ability to engage learners in true interaction; users do not
take advantage of network opportunities.
Empirical studies

17. STUDENT ATTITUDES
TOWARDS THE ROSETTA
STONE PROGRAM
Criterion 3:

18. • Why did you volunteer for the Rosetta Stone class-
replacement option?
o I heard a lot of good things about RS and wanted to
try it.
o Can better manage my time and schedule and move
more at my own pace without dealing with class.
o Sounded more beneficial.
o I was going to use my own to supplement education
anyway.
Background questionnaire

19. PERCEIVED QUALITY OF THE
ROSETTA STONE MATERIALS
Criterion 4:

20. • Comments from English interviews coded for
mentions of the Rosetta Stone materials
o 42/181 comments in RS group
o 80/124 comments in RS+C group
o Difficult to classify as categorically positive or
negative
• Main themes:
o Technology [Flexibility, Ease of use, Glitches]
o Approach to language learning
Student Impressions of
Materials

21. • Like you’re able to kind of
do it like on your own time,
you know, I’m not like
restricted.
• It’s nice not to go to class.
• I have always preferred to
learn language, like, on my
own.
• It’s more flexible with my
schedule.
Sample student comments
(Flexibility)

22. • Because it’s a lot of visual stuff,
and I feel like I’m a visual
learner.
• It’s pretty easy to use.
• World is good, I do the games.
• Oh, and you can do it on your
iPad … so I’d do that a lot, like
take it to people’s houses, and
they’ll all be like watching TV,
and I would be like, doing my
Rosetta Stone.
• I like how it’s like uh, more like
a game, so I’m more willing to
actually do it…
Sample student comments
(Ease of Use)
• It was just kind of a lot harder [to
use] than I expected.
• I just didn’t show what words I
needed to use before it.
• I [don’t like] the lack of human
interaction.
• Sometimes it’ll show the person
speaking, and sometimes it’ll say like
he or she, and sometimes it’ll be I.
And I couldn’t tell the difference.
• …but it is tedious, a lot of it.

23. • I’m still having problems
with the computer.
• I can’t get the
microphone to work…
• I was doing my Studio
session and… I had no
audio, like, I could hear
them, but they couldn’t
hear me the whole time.
Sample student comments
(Tech glitches)

24. • I feel like it’s more like how you
naturally learn the language
instead of like, “These are … your
vocabulary words this week”.
• I think the Studio session was the
most interesting thing.
• Speaking it for sure [is hardest],
because when you see it on paper
it’s a little bit… but when it gives
you four options, it’s easier to be
like, it’s one of these four.
• The program is really good with
like teaching like vocabulary.
• With like vocabulary, it’s like really
good, and you get by.
Student Impressions
(Language learning in general)
• It’s just like the grammar, and how to
like, put it together.
• You can’t ask questions.
• You don’t get any writing, and then all
of a sudden there’s one writing thing.
• You have that whole grammar and
conjugation issue on Rosetta, because
they don’t really explain it.
• You really need to have communication
with a real person.
• I would enjoy getting more grammar
lessons … just to get a foundation of
knowledge, then building up on that.
• Rosetta Stone doesn’t like, I don’t know,
hammer it down for you like they
would in a normal grammar class.

25. AFFECTIVE FACTORS AND
EMOTIONAL REACTIONS TO
THE ROSETTA STONE
PROGRAM
Criterion 5:

26. • Comments from English interviews coded for any
mention of affect/emotion
o Related to using the materials, learning, etc. and how
the learners felt in that respect
• 33/181 comments in RS group
o 8 positive towards RS, 8 negative, 17 unrelated
• 24/123 comments in RS+C group
o 2 positive towards RS, 9 negative, 13 unrelated
Affective factors

27. • I don’t have a problem
with staying focused on
my computer at all.
• I like not having a
textbook to worry
about.
• I like it a lot.
Student reactions/emotions
• I’m kind of struggling.
• I didn’t like the stories.
• I’m always just frustrated
because I’m like, I don’t
really understand it.
• I feel like it should be
more structured.

28. LINGUISTIC OUTCOMES OF
THE ROSETTA STONE
PROGRAM
a) Self-perceived communicative abilities
Criterion 6:

29. • Comments from English interviews coded for any
mention of perceived learning or outcomes
• 34/181 comments in RS group
o 9 positive assessments, 22 negative, 3 unrelated to
materials
• 14/123 comments in RS+C group
o 8 positive assessments, 3 negative, 3 unrelated
Perceived learning

30. • I understood like the vocab.
• I feel like I’m slowly learning
and progressing to learn the
language.
• I do like the studio sessions…
When I do do them, like I feel
like it helps me more.
• I can understand almost
always what is being said.
• Learning a language just like
on your computer as opposed
to like in person can actually
be effective.
Perceived learning
• Rosetta Stone doesn’t give you too
much, like, actual instruction so you
don’t learn.
• I don’t know how to use words in
Spanish.
• I’m just kind of like nervous about
going back to class.
• I don’t feel as confident as I did at
that last meeting, honestly.
• We didn’t ever have to like memorize
the conjugations, so I don’t know
them, so I can’t say that it was super
effective.
• It was just kind of a lot harder than I
expected.

31. Significant decrease among RS on item #11:
“Interacting via chat or telephone is comparable
to interacting face-to-face.”
Significant increase among RS and RS+C on item #19:
“I would prefer to learn a language on my own
time and at my own pace than in a group or
classroom setting.”
Perceived learning
(pre-post changes on relevant
items from attitude survey)

32. LINGUISTIC OUTCOMES OF
THE ROSETTA STONE
PROGRAM
b) Empirical measurements of communicative abilities
Criterion 6:

33. CLEP test
Average scores (converted to %)
38.65 39.17
47.50
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Classroom Rosetta Stone RS+class
p = 0.165
RS = Classroom = RS+Class

34. 27.08 26.25
20.00
0.00
10.00
20.00
30.00
40.00
50.00
60.00
70.00
80.00
90.00
100.00
Classroom Rosetta Stone RS + Class
Versant test
Average scores (converted to %)
p = 0.620
RS = Classroom = RS+Class

35. • No significant difference
between groups or
test times.
• Classroom males show
slight trend toward
more native-like F1
values in final interview.
Pronunciation
Acoustic analysis of vowel /e/
RS = Classroom = RS+Class ?

36. LINGUISTIC OUTCOMES OF
THE ROSETTA STONE
PROGRAM
c) Discourse analysis of communicative abilities
Criterion 6:

37. INTERVIEWER: Cuéntame, ¿qué te gusta hacer en tu tiempo libre, o los fines de semana?
SL: Repitas, please.
INTERVIEWER: ¿Qué te gusta hacer?
SL: Qué te gusta hacer…
INTERVIEWER: ¿Te gusta ir a películas? ¿Te gusta escuchar música?
SL: Uh, ¿fin de semana?
INTERVIEWER: Sí.
SL: Uh, sí. En fin de semana, yo… yo estudio, uh, mucho.
INTERVIEWER: ¿Sí?
SL: Uh, para mis exámenes. Sí. Yo tengo muchos examines en química orgánica, biología, y laboratorio. Uh, sí. Mucho, uh… no,
muy ocupado. So, no películas, no, uh, deportes.
INTERVIEWER: ¿Cuál fue la última película que viste?
SL: Cuál te…
INTERVIEWER: La última vez, the last time, que viste una película.
SL: Phew… Hmm. Let’s see… dos menses.
INTERVIEWER: Meses, mhm.
SL: Meses. Ago. ¿Cómo se dice “ago”?
INTERVIEWER: Hace. Hace dos meses.
SL: Hace, sí.
INTERVIEWER: Wow.
SL: Yo no… yo no veo muchas películas en Gainesville.
INTERVIEWER: ¿Qué película fue esa, hace dos meses? ¿Cómo se llamaba?
SL: Uh, el pelí—la película… ¿cómo se dice “was”?
INTERVIEWER: Era, o fue.
SL: Era. La película era…
INTERVIEWER: ¿No te acuerdas?
SL: Yo no… sí.
Final interview - Classroom

38. INTERVIEWER: Mhm, ¿y qué haces en Gainesville?
SH: Um… you’re going to have to forgive me, my mind’s like blown… Um, yo estoy estudiar.
INTERVIEWER: ¿Tú estudias? ¿Y qué más?
SH: Yo trabajo en un restaurante de Dragonfly.
INTERVIEWER: Y, ¿con mucha frecuencia, vas de compras?
SH: Yo no entendí, repetirlo, por favor.
INTERVIEWER: ¿Con mucha frecuencia, vas de compras? “Ir de compras” significa go shopping.
SH: All right, say that one more time, please.
INTERVIEWER: ¿Con mucha frecuencia, vas de compras?
SH: Uh, no, uh, no voy a… what did you, how did you say “to go shopping”?
INTERVIEWER: Ir de compras.
SH: No voy de compras.
INTERVIEWER: Y, ¿qué vas a hacer este verano?
SH: Este verano, yo voy a visitar Brazil.
INTERVIEWER: Vas a visitar Brazil, y ¿vas a estudiar en Brazil?
SH: No, um, yo voy a trabajar en Brazil.
INTERVIEWER: Y, em, ¿qué más vas a hacer en Brazil? ¿Vas a leer, vas a jugar deportes?
SH: What am I going to do in Brazil? I thought I just answered that.
INTERVIEWER: ¿Solo trabajar?
SH: I don’t know, I’m going on a missions trip, I don’t know how to express that in Spanish, but…
INTERVIEWER: Pues, buena suerte, muchas gracias.
Final interview – Rosetta Stone

39. Ratio of L1/L2 words
0.26
0.83
0.68
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
Ratio of English-to-Spanish words used, by group
Control Average
RS + class Average
Rosetta Stone Average
0 = no English words produced
1 = 1 English word produced for every Spanish word

40. Assistance requests
2.06
1.03
0.11
1.75
0.61
3.24
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
# Clarification requests in Spanish # Clarification requests in English
Average # of clarification / assistance requests by group
Control Average RS + class Average Rosetta Stone Average

41. SUMMARY

42. 1. Professional assessments
2. Empirical studies
3. Student attitudes (a priori)
4. Perceived quality of materials
5. Affective factors
6. Outcomes:
a) Self-perceived communicative abilities
b) Empirical measurements of communicative
abilities
c) Discourse analysis reflecting communicative
abilities
Myth or Reality?

43. GROUP
Completion
Rate
Average
Score
Total Course
Usage (hours)
Total Class
Time (hours)
Classroom 96.99% 90.77% 70.00 39.00
RS+C 93.67% 98.63% 32.81 37.25
RS 97.67% 95.88% 30.69 NA
What about time on task?

44. But…

45. IN CONCLUSION…

46. • Further professional assessments of these types of
programs, as new tools and new features are
developed
• Continuing analysis of current data
• Empirical studies of outcomes and effectiveness
o Attitudes and reactions
o Linguistic outcomes
o Larger, more diverse populations
o Consider culture, sociocultural aspects
1. More research is needed.

47. In certain circumstances:
o Introduction and exposure to new languages
o Refresh skills previously acquired
o Excellent vocabulary presentation and practice
o Flexible for varied populations
o Highly motivated/diligent learners more apt to
make progress
… and are undoubtedly better than nothing!
2. Online (stand-alone)
programs can be effective.

48. Based on multiple data sources, programs like Rosetta
Stone do not appear to be necessarily any more …
than other methods, such as common classroom-based
approaches, immersion programs, etc.
3. BUT…

49. 4. As language experts, then,
we should…
 … not believe everything the ads promise!
o There is a significant divide between what the ads
promise and what the programs seem capable of
 … keep an open mind.
o New tools and capabilities emerge daily
o Knowledge of another language is always valuable
 … work to develop criteria for assessing these
programs.
o [at tomorrow’s workshop!]

50. Special thanks to
• UF College of Liberal Arts & Sciences
• UF CLAS Humanities Scholarship
Enhancement Fund
• Laura Bradley, Lisa Frumkes
(Rosetta Stone)
• Caroline Reist, Brandon Shufelt, Keegan
Storrs, Diana Wade (RAs)
• Carlos Enrique Ibarra (statistics)
Thank you
glord@ufl.edu

51. DeWaard, L. (2013). “Is Rosetta Stone a viable option for L2 learning?” Forthcoming in ADFL Bulletin.
Godwin-Jones, R. (2007). “Emerging technologies; Tools and trends in self-paced language
instruction. Language Learning and Technology,” 11(2), 10-17. Retrieved 26 September 2012 from
http://llt.msu.edu/vol11num2/emerging/
Godwin-Jones, R. (2009). “Emerging technologies: Speech tools and technologies. Language Learning
and Technology,” 13(3), 4-11. Retrieved 26 September 2012 from
http://llt.msu.edu/vol13num3/emerging.pdf
Lafford, B. & Sykes, J. (2007). “Entre dicho y hecho …: An assessment of the application of research
from second language acquisition and related fields to the creation of Spanish CALL materials for
lexical acquisition.” CALICO Journal, 24(3), 427-529.
Nielson, K. B. (2011). “Self-study with language learning software in the workplace.” Language
Learning and Technology, 15(3), 110-129. Retrieved 26 September 2012 from
http://llt.msu.edu/issues/october2011/nielson.pdf
Santos, V. (2011). “Review of Rosetta Stone Portuguese (Brazil) levels 1, 2, & 3.”CALICO Journal, 29(1),
177-194.
Stevenson, M. P. & Liu, M. (2010). “Learning a language with web 2.0: Exploring the use of social
networking features of foreign language learning websites.” CALICO Journal, 27(2), 233-259
Vesselinov, Roumen. Measuring the Effectiveness of Rosetta Stone.
http://resources.rosettastone.com/CDN/us/pdfs/Measuring_the_Effectiveness_RS-5.pdf.
Works Cited

53. Sample CLEP test items

54. Versant proficiency test

55. Instrument has been tested for reliability and validity
Versant proficiency test

56. Rosetta Stone interface

57. Rosetta Stone interface
(vocabulary)

58. Rosetta Stone interface
(grammar)

59. Rosetta Stone interface
(pronunciation)

60. Rosetta Stone interface
(World – “play”)

61. Rosetta Stone interface
(World – “talk”)

62. Rosetta Stone interface
(World – “explore”)

63. Rosetta Stone interface
(Studio)

64. Attitude survey