Comparative and non-comparative evaluation in educational technology
J Raphael Holmes MS Thesis Defense
1. Technology-Assisted Independent
Study of English as a Second Language
J. Raphael Holmes
October 16, 2012
Thesis Advisory Committee: Dr. Ross Perkins, Dr. Dazhi Yang, Dr. Arturo Rodriguez
2. Presentation outline
• Definitions
• Research problem and purpose
• Research questions and significance
• CALL overview and background information
• Gaps in research
• Study design, instruments, and data analysis
• Findings- who participated
• Findings- answers to research questions
• Implications
• Recommendations
3. Research in brief:
Independent, in-service ESL learners were interviewed and
surveyed to learn about their use of CALL technology.
Image from flickr.com/photos/bartelomeus/4184705426/
4. Acronyms
CALL – Computer assisted language learning (umbrella term,
includes CALI, TELL, MALL, CAPT)
ESL – English as a second language (studied in an English-
speaking country)
EFL – English as a foreign language (studied in a non-English-
speaking country)
ICT – Information and communications technology
NLP – Natural language processing
5. Definitions
Independent learner – A learner who is not enrolled in a
formal class or taking private lessons
In-service learner – A learner who is using English on a
daily basis, for professional, educational, or personal reasons
Learner autonomy – A learner’s ability to take charge of his
or her own learning (Holec, 1981), encompassing initiative,
persistence, and resourcefulness (Macaskill & Taylor, 2010)
6. Research problem
Independent, in-service ESL learners- insufficient returns on
effort, particularly acute with modern ICT developments.
“The web for language learning remains an interconnected quagmire of
unorganized opportunities” (Howard, 2010, p. 196).
“[Tertiary EFL students] reported limited knowledge of how various
technologies could be used for language learning” (Lai & Gu, 2011, p. 324).
“…students tend not to have adequate access to or literacy in using
specialized tools that are often necessary for CALL” (Winke & Goertler,
2008, p. 482).
7. Purpose of the study
• to explore the use of CALL technology by these learners
• to examine what technologies they use and what aspects of
independent language learning are aided by technology
• to examine the relationship between CALL technology use
and learner autonomy
8. Research questions (in brief)
1. What do they use?
2. What do they use it for?
3. How does this relate to learner autonomy?
9. Research questions
1. What technologies do adult, independent, in-service learners
use to engage in English language study?
2. Out of seven key components of language learning in general a
and four that are specific to independent language learningb,
which are facilitated by these learners’ uses of technology?
3. What is the relationship between the nature of these
learners’ uses of technology and their levels of learner
autonomy?
a
Reading, writing, listening, speaking, grammar, vocabulary, and pronunciation
b
Motivation, management, reflection, and feedback
10. Who cares?
• Independent language learners
• Language education, independent learning, & CALL
researchers.
• Creators of CALL technology
• Classroom instructors
11. Multiple ways to organize CALL
• A series of historical phases or approaches
• A collection of language learning technologies
• Essential components of language learning being provided by
technology
12. Historical phases / approaches
• Historical phases (Warschauer, 1996)
• behaviorist
• communicative
• integrative
• Approaches: (Bax, 2003)
• restricted
• open
• integrated
• Final phase and approach both imply
normalization.
Image from humsci.stanford.edu/faculty/discoveries/
13. What does CALL look like?
(This connects to RQ1, how CALL manifests itself for participants.)
• Media
• Communication technology
• Software
• Internet resources/tools/environments
• Old or new
• Maybe specific to language learners, maybe not
14. Components of language learning
(These form the basis of RQ2, what CALL is used for.)
• Reading, writing, listening, speaking- universally agreed upon
• Grammar, vocabulary, pronunciation- also broad consensus
on these secondary components (e.g. Lasagabaster & Sierra,
2003; Levy & Hubbard, 2005)
15. Components of independent learning
(These form the basis of RQ2.)
• Motivation- “fuel” driving learning endeavors (Garrison,
1997). Close ties to language learning (Ushioda, 2011) and
technology use (Ringstaff & Kelley, 2009).
• Management- selection of objectives, materials, activities
(Garrison, 1997; Pilling-Cormick, 1997; Chan et al., 2002).
• Reflection- metacognition, metalinguistic awareness
(Schwienhorst, 2008; Garrison, 1997; Pilling-Cormick, 1997;
Tarvin & Al-Arishi, 1991).
• Feedback- from technology or facilitated by technology
(Nagata, 1993; Pilling-Cormick, 1997; Alm, 2006)
16. More on learner autonomy
(This is the subject of RQ3, measured with an instrument.)
• As with other similar terms (e.g. self-directed learning, self-
access learning, andragogy) can describe a behavior, a
perspective, or an ability
• Thought to contribute to independent learning success
(Ponton & Carr, 2000; Benson, 2007)
• Frequently studied in the context of classroom instruction
• Special relationship with language learning
17. Gaps in research- what’s missing?
Image from jamesdawson.blogspot.com/2009_09_01_archive.html
18. Gaps in research- what’s missing?
• Research on non-undergraduate learners
• Research on independent learning
• Studies with broad focus, rather than one or two aspects of
language learning
• Studies addressing affective components (e.g. motivation)
• Studies addressing pronunciation
• Research connecting theoretically generalized learner autonomy
and actual attitudes/behaviors
• Longitudinal studies
• Research on non-adult learners
• Research on languages other than English
(Zhao, 2003; Gan, 2004; Hurd, 2008; Validvia et al., 2011; Tanner & London, 2009)
19. Research design
• Semi-structured interview (10 participants)
• Electronic questionnaire (112 general participants, 40 specific
to independent language learning website, antimoon.com)
• Mixed interview and survey design allowed for
• triangulation
• deeper interpretation of questionnaire results
• revision of questionnaire
20. Brief overview of instruments
• Interview: 30 minutes, 28 questions in five categories:
• background info
• English studying
• technology
• independence
• two hypotheticals
• Questionnaire: 10 minutes, four sections:
• demographic info
• brief ESL self-assessment
• technology use questions
• learner autonomy instrument
21. Data analysis
• Interview- transcripts coded, then codes merged into six
themes
• Questionnaire- Descriptive statistics for means and variances
• Questionnaire- Learner autonomy scores checked for
correlation with other responses, looking for Pearson’s
correlation coefficients > 0.3
22. Findings- who participated?
• Interview
• six native languages
• narrow age range
• mostly professional/technical occupations
• all participants feeling that English was important to their
professional goals
• Questionnaire
• 37 native languages
• broad age range
• spread of primary reasons for studying English
• some differences between two sample groups
23. Findings- 3 interview themes
• Linguistic isolation
Nobody corrects me. And I don’t have many chances to speak.
If there are no American friends, I cannot expand my vocabulary.
• Motivating factors
Sometimes Americans laugh a lot, and I want to understand why they
laugh.
…I cannot express my opinion, or what I was thinking. That’s the big
motivation.
• Continuous learning
If I’m not sleeping, I think I’m learning English.
…I feel [studying is] something natural… I wouldn’t explicitly sit down to it.
25. Findings- research question 1
• What CALL technologies do they use?
• Media
• Communication tools
• Reference tools
• What don’t they use?
• Language learning software
• Language learning websites
26. Findings- research question 2
• What do they use CALL technologies for?
• Reading
• Listening
• Vocabulary
• Feedback*
• What don’t they use CALL for?
• Speaking
• Learning management
• Feedback*
27. Findings- research question 3
Is there a relationship between their use of CALL
technology and their learner autonomy?
• Yes. Learner autonomy scores were correlated with
• mobile/smart phone use*
• use of spell check/grammar check
• use of automatic translation
• use of technology for grammar, reflection*, management*
The *’s denote correlations with p ≤ 0.01. Even with a large
number of correlations checked, this is highly unlikely to occur by
chance alone. All correlations were positive, as expected.
28. Findings- research question 3
Is there a relationship between their use of CALL
technology and their learner autonomy?
• No.
• Most tools and attitudes show no correlation with learner
autonomy
• No overlap in correlations between two survey samples
• Interview data contradict the potential for a relationship
The study doesn’t provide a compelling answer to this question,
but the findings raise interesting additional questions.
29. Implications
• CALL developments may not reach these learners
• Some CALL tools may match these learners’
“study” habits better than others
• Sophisticated NLP tools may impact both these
issues
Image from Edge et al., 2011
30. Recommendations
• Find out more about these learners with more
rigorous data collection.
• Consider how CALL is serving these learners.
• Explore relationship between CALL and learner
autonomy further. Address specific practices.
• Explore the degree to which generalizable
learner autonomy applies to advanced language
learning.
31. References
Alm, A. (2006). CALL for autonomy, competence and relatedness: Motivating language learning environments in Web
2.0. The JALT CALL Journal, 2(3), 29-38.
Bax, S. (2003). CALL–past, present and future. System, 31(1), 13-28.
Benson, P. (2007). Autonomy in language teaching and learning. Language Teaching, 40, 21-40.
doi:10.1017/S0261444806003958
Chan, V., Spratt, M., & Humphreys, G. (2002). Autonomous language learning: Hong Kong tertiary students’ attitudes
and behaviours. Evaluation & Research in Education, 16(1), 1-18.
Edge, D., Searle, E., Chiu, K., Zhao, J., & Landay, J. A. (2011). MicroMandarin: Mobile language learning in context.
Proceedings of the 2011 annual conference on Human factors in computing systems (pp. 3169–3178).
Vancouver, CA. Retrieved from http://dl.acm.org/citation.cfm?id=1979413
Gan, Z. (2004). Attitudes and strategies as predictors of self-directed language learning in an EFL context. International
Journal of Applied Linguistics, 14(3), 389-411. doi:10.1111/j.1473-4192.2004.00071.x
Garrison, D. R. (1997). Self-directed learning: Toward a comprehensive model. Adult Education Quarterly, 48(1), 18-33.
doi:10.1177/074171369704800103
Holec, H. (1981). Autonomy and foreign language learning. Oxford: Pergamon.
Howard, C. D. (2011). Web 2.0 sites for collaborative self-access: The learning advisor vs. Google®. Studies in Self-
Access Learning Journal, 2(3), 159-211.
Hurd, S. (2008). Affect and strategy use in independent language learning. In: S. Hurd & T. Lewis (Eds.), Language
learning strategies in independent settings (pp. 218-236). Bristol, UK: Multilingual Matters, pp. 218–236. Retrieved
from http://http://oro.open.ac.uk/10049/
32. References (cont’d)
Lai, C., & Gu, M. (2011). Self-regulated out-of-class language learning with technology. Computer Assisted Language
Learning, 24(4), 317-335. doi:10.1080/09588221.2011.568417
Lasagabaster, D., & Sierra, J. M. (2003). Students’ evaluation of CALL software programs. Educational Media
International, 40(3-4), 293-304. doi:10.1080/0952398032000113211
Levy, M., & Hubbard, P. (2005). Why call CALL “CALL”? Computer Assisted Language Learning, 18(3), 143-149.
doi:10.1080/09588220500208884
Nagata, N. (1993). Intelligent computer feedback for second language instruction. Modern Language Journal, 77(3), 330-
339.
Pilling-Cormick, J. (1997). Transformative and self-directed learning in practice. New Directions for Adult and Continuing
Education, 74, 69-77.
Ponton, M.K., & Carr, P.B. (2000). Understanding and promoting autonomy in self-directed learning. Current Research in
Social Psychology, 5(19), 271-284.
Ringstaff, C., & Kelley, L. (2002). The learning return on our educational technology investment: A review of findings
from research. Washington, DC: Office of Educational Research and Improvement.
Schwienhorst, K. (2008). Learner autonomy and CALL environments. Routledge studies in computer assisted language
learning. New York: Routledge.
Stockwell, G. (2007). Vocabulary on the move: Investigating an intelligent mobile phone-based vocabulary tutor.
Computer Assisted Language Learning, 20(4), 365-383. doi:10.1080/09588220701745817
Tanner, M. W., & Landon, M. M. (2009). The effects of computer-assisted pronunciation readings on ESL learners’ use
of pausing, stress, intonation, and overall comprehensibility. Language Learning & Technology, 13(3), 51-65.
33. References (cont’d)
Tarvin, W. L., & Al-Arishi, A. Y. (1991). Rethinking communicative language-teaching: Reflection and the EFL classroom.
TESOL Quarterly, 25(1), 9-27.
Ushioda, E. (2011). Language learning motivation, self and identity: Current theoretical perspectives. Computer Assisted
Language Learning, 24(3), 199-210. doi:10.1080/09588221.2010.538701
Valdivia, S., McLoughlin, D., & Mynard, J. (2011). The importance of affective factors in self-access language learning
courses. Studies in Self-Access Learning Journal, 2(2), 91-96.
Warschauer M. (1996). Computer assisted language learning: An introduction. In S. Fotos (ed.), Multimedia language
teaching (pp. 43-62). Tokyo: Logos International. Retrieved from http://www.ict4lt.org/en/warschauer.htm
Winke, P., & Goertler, S. (2008). Did we forget someone? Students’ computer access and literacy for CALL. CALICO
Journal, 25(3), 482–509.
Zhao, Y. (2003). Recent developments in technology and language learning: A literature review and meta-analysis.
Calico Journal, 21(1), 7-28.