Using mobile phones for language assessment and oral skills development in secondary education. A pilot project tested using mobile phones to assess students' oral English skills through speaking tasks and questions. Teachers and students provided feedback, noting technical issues, question design, and concerns about costs. Overall, mobile phones showed potential as a tool for oral assessment practice, though in-person testing may be preferred and costs remain an important consideration.

1. Using mobile phones for
the assessment of and for
oral skills development in
secondary education.
Ton Koenraad
Hogeschool Utrecht, University of Applied
Sciences, Faculty of Education
TELLConsult

7. Overview
 Project context & goals
 Pre-Pilot: set-up & research questions
 Data collection
 Results & conclusions
 Next phase

8. English for Kids Foundation
“Voice for children”
A, not -for-profit organisation

9. " Voice for children "
Every kid has a right to communicate.
English is a good tool to communicate
internationally.
The English for Kids Foundation wants to
promote the use of English
to communicate internationally.
We want to give children 'a voice'.

10. Projects are to meet the
following requirements:
> long term target (3-5 yrs)
> aimed at children
> close co-operation with
local organisations in the
project countries so that
activities can be embedded
and be allowed to proceed

11. EFKF projects:
Nicaragua
Very Young Cambodia
Learners
India
Surinam
EDS
South Africa
Elandsdoorn
Kenia Teacher
support
Gobabis
Namibia

12. www.englishforkidsfoundation.org
www.efkf.org

13. Anglia Network Europe
www.anglianetwork.eu

14. International context
15 European countries

15. Step by step

16. Examinations
- 28 January
- 15 April
- 20 May
- 24 June
- Speaking Tests

17. Why this Project?
 Anglia: flexibility / assessment of
- sharing innovative speaking practice
materials
- delivery of formal language
assessments
English for Kids: mobile as infrastructure,
& schools assessment for:
washback effect of testing

18. Issues to be researched
 general organisation
 task and test design,
 teacher competences
 face & content validity aspects
 system usability
 Research design / instrument
development

20. Natural interface.
Widely available.
Familiar to students.
Technology used as a tool –
not “accessorizing education”.
Also available on iPod Touch & Skype.
Why use a phone? 20 20

21. How it works
Teachers:
Personal online workspace to set spoken
questions/tasks.
Questions are easy to set online by using a
microphone or uploading audio files.
Audio player allows teachers to review spoken
work and leave feedback.
Over time teachers establish a digital portfolio
of student work.
21

22. How it works
Students:
Connect using:
Mobile phones
iPod Touch
Skype
Landline
Computer
Access spoken exercises & leave voice responses.
Connect with other students for role play.
Personal online workspace to store work, listen & receive
feedback.
Listen to exemplar and sample questions posted by teachers.
22

23. Pilot
 2 secondary ed. EFL teachers
 2 Anglia member schools
 Volunteer students (n= 20)
 Assessment: asynchronous, interview
format
 Oral presentation skills

24. Data Collection
Instruments Pupils Teachers
Pre- + +
Questionnaire
Recordings + +
+ Test scores
Post- +
Questionnaire
Structured N=2 +
Interview
Reflections Developers

25. Pre-Questionnaire:
Learner Profiles (1)
Aspect Group A Group B
Group size 12 8
Girls 5 4
Boys 7 4
Avg. Age 13.5 13.5
Years of English 3 3
Attitude to Fairly
Learning English positive Positive
Average score 6.37 7.6
at Secondary

26. Pre-Questionnaire:
Learner Profiles (2)
Aspect Group A Group B
Speaking Skills: Fairly good Good
(Self reported)
Like speaking in So, so Definitely
class
Actual speaking hardly Very frequently
Telecollaboration at n/a Slightly more than
school once
Tel. Experience Very occasionally
in projects n/a
Tel. Experience seldom seldom
IRL

27. Pre-Questionnaire:
Learner perceptions:
L2 in class & IRL
Aspect Group A Group B
L2 in lessons Once in 3 Practically every
lessons lesson
Answers /Discussion Only now (very)
and then frequently
Pairwork Hardly ever sometimes
Use of English IRL 1. Chat in 1. Holidays,
games 2. Chat in
2. Holidays games
3. Skype 3. Txt chat

28. Assessment of Oral skills
Aspect Group A Group B
Assessed Tasks in All: 1 All: 3.5
2010
School reports: Oral 35% ? 70% ?
skills included?
Expectations: Yes: 35% Yes: 60%
Is tele-testing valid? ?: 55% ?: 40 %
Post: Valid Yes: 60% 40%

29. Topics
 Introduction/warming up
 Your holiday this year
 A good school
 Social networks
 A million euros
 The climate

30. Post-Questionnaire (1)
Aspect Group A Group B
Technically OK? Yes So, so
Read Yes Sure
Instructions
Different from Yes Yes
expectation
Questions: Hard to Idem, but
complexity, remember, Speed OK
speed, Speed bit fast,
loudness, Not loud enough

31. Post-Questionnaire (2)
Aspect Group A Group B
Answer time left Yes Yes
Expected Mark Just sufficient O.K
Problem Topics Networks; 1M Euros Good School; Climate
Test Location School School + home
O.K. to do Yes, quite Yes
More pleasant No No
without teacher

32. Pupils’ Comments
 Time constraint is unnatural
 Was interrupted: new session needed
 Retries: worries about costs
 Questions could be louder
 Questions: peer voice is more inviting

33. Teacher Perceptions
 System usability
-System: fairly user-friendly
 Topics
- More alignment with pupils‟ interests might be
needed
 Validity
-Content measured in time is less suitable as
criterion when no interaction is possible
- computer-based testing, as such, not perceived
as unusual or unfriendly.
- Retry option?

34. Reviewing work online
34

35. Teacher Perceptions
 Teacher competencies:
- Knowledge of CEFR -> difficult, training
/ practice needed
- Evaluation categories (content,
accuracy, complexity, fluency.) useful;
scoring doable in one session.
But …would prefer a grading scale that
results in a CEF-level:
better match to Dutch current grade
system

36. Teacher perceptions
 Implementation
- Use as practice material and preparation for
speaking test. Actual testing: rather face-to-
face
- Gives students the opportunity to practice
outside the classroom, extra practice
- Chances for providing individualised
feedback
- May help reduce anxiety of insecure & shy
students:

37. Conclusions / next steps
 Improve briefing (demo, online tutorial) +
raise awareness implications of re-tries
 Redesign questions (granularity)
 Try-out alternatives:
- system access (landlines, computers)
- content aligned to syllabus / textbook

38. Conclusions
 Pupils, teachers & management have
concerns about costs
 Also found in other projects:
[…] cost to the end user is a major
consideration and can be a barrier to
successful uptake when using mobile
devices (Kukulska-Hulme & Shield, 2007)

39. Thank you for
your attention.
Comments, Questions?
www.koenraad.info
Ton.Koenraad@gmail.com

40. Annex
Literature Selection
Research data
Learnosity

41. Literature selection
 Collins, T. (2005). „English Class on the air: Mobile
Language Learning with CellPhones‟, Proceedings of the
Fifth IEEE International Conference on Advanced Learning
Technologies (ICALT‟05).
 http://csdl2.computer.org/comp/proceedings/icalt/2005/23
38/00/23380402.pdf
 Fallahkair, S., Pemberton, L. & Griffiths, R. 2007.
„Development of a cross-platform ubiquitous language
learning service via mobile phone and interactive
television‟. Journal of Computer Assisted Learning, 23 (4),
312-325.
 Kukulska-Hulme, Agnes; Sharples, Mike; Milrad, Marcelo;
Arnedillo-Sanchez, Inmaculada and Vavoula, Giasemi
(2009). Innovation in Mobile Learning: A European
Perspective. International Journal of Mobile and Blended
Learning, 1(1), pp. 13–35.

42. Literature selection (2)
 Naismith, L., Lonsdale, P., Vavoula, G. & Sharples, M.
(2004). „Literature Review in Mobile Technologies and
Learning‟. FutureLab Report 11.
http://www.futurelab.org.uk/resources/documents/lit_revie
ws/Mobile_Review.pdf.
 Sharples, M. (Ed.). (2006). Big issues in mobile learning.
Report of a workshop by the Kaleidoscope Network of
Excellence Mobile Learning Initiative, University of
Nottingham, UK.
 Shield, Lesley and Kukulska-Hulme, Agnes (2008). Special
issue of ReCALL on Mobile Assisted Language Learning.
Cambridge University Press.
 Thornton, P. & Houser, C. (2005). „Using mobile phones in
English education in Japan‟. Journal of Computer Assisted
Learning, 21, (3): 217-228.

43. Pre-Questionnaire:
Learner Profiles (1)
Aspect Group A Group B
Group size 12 8
Girls 5 4
Boys 7 4
Years of English 3 3
Positive Attitude to 2.42 3.11
Learning English STD: 0.51 STD: 0.33
Average score 6.37 7.6
at Secondary STD: 0.9 STD: 1.7
Speaking Skills: 6.58 7.55
(Self reported) STD: 0.9 STD 0.68

44. Pre-Questionnaire:
Learner Profiles (2)
Aspect Group A Group B
Like speaking in 2.67 3.78
class STD: 0.78 0.44
Actual speaking 1.5 3.56
STD: 0.52 STD: 0.53
Telecollaboration at n/a 2.0
school 0.0
Tel. Experience 1.38
in projects n/a STD: 1.0
Tel. Experience 1.83 1.44
IRL STD: 1.19 STD: 0.73

45. Pre-Questionnaire:
Learner perceptions: Oral L2 in
class
Aspect Group A Group B
L2 in lessons 2.1 5.0
STD: 0.50 0.0
Answers /Discussion 2.2 / 1.6 3.9 / 3.2
Pairwork 1.6 1.9
English IRL 1. Chat in 1. Holidays,
games 2. Chat in
2. Holidays games
3. Skype 3. Txt chat

46. Assessment of Oral skills
Aspect Group A Group B
Assessed Tasks in All: 1 All: 3.5
2010
Formal reports: Oral 35% ? 70% ?
skills included?
Is tele-testing valid? Yes: 35% Yes: 60%
? : 55% ?: 40 %
Post: Valid Yes: 60% 40%

47. Post-Questionnaire (1)
Aspect Group A Group B
Likert scale Disagree 1 Agree 4
Technically OK? 2.43 1.8
STD: 1.13 0.84
Read 2.8 3.0
Instructions STD: 0.7 0.7
Different from 2.7 2.8
expectation STD: 0.76 1.3
Questions: Not loud enough idem
complex, speed, Hard to Speed OK
loudness, remember

48. Post-Questionnaire (2)
Aspect Group A Group B
Likert scale Disagree 1 Agree 4
Answer time left 3.4 3.2
STD: 0.5 1.1
Expected Mark Just sufficient O.K
STD: 1.4 1.1
Problem Topics Networks; 1M Euros Good School; Climate
Fun to do 2.8 2.4
STD: 0.4 0.9
More pleasant 2.07 2.0
without teacher STD: 0.6 1.0

49. Speak. Listen. Learn.
www.learnosity.com
Twitter @learnosity