1. REALL: Rubric for the
Evaluation of Apps in
Language Learning
Elena Martín Monje (UNED), Jorge Arús Hita (UCM),
Pilar Rodríguez Arancón (UNED) & Cristina Calle Martínez (UCM)
ATLAS Research Group
Logroño, 2-3 May 2013

2. Table of contents
 Introduction
 Pedagogic assessment of mobile learning apps for EFL
◦ Stage 1: Analysis and categorization of EFL apps available
◦ Stage 2: Design of an evaluation rubric for the pedagogic
assessment of EFL apps
◦ Stage 3: Rubric for the evaluation of apps in language
learning (REALL)
 Conclusion
 References

3. Introduction
 Mobile learning
 Mobile-Assisted Language Learning (MALL)
 SO-CALL-ME project
 Stages in this research study:
Stage 1:
Analysis and
categorization
of EFL apps
available
Stage 2:
Design of
rubric for
pedagogic
assessment of
EFL apps
Stage 3:
Creation of
REALL (Rubric
for the
Evaluation of
Apps in
Language
Learning

4.  Previous rubrics & categorizations
Taken from http://langwitches.org/blog/2012/03/31/ipad-apps-and-blooms-taxonomy/

5. Taken from Tony Vincent’s learninginhand.com

6. Stage 1: Apps available
 3 criteria: apps’ cognitive value, similarity/
complementarity with SO-CALL-ME
 67 apps assessed
 Frequent technical problems downloading/ starting apps
 Vast majority for Apple devices
 Most expensive: dictionaries, grammar
 Many for 2-3€ or with initial free sample pack
 Categories:
◦ Games
◦ App versions of printed materials (e.g. coursebooks)
◦ Apps for pronunciation/ grammar/vocabulary

7. Stage 2: Rubric for pedagogic
assessment
 Quality guide for digital learning objects (Fernández-
Pampillón et al., 2011)
• Cognitive value and pedagogic
coherence (1)
• Content quality (2)
• Capacity to generate learning (3)
• Interactivity and adaptability (4)
• Motivation (5)
Pedagogical
criteria
• Format and layout (6)
• Usability (7)
• Accessibility (8)
• Visibility (9)
• Compatibility (10)
Technical
criteria

8.  Rubric with descriptors and scale 1-5.
 E.g. Criterion 3 in rubric for pedagogic assessment:
 The 5 top EFL apps of Stage 1 assessed with this rubric:
1 2 3 4 5
3. Capacity to
generate
learning
Contents do not
help to achieve
learning goals
or autonomous
learning
Contents help
autonomous
learning but not
clearly the
achievement of
the initial
learning goals
Contents help
to achieve the
learning goals
but neither
autonomous
learning nor
relating old
knowledge to
new knowledge
Contents help
to achieve the
learning goals
but not
autonomous
learning OR not
relating old
knowledge to
new knowledge
Contents help
to achieve the
learning goals,
autonomous
learning and
relating old
knowledge to
new knowledge

10. Stage 3: Rubric for linguistic
assessment
 Common European Framework of Reference for Languages
(Council of Europe, 2001)
 Focus on levels A2-B2
 Initial rubric for oral
comprehension
 Categories:
 Level
 Type of texts
 Topics
 Delivery
 E.g. Delivery in rubric for linguistic assessment:
1 2 3 4 5 N/A
Delivery
Language
difficulty, clarity
and speed mix
different levels.
If adaptive,
delivery is
clearly not well
adapted
Language difficulty,
clarity and speed
rarely belong to the
same level. If
adaptive, delivery
rarely corresponds
to the right level
Language difficulty,
clarity and speed
tend to belong to
the same level. If
adaptive, delivery
more often than not
corresponds to the
right level
Language difficulty,
clarity and speed
usually belong to
the same level. If
adaptive, delivery
usually
corresponds to the
right level
Language
difficulty, clarity
and speed belong
to the same level.
If adaptive,
delivery
corresponds to
the right level

11.  Parallel process in stage 2 & stage 3: 1 rubric, 2 evaluators
• Stage 2: 91/100
• Stage 3: 29/40
• Stage 2: 89/100
• Stage 3: 37/40
• Stage 2: 82/100
• Stage 3: 13/40
• Stage 2: 81/100
• Stage 3: 19/40
• Stage 2: 64/100
• Stage 3: 14/40

12.  The most comprehensive apps assessed were Speakingpal
and Learn English Elementary Podcasts

13. Conclusions
 Pedagogic & technical quality ≠ linguistic value &
adequacy for EFL teaching & learning
 Importance of all three dimensions:
◦ Pedagogical
◦ Technical
◦ Linguistic
 Piloting of stage 2 & stage 3 rubrics enables a future
study on a larger scale
 Future theoretical framework for successful,
pedagogically & linguistically sound EFL apps

14. References
 Arús-Hita, Jorge; Pilar Rodríguez-Arancón and Cristina Calle-
Martínez (in press) “A pedagogic assessment of mobile learning
applications”. In Proceedings of ICDE 2013, Mobilizing Distance
Education, UNED, Madrid.
 Council of Europe (2001). Common European Framework of
Reference for Languages. Cambridge: Cambridge University
Press.
 Fernández-Pampillón Cesteros, Ana; Elena Domínguez Romero
and Isabel de Armas Ranero (2011) Herramienta para la revisión
de la Calidad de Objetos de Aprendizaje Universitarios (COdA):
guía del usuario. v.1.1. Madrid: e-prints Complutense. Retrieved
from http://eprints.ucm.es/12533/ (accessed 18/04/2013).

15. THANK YOU
emartin@flog.uned.es