This document discusses ways to make computer-assisted language learning (CALL) environments more intelligent by bridging the gap between closed and open item types. It defines classical closed and open item types and their limitations. It then proposes several new item types, including half-closed items like select text and dictation that provide more freedom in learner output while still allowing automated correction. Half-open items like translation and reformulation are also suggested to provide more freedom than closed items but with more limited and predictable answers. Finally, supported open items are proposed which are fully open-ended but provide automated feedback and correction based on model answers, keywords, and blacklists/whitelists of acceptable answers. Challenges for future intelligent language learning environments

1. Bridging the gap between closed and open items
or
how to make CALL more intelligent
Piet Desmet & Bert Wylin
Fleat VI Harvard University
August 11-15, 2015

2. 1. Item-based learning & testing environments (ILTE): definition
2. CALL, SLA & LT: different views on a “classical” ILTE
3. Beyond the closed & open items in an ILTE
4. Half-closed items
5. Half-open items
6. Supported open items
7. Challenges for ILTEs
8. Conclusion

3. 1. Item-based learning & testing environments
(ILTE): definition
1.1. Definition of an item
“A digital item asks the learner to react to a given input,
leading to an output that is treated by the system”.
Typically, items are
• part of a series (or stand on themselves)
• structured (organized),
• (minimally) metadated,
• reusable,
• multimedial,
• stored in an item bank

4. 1.2. “Classical” items: closed or open
CLOSED OPEN
Learner output
level of freedom limited totally free
# correct answers limited to 1 or a few many
predicatibility answers maximal very limited
Output treatment
correction type automated manual
reliability high
Examples
closed: multiple choice, multiple answer, drag & drop, order, fill gaps, etc.
open: upload text file, audio or video-recording (without correction)

5. 2. CALL, SLA & LT:
Different views on “classical” ILTEs
2.1. Within CALL: tutor vs tool
Computer as a tutor (tutorial CALL):
ILTEs still crucial today although need for improvement
“Many programs being produced today feature little more than visually
stimulating variations on the same gap-filling exercises used 40 years ago”
(Beatty 2003: 11)
vs
Computer as a tool (multimedia, CMC, web 2.0, etc.):
ILTEs less important since main focus is on
CMC, social media, immersive virtual worlds, etc.
allowing for communicative activities and tasks

6. Tutorial CAL not even on the
Hype cycle for education (Gartner, 2013)

7. 2.2. Within SLA: cognitive vs socio-cultural
Different perspectives on SLA:
cognitive perspective: cognitive processing by the learner
(noticing, motivation, etc.)
socio-cultural perspective: impact of social environment of the learner
(collaboration between learners, scaffolding by interlocutor, etc.)
-> ILTEs are more crucial within a cognitive framework

8. 2.3. Within language teaching:
behavioral vs communicative/task-based
° Different methods:
grammar-translation
direct methods
communicative approach
task-based language teaching (TBLT)
etc.
-> ILTEs are considered to be less crucial in TBLT than before (cf. “drill & kill”)
° Different focus:
focus on form vs focus on meaning
rule-based vs usage-based
knowledge-oriented vs skills-oriented
teacher-centered vs learner-centered
-> ILTs are mainly associated with the left focuses

9. 3. Beyond the closed & open items in an ILTE
3.1. Limitations of “classical” closed items
(a) too limited freedom at the level of the learner output
(b) too limited cognitive complexity
(c) limited number of item types
(d) less suited for advanced learners
-> need for more “intelligent” CALL

10. 3.2. Old wine in new bottles…
Till recently only technological innovation
floppy disk (DOS only)
cd-rom (Windows)
website
platforms
CMS LMS learning platform testing platform
SPOC MOOC

11. 3.3. “Our” solution:
bridging the gap between closed and open items
= pedagogical innovation
still automated correction with high reliability
BUT:
Learner output: more freedom
more correct answers
less predictability
www.edumatic.com

12. http://www.delta-associates.com/what-about-the-old-advice-dont-reinvent-the-wheel-is-it-stupid-or-smart/

13. 4. Half-closed items
CLOSED HALF-CLOSED
Learner output
level of freedom limited more free
# correct answers limited to 1 or a few limited
predicatibility answers maximal maximal
Output treatment
correction type automated automated
reliability high high
Examples
(1) select text
(2) dictation
4.1. Definition

14. 4.2. Select text
Learner output: selection of relevant passage in a text
The locus of the points of interest is not given beforehand
-> more freedom at the level of the learner output
Mechanism behind these items:
° mark the keyword(s) in a given text (sentence or paragraph)
& link/group these keywords
° define ranges for selection
(ranges as such don’t influence the score)
° prepare feedback for correct and wrong keywords

16. Bert: TE VERVANGEN DOOR VB VOOR TAAL!

17. 4.3. Dictation
Learner output: transcription of a (bookmarked) audio file
Learner doesn’t know what are the possible points of interest
Learner can decide not to transcribe certain parts (without impact
on the correction mechanism)
-> more freedom at the level of the learner output
Mechanism behind these items:
Approximate string matching

18. Approximate String Matching @ Edumatic
• Normalization of input (or not)
• caps
• interpunction
• accents
• algorithm based on best match with input
I inform you to XXX the (…) tomorrow (XXX).
• 3 codes: delete, insert, substitute (error)
• Attempts model:
attempt – feedback – attempt – (…) – solution model

19. Approximate String Matching @ Edumatic
• “Brackets” model
[[In the/Every] morning, Mary listens to the radio./Mary listens to the radio [in
the/every] morning.]
• not only feedback,
also show solutions based on best match with student’s input
showing non matching solutions is an option

20. Bert: VB van gecorrigeerd dictee toevoegen

21. 5. Half-open items
HALF-CLOSED HALF-OPEN
Learner output
level of freedom more free more free
# correct answers limited to 1 or a few many
predicatibility answers maximal limited
(but feasable and
progressive build up)
Output treatment
correction type automated automated
reliability high average to high
Examples
(1) translate
(2) reformulate
(3) correct
5.1. Definition

22. 5.2. Translate
xxx = substitute
(…) = insert
(xxx) = delete

23. 5.2. Translate 2.0
Correction on the letter level

24. BE-ODL
21 maart 2006

25. 5.3. Reformulate/correct

26. 6. Open supported
HALF-OPEN OPEN SUPPORTED
Learner output
level of freedom more free free
# correct answers many many
predicatibility answers limited even more limited
Output treatment
correction type automated automated
reliability average to high average to high
6.1. Definition

27. 6.2. Mechanism
• open question with free learner input
• with due date
• generation of feedback on the basis of:
model answer
keyword matching
• white list (+ score)
• and
• if
• if then
• black list (0 or – score)
• negations (and range)

28. 4 functions of supported open item type:
1) Creation of open question
with model answer, black list, white list, elaborated feedback, etc.
2) Publication of this item
fix due date, select student groups, follow-up received
answers, etc.
3) Half-automated correction of the answers
correction proposal on the basis of the available info
manual correction of scores and adaptation of
black list & white list (-> update of automatic scores)
4) Generation of feedback report
individualised feedback, fix scores, add personal comments
notify all users by automatically generated mail

29. Item Input: create New item

30. Item Input: create New item
Add original
text in
“logical
units”
(paragraph
or
sentences)
Add
instruction

31. Students make translations
•Use quick codes to have alternative
correct solutions
• Eg. [on passe/on passera/on fera/on effectuera/sera
passée/sera prise/l'infirmière glissera]
•Decide about keyphrases
•Add scores per keyphrase
•Add feedback per keyphrase
• including error specific feedback

32. Student/candidate response
With or without
Correction button
(practice vs.
exam)
Student/candidate
types answer

33. Students make translations
•While correcting student input,
• Add more options
• Update all existing corrections
constantly
•See the effect of the updates in new
student input:
• less and less corrections to make
• more and more keyphrases
recognized (both correct and wrong
answers)

34. Item Input: create New item
Add translation
keywords and
keyphrases

35. Item Input: create New item
Option: set
options for
spell checker
Option: provide
model answer
(for feedback)

36. Update translations/scores
Update, add,
delete translations
System asks to
apply changes in
translations to all
students

37. Final reporting
Based on updated
translations and
scores
See
individual and
group results

38. Bert: TE VERVANGEN DOOR VB VOOR TAAL!

39. • Use of supported open exercises in three steps
• Step 1 : try out
as a marking and feedbacktool (aid) used by teaching staff
-> human verification and improvement of the black & white list is
necessary
• Step 2 : learning
result of scenario 1 can be used as an exercise with full automatic corrective
and elaborated feedback (with human intervention!)
-> human verification
and e-mail feedback
• Step 3 : exam simulation
results of scenario 2 can be used as an exercise with full immediate
automatic corrective and elaborated feedback (without human
intervention!)

40. •!
Supported open exercises are
not limited to languages
•Excellent experiences in
•Law faculty
•Medical faculty

41. 7.1. Adaptivity
-> frontend: e.g. adaptive item sequencing
adaptive feedback
7.2. Gamification
-> frontend: e.g. Badges & rankings
Collaboration & competition
7.3. Flexible delivery mode
-> frontend e.g. Integration in App or digital textbook
Integration in skills oriented learning environment
7.4. Output correction through NLP
-> from backend to frontend: e.g. parsing half-open input
7.5. Analysis of tracking & logging data
-> from backend to frontend: e.g. reporting
7. Challenges for ILTEs

42. http://ingvihrannar.com/wp-content/uploads/2014/02/testing_cartoon.jpg
7.1. Adaptivity

43. 4D-model of adaptive instruction
Vandewaetere, Desmet & Clarebout 2011 / Vandewaetere & Clarebout, 2012
Cognition
(e.g. prior knowledge)
Affect
(e.g. motivation)
Behavior
(e.g. need for help)
What elements in the
environment to
adapt?
Adapt during interaction,
between interactions, prior to
interaction?
Who’s in control?
Learner vs. instructor decides
what/when/how to adapt?
Or both?

44. http://www.slideshare.net/piet_desmet/2015-0522-presentatiecalicodesmet-vandewaetere-def

45. 7.2. Gamification
MindSnacks ‘Swell’

46. Using gameplay mechanics for non-game applications
- Challenges embedded in a compelling story
- Various layers or levels & character upgrades
- Rewards (scores & badges)
- Social interacton & peer motivation through competition
http://www.playwarestudios.com/wp-content/uploads/2013/07/gbl-cartoon.jpg

47. 7.3. Flexible delivery mode
“Classical” delivery mode
Items
(in Activities)
from: Horton, William, E-Learning by Design, Wiley, 2011

48. (a) From a technological point of view
ILTE as a
- smartphone app
- daily small interactive e-mail or sms
- micro-series of items, embedded in a digital textbook
- etc.
More flexibility

49. (b) From a pedagogical point of view
“Skinning” of item types to be integrated in a skills oriented environment
e.g. multimedia learning environment focusing on audio-visual comprehension
e.g. situational judgment test / inbox exercises
www.franel.eu Nedbox

50. 7.4. Output correction through NLP or statistical methods
NLP ASM
- by definition language dependent
- high R&D effort
+ by definition language independent
+ lower R&D effort
- unequal availability and quality of existing
algorithms and tools
- technologies not easily transferable to new
tools/environments
- slow
+ high availability of existing ASM algorithms
+ easily reusable algorithms
+ higher speed
+ better granularity (fineness with which
input can be analyzed)
- highly depending on teacher’s input
(number of correct answers predicted by
teacher)
+ language specific intelligent feedback
generation by the algorithm (cf. E-Tutor T.
Heift)
- no automatic language specific feedback
generation

51. NLP: lemmatisation -> tagging -> parsing (-> semantic analysis?)
Statistical methods: combine advantages of ASM & NLP!
Statistical error detection:
training a classifier based on a corpus of corrected utterances
with feedback
(cf. PhD Ruben Lagatie)

52. 7.5. Analysis of tracking & logging data
From manually entering data to online massive storage
From self-reporting data to behavioral data
From single measurements to longitudinal measurements
From inaccessible to everywhere
From big data to rich data…

53. Not the data, but the views on the data make it interesting…
For the user: - detailed reporting (from generic to specific!)
- advice on next steps
For the teacher: - reporting at individual and group level
- item analysis

54. For the user: detailed reporting (from generic to specific reports)

55. For the user: advise on next steps

56. For the teacher: reporting at group level
Bert: illustratie invoegen!

57. For the teacher/content author: item analysis

58. http://ayende.com/blog/2421/when-does-it-
make-sense-to-reinvent-the-wheel
8. Conclusion

59. CLOSED HALF-CLOSED HALF-OPEN OPEN
SUPPORTED
OPEN
Learner output
level of
freedom
limited more free more free free totally
free
# correct
answers
limited to 1
or a few
limited many many many
predicatibilit
y answers
maximal maximal limited very
limited
very
limited
Output treatment
correction
type
automated automated automated automated manual
reliability high high average to
high
average to
high

61. More info
Piet Desmet Bert Wylin
Piet.Desmet@kuleuven.be Bert.Wylin@kuleuven.be
B. Wylin@televic.com
www.linkedin.com/in/pietdesmet www.linkedin.com/in/bertwylin
@PietDesmet
ITEC
www.kuleuven.be/itec