Presented at the CALRG Annual Conference, June 2015. http://www.open.ac.uk/research-groups/calrg/conferences/annual-conference Abstract Over the past three years we have been developing a framework intended to facilitate both the occurrence and evaluation of incidental learning. This work has been part of the MASELTOV project, supported by the European Commission, eInclusion programme FP7-ICT-7. The project intends to exploit the potential of mobile services for promoting integration and cultural diversity in Europe, and is focusing on support for immigrants with particular needs e.g. those who have not learned foreign languages, and who have a cultural background that contrasts with that of their host country.   The goal of the Incidental Learning Framework is to facilitate the creation of technology rich learning opportunities which emanate from incidental learning i.e. learning that is spontaneous and unplanned. As a design tool its use should encourage links and triggers to structured and reflective learning to back up and deepen learning that happens incidentally. The framework is a descriptive mechanism that enhances analysis and evaluation of incidental learning, and a generative tool to support discussions around software system design. It facilitates the communication of learning design ideas both visually and textually.   Initial work was presented at the CALRG conference in 2012, and the framework has been under development since that time (Brasher et al., 2012, Gaved et al., 2013, Kukulska-Hulme et al., in press). Development of the framework has included expert evaluation, and changes in response to data acquired from field trials of a mobile app intended to support language and cultural learning. In this paper we describe the development, use and evaluation of the framework to date, and reflect on lessons learnt.

1. www.maseltov.eu
Designing and evaluating
incidental learning
Andrew Brasher, Ann Jones, Agnes Kukulska-
Hulme, Mark Gaved, Eileen Scanlon, Lucy Norris
IET, The Open University

2. www.maseltov.eu Maseltov project
“Mobile Assistance for Social Inclusion and
Empowerment of Immigrants with Persuasive
Learning Technologies and Social Network
Services” www.maseltov.eu
Joanneum Research, Austria
CURE - Centre for Usability Research &
Engineering, Austria
University of Applied Sciences, FH Joanneum,
Austria
Athens Information Technology, Greece
busuu.com -, Spain
Universitat Oberta de Catalunya, Spain
Open University, UK
Coventry University, UK
Czech Technical University, Czech Republic
Telecom Italia SpA, Italy
Fluidtime Data Service GmbH, Austria
Fundacian Desarrollo Sostenido, Spain
Verein Danaida, Austria
Migrants Resource Centre, UK
Pearson Publishing, UK

3. www.maseltov.eu Contents
• Overview of the Incidental Learning Framework
• Evaluation & use
– Workshop at plenary meeting
– expert researchers
– Iliinsky’s work
• Challenges
–Use in evaluation: analysis of MK trial data
• Conclusions

4. www.maseltov.eu Research questions
Research question driving the development of the Incidental Learning Framework

5. www.maseltov.eu
Goals of the Incidental
Learning Framework
• to facilitate the creation of technology rich learning
opportunities which emanate from incidental
learning
i.e. learning that is spontaneous and unplanned;
• to encourage links and triggers to structured and
reflective learning to back up and deepen learning
that happens incidentally;
• to enhance analysis and evaluation of incidental
learning;
• to support discussions around software system
design;
• to facilitate the communication of learning design
ideas both visually and textually.

6. www.maseltov.eu Review of frameworks
• Design frameworks
–to support the planning and instantiation of new
learning events and situations
• Analytical frameworks
–for analysing, understanding and evaluating
learning events
• Learner models
–to describe the state of the learner so that a
system can react appropriately

7. www.maseltov.eu Review of frameworks
• Design frameworks
• Analytical frameworks
• Learner models
Design framework
Analytical framework

8. www.maseltov.eu Examples of frameworks
Analytical
• Theory of Learning for
the Mobile Age
(Sharples et al., 2007)
• A Pedagogical
Framework for Mobile
Learning
(Park, 2011)
Design
• Ecology of Resources
design framework
(Luckin, 2010)
• Four dimensional
framework
(de Freitas et al., 2010)
• Language learning
defined by time and
place
(Kukulska-Hulme, 2012)
Learner model
• Advances in learner &
skill modeling in
intelligent learning
environments
(Desmarais and Baker, 2011)

9. www.maseltov.eu Kolb’s learning cycle

10. www.maseltov.eu ILF version 1
Outcomes
Social
Tasks
Tools
Plac
e
Learner’s journey
Time
Incident:
understandi
ng notice
board
Language
tool
Start
TextLens
Task:
finding
station
Task: asking
directions
Reflect/plan
Structured
learning
Language
Tool/Social
Task:
navigating
Navigation
tool
Language
Tool
Language
Tool/Social

11. www.maseltov.eu Example, stage 1

12. www.maseltov.eu Example, stage 2

13. www.maseltov.eu Evaluation
• Workshop: adding detail to scenarios
• Expert focus group

14. www.maseltov.eu Evaluation
The framework
• “generates helpful discussions about the learner
journeys which immigrants might make”
• Exposes “misunderstandings and different
interpretations of the same problem space
through different domain experts views”
• The current representation was sufficient to
engage participants and encourage discussion.
(Gaved et al., 2013)

15. www.maseltov.eu Evaluation
• Ordering of dimensions was questioned
• Outer dimension of “learner’s journey” was
debated:
- too general?
- how to show e.g. social inclusion in to a new
community
(Gaved et al., 2013)

16. www.maseltov.eu Extension ideas (1)
Visualisation of specific learner journeys, showing opportunities for planned and
incidental language learning
Gaved et al. (2012)

17. www.maseltov.eu
Extension ideas (2)
Visualisation of the range of affordances that may be associated with particular places
that a MASELTOV service user is likely to encounter during their daily activities

18. www.maseltov.eu Extension ideas (3)
Visualisation of the range of dimensions associated with motivation and concerns
that might be felt by the service user while engaging with mobile situated language
learning

19. www.maseltov.eu Choosing visual properties
Iliinsky, N. (2013)

20. www.maseltov.eu Place
• Not just a location - also contextual information.
• ‘A Place is a geographical and historical context for
Names and Locations’ (Gillies, 2011).
• a ‘Place’ has a name and a location, and each may
be applicable for a particular time period.
“Andrew’s workplace”
currently has
- Name: “Jennie Lee
Building”
- Location: map
reference SP 88626
37058

21. www.maseltov.eu Learning outcomes
Categorical Related

22. www.maseltov.eu Linear view
Place
Task
Tools
Outcomes
Social
Time Evening
Language learning
Practise work related vocabulary
Understand work related vocabulary
Sleep/Break
19.00 23.00 07.00
Language learning
Practise past tense
Communicate
successfully in work
related situation
Breakfast
08.00

23. www.maseltov.eu Data gathering
• Pre trial questionnaire:
paper
• Post trial interviews:
face-to-face
E.g.
Lesson
access
duration:
2’39s
Synchronous
& located
with MApp use
• Event data from
the MApp
• Usage of other
apps
• Geo contextual
data
Asynchronous
& not located
with MApp useWeekly survey:
online

24. www.maseltov.eu
Participant’s use of Mapp
services
I have really liked the English course,
it is very practical for the times when
you have to use English for different
occasions..it has helped me a lot The
second is the translation tool
because now I take a photo of
everything I don't understand and at
that moment I know what it says ... It
has helped me a lot.
I have
learned a
lot more
English
language
I have learned
to develop
myself in
different aspects
of speaking
English with the
help of the
course

25. www.maseltov.eu Use in analysis
• Example language lesson data
key value timestamp
lesson transport 22/01/2015 13:22
publication mas_trav 22/01/2015 13:22
score 90.90908813 22/01/2015 13:22
lesson plan_journey 24/01/2015 18:41
publication mas_trav 24/01/2015 18:41
score 0 24/01/2015 18:41
lesson plan_journey 24/01/2015 18:41
publication mas_trav 24/01/2015 18:41
score 0 24/01/2015 18:41
lesson transport_chaos 24/01/2015 20:07
publication mas_trav 24/01/2015 20:07
score 100 24/01/2015 20:07
lesson body_and_health 24/01/2015 20:59
publication mas_health 24/01/2015 20:59
score 85.7142868 24/01/2015 20:59

26. www.maseltov.eu Conclusions
• Development of ILF as a design tool
–Work to do is guided by literature and
experimental findings
• Development of ILF as an analytical tool
–Data available complex and multi-faceted;
–Initial experiments started;
–Transfer of design view not straightforward?
–Learner models?;
–Suggestions welcome!
Levels of
abstraction?
Level of
detail?

27. www.maseltov.eu To do
• Include stuff from conclusions slide from 2012 Calrg
presentation
• …feedback from eval D7.12.
• Other stuff….
• Issues – learning outcomes can be shown at gps
locations, but need to show learning journey
• To point to new developments – show new version

28. www.maseltov.eu Extensions
• D7.1.2 i.e.
- SCAMP
- Learning journey

29. www.maseltov.eu References
• de Freitas, S., Rebolledo-Mendez, G., Liarokapis, F., Magoulas, G. & Poulovassilis, A. 2010.
Learning as immersive experiences: Using the four-dimensional framework for designing and
evaluating immersive learning experiences in a virtual world. British Journal of Educational
Technology, 41, 69-85.
• Desmarais, M. C. & Baker, R. S. J. D. 2011. A review of recent advances in learner and skill
modeling in intelligent learning environments. User Modeling and User-Adapted Interaction,
22, 9-38.
• Kukulska-Hulme, A. 2012. Language learning defined by time and place: A framework for
next generation designs. In: D´ıAZ-VERA, J. E. (ed.) Left to My Own Devices: Learner
Autonomy and Mobile Assisted Language Learning.
• Iliinsky, N. (2013). Choosing visual properties for successful visualizations. Retrieved
24/4/2014, from http://bit.ly/successfulvis
• Luckin, R. 2010. Re-Designing Learning Contexts: Technology-Rich, Learner-Centred
Ecologies (Foundations and Futures of Education). 208.
• Park, Y. 2011. A Pedagogical Framework for Mobile Learning: Categorizing Educational
Applications of Mobile Technologies into Four Types. International Review of Research in
Open and Distance Learning, 12, 78-102
• Sharples, M., Taylor, J. & Vavoula, G. 2007. A Theory of Learning for the Mobile Age. In:
Andrew, R. & Haythornwaite, C. (eds.) The Sage Handbook of Elearning Research. London:
Sage.