The document discusses methodological challenges in evaluating learning and technology use in informal mobile settings and proposes a task model based on cultural historical activity theory to systematically represent user activity and capture the complexity of these settings. Case studies from two mobile learning projects are analyzed using the task model to identify how the technological and semiotic layers interact and where they provide support or cause conflicts.

1. Methods for studying learning,
collaboration and technology
use in mobile environments
Josie Taylor,
Professor of Learning Technology
Centre for Research in Computing
Institute of Educational Technology
The Open University
Zurich July 2006

2. Aim of talk
• Key issue:
– methodological challenges of trying to satisfy various
stakeholders when evaluating learning and technology use in
informal settings.
• A method
– for representing user activity (practices, strategies and
conflicts) that emerge when interacting with technological
systems in an informal mobile learning setting (learning?)
– semiotic and technological space.
– cultural historical activity theory, and develops Engestrom's
(1987) extended model of human activity.
– Task Models
Zurich July 2006

3. Acknowledgements
• Mike Sharples
• Patrick McAndrew
• Giasemi Vavoula
• Claire O’Malley
Zurich July 2006

4. Model of Mobile Learning:Research
from MOBIlearn
• Sharples, Taylor & Vavoula (in press) propose a
theory of learning in the mobile age as:
‘the processes of coming to know through
conversations across multiple contexts amongst
people and personal interactive technologies’
Zurich July 2006

5. A theory of learning in the mobile age
• “Conversation and context are essential
constructs for understanding mobile learning,
and offering implications for the ownership of
learning and the integration of mobile learning
with conventional education.”
• Draws on Pask’s (1976) Conversation Theory,
as does Laurillard (2002)
Zurich July 2006

6. Conversational Framework, Laurillard 2002
Partner provides facility for mediating agreements
Level of
Descriptions
Offers theories and ideas
Re-describes theories
Partner demonstrates or elicits Learner demonstrates
models and problem solutions Why questions and responses understanding of models and
problem solutions
Offers conceptions and explanations
Re-describes conceptions
Sets goals
Adjusts model
Partner acts to build models Learner acts to build models
and solve problems How questions and responses and solve problems
Acts
Modifies actions
Level of Actions
Partner provides facility for practical model building and
problem solving
Zurich July 2006

7. A theory of learning in the mobile age
• The focus of our investigation is not the learner, nor
their technology, but the communicative interaction
between these to advance knowing.
• Conversation is the driving process of learning. It is the
means by which we negotiate differences, understand
each other’s experiences and form transiently stable
interpretations of the world.
• Sharples, M., Taylor, J., and Vavoula, G., (in press) A Theory of Learning for the Mobile Age, in
R. Andrews and C. Haythornthwaite, Handbook of e-learning Research, Sage Publications.
Zurich July 2006

8. Research in progress
 Ways of capturing and analysing this
learning
– Based in activity theory
– Use scenarios to develop mobile learning
activities
– Run trials to test both the activity and the
technology
Zurich July 2006

9. Additional Issues
• Communication in large multi-site, multi-
national projects building systems (Carroll:
1995 – Scenarios; Taylor & Evans, 2005)
• Evaluators need to move between
stakeholders, responding appropriately to
each, as well as serving the needs of the end
users
• Stakeholders: funders, system designers,
system builders, educators, domain specialists,
curators, teachers, companies, end users…
Zurich July 2006

10. MOBIlearn Project
• Project Info:
– 33 month EU-funded with 15+ partners
– Circa 8m Euros
– 200+ members of SIG
• Goals:
– Designing an architecture for pedagogically sound
mobile learning environments
– Implementing an instantiation of the architecture with
current technologies
– Evaluate
Zurich July 2006

11. 3 MOBIlearn Scenarios:
3 types of learning
• Museum
– visitors to Art Gallery (informal)
• MBA
– professionals in full time work engaged in study
(formal curriculum, work based)
• First Aid Training
– volunteers in full time work needing training in
situ (voluntary, curriculum, work-based)
Zurich July 2006

12. Externally
initiated
Formal MBA
learning - not Resource
within scope for based learning
MOBIlearn
Externally Internally
structured structured
First Aid Museum
Voluntary Informal
learning learning
Internally initiated
Adapted from Livingstone 2001
Zurich July 2006

13. Scenario Development
Process
Zurich July 2006

14. Software Development Process
•Pilot studies
Literature
•Qnr studies
review
Scenarios •Interviews
•Diary studies
Guidelines Functional Non functional
Requirements
requirements
UML
System Design
Trials
Zurich July 2006

15. Development of Task Model
• Based in AT – a tool to:
– Represent learning in mobile settings
– Capture the complexity of the setting and
situation
– Capture aspects of the dialectical process of
appropriation both technical and semiotic
– Communicate with various stakeholders as
part of evaluation
Zurich July 2006

16. Activity Theory (AT)
• Cultural Historical Activity Theory (Leont’ev,
1978)
• Grounded in Vygotsky’s (1978) theorising
about the social-cultural development of
human mind (developmental studies of
higher mental functioning)
Zurich July 2006

17. Extended model of human activity
(Engeström, 1987)
Tools
Subject Object Outcome
Rules Division
Community of Labour
Zurich July 2006

18. Community Division of
Rules
labour
Reflecting social Not just human Negotiating both
rules and control in community but with other humans
technological also technological and technology
sense setting
Control Context Communication
Zurich July 2006

19. Semiotic Layer – Human/task focused
Zurich July 2006

20. Technological Layer –
Implementation focused
Zurich July 2006

21. Interaction between semiotic
and technological layers
Zurich July 2006

22. A pretty version
Zurich July 2006

23. Purpose:
• To hold a very complex situation in tension
– Looking for conflicts between layers
– Looking for support between layers
– Identifying relationships and interactions
– Allowing systematic manipulation
– Allowing multiple views to be represented
Zurich July 2006

24. Example: First Aid Training
• First aiders at the Open University
• Need to keep training up to date and
refreshed, but not many opportunities to
practise skills
• All are full-time workers, so training could
take place in workplace
• Communication/collaboration is as
important as access to content of training
Zurich July 2006

25. First Aid Training Scenario:
players
• Task leader:
– Configures task
– Interacts with participants
– Provides guidance
• Task participants
– act singly and in pairs to complete task
– Briefed and de-briefed as a whole group
Zurich July 2006

26. Task: Participant view
• Initial on-line chat (group)
• On-line Quiz (individual)
• Challenge is set (pairs)
• G1: On-line Brainstorming (pairs)
• G2: On-line Brainstorming (pairs)
• Get together on-line and vote (group)
Zurich July 2006

27. Agenda for the First Aid Trial
Zurich July 2006

28. Task: Task Leader view
• Set up content and direct participants to it
• Construct on-line agenda
• Oversee activity during chats
• Set tools to appropriate states (e.g. close
the vote at the end of voting)
Zurich July 2006

29. Task Model for First Aid Scenario
Technological Tool
(PDA and mobile phones)
Semiotic Tool
(First Aid Training)
Technological Object
(access to information)
Semiotic Object
Technological Subject
( knowledge & skills)
(User)
Semiotic Subject
Changed Object
(First Aider)
(revised knowledge
& skills)
Control Context Communication
Technological
Technological Technological
(network facilities, wireless)
(usability of device) Semiotic (texting, uploading pictures & text)
(community of first aiders)
Semiotic Semiotic
(constraints & protocols in first aid) (conversation; support of co-workers)
Zurich July 2006

30. Task: Evaluator view
• Developing scenario, constructing task
models and feeding into requirements or
design
• Designing evaluation instruments (e.g.
observation schedules, questionnaires etc)
Zurich July 2006

31. Iterative feedback
Generic
Requirements
Trials
Scenario
Semiotic Technical
Model Model
Design
Feedback
Zurich July 2006

32. Semiotic model for First Aid Scenario:
brainstorming
Semiotic Tool
(Collaborative: Working together to
assess situations)
Semiotic Object
(G1: online brainstorming)
Semiotic Subject
Changed Object
(First Aider)
(improved
performance leading
to G2)
Control Context Communication
Semiotic
Semiotic Semiotic
(partner for activity; larger group)
(Share; don’t diss (conversation; discussion;
other person’s pov; be Agreement; resolution of argument)
constructive)
Zurich July 2006

33. Technical Model for First Aid Trial
Technological Tool
Workspaces: agenda,
brainstorm, chat, vote
Technological Subject Technological Object
User at work Work through agenda
Control Context Communication
Moderated, user selections, Many
Wireless, fixed location, group
choices, different levels,
work, divided groups Guided, group and moderator, chats
task-based
Zurich July 2006

34. Post Trial Findings
• Technical:
– Tool choice allows unified working
– Usability issues are reduced by providing a
task focus
– Performance on MOBIlearn system is
adequate when working on real tasks
– Task-based control is available but does not
ensure shared views
Zurich July 2006

35. Post Trial Findings
• Semiotic:
– Moderating requires control information to and from
other users
– Alerting is needed to bring synchronicity
– Mobility needs the user to be mobile not just the
tools
Zurich July 2006

36. From trial: Support from technology
Technological Tool
Workspaces: agenda,
brainstorm, chat, vote
Technological Subject Technological Object
User at work Work through agenda
Control Context Communication
Moderated, user selections, Many
Wireless, fixed location, group
choices, different levels,
work, divided groups Guided, group and moderator, chats
task-based
Zurich July 2006

37. From trial: Conflicts in technology
Technological Tool
Workspaces: agenda,
brainstorm, chat, vote
Technological Subject Technological Object
User at work Work through agenda
Control Context Communication
Moderated, user selections, Many
Wireless, fixed location, group
choices, different levels,
work, divided groups Guided, group and moderator, chats
task-based
Zurich July 2006

38. From trial: Conflicts in technology
Technological Tool
Workspaces: agenda,
brainstorm, chat, vote
Technological Subject Technological Object
User at work Work through agenda
Control Context Communication
Moderated, user selections, Many
Wireless, fixed location, group
choices, different levels,
work, divided groups Guided, group and moderator, chats
task-based
Zurich July 2006

39. Task Issues for First Aid Scenario
Semiotic Tool
(First Aid Training)
Semiotic Object
( knowledge & skills)
Semiotic Subject
Changed Object
(First Aider)
(revised knowledge
& skills)
Control Context Communication
Mobility
Alerting Moderator role
Zurich July 2006

40. Museum Scenario
• Uffizi museum in Florence
• Groups of users looking at Botticelli
paintings
• Variety of handheld devices and phones
– Tablet PCs; pocket PCs; phones; PDAs
• Data collected by Italian partners
– Observation studies
– Videotape of movement through gallery
– Pre/post questionnaires (satisfaction) 40
Zurich July 2006

41. Museum Scenario data
• Good multimedia facilities and high levels
of usability in the tablet computers
facilitated the semiotic level:
– associations with other personal ‘devices’ (i.e.
diaries and calendars)
• Semiotic context has within it relatively
large numbers of young women
– levels of communication and sharing
increased
Zurich July 2006

42. Task Model Example: Museum Scenario
Technological Tool:
(PDA and mobile phones,
pocket PCs, tablet PCs)
Semiotic Tool
(Learning about paintings)
Technological Object
(access to information)
Semiotic Object
Technological Subject
( knowledge & skills)
(User)
Semiotic Subject
Changed Object
(Museum Visitor) (revised knowledge
& skills)
Control Context Communication
Technological
Technological Technological
(network facilities, wireless)
(usability of device) Semiotic (texting, uploading pictures & text)
(community of visitors)
Semiotic Zurich July 2006
Semiotic

43. Museum Scenario data
• Younger participants were keen on the idea of using
the Chat service to save, download and print
conversations so that they had a textual photograph to
remind them of their visit to the museum.
• Technological aspects of the scenario were supporting
and augmenting the semiotic activities, providing a
much richer experience for visitors.
Zurich July 2006

44. Support in the Museum Scenario
Technological Tool
(Pocket PC/ Notebook)
Semiotic Tool
(Learning about
Technological Object
paintings)
(access to information)
Technological Subject Semiotic Object
(User) ( knowledge & skills)
Semiotic Subject Changed Object
(museum visitor) (revised
knowledge &
skills)
Control Context Communication
Technological Technological Technological
(usability of device - good) (interesting content well (good comms facilities; chat)
Semiotic presented) Semiotic
(Social Rules: diary Semiotic (exchange of information/opinion)
functions/increased sharing) (Young people/young women)
Zurich July 2006

45. Conflict in the Museum Scenario
• In the case of the other technologies
(mobile telephones and PDA’s) we find
that the relative lack of usability in the
technological domain inhibits all these
developments in the semiotic.
Zurich July 2006

46. Conflict in the Museum Scenario
Technological Tool
(mobile phone/PDA)
Semiotic Tool
(Learning about
Technological Object
paintings)
(access to information)
Technological Subject Semiotic Object
(User) ( knowledge & skills)
Semiotic Subject Changed Object
(museum visitor) (revised
knowledge &
skills)
Control Context Communication
Technological Technological Technological
(usability of device -poor) (difficult interface, poor (good comms; chat)
Semiotic search) Semiotic
(Social Rules: little interaction Semiotic (communications not used)
with exhibits, system or other (community: no engagement)
visitors) Zurich July 2006

47. Conclusions
• Work-in-progress
• The task model helps to provide a structured
space within which to investigate success or
failure
• Enables evaluators :
– to disambiguate specific technology from
functionality
– to separate human issues from technological
issues, and put them back together again
– to try several different implementation
solutions in comparable circumstances
Zurich July 2006

48. Conclusions (2)
• Understanding the dialectic is very
important
– Process level – appropriation and its impact
on behaviour
– Specific technology level – what the tools are
offering in terms of functionalities and how the
user makes the best use of them to achieve
their goal (or not!)
• Focus on the activity keeps the learner
to the fore Zurich July 2006

49. Conclusions (3)
• So far the task models are proving useful
for evaluators
• Enables us to say more about what might
be going on in a given activity
• Enables us to pinpoint potential sources of
conflict within and between levels
• Provides a way of representing learning
and interaction with technology
Zurich July 2006

50. Future Work
• Computer support for diagrams
• Drilling deeper into scenarios
– How deep is deep enough?
– Is more useful data yielded?
– Are triangles the best representation?
– If so, why exactly?
Zurich July 2006