The document outlines plans for developing an evaluation framework for a project involving games and sensors for financial decision training. It discusses holding stakeholder workshops to develop shared definitions and an evaluation framework involving research questions, methods, and timelines. The framework will use a participatory, iterative approach drawing on utilization-focused evaluation, participatory design, and learning design principles. Key project milestones are identified relating to workshops, developing the framework, and conducting evaluation studies throughout the project.

1. Game and Sensor Solutions for Financial Decision Training and
Support
Work package 6
Kick-Off Meeting, CIMNE
March 9-11, 2009
[The Institute of
Educational Technology,
The Open University]
[The Institute of
Educational Technology,
The Open University]

2. Objectives
 To reach agreement on the project’s overall
evaluation framework, which embeds
formative evaluation at each stage of the
project
 To ensure that all participants start from a
common basis and make use of the latest
developments in the different RTD areas
[wp 6][wp 6]
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

3. Objectives and Aims
 The main objective however is the
development of a sound evaluation
framework within which RTD activities can
be effectively organised and executed.
 The framework provides the vehicle to ensure
that comprehensive, ongoing evaluation is
built into all facets of the project and that
evaluation findings feed back into the
ongoing development activities of the project

4.  Project Stakeholder Workshops
 (m4, m33)
 Evaluation Framework
 (m9)
 Research questions and instruments
 (Throughout)
 Evaluation studies
 (m12, m24, m36)
[WP 6][WP 6]
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

5.  Milestones
 M6.1 Initial stakeholder workshop has taken place (m4)
 M6.2 xDELIA evaluation framework initial draft
established (m4)
 M6.3 First iteration of pilot studies undertaken (m24)
 M6.4 Second iteration of pilot studies undertaken (m33)
 M6.5 Final stakeholder workshop (m33)
[WP 6][WP 6]
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

6.  Need to work closely with all partners
 Importance of getting a shared understanding and
terminology for the project
 Evaluation needs to iteratively feed into the R&D
activities and vice versa
 State of the art report in m 6 is a collation of reports
from wp 2-5
wp 6wp 6
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

7.  Methods of working
 Tools to be used
 What do partners currently use?
 Means of communication
 What do partners currently use?
 Project website, blogs, twitter etc?
 Role of OU evaluation research fellow
 Coordinating, contributing?
[WP 6][WP 6]
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

8. Approach and theoretical perspectives
 Approach
 Participatory
 Iterative
 Useful
 Draws on the following:
 Utilization-focused evaluation
 Participatory design
 Learning design

9. Evaluation - definition
 Evaluation is the systematic acquisition and
assessment of information to provide useful
feedback about some object
http://www.socialresearchmethods.net/kb/intreval.htm
 Generic goal is to provide useful feedback to
stakeholders to influence decision-making or policy
formulation through the provision of empirically-
driven feedback
 Many different types, our approach is ‘participant
oriented’ and formative (i.e. feeds into the project
developments on an ongoing basis)

10. Utilization-focused evaluation
 Evaluation should be judged by their utility and
actual use
 Evaluation designed with use in mind i.e. how real
people in the real world will apply evaluation
findings and experience the evaluation process
 Intended users are more likely to use evaluations if
they understand and feel ownership of the
evaluation process and findings; they are more likely
to understand and feel ownership if they've been
actively involved

11. Participatory design
 Active engagement of the end users in the design
process to ensure it meets their needs
 Take account of the context of use and the users
 Move away from computerisation of human skills to
giving user better tools to work with
 Users perceptions and feelings about technology are
taken account of
 Views technology in context; as processes rather
than products

12. Learning design
 A new research field that has emerged in recent
years, partly in response to a desire to see better
use of technologies to support learning
 Concerned with the development of tools, models
and schema for supporting and making explicit
design decisions in the creation of learning
interventions/activities

13. Design and evaluation framework
Formulation
of intervention
Design
of intervention
Implementation
of intervention
Formulation
of questions
Data
collection
Data
analysis
Utilization
of results
Reflection
On results
Evaluation
Design
Adapted from
www.socialresearchmethods.net/kb/pecycle.php

14. Design and evaluation framework
Formulation
of activity
Design
of activity
Implementation
of activity
Formulation
of questions
Data
collection
Data
analysis
Utilization
of results
Reflection
On results
Evaluation
Design

15. Example
 Design
 Formulation: Agree
outline for a game for
use inWP2 to address
specific research
questions
 Design: Creation of the
game
 Implementation: Use in
a specific context &
collection of data
 Reflection: On findings
 Evaluation
 Formulation: Agree
which evaluation
questions are to be
considered
 Data collection: Agree
appropriate methods
 Data analysis: Analyse
findings
 Utilization: Feedback
and application

16. Examples
 PI stakeholder design workshop
 Participants: children, teachers,
local centre representatives
 Structured day – outline of
problem, presentation of inquiry-
model, brainstorm design ideas in
groups, reflections
 Representing pedagogy
 Use of the CompendiumLD tool to
articulate a design

17. Examples continued…
 Brainstorming and refining
research questions
 Use of mind mapping to develop a
shared, collective set of research
questions
 Design interviews
 Stakeholders design perceptions
 Teachers (how do they design,
where do they get ideas, how do
they represent/share designs, how
do they evaluate effectiveness?)

18. Examples continued…
 ‘Cloudfests’
 Agile development, iterative
presentation and feedback
 Design challenge workshop
 Design a short course in a
day, participants work in
teams supported by resource
stalls
 Evolving understanding
 Working papers, and a project
definition wiki

19. Research
questions
Research
questions
StakeholdersStakeholders MethodsMethods
Activity

20. Instructions
 Write down
 evaluation research questions
 potential data collection methods
 stakeholders and their interests
 Write one item per post-it note and stick up
 Read other people’s post-its
 Cluster together those that are related
 Draw connections between questions,
methods and stakeholders

21. Post its
 Write down evaluation research questions you would
like to see addressed in the project
 Write down suggestions for how the question might
be answered – what data collection methods could
be used?
 Browse the LTDI evaluation cookbook for ideas
www.icbl.hw.ac.uk/ltdi/cookbook/contents.html
 Who are the stakeholders for the evaluation and
what are they interested in? What methods have you
used before?

22. Division of labour
CENTRALTEAM
 Capturing initial vision
statements
 Development of D & E
framework
 Ongoing evaluation of
XDelia processes
 Support and advice
 Synthesis of local
evaluations
LOCALTEAMS
 Input into the development
of the D & E framework
 Design of local
interventions
 Data collection and
analysis of findings
 Contribute to evaluation of
XDelia processes
 Production of trial reports

23.  Agree focus for stakeholder workshop
 To agree a cross-project design and evaluation framework
 Agree intended outcomes
 Shared understanding and clear definitions
 A design and evaluation framework of research questions,
methods and indicative timescales
 Arrange date for stakeholder workshop
[WP 6][WP 6]
Kick-Off Meeting, CIMNE Barcelona, 9-11 March 2009

24. Useful references
 Oliver, Harvey, Conole and Jones (2007), Evaluation, in G. Conole and
M. Oliver (Eds), Contemporary perspectives in e-learning research,
RoutledgeFalmer: London
 LTDI evaluation cookbook
http://www.icbl.hw.ac.uk/ltdi/cookbook/contents.html
 Research methods knowledge base
http://www.socialresearchmethods.net/kb/evaluation.php
 Patton, M. (1997), Utilization-focused evaluation, 3rd
Edition, Sage
Publications
 Schuler, D. and Namioka, A. (1993), Participatory design, Lawrence
Erlbaum Associates
 Lockyer, L., Bennett, S., Agostinho, S. and Harper, B. (Eds), in Handbook
of Research on Learning Design and Learning Objects: Issues,
Applications and Technologies, Hersey PA: IGI Global