Menu/title will go here
OER Impact Studies Workshop
Dr. Rob Farrow
1-5 Dec 2014
Penang, Malaysia

ROER4D Impact Studies Workshop
Day 1

Introduction
Rob Farrow

@philosopher1978

oerresearchhub.org
#oerrhub

Goals of workshop facilitation
 Shared understanding of ROER4D-IS
 Harmonization of impact studies
 Sharing the OERRH experience
 Refinement of ROER4D-IS proposals

Methods for workshop facilitation
 Critical discussion of existing research
 Peer review
 Facilitating reflection on methods and claims
 Exploration of key concepts
 Making explicit what is assumed or implied
 Identifying problematic areas
 Effective planning

Things to avoid
✖ Dictating what methodology should be
✖ Being disrespectful or patronising
✖ Pleasing me

Icebreaker
Where in the world?
 Name, institution, country
 One key question
 Swapping places to present partner institutions

Overview of ROER4D-IS (CHW)
 Overview of objectives, activity, progress
 Expectations of impact studies
[see other slide deck]

ROER4D Objectives
 Build empirical knowledge base
 Developing research capacity
 Building scholarship networks
 Open curation of research
 Communicate research to influence
policy

ROER4D Strategies
 Knowledge building (degrees of openness, OA)
 Building research capacity (harmonization)
 Build network through conference, workshops, etc.
 Open curation (repositories, social media)
 Collaborative, supportive approach to leadership
 Seeking out creative synergies
 Effective (agile?) methods for collaboration
 Iterative evaluation

Expectations of ROER4D-IS
 Case studies provide detail relative to broad understanding
of the Global South developed through survey work and
ROER4D as a whole
 Balancing needs of network with individual needs
 Open by default: CC-BY, open data, OA publishing

Lunch, then
Presentation by grantees

AVU / Teacher Education in Sub-Saharan Africa
 Need for trained teachers and updated curriculum
 OER offer promise of addressing issues of access, quality, cost
 AfDB / UNDP resources in core subjects (Teacher Education)
 Fullan (2006) theory of change underpins change knowledge
 Examination of the conditions that sustain OER use
 Comparative analysis across 12 institutions
 Participatory approach to the research; qualitative data;
phenomenology

Darakht-e Danesh / Afghanistan
 Conflict has destroyed educational infrastructure
 OER gives educators independent access to content
 OER supports much needed adaptation and localization
 DD Library accessed via web, e-learning lab and mobile
 “Effective measurement” of impact on teaching quality
 Assumption that access to CPD resources will improve learning
outcomes (via improved literacies/competences)
 Survey based approach (which questions?) supported by analytics
from learning lab and website access; student records
 Theory of change: how is openness playing a role?

OER Impact in Asian non-formal ed. / Mongolia, India
 Plurality of ‘impacts’ (knowledge, skills, aspirations, attitudes) on
learners and trainers from various OER types
Focused on strategies for collaboration and sharing between formal
and non-formal learning providers
 Identify policies that improve quality and affordability of learning
 Using Bennett’s (1979) hierarchy of outcomes to evaluate impact
 Performance indicators = quantitative, qualitative, financial
 Open = openly licensed? (If not, what?)

OER in teacher education / OU Sri Lanka
 Action Research methodology (communities of practice)
 Fullan (1993) as a framework for understanding change
 4 hypotheses: changing pedagogical beliefs & practices; reduce cost
of learning; improve the quality of learning
 Running workshops to raise awareness
 Stakeholders: learners, teachers from six provinces & various levels of
study, subjects, etc. (nb. teachers as learners)
 Interpretative Phenomenological Analysis (IPA) as organising
framework for qualitative data collected – emerging themes / meanings

OU UK / Teacher Education in E. Africa
 Some research suggests that ‘quality’ teachers improve learning
 National policies advocate ‘learner-centred’ education but this is vague
 Focus on co-construction of knowledge as feature of openness
 TESSA is a consortium of OER producing universities & other
organizations who developed a repository of OER for teacher learning
 Practitioner responses to OER – attitudinal? Wider changes?
 5 institutions: qualitative data; interpretation; phenomenology
 Ontological & epistemological ‘shifts’ – is this clear?
 How precise a conception of openness is appropriate here?

Practices and Openness in African HE / UCT
 Global South tends to be seen as a recipient rather than provider
 UCT has several MOOC available or in production (FutureLearn)
 Various dimensions of openness: access, licensing, instruction
 Impact of MOOC on educator and student practice & view of open
 Impact of MOOC on valuing and repurposing of OER
 How MOOC initiate OER use and creation
 Methods: surveys, interviews, learning analytics, case studies
 Attempt to map research questions to MOOC development cycle

Cost-Effectiveness Analysis of OER / U Philippines OU
 Comparison of open vs non-open course development costs
 Quasi-experimental research design
 Participants chosen randomly from three disciplines (education,
health, management)
 Strict separation of OER vs standard groups
 Measuring: teacher competence; learner performance; quality of
materials – but how? Key indicators around savings per unit, efficacy

Virtual University Pakistan / Impact of OER in Pakistan
 Study split between two institutions
 Target of 88% ‘literacy’ by 2015 – only 60% at the moment
 Internet access and use is rising (nb. laptop scheme)
 Focus on lecture delivery; student performance; policy
 Large scale survey augmented by interviews
 Using Fullan’s theory of change
 COUP framework to assess cost difference and impact on student
outcomes (http://openedgroup.org/coup /
/http://jime.open.ac.uk/article/view/252)

ROER4D Impact Studies Workshop
End of Day 1

ROER4D Impact Studies Workshop
Day 2

Enhancing Research Value
Between OER Practitioners
across the Global
North/South Divide Through
Open Collaboration
Dr. Rob Farrow

• Research project at The Open University (UK)
• Funded by William & Flora Hewlett Foundation for two years
• Tasked with building the most comprehensive picture of OER impact
• Organised by eleven research hypotheses
• Collaboration model works across different educational sectors
• Global reach but with a USA focus
• Openness in practice: methods, data, dissemination
OER Research Hub
oerresearchhub.org
#oerrhub

Project Co-PILOT

Keyword Research Hypothesis
Performance OER improve student performance/satisfaction
Openness People use OER differently from other online materials
Access OER widen participation in education
Retention OER can help at-risk learners to finish their studies
Reflection OER use leads educators to reflect on their practice
Finance OER adoption brings financial benefits for students/institutions
Indicators Informal learners use a variety of indicators when selecting OER
Support Informal learners develop their own forms of study support
Transition OER support informal learners in moving to formal study
Policy OER use encourages institutions to change their policies
Assessment Informal assessments motivate learners using OER
‘Evidence’ is only evidence in relation to a claim or hypothesis:
the project hypotheses form the core of the metadata model.

OER Evidence Report
2014http://tinyurl.com/o
erevidence

OER Impact Map
http://oermap.org

• Research instruments applied
consistently across collaborations:
surveys, interview questions,
focus groups, etc.
• Supplemented by integration of
secondary research
• ‘Agile’ research, sprinting
• Thematic and methodological
cohesion provided by research
hypotheses
Research Process

• Synthesis and aggregation of other
case studies
• Sharing networks, resources and
experiences
• Comparisons with Global North
• Initial agile enquiry through OLnet,
SCORE and OERRH fellows
networks
• Capacity for further, responsive
research
Essence of the proposal

Synthesis
Synthesis Methods
• Isolating data by hypothesis, sector, country, or any combination
• Collaborative curation of research data
• Data visualization, reporting
• Editorial quality control exercised centrally
Validation
• Iteration through current and future patterns of evidence
• Open citation trails allow public auditing of evidence
• Community voting

in serv