1. The document discusses open education and calls for more critical evaluation and analysis within the open education community. 2. It notes that few studies have rigorously evaluated the impact of open education on learning outcomes and that the existing studies show biases. 3. The author proposes several methods for more critically and systematically analyzing research in open education such as citation network analysis, evaluating publication biases, and acknowledging its intellectual history.

1. Open. But not for criticism?
Dr Vivien Rolfe BSc PhD PFHEA
National Teaching Fellow
@vivienrolfe
University of the West of England,
Bristol, UK

2. I am calling for a (radical?)
pedagogy caucus, a core,
self-identified group
committed to placing
pedagogy at the center of
the OpenEd movement.
(Robin De Rosa)
My sense of #OpenEd2015 is that there was widespread
interest in ambitions beyond open textbooks but, lacking a
clearly articulated ladder of ambition, there wasn’t a lot of focus
on it. http://mfeldstein.com/is-open-education-a-movement/

3. This is important, because a community of practice is a
shared history of that practice.
http://followersoftheapocalyp.se/keep-the-fire-notes-on-my-opened15-presentation/

5. Two interesting aspects of OpenEd 2015…
The trust and willingness to be open to criticism within the
space of the community…
…but a perceived lack of criticality within it.

6. Critical, criticism, criticality…..
At the heart of innovation is the reuse of knowledge and
ideas and ability to critically reflect and reject old solutions
(Kuhn 1970).
Criticality can be viewed as a pedagogical outcome with three
interrelated elements: critical knowledge, critical thinking
skills and critical spirit (James 2001).
Are we being critical? Where are
we being critical? Is this enough?

7. Meta-research approaches
Citation network analysis
Analysis of inter-disciplinarity
History and legacy
What are we
publishing and where,
to theorise about
openness?
Are we critical in our
writing and thinking?
Sociospatial and
historical aspects?
Retrieve a non-biased sample of papers through
quasi-systematic approach: #search for ‘open*’
‘student’ ‘learning outcomes’
Analysis of article outcomes
Citation analysis
Other biases?
(References – see end)

8. Retrieval
(Pubmed MeSH + ERIC Thesaurus
+open keyword searching)
Results
186 (Pubmed) + 627 (ERIC) retrieved
REVIEW OF TITLES AND ABSTRACTS TO
EXCLUDE
● Reports and other article types
● Interventions that weren’t “open”
● Those not an evaluation of learning
(but satisfaction)
REVIEW OF FULL PAPERS
● Interventions that weren’t “open”
● Not an evaluation of learning
53 articles
5 articles did evaluate the impact of
open education on learning and
learner outcomes
1. There are few
evaluations of the impact
of open on learning.
2. A good proportion of
abstracts didn’t contain
the detail to judge the
quality and content of the
paper.
Search results

9. 1. Publication bias evident (within
small sample).
2. Some evidence of critical
reflection and writing.
3. Citation bias - introductions
contained affirmatory articles.
4. Ball (2015) noted in his science
study, 2.4% citations in papers
were negative.
All 5 presented positive findings as
primary and secondary outcome
e.g. learning gain, test results
All stated limitations of their
methodology and approaches.
Of 62 citations within the
introduction sections, 6 were of a
negative critical context (9%).
5 articles did evaluate the impact of
open education on learning and
learner outcomes

10. 0
1
2
3
4
5
6
7
8
9
10
0
5
10
15
20
25
30
35
Not evaluation of
learning / open (i.e.
satisfaction)
Not open Case study (not
evaluation)
Categories of articles retrieved / exclusion criteriaNumbersofarticlesretrieved
Weller 2016 – OER Knowledge Cloud analysis = Policy, Practitioner, OER in developing nations,
Pedagogy, Open data/practice/access

11. From the 3 Rs to the 5 Rs
0
20
40
60
80
100
120
140
160
180
1970-74 1975-79 1980-84 1985-89 1990-94 1995-99 2000-04 2005-09 2010-15 2016
Numbersofarticlesretrieved
Number of articles retrieved over time

12. “Open education is used here to designate a general approach to teaching
and learning which presumes the child's right and competence to make
important decisions; views the teacher more as a facilitator of learning than
a transmitter of knowledge, and abundant alternatives and choice for
students”.
Barth 1971
Well cited
“Open education movement (Katz 1972)…commitment to humanistic values
including self-determination, freedom of children and aesthetic appreciation”.
Cited once in 2013 and not since 1984
“While the open education movements and educational technology are often
seen as mutually hostile, the challenge in education for the future is to find
ways to develop the full range of each individual’s capacities.”
Resnick 1972
Cited once in 1996 and not since 1970’s

13. Laura Czerniewicz talks about global knowledge inequalities – financial, social, xxx
http://www.slideshare.net/laura_Cz/laura-czerniewicz-open-repositories-conference-2016-dublin
= Paper
= Citation

14. = Paper
= Citation
Does the discipline lack hetergeneity, (it showed confirmatory bias of citing many papers from that journal)
and does this represent egotism of some scholarly work? Lawani 1982

15. Discussion
1. Publishing and citation creates a footprint and is our legacy.
2. There was more theorising in the 1970’s than today. This history
goes largely unrecognised despite parallels with our present
humanistic approaches and shared values.
3. The open education community is critical within itself but not of
itself. There are few robust evaluations within my chosen topic,
and biases not dissimilar to research in general.
4. This small sample may suggest an element of geographical
inequality, and subject differences in citation hetergeneity.
5. We need evidence and legacy for creating persuasive
arguments.

16. BHAGs?
Shared spirit and ethos of open
Altruistic community
Lowering costs
A successful movement
Theoretizing openness
My sense of #OpenEd2015 is that there was
widespread interest in ambitions beyond open
textbooks but, lacking a clearly articulated ladder of
ambition, there wasn’t a lot of focus on it.
http://mfeldstein.com/is-open-education-a-movement/

17. BHAGs?
Shared spirit and ethos of open
Altruistic community
Lowering costs
A successful movement
Theoretizing openness
Evaluations?
Critical
Unbiased
Published
Cited
My sense of #OpenEd2015 is that there was
widespread interest in ambitions beyond open
textbooks but, lacking a clearly articulated ladder of
ambition, there wasn’t a lot of focus on it.
http://mfeldstein.com/is-open-education-a-movement/

18. BHAGs?
Shared spirit and ethos of open
Altruistic community
Lowering costs
A successful movement
Theoretizing openness
Evaluations?
Critical
Unbiased
Published
Cited
My sense of #OpenEd2015 is that there was
widespread interest in ambitions beyond open
textbooks but, lacking a clearly articulated ladder of
ambition, there wasn’t a lot of focus on it.
http://mfeldstein.com/is-open-education-a-movement/
Criticality
Shared history
Acknowledged and
built-upon

19. BHAGs?
Shared spirit and ethos of open
Altruistic community
Lowering costs
A successful movement
Theoretizing openness
Evaluations?
Critical
Unbiased
Published
Cited
Criticality
Shared history
Acknowledged and
built-upon
Community
Critical spirit
Multidisciplinary
bridging communities
My sense of #OpenEd2015 is that there was
widespread interest in ambitions beyond open
textbooks but, lacking a clearly articulated ladder of
ambition, there wasn’t a lot of focus on it.
http://mfeldstein.com/is-open-education-a-movement/

20. BHAGs?
Shared spirit and ethos of open
Altruistic community
Lowering costs
A successful movement
Theoretizing openness
Evaluations?
Critical
Unbiased
Published
Cited
Critical knowledgeCritical skillsCritical spirit
My sense of #OpenEd2015 is that there was
widespread interest in ambitions beyond open
textbooks but, lacking a clearly articulated ladder of
ambition, there wasn’t a lot of focus on it.
http://mfeldstein.com/is-open-education-a-movement/
Criticality
Shared history
Acknowledged and
built-upon
Community
Critical spirit
Multidisciplinary
bridging communities

21. So get thinking!!
Felix by @mdvfunes https://www.flickr.com/photos/97994829@N03/29650137712 CC BY-NC-SA

22. Background
OpenEd2015 Archive. http://openedconference.org/2015/index.html%3Fp=368.html
Including:
Croon A (2015). Is that what we meant? http://adamcroom.com/2015/11/is-that-what-we-meant/
De Rosa R (2015). Open textbooks. Ugh.http://robinderosa.net/uncategorized/open-textbooks-ugh/
Barth RS (1972). Open Education and the American School.
Czerniewicz L (2016). Open Repositories Conference Dublin. Available:
http://www.slideshare.net/laura_Cz/laura-czerniewicz-open-repositories-conference-2016-dublin
Lawani, Stephen M. "On the heterogeneity and classification of author self-citations." Journal of the
American society for Information Science 33.5 (1982): 281.
James N (2001). Criticality, Critical Pedagogy and a Critical Legal Education. WG Hart 2001 Legal
Workshop.
Resnick LB (1972). Open Education: Some Tasks for Technology." Educational Technology 12(1), 70-76.
Katz L G (1972). Research on Open Education: Problems and Issues.
Kuhn T (1970). Scientific Revolutions (2nd. ed., Enlarged), Chicago: The University of Chicago Press.
Weller M (2016). Different aspects of the emerging open education discipline.
https://altc.alt.ac.uk/2016/sessions/different-aspects-of-the-emerging-open-education-discipline-1283/

23. Methods
Higgins JPT and Green S eds (2008). Cochrane handbook for systematic reviews of interventions.
Vol. 5. Chichester: Wiley-Blackwell. (Guide to systematic searching)
Gasevic D et al (2014). Where is research on massive open online courses headed? A data
analysis of the MOOC Research Initiative. The International Review of Research in Open and
Distributed Learning 15(5). (Details of content and network analysis)
Rodrigues V (2013). Publication and reporting biases and how they impact publication of research.
Editage.com. Available: http://www.editage.com/insights/publication-and-reporting-biases-and-
how-they-impact-publication-of-research (Introduction to biases in publishing)
Ioannidis JPA, et al (2015). Meta-research: evaluation and improvement of research methods and
practices. PLoS Biol 13(10). (Meta-research techniques for evaluating research practice).
EPPI Centre (2016). Reviewing tools: keyword strategy. Available:
https://eppi.ioe.ac.uk/cms/Default.aspx?tabid=184. (Toolkits for systematic reviews in education –
keywords, study quality evaluation).
Ball P (2015). Science papers rarely cited in negative ways. Nature News.
Knoth P and Herrmannova D (2016). Semantometrics http://semantometrics.org/
Neylon C (2016) What constitutes research data? What is citation?
https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/