Joint presentation by David Kernohan and Viv Rolfe at #OER16 Conference in Edinburgh 2016. They took a critical look at the open education publishing community including some interesting insights into citation metrics.

1. Open education: "Runnin' with the Devil"
Vivien Rolfe and David Kernohan
@vivienrolfe @dkernohan

2. I am calling for a (radical?)
pedagogy caucus, a core,
self-identified group
committed to placing
pedagogy at the center of
the OpenEd movement.
(Robin De Rosa)
#opened15

3. At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old
solutions (Kuhn 1970).
Are we being critical enough? Where are
we being critical?
What are we
publishing and
where, to explore
facets of openness?
Are we critical in our
thinking and writing?
How do we measure
this?

4. su("Open education" OR "Open source
technology" OR "Open university") AND
su(Learning OR "Outcomes of education")
AND su(Students OR "Adult students" OR
"Independent study")
Systematic literature retrieval
(Pubmed MeSH + ERIC Thesaurus)
186 (Pubmed) + 627 (ERIC) retrieved
REVIEW OF TITLES AND ABSTRACTS TO
EXCLUDE
● Reports and other article types
● Interventions that weren’t “open”
● Those not an evaluation of learning
(but satisfaction)
REVIEW OF FULL PAPERS
● Lost interventions that weren’t “open”
● Not an evaluation of learning (but
satisfaction)
53 articles
5 evaluations of the impact of open
education innovation on learning
Albeit a ‘pilot’ search -
indications success there is
not an established culture
of evaluation.
A good proportion of
abstracts didn’t contain the
detail to judge the quality
and content of the paper.

5. 5 evaluations of the impact of open
education innovation on learning
1) Publication bias.
2) Evidence of critical reflection
and writing.
3) Citation bias - many
introductions are just ‘shopping
lists’ of past research.
4) Ball (2015) noted in his science
study, 2.4% citations in papers
were negative.
All 5 presented positive findings
(main outcome, second outcome
e.g. learning gain, test results)
All stated limitations of their
methodology. Conclusions were
positive but placed in context.
Of 62 citations within the
introductions, 6 were within a
negative critical context.

6. From 3Rs to the 5Rs!

7. At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old
solutions (Kuhn 1970).
Are we being critical enough? Where are
we being critical?
What are we
publishing and
where, to explore
facets of openness?
Are we critical in our
thinking and writing?
How do we measure
this?

8. “I found the simple life ain’t so simple”
- Other ways of examining the literature.

9. Hypotheses:
1. Most of the *really good stuff*
is in the blogs - blog citations
are a significant part of the
OER literature
2. Blog citations can be
measured via the parts of the
scholarly graph we have
access to.
3. Citations of blog posts will be
in papers concerning the kinds
of topics that bloggers write
about.

10. Hypotheses:
1. Most of the *really good stuff*
is in the blogs - blog citations
are a significant part of the
OER literature
2. Blog citations can be
measured via the parts of the
scholarly graph we have
access to.
3. Citations of blog posts will be
in papers concerning the kinds
of topics that bloggers write
about.
Methods:
1. Collect citations and basic
stats from Google Scholar
using Harzing’s “Publish or
Perish” tool (H1, H2)
2. Measure word frequency in
citing paper titles using Voyant
tools.
3. Measure word frequency in
each blog corpus (via rss)
using Timdream Wordcloud.
4. Compare (H3)

11. Blogs examined
David Wiley
http://www.opencontent.org/blog
George Siemens
http://elearnspace.org/blog
Audrey Watters
http://www.hackeducation.com
Martin Weller
http://nogoodreason.typepad.co.uk
Also
“Edupunk” (term)
“connectivism.ca” (former blog)
Notable findings.
1. The papers that cite George Siemens’
blog have a h-index of 38! (this is better
than most education researchers)
2. The papers that cite Connectivism.ca, a
former blog of his, has a still impressive h-
index of 29.
3. Large amounts of the “edupunk” literature
are in Spanish.
4. Siemens and Wiley are most likely to have
their blogs cited together.

12. Semantometrics and citation metrics
Image from http://Semantometrics.org - no license indicated but assumed CC-BY
due to funder requirements.

13. What is a citation anyway?
Citations serve a range of purposes, both technical and social. The reasons for citation are contested and
no comprehensive social theory of citation exists. Broadly there are two sets of theories around the
motivations for citation, normative, and social-constructivist.
Normative theories hold that citations are a means of expressing norms of the research community,
primarily the norm of assigning credit where it is due, and the norm of showing evidence
transparently. Normative theories focus on shared practice and purposes that are generally held to
be transparent and well understood.
Social constructivist theories focus on reasons for citation that are social or cultural, often differing
across communities, that are not (necessarily) associated with the normative aspects of
acknowledging intellectual debt.
https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/
- Cameron Neylon

14. Semantic distance: data

15. Semantic distance: grouping

16. Semantic prediction
How good are the top five words from each blog at predicting the titles of citing
papers?
David Wiley 0.7038
George Siemens 0.77373
Connectivism.ca 0.58621
Audrey Watters 0.12
Martin Weller 0.57639
Total instances of top five words in titles
Number of titles
Lower number = less predictive
Therefore a greater
semantic distance between
the blog and citing papers.

17. Summary
The community called for critical debate on the direction of Open Education, but
where do we do this? Robust research is lacking (as always identified by
systematic approaches), and we are subject to publication and citation bias.
Social media brings critical discussions into the open as a new academic space.
Some blogs are cited in formal literature, either within the literature they would be
seen as a part of, or in wider literature.
Systematic review approaches, semantometrics and citation metrics are all
different ways of examining a literature, each with strengths and weaknesses.
Future work should draw on a multiplicity of methodologies.

18. In answer to Adam and Robin, “open textbooks ugh” - “is that what we meant”?
The comment reflects a common assertion that specificity must necessarily, in
and of itself, betray the spirit of openness and informality. (Katz 1972)
At times, revolutions are misleading. Consider the spreading educational
innovation which has come to be known as the British Infant School.The later is a
concept which stands for a collection of educational approaches, all using the
open classroom as a learning environment. Note that it is a collection of
approaches, not just one theoretical or practical idea. (Nelson 1972)

19. Further Reading
Ball P (2015). Science papers rarely cited in negative ways. Nature News.
Croon A (2015). Is that what we meant? http://adamcroom.com/2015/11/is-that-what-we-meant/
Katz L G (1972). Research on Open Education: Problems and Issues.
Kuhn T (1970). Scientific Revolutions (2nd. ed., Enlarged), Chicago: The University of Chicago Press.
Nelsen J (1972). Open Minded, Thought-Filled Education.
De Rosa R (2015). Open textbooks. Ugh.http://robinderosa.net/uncategorized/open-textbooks-ugh/
Knoth P and Herrmannova D (2016) Semantometrics http://semantometrics.org/
Neylon C (2016) What constitutes research data? What is citation?
https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/