Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open education: a critical appraisal.

138 views

Published on

Joint presentation by David Kernohan and Viv Rolfe at #OER16 Conference in Edinburgh 2016. They took a critical look at the open education publishing community including some interesting insights into citation metrics.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Open education: a critical appraisal.

  1. 1. Open education: "Runnin' with the Devil" Vivien Rolfe and David Kernohan @vivienrolfe @dkernohan
  2. 2. I am calling for a (radical?) pedagogy caucus, a core, self-identified group committed to placing pedagogy at the center of the OpenEd movement. (Robin De Rosa) #opened15
  3. 3. At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old solutions (Kuhn 1970). Are we being critical enough? Where are we being critical? What are we publishing and where, to explore facets of openness? Are we critical in our thinking and writing? How do we measure this?
  4. 4. su("Open education" OR "Open source technology" OR "Open university") AND su(Learning OR "Outcomes of education") AND su(Students OR "Adult students" OR "Independent study") Systematic literature retrieval (Pubmed MeSH + ERIC Thesaurus) 186 (Pubmed) + 627 (ERIC) retrieved REVIEW OF TITLES AND ABSTRACTS TO EXCLUDE ● Reports and other article types ● Interventions that weren’t “open” ● Those not an evaluation of learning (but satisfaction) REVIEW OF FULL PAPERS ● Lost interventions that weren’t “open” ● Not an evaluation of learning (but satisfaction) 53 articles 5 evaluations of the impact of open education innovation on learning Albeit a ‘pilot’ search - indications success there is not an established culture of evaluation. A good proportion of abstracts didn’t contain the detail to judge the quality and content of the paper.
  5. 5. 5 evaluations of the impact of open education innovation on learning 1) Publication bias. 2) Evidence of critical reflection and writing. 3) Citation bias - many introductions are just ‘shopping lists’ of past research. 4) Ball (2015) noted in his science study, 2.4% citations in papers were negative. All 5 presented positive findings (main outcome, second outcome e.g. learning gain, test results) All stated limitations of their methodology. Conclusions were positive but placed in context. Of 62 citations within the introductions, 6 were within a negative critical context.
  6. 6. From 3Rs to the 5Rs!
  7. 7. At the heart of innovation is the reuse of knowledge and ideas and ability to critically reflect and reject old solutions (Kuhn 1970). Are we being critical enough? Where are we being critical? What are we publishing and where, to explore facets of openness? Are we critical in our thinking and writing? How do we measure this?
  8. 8. “I found the simple life ain’t so simple” - Other ways of examining the literature.
  9. 9. Hypotheses: 1. Most of the *really good stuff* is in the blogs - blog citations are a significant part of the OER literature 2. Blog citations can be measured via the parts of the scholarly graph we have access to. 3. Citations of blog posts will be in papers concerning the kinds of topics that bloggers write about.
  10. 10. Hypotheses: 1. Most of the *really good stuff* is in the blogs - blog citations are a significant part of the OER literature 2. Blog citations can be measured via the parts of the scholarly graph we have access to. 3. Citations of blog posts will be in papers concerning the kinds of topics that bloggers write about. Methods: 1. Collect citations and basic stats from Google Scholar using Harzing’s “Publish or Perish” tool (H1, H2) 2. Measure word frequency in citing paper titles using Voyant tools. 3. Measure word frequency in each blog corpus (via rss) using Timdream Wordcloud. 4. Compare (H3)
  11. 11. Blogs examined David Wiley http://www.opencontent.org/blog George Siemens http://elearnspace.org/blog Audrey Watters http://www.hackeducation.com Martin Weller http://nogoodreason.typepad.co.uk Also “Edupunk” (term) “connectivism.ca” (former blog) Notable findings. 1. The papers that cite George Siemens’ blog have a h-index of 38! (this is better than most education researchers) 2. The papers that cite Connectivism.ca, a former blog of his, has a still impressive h- index of 29. 3. Large amounts of the “edupunk” literature are in Spanish. 4. Siemens and Wiley are most likely to have their blogs cited together.
  12. 12. Semantometrics and citation metrics Image from http://Semantometrics.org - no license indicated but assumed CC-BY due to funder requirements.
  13. 13. What is a citation anyway? Citations serve a range of purposes, both technical and social. The reasons for citation are contested and no comprehensive social theory of citation exists. Broadly there are two sets of theories around the motivations for citation, normative, and social-constructivist. Normative theories hold that citations are a means of expressing norms of the research community, primarily the norm of assigning credit where it is due, and the norm of showing evidence transparently. Normative theories focus on shared practice and purposes that are generally held to be transparent and well understood. Social constructivist theories focus on reasons for citation that are social or cultural, often differing across communities, that are not (necessarily) associated with the normative aspects of acknowledging intellectual debt. https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/ - Cameron Neylon
  14. 14. Semantic distance: data
  15. 15. Semantic distance: grouping
  16. 16. Semantic prediction How good are the top five words from each blog at predicting the titles of citing papers? David Wiley 0.7038 George Siemens 0.77373 Connectivism.ca 0.58621 Audrey Watters 0.12 Martin Weller 0.57639 Total instances of top five words in titles Number of titles Lower number = less predictive Therefore a greater semantic distance between the blog and citing papers.
  17. 17. Summary The community called for critical debate on the direction of Open Education, but where do we do this? Robust research is lacking (as always identified by systematic approaches), and we are subject to publication and citation bias. Social media brings critical discussions into the open as a new academic space. Some blogs are cited in formal literature, either within the literature they would be seen as a part of, or in wider literature. Systematic review approaches, semantometrics and citation metrics are all different ways of examining a literature, each with strengths and weaknesses. Future work should draw on a multiplicity of methodologies.
  18. 18. In answer to Adam and Robin, “open textbooks ugh” - “is that what we meant”? The comment reflects a common assertion that specificity must necessarily, in and of itself, betray the spirit of openness and informality. (Katz 1972) At times, revolutions are misleading. Consider the spreading educational innovation which has come to be known as the British Infant School.The later is a concept which stands for a collection of educational approaches, all using the open classroom as a learning environment. Note that it is a collection of approaches, not just one theoretical or practical idea. (Nelson 1972)
  19. 19. Further Reading Ball P (2015). Science papers rarely cited in negative ways. Nature News. Croon A (2015). Is that what we meant? http://adamcroom.com/2015/11/is-that-what-we-meant/ Katz L G (1972). Research on Open Education: Problems and Issues. Kuhn T (1970). Scientific Revolutions (2nd. ed., Enlarged), Chicago: The University of Chicago Press. Nelsen J (1972). Open Minded, Thought-Filled Education. De Rosa R (2015). Open textbooks. Ugh.http://robinderosa.net/uncategorized/open-textbooks-ugh/ Knoth P and Herrmannova D (2016) Semantometrics http://semantometrics.org/ Neylon C (2016) What constitutes research data? What is citation? https://rdmetrics.jiscinvolve.org/wp/2016/04/12/what-constitutes-research-data-what-is-citation/

×