Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Difficulties Evaluating cMOOCS (Open Education Conference 2013)


Published on

A presentation on various ways one might try to evaluate the effectiveness of cMOOCs, and some questions and concerns about each one, ending with a question: how best should we do this?

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

Difficulties Evaluating cMOOCS (Open Education Conference 2013)

  1. 1. Difficulties evaluating cMOOCs: Negotiating Autonomy and Participation #DiffCMOOC Christina Hendricks University of British Columbia, Vancouver Open Education Conference, November 2013 Presentation licensed CC-BY
  2. 2. Connectivist MOOCs Network: Facilitating connections between people and information, ideas (not transmitting knowledge from central source) (Siemens 2012 ) Distributed: Takes place in multiple spaces (blogs, wikis, tweets, discussion boards, webinars, etc.): “A MOOC is a web, not a website” (Downes 2013a ) #OOE13 Open Online Experience 2013-2014
  3. 3. Connectivist MOOCs when & how to participate; Autonomy: Participants decide create own learning goals, choose own paths through course (McAuley et al. 2010; Downes 2009 Open: free access available to anyone with reliable internet connection; curriculum open to alterations by participants (Downes, 2013b ) From a video on MOOcs by Dave Cormier & Neal Gillis (licensed CC-BY)
  4. 4. Evaluating cMOOC effectiveness Do they achieve goals? Which goals? • Of designers • Of participants • Connectivism: making connections w/people & information • What sort of entities cMOOCs are & whether fulfill purposes (Downes)
  5. 5. Goals of cMOOC designers Participant autonomy: •What happens in course depends on what participants do: “learners are expected to actively contribute to the formation of the curriculum through conversations, discussions, and interactions” (Cormier & Siemens, 2010 ) Hub & Spoke, flickr photo by Antony_Mayfield, licensed CC-BY •course may be successful (or fail) in ways designers never envisioned
  6. 6. Goals of participants E.g., Lane, 2013 •participants may have goals that don’t fit course; a problem if not fulfilled? • may not have any goals; just want to see what happens • course may have other benefits not captured in participants’ goals; may miss this if focus on their goals • benefits may take a long time to realize Huma Bird tweet analysis, #whyopen
  7. 7. Goals of participants What one might do: •Ask participants at the end what they got out of the course, with or without reference to their original goals •Return to them six months or more later to ask again-perhaps see longer-term effects •Consider how to support learners in being self-directed, working to achieve own goals in a cMOOC (e.g., Kop, Fournier and Mak, 2011 )
  8. 8. Connectivism: connections among people & info • “Knowledge is defined as a particular pattern of relationships and learning is defined as the creation of new connections and patterns as well as the ability to maneuver around existing networks/patterns." (Siemens 2008 ) • "At its heart, connectivism is the thesis that knowledge is distributed across a network of connections, and therefore that learning consists of the ability to construct and traverse those networks." (Downes, 2007
  9. 9. Connectivism: connections among people & info Participation rates: •surveys of participants: Milligan, Littlejohn and Margaryan, 2013 •log data from P2PU platform: Ahn, Weng and Butler, 2013 •mixed methods: • Waite, Mackness, Roberts and Lovegrove, 2013 • Kop, 2011
  10. 10. Connectivism: connections among people & info Negotiating autonomy & participation •Tension: need at least some active participation, but participants must have autonomy •Lurkers valued? Just b/c may become active participants? #ds106zone May 25-June 6, 2013
  11. 11. Purposes of cMOOCs themselves Downes 2013b •look at what sorts of entities cMOOCs are, what purposes they serve, whether designed well for those purposes (rather than how they’re used) •To evaluate a cMOOC, consider: “what a successful MOOC ought to produce as output, without reference to existing (and frankly, very preliminary and very variable) usage.” (Ibid.) • output: “emergent knowledge”
  12. 12. Emergent knowledge: See something or say something: Jakarta, Flickr photo shared by Eric Fischer, licensed CC-BY Blue dots tweets; red dots Flickr, white dots both In a successful cMOOC, “the structure of the interactions produces new knowledge, that is, knowledge that was not present in any of the individual communications, but is produced as a result of the totality of the communications, in such a way that participants can through participation and immersion in this environment develop in their selves new (and typically unexpected) knowledge relevant to the domain.” (Downes, 2013b; emphasis added)
  13. 13. Networks that tend to produce emergent knowledge 1.Autonomy 2.Diversity 3.Openness 4.Interactivity/Connectedness (Downes, 2013b) Anek Rang, Ek Sang, Flickr photo shared by Sanjay, licensed CC-BY
  14. 14. How evaluate cMOOCs acc to these criteria? • Don’t measure each aspect of a cMOOC against these as if a checklist; rather, consider cMOOCs a “language” and a course as an expression in it • These criteria should be considered “an aid, used to assist a person who is already fluent in MOOC design (or at least in the domain or discipline being studied) [to] recognize the quality (or lack of quality) of a MOOC” (Downes, 2013b).
  15. 15. Questions & concerns about this approach • How can we determine if emergent knowledge has been produced? Where would we look? Whom would we ask? • Seemingly exclusive focus on design and purpose of cMOOCs-doesn’t consider the experiences of participants E.g., participant experiences in a cMOOC: Mackness, Mak and Williams, 2010 Crowd, Flickr photo by James Cridland, lic ensed CC-BY (altered)
  16. 16. Footprints of emergence • Williams, Karousou and Mackness, 2011 • Emergent and prescriptive learning--need balance Williams, Mackness and Gumtau, 2012 - Draw “footprints” of courses to map degrees of prescriptive and emergent learning
  17. 17. Footprint for CCK08 published in Williams, Mackness & Gumtau 2012 (licensed CC-BY) Centre: prescriptive learning Light area: apex of emergent learning Periphery: “edge of chaos” Map points based on 24 factors, in four clusters See wiki for factors, how to draw footprints, and more: http://footprints-of-emergence.wikispac /
  18. 18. Full circle • Footprints are not meant to provide evaluations of courses by themselves • Instead, provides way to evaluate if course fits purposes (Footprints of emergence wiki: ) • After have footprint, ask: “Is this appropriate, or fit, for the purpose and context of the course and for you, and/or the particular learners?” Back to the beginning...
  19. 19. What now? Suggestions?
  20. 20. THANK YOU! Twitter: @clhendricksbc Blog: Slides, video and bibliography at: