A presentation on various ways one might try to evaluate the effectiveness of cMOOCs, and some questions and concerns about each one, ending with a question: how best should we do this?

1. Difficulties evaluating
cMOOCs: Negotiating
Autonomy and Participation
#DiffCMOOC
Christina Hendricks
University of British Columbia, Vancouver
Open Education Conference, November 2013
Presentation licensed CC-BY

2. Connectivist
MOOCs
Network: Facilitating
connections between
people and information,
ideas (not transmitting
knowledge from central
source) (Siemens 2012
http://is.gd/K5JfXK )
Distributed: Takes place
in multiple spaces (blogs,
wikis, tweets, discussion
boards, webinars, etc.):
“A MOOC is a web, not a
website” (Downes 2013a
http://www.downes.ca/presentation/327
)
#OOE13 Open Online Experience 2013-2014
http://www.ooe13.org

3. Connectivist
MOOCs when & how to participate;
Autonomy: Participants decide
create own learning goals, choose own paths through course
(McAuley et al. 2010 http://is.gd/6j1X1k; Downes 2009 http://is.gd/AYc84B)
Open: free access
available to anyone
with reliable internet
connection;
curriculum open to
alterations by
participants
(Downes, 2013b
http://is.gd/Downes2013 )
From a video on MOOcs by Dave Cormier & Neal Gillis (licensed CC-BY)
http://is.gd/cQwOSP

4. Evaluating cMOOC
effectiveness
Do they achieve goals?
Which goals?
• Of designers
• Of participants
• Connectivism: making connections w/people &
information
• What sort of entities cMOOCs are & whether fulfill
purposes (Downes)

5. Goals of cMOOC
designers
Participant autonomy:
•What happens in course depends on
what participants do: “learners are
expected to actively contribute to the
formation of the curriculum through
conversations, discussions, and
interactions” (Cormier & Siemens, 2010
http://is.gd/nqTED )
Hub & Spoke, flickr photo by
Antony_Mayfield, licensed CC-BY
•course may be successful (or fail) in
ways designers never envisioned

6. Goals of participants
E.g., Lane, 2013 http://is.gd/W0360s
•participants may have goals that don’t fit course; a
problem if not fulfilled?
• may not have any goals; just want to
see what happens
• course may have other benefits not
captured in participants’ goals; may
miss this if focus on their goals
• benefits may take a long time to
realize
Huma Bird tweet analysis,
#whyopen http://is.gd/4h5CFq

7. Goals of participants
What one might do:
•Ask participants at the end what they got out of the
course, with or without reference to their original goals
•Return to them six months or more later to ask again-perhaps see longer-term effects
•Consider how to support learners in being self-directed,
working to achieve own goals in a cMOOC (e.g., Kop, Fournier
and Mak, 2011 http://is.gd/KopEtAl2011 )

8. Connectivism:
connections among
people & info
• “Knowledge is defined as a particular pattern of
relationships and learning is defined as the creation of new
connections and patterns as well as the ability to maneuver
around existing networks/patterns." (Siemens 2008
http://www.connectivism.ca/?p=116 )
• "At its heart, connectivism is the thesis that knowledge is
distributed across a network of connections, and therefore
that learning consists of the ability to construct and
traverse those networks." (Downes, 2007 http://is.gd/Downes2007)

9. Connectivism:
connections among
people & info
Participation rates:
•surveys of participants: Milligan, Littlejohn and Margaryan,
2013 http://is.gd/MilliganEtAl2013
•log data from P2PU platform: Ahn, Weng and Butler, 2013
http://is.gd/AhnEtAl2013
•mixed methods:
• Waite, Mackness, Roberts and Lovegrove, 2013
http://is.gd/WaiteEtAl2013
•
Kop, 2011 http://is.gd/Kop2011

10. Connectivism:
connections among
people & info
Negotiating autonomy
& participation
•Tension: need at least
some active
participation, but
participants must have
autonomy
•Lurkers valued? Just b/c
may become active
participants?
#ds106zone May 25-June 6, 2013
http://is.gd/o27mvc

11. Purposes of cMOOCs
themselves
Downes 2013b http://is.gd/Downes2013
•look at what sorts of entities cMOOCs are, what purposes
they serve, whether designed well for those purposes (rather
than how they’re used)
•To evaluate a cMOOC, consider: “what a successful
MOOC ought to produce as output, without reference to
existing (and frankly, very preliminary and very variable)
usage.” (Ibid.)
• output: “emergent knowledge”

12. Emergent knowledge:
See something or
say something: Jakarta, Flickr photo shared by
Eric Fischer, licensed CC-BY
Blue dots tweets; red dots Flickr,
white dots both
In a successful cMOOC, “the
structure of the interactions
produces new knowledge, that is,
knowledge that was not present
in any of the individual
communications, but is produced
as a result of the totality of the
communications, in such a way
that participants can through
participation and immersion in
this environment develop in their
selves new (and typically
unexpected) knowledge relevant
to the domain.” (Downes, 2013b;
emphasis added)

13. Networks that tend to
produce emergent
knowledge
1.Autonomy
2.Diversity
3.Openness
4.Interactivity/Connectedness
(Downes, 2013b)
http://is.gd/Downes2013
Anek Rang, Ek Sang, Flickr photo
shared by Sanjay, licensed CC-BY

14. How evaluate
cMOOCs acc to
these criteria?
• Don’t measure each aspect of a cMOOC against these
as if a checklist; rather, consider cMOOCs a
“language” and a course as an expression in it
• These criteria should be considered “an aid, used to
assist a person who is already fluent in MOOC
design (or at least in the domain or discipline being
studied) [to] recognize the quality (or lack of quality)
of a MOOC” (Downes, 2013b).

15. Questions & concerns
about this approach
• How can we determine if emergent knowledge has been
produced? Where would we look? Whom would we ask?
• Seemingly exclusive focus on
design and purpose of cMOOCs-doesn’t consider the experiences
of participants
E.g., participant experiences in a
cMOOC: Mackness, Mak and
Williams, 2010
Crowd, Flickr photo by James Cridland, lic http://is.gd/MacknessEtAl2010
ensed CC-BY (altered)

16. Footprints of
emergence
•
Williams, Karousou and Mackness, 2011
http://is.gd/WilliamsEtAl2011
•
Emergent and prescriptive learning--need balance
Williams, Mackness and Gumtau, 2012
http://is.gd/WilliamsEtAl2012
-
Draw “footprints” of courses to map degrees of
prescriptive and emergent learning

17. Footprint for CCK08
published in Williams, Mackness & Gumtau
2012 http://is.gd/WilliamsEtAl2012
(licensed CC-BY)
Centre: prescriptive learning
Light area: apex of emergent
learning
Periphery: “edge of chaos”
Map points based on 24
factors, in four clusters
See wiki for factors, how to draw
footprints, and more:
http://footprints-of-emergence.wikispac
/

18. Full circle
• Footprints are not meant to provide evaluations of
courses by themselves
• Instead, provides way to evaluate if course fits purposes
(Footprints of emergence wiki: http://footprints-of-emergence.wikispaces.com/ )
• After have footprint, ask: “Is this appropriate, or fit, for
the purpose and context of the course and for you,
and/or the particular learners?”
Back to the beginning...

19. What now?
Suggestions?

20. THANK YOU!
Twitter: @clhendricksbc
Blog: http://blogs.ubc.ca/chendricks
Slides, video and bibliography at:
http://is.gd/HendricksOpenEd2013