SlideShare a Scribd company logo
1 of 29
Download to read offline
A Novel Model of Cognitive Presence
Assessment Using Automated
Learning Analytics Methods
Vitomir Kovanovic
School of Informatics
The University of Edinburgh
http://vitomir.kovanovic.info
v.kovanovic@ed.ac.uk
#vkovanovic
2
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Global Grand Challenges
Shifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges
3
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Four challenges for next decade
1. Affordable, sustainable energy.
2. Cures for HIV and neurodegenerative diseases like Alzheimer's.
3. Protection from future health epidemics.
4. Tools to provide a world-class education to all students.
4
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 GLOBAL NEED
FOR EDUCATION
Significant Challenges in Higher Education
2 INCREASE
STUDENT
ENGAGEMENT
3 DECREASING
UNIVERSITY
FUNDING
MORE AND MORE
UNIVERSITIES TURNING
TO TECHNOLOGY
MORE AND MORE
UNIVERSITIES TURNING
TO TECHNOLOGY
5
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
ONLINE &
BLENDED
Blended is the new norm.
Increasing interest in online learning.
MOOCs and new forms of delivery.
LEARNING
DATA
Large amounts of data collected.
Improve student experience.
Understand learning processes.
NEW MARKETS
AND MODELS
Workspace learning.
Lifelong learning.
Developing world.
Technology & Data Trends
Three important ways in which technology is shaping the future of education
6
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
CFP for the First Learning Analytics and Knowledge Conference
https://tekri.athabascau.ca/analytics/
What is Learning Analytics?
Learning Analytics
Making sense of the available learning data
Measurement Collection Analysis Reporting
7
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
What can Learning Analytics bring
Improvement in teaching quality, student retention, learning outcomes, and understanding of learning processes
8
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Traditional Online Learning
How can we use the educational data to understand student
learning?
Massive Open Online Courses
How to adapt existing models of online learning in MOOC
settings?
What are the challenges of MOOC pedagogies?
Learning Analytics for Online Learning
9
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 Pedagogy
The use of Learning analytics must be aligned with the existing
pedagogical approaches used in particular learning context.
Driving Principles
Foundations for my learning analytics research
2 Data analysis
Use automated data mining techniques to process large amounts of
student-generated data to understand how student learn in online
setting
3 Tools
Develop automated learning analytics tools that can be used to assess
student learning and improve research in online and distance
education.
10
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Community of Inquiry Framework
Dimensions of student online learning experience
Social-constructivist model of learning
- Students construct knowledge rather than receive it
- Discussion is an essential part of learning
- Inquiry basic learning activity
Widely used in “traditional” online
and distance education
- Strong teacher presence
- Up to ~ 30 students
Cognitive
Presence
Teaching
Presence
Social
Presence
Student
Experience
Development of critical
and deep thinking skills
Social climate in the course
1. Group cohesion
2. Interactivity
3. Group affectivity
Instructor’s role in the course:
1. Design & organization,
2. Facilitation,
3. Direct instruction
(Garrison, Anderson, and Archer, 1999)
11
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive Presence
Operationalization of critical and deep thinking skills
• Exploring and
testing
solutions
• Dilemma or
problem
identified
• Synthesis of
relevant
information
• Brainstorming
• Exploring
ideas
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of reflection
“an extent to which the participants
in any particular configuration of a
community of inquiry are able to
construct meaning through
sustained Communication”
(Garrison, Anderson, and Archer,
1999, p. 89)
Definition
12
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 Content analysis
Coding scheme for each
presence
Assessment of three presences
How do we measure levels of cognitive, social and teaching presence?
2 Self-reported survey
34 items, 1-5 Likert scale
questions
• Experience with the coding scheme
• Inter-rater reliability issues
• Time-consuming
• Non-real time
• Primary use: research
• Fait accompli: results known after the
course is over
• Self-selection bias
• Invasive
(Arbaugh et al., 2008).
(Garrison, Anderson,
and Archer, 1999)
13
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
1 From the online
discussions
Assessment based on Learning Analytics
Automate as much as possible
2 Based on the use
of educational
technology
• Identifying different study
approaches
• How those approaches relate
to development of cognitive
presence?
• Clustering based on trace
data
Shared world learning
Private world learning
• Automating CoI coding
scheme for cognitive
presence
• Real-time analysis of student
cognitive presence
development
• Easier adoption and research
• Text mining of student online
discussion messages
1 Traditional online
courses
2 MOOCs
Athabasca University
fully online course
Delft University of
Technology MOOC
data
Types of data and analytics Contexts
14
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive presence assessment framework
Driven by Evidence-Centered Design (ECD) framework
ASSESSMENT IMPLEMENTATION
Automated cognitive presence classification system
Technology use profiling system
ASSESSMENT FRAMEWORK
Evidence-centered design (ECD) assessment
framework: Student model, Evidence model and Task
model
EDUCATIONAL TECHNOLOGY
Technology use trace data and online discussion
data from Learning Management Systems (LMS)
and MOOC platforms:
THEORY & PEDAGOGY
Social constructivist learning & Community of Inquiry
model
Kovanovic et al. (in-press)
15
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Cognitive presence classification system
Five-class text classification problem:
1 – Triggering event,
2 – Exploration,
3 – Integration,
4 – Resolution,
0 – Other (non cognitive).
Data from six offers of a fully online course
1,747 messages coded for cognitive presence
Developed random forest classifier
Extracted features:
Linguistic Inquiry Word Count features (LIWC):
93 different counts indicative of different psychological processes (e.g.,
affective, cognitive, social, perceptual)
Coh-Metrix features:
108 metrics of text cohesion
LSA coherence:
Average LSA similarity of message’s paragraphs to each other.
LSA space is built from Wikipedia articles related to concepts extracted
from the topic start message (using TAGME).
Named entity count:
Number of concepts related to DBPedia computer science category (using
DBPedia spotlight)
Context features:
Number of replies
Message depth
Cosine similarity to previous/next message
Thread start/end Boolean indicators
Kovanovic et al. (2016)
16
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Performance evaluation
• We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ.
• Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014)
and Waters et al. (2015) studies.
• The feature space ~ 100x smaller
• The feature space is also more generalizable
• We provided more detailed operationalization of the cognitive presence coding scheme.
Current work:
– Analysis of discussion messages from “Functional Programming” MOOC by TU Delft
– Investigation of the suitability of the same classification approach in various contexts
(Essay analysis, Twitter messages)
Kovanovic et al. (2016)
17
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Feature importance
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
* MDG - Mean decrease Gini impurity index, FPS - first person singular
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
Kovanovic et al. (2016)
18
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
The higher the cognitive presence…
• The longer the message, the more paragraphs and
sentences.
• More concepts mentioned (more named entities).
• The lower the lexical diversity (both at content level
and in general).
• The later its position in the thread. Except non-
cognitive messages, they tend to occur closer to the
end as well.
• The fewer the question marks (except non-cognitive,
they have fewest question marks)
• The higher the average length of sentence and their
similarity to each other.
• The more money-related terms.
Operationalization of cognitive presence
• Exploring and
testing
solutions
• Dilemma or
problem
identified
• Synthesis of
relevant
information
• Brainstorming
• Exploring
ideas
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of reflection
• Lowest message
readability
• Syllabi count
inconsistent
• Most replies
• Low similarity with
the next message
• More replies than
exploration and
resolution
• Least replies
• Question marks
more frequent
than integration
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of reflection
Kovanovic et al. (2016)
19
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Technology use profiling system
• Used 200,000 LMS trace data records of student learning activities
• Hierarchical clustering (Ward method, Euclidean distance)
• MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis
• We identified six different technology use profiles.
• Large number of students poorly regulated technology use.
• Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We
observed large effect size (partial eta2 = .54).
• Quality of interactions is more important than the quantity for the development of cognitive presence.
Kovanovic et al. (2014)
20
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
In progress: Communities of Inquiry for
Massive Open Online Courses
Used the data from five TU Delft MOOCs
2,446 survey responses
5-item Likert scale responses
Originally, CoI studies found 3-factor structure to best describe CoI survey instrument
Goal: can we replicate those findings in MOOCs?
21
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Factor analysis
Original model holds in the MOOC context
CP6: Online discussions were valuable in helping me
appreciate different perspectives
Affectivity
Resolution
Course organization
& design
Better fit: 6 factor model:
22
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
How are they interacting
Structural equation model between three presences
• Direct effect of Teaching Presence on Cognitive presence, and
• Mediating effect of social presence on teaching presence and cognitive presence relationship
Cognitive
presence
Social
presence
Teaching
presence
Cognitive
presence
Social
presence
Teaching
presence
Original SEM model MOOC model
0.52
0.51 0.40 0.60 0.22
0.35
23
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Technology-use in MOOCs
• Adopted same methodology as in “traditional” online setting
• Significant MANOVA, (p=0.038).
• Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )
24
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary
• Important to realize the tremendous potential educational data has for improving teaching and learning.
• How this potential of the data fits the existing models of online and blended learning?
• How can we design analytics that are flexible enough and provide meaningful insights about student learning?
• Can we better support research in online education by automating existing approaches used by researchers?
• Can we use the existing pedagogical models in MOOC setting?
• How can we make student learning in MOOCs les solitary?
25
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary: cognitive presence assessment
framework
• We developed a framework for assessing the levels of cognitive presence
• We focused on two types of insights: from trace data & discussion data and
• We focused on two contexts: traditional online courses and MOOCs
• We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines
• Automated existing tool for measuring cognitive presence
• Enables easier adoption of Community of inquiry model
• Better operationalization of the phases of cognitive presence
• We developed a student profiling method that can be used to identify different profiles of student technology use
• Better understand online learning, especially related to student self-regulation of learning
• Insights into their private solitary learning not displayed in online discussion transcripts
26
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Summary: Cognitive presence in MOOCs
• Explored the use of CoI model in MOOC context and use existing pedagogies at scale
• We conducted a factor analysis of Community of Inquiry model in the MOOC context:
• Original CoI model holds but with certain factors being more emphasized than others
• The role of social group is much weaker in MOOCs
• We identified profiles of students based on their technology use
• Identified profiles differ in the levels of their cognitive and social presence
• The differences are much smaller, primarily due to the use of survey-based instrument
27
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
Current and future work
• Enable provision of formative feedback based on their use of educational systems
• Possible to support instructional interventions by embedding the developed analytics into the learning platform
• We are currently working on a MOOC experimentation platform that will include cognitive presence analytics
• At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context
• Also, exploring the use of the same classification methodology for similar problems
Student reflection in discussions (OU)
Essay grading (USF)
Twitter and YouTube comments (QUT)
28
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
The End
THANK YOU FOR YOUR TIME
Q/A
29
Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info
REFERENCES
Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry
framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136.
Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer
Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105.
Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using
Automated Learning Analytics Methods (Analytics4Learning). SRI Education. Available at:
http://vitomir.kovanovic.info/publications/
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content
analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning
Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of
learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–
89. https://doi.org/10.1016/j.iheduc.2015.06.002
Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In
Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics
and Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/

More Related Content

What's hot

Admixture of Poisson MRFs: A New Topic Model with Word Dependencies
Admixture of Poisson MRFs: A New Topic Model with Word DependenciesAdmixture of Poisson MRFs: A New Topic Model with Word Dependencies
Admixture of Poisson MRFs: A New Topic Model with Word Dependencies
David Inouye
 
SemTech Survey - Web Science 2009 Conference
SemTech Survey - Web Science 2009 ConferenceSemTech Survey - Web Science 2009 Conference
SemTech Survey - Web Science 2009 Conference
thanassis
 

What's hot (6)

Admixture of Poisson MRFs: A New Topic Model with Word Dependencies
Admixture of Poisson MRFs: A New Topic Model with Word DependenciesAdmixture of Poisson MRFs: A New Topic Model with Word Dependencies
Admixture of Poisson MRFs: A New Topic Model with Word Dependencies
 
Microblogging in the classroom. From information to participation
Microblogging in the classroom.From information to participationMicroblogging in the classroom.From information to participation
Microblogging in the classroom. From information to participation
 
Expert Perceptions of the Feasibility of MOOCs
Expert Perceptions of the Feasibility of MOOCsExpert Perceptions of the Feasibility of MOOCs
Expert Perceptions of the Feasibility of MOOCs
 
SemTech Survey - Web Science 2009 Conference
SemTech Survey - Web Science 2009 ConferenceSemTech Survey - Web Science 2009 Conference
SemTech Survey - Web Science 2009 Conference
 
Lilac 2012 Essential information skills for researchers: A collaborative proj...
Lilac 2012 Essential information skills for researchers: A collaborative proj...Lilac 2012 Essential information skills for researchers: A collaborative proj...
Lilac 2012 Essential information skills for researchers: A collaborative proj...
 
LDA Beginner's Tutorial
LDA Beginner's TutorialLDA Beginner's Tutorial
LDA Beginner's Tutorial
 

Viewers also liked

Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Vitomir Kovanovic
 

Viewers also liked (7)

Penetrating the black box of time-on-task estimation
Penetrating the black box of time-on-task estimationPenetrating the black box of time-on-task estimation
Penetrating the black box of time-on-task estimation
 
Inquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & OpportunitiesInquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & Opportunities
 
Automated System for Cognitive Presence Coding
Automated System for Cognitive Presence CodingAutomated System for Cognitive Presence Coding
Automated System for Cognitive Presence Coding
 
What does effective online/blended teaching look like?
What does effective online/blended teaching look like?What does effective online/blended teaching look like?
What does effective online/blended teaching look like?
 
MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
 
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 TutorialTopic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
 

Similar to A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods

Web Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdfWeb Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdf
Stefanie Panke
 
Semantic Technologies in HE Seminar - Learning Societies Lab
Semantic Technologies in HE Seminar - Learning Societies LabSemantic Technologies in HE Seminar - Learning Societies Lab
Semantic Technologies in HE Seminar - Learning Societies Lab
thanassis
 
CEMCA EdTech Notes: Massive Open Online Courses
CEMCA EdTech Notes: Massive Open Online CoursesCEMCA EdTech Notes: Massive Open Online Courses
CEMCA EdTech Notes: Massive Open Online Courses
CEMCA
 

Similar to A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods (20)

A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...
 
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
 
Web Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdfWeb Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdf
 
Tcc keynote Hawaii april 2013
Tcc keynote Hawaii april 2013Tcc keynote Hawaii april 2013
Tcc keynote Hawaii april 2013
 
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid. ...
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid. ...VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid. ...
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid. ...
 
Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...
 
EMOOCs 2014 Proceedings of the European MOOC Stakeholder Summit 2014
EMOOCs 2014 Proceedings of  the European  MOOC Stakeholder  Summit 2014EMOOCs 2014 Proceedings of  the European  MOOC Stakeholder  Summit 2014
EMOOCs 2014 Proceedings of the European MOOC Stakeholder Summit 2014
 
Mad about MOOCs
Mad about MOOCsMad about MOOCs
Mad about MOOCs
 
Professor Grainne Conole Disruptive Education AIT
Professor Grainne Conole  Disruptive Education AITProfessor Grainne Conole  Disruptive Education AIT
Professor Grainne Conole Disruptive Education AIT
 
Open Educational Resources: Experiences of use in a Latin-American context
Open Educational Resources: Experiences of use in a Latin-American contextOpen Educational Resources: Experiences of use in a Latin-American context
Open Educational Resources: Experiences of use in a Latin-American context
 
MOOCs and personal learning: reality or myth?
MOOCs and personal learning: reality or myth?MOOCs and personal learning: reality or myth?
MOOCs and personal learning: reality or myth?
 
MOOCs, Higher Education and NEP 2020
MOOCs, Higher Education and NEP 2020MOOCs, Higher Education and NEP 2020
MOOCs, Higher Education and NEP 2020
 
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
2022_12_16 «Informatics – A Fundamental Discipline for the 21st Century»
 
Old wine in new bottles? Exploring MOOCs
Old wine in new bottles? Exploring MOOCsOld wine in new bottles? Exploring MOOCs
Old wine in new bottles? Exploring MOOCs
 
2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - How it s...
2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - How it s...2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - How it s...
2018-07-13 MOOQ Conference in Athens MOOQ and the Quality of MOOCs - How it s...
 
«Ejemplos de herramientas que nos facilitan las analíticas de aprendizaje en ...
«Ejemplos de herramientas que nos facilitan las analíticas de aprendizaje en ...«Ejemplos de herramientas que nos facilitan las analíticas de aprendizaje en ...
«Ejemplos de herramientas que nos facilitan las analíticas de aprendizaje en ...
 
Design Patterns for Digital Identity
Design Patterns for Digital IdentityDesign Patterns for Digital Identity
Design Patterns for Digital Identity
 
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid L...
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid L...VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid L...
VII Jornadas eMadrid "Education in exponential times". Mesa redonda eMadrid L...
 
Semantic Technologies in HE Seminar - Learning Societies Lab
Semantic Technologies in HE Seminar - Learning Societies LabSemantic Technologies in HE Seminar - Learning Societies Lab
Semantic Technologies in HE Seminar - Learning Societies Lab
 
CEMCA EdTech Notes: Massive Open Online Courses
CEMCA EdTech Notes: Massive Open Online CoursesCEMCA EdTech Notes: Massive Open Online Courses
CEMCA EdTech Notes: Massive Open Online Courses
 

More from Vitomir Kovanovic

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
Vitomir Kovanovic
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
Vitomir Kovanovic
 

More from Vitomir Kovanovic (14)

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
 
Understand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsUnderstand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analytics
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
 
SFU SIAT Comprehensive Examination
SFU SIAT Comprehensive ExaminationSFU SIAT Comprehensive Examination
SFU SIAT Comprehensive Examination
 

Recently uploaded

一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
pyhepag
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Riyadh +966572737505 get cytotec
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
pyhepag
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Valters Lauzums
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
pyhepag
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
RafigAliyev2
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
pyhepag
 

Recently uploaded (20)

Artificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdfArtificial_General_Intelligence__storm_gen_article.pdf
Artificial_General_Intelligence__storm_gen_article.pdf
 
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPsWebinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
Webinar One View, Multiple Systems No-Code Integration of Salesforce and ERPs
 
2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting2024 Q2 Orange County (CA) Tableau User Group Meeting
2024 Q2 Orange County (CA) Tableau User Group Meeting
 
AI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdfAI Imagen for data-storytelling Infographics.pdf
AI Imagen for data-storytelling Infographics.pdf
 
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
一比一原版(Monash毕业证书)莫纳什大学毕业证成绩单如何办理
 
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
Data Visualization Exploring and Explaining with Data 1st Edition by Camm sol...
 
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotecAbortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
Abortion pills in Dammam Saudi Arabia// +966572737505 // buy cytotec
 
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
一比一原版加利福尼亚大学尔湾分校毕业证成绩单如何办理
 
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
Data Analytics for Digital Marketing Lecture for Advanced Digital & Social Me...
 
Slip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp ClaimsSlip-and-fall Injuries: Top Workers' Comp Claims
Slip-and-fall Injuries: Top Workers' Comp Claims
 
一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理一比一原版西悉尼大学毕业证成绩单如何办理
一比一原版西悉尼大学毕业证成绩单如何办理
 
Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)Atlantic Grupa Case Study (Mintec Data AI)
Atlantic Grupa Case Study (Mintec Data AI)
 
Pre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptxPre-ProductionImproveddsfjgndflghtgg.pptx
Pre-ProductionImproveddsfjgndflghtgg.pptx
 
How I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prisonHow I opened a fake bank account and didn't go to prison
How I opened a fake bank account and didn't go to prison
 
Fuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertaintyFuzzy Sets decision making under information of uncertainty
Fuzzy Sets decision making under information of uncertainty
 
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflictSupply chain analytics to combat the effects of Ukraine-Russia-conflict
Supply chain analytics to combat the effects of Ukraine-Russia-conflict
 
社内勉強会資料  Mamba - A new era or ephemeral
社内勉強会資料   Mamba - A new era or ephemeral社内勉強会資料   Mamba - A new era or ephemeral
社内勉強会資料  Mamba - A new era or ephemeral
 
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdfGenerative AI for Trailblazers_ Unlock the Future of AI.pdf
Generative AI for Trailblazers_ Unlock the Future of AI.pdf
 
basics of data science with application areas.pdf
basics of data science with application areas.pdfbasics of data science with application areas.pdf
basics of data science with application areas.pdf
 
一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理一比一原版阿德莱德大学毕业证成绩单如何办理
一比一原版阿德莱德大学毕业证成绩单如何办理
 

A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods

  • 1. A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods Vitomir Kovanovic School of Informatics The University of Edinburgh http://vitomir.kovanovic.info v.kovanovic@ed.ac.uk #vkovanovic
  • 2. 2 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Global Grand Challenges Shifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges
  • 3. 3 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Four challenges for next decade 1. Affordable, sustainable energy. 2. Cures for HIV and neurodegenerative diseases like Alzheimer's. 3. Protection from future health epidemics. 4. Tools to provide a world-class education to all students.
  • 4. 4 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info 1 GLOBAL NEED FOR EDUCATION Significant Challenges in Higher Education 2 INCREASE STUDENT ENGAGEMENT 3 DECREASING UNIVERSITY FUNDING MORE AND MORE UNIVERSITIES TURNING TO TECHNOLOGY MORE AND MORE UNIVERSITIES TURNING TO TECHNOLOGY
  • 5. 5 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info ONLINE & BLENDED Blended is the new norm. Increasing interest in online learning. MOOCs and new forms of delivery. LEARNING DATA Large amounts of data collected. Improve student experience. Understand learning processes. NEW MARKETS AND MODELS Workspace learning. Lifelong learning. Developing world. Technology & Data Trends Three important ways in which technology is shaping the future of education
  • 6. 6 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. CFP for the First Learning Analytics and Knowledge Conference https://tekri.athabascau.ca/analytics/ What is Learning Analytics? Learning Analytics Making sense of the available learning data Measurement Collection Analysis Reporting
  • 7. 7 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info What can Learning Analytics bring Improvement in teaching quality, student retention, learning outcomes, and understanding of learning processes
  • 8. 8 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Traditional Online Learning How can we use the educational data to understand student learning? Massive Open Online Courses How to adapt existing models of online learning in MOOC settings? What are the challenges of MOOC pedagogies? Learning Analytics for Online Learning
  • 9. 9 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info 1 Pedagogy The use of Learning analytics must be aligned with the existing pedagogical approaches used in particular learning context. Driving Principles Foundations for my learning analytics research 2 Data analysis Use automated data mining techniques to process large amounts of student-generated data to understand how student learn in online setting 3 Tools Develop automated learning analytics tools that can be used to assess student learning and improve research in online and distance education.
  • 10. 10 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Community of Inquiry Framework Dimensions of student online learning experience Social-constructivist model of learning - Students construct knowledge rather than receive it - Discussion is an essential part of learning - Inquiry basic learning activity Widely used in “traditional” online and distance education - Strong teacher presence - Up to ~ 30 students Cognitive Presence Teaching Presence Social Presence Student Experience Development of critical and deep thinking skills Social climate in the course 1. Group cohesion 2. Interactivity 3. Group affectivity Instructor’s role in the course: 1. Design & organization, 2. Facilitation, 3. Direct instruction (Garrison, Anderson, and Archer, 1999)
  • 11. 11 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Cognitive Presence Operationalization of critical and deep thinking skills • Exploring and testing solutions • Dilemma or problem identified • Synthesis of relevant information • Brainstorming • Exploring ideas 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection “an extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained Communication” (Garrison, Anderson, and Archer, 1999, p. 89) Definition
  • 12. 12 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info 1 Content analysis Coding scheme for each presence Assessment of three presences How do we measure levels of cognitive, social and teaching presence? 2 Self-reported survey 34 items, 1-5 Likert scale questions • Experience with the coding scheme • Inter-rater reliability issues • Time-consuming • Non-real time • Primary use: research • Fait accompli: results known after the course is over • Self-selection bias • Invasive (Arbaugh et al., 2008). (Garrison, Anderson, and Archer, 1999)
  • 13. 13 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info 1 From the online discussions Assessment based on Learning Analytics Automate as much as possible 2 Based on the use of educational technology • Identifying different study approaches • How those approaches relate to development of cognitive presence? • Clustering based on trace data Shared world learning Private world learning • Automating CoI coding scheme for cognitive presence • Real-time analysis of student cognitive presence development • Easier adoption and research • Text mining of student online discussion messages 1 Traditional online courses 2 MOOCs Athabasca University fully online course Delft University of Technology MOOC data Types of data and analytics Contexts
  • 14. 14 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Cognitive presence assessment framework Driven by Evidence-Centered Design (ECD) framework ASSESSMENT IMPLEMENTATION Automated cognitive presence classification system Technology use profiling system ASSESSMENT FRAMEWORK Evidence-centered design (ECD) assessment framework: Student model, Evidence model and Task model EDUCATIONAL TECHNOLOGY Technology use trace data and online discussion data from Learning Management Systems (LMS) and MOOC platforms: THEORY & PEDAGOGY Social constructivist learning & Community of Inquiry model Kovanovic et al. (in-press)
  • 15. 15 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Cognitive presence classification system Five-class text classification problem: 1 – Triggering event, 2 – Exploration, 3 – Integration, 4 – Resolution, 0 – Other (non cognitive). Data from six offers of a fully online course 1,747 messages coded for cognitive presence Developed random forest classifier Extracted features: Linguistic Inquiry Word Count features (LIWC): 93 different counts indicative of different psychological processes (e.g., affective, cognitive, social, perceptual) Coh-Metrix features: 108 metrics of text cohesion LSA coherence: Average LSA similarity of message’s paragraphs to each other. LSA space is built from Wikipedia articles related to concepts extracted from the topic start message (using TAGME). Named entity count: Number of concepts related to DBPedia computer science category (using DBPedia spotlight) Context features: Number of replies Message depth Cosine similarity to previous/next message Thread start/end Boolean indicators Kovanovic et al. (2016)
  • 16. 16 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Performance evaluation • We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ. • Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014) and Waters et al. (2015) studies. • The feature space ~ 100x smaller • The feature space is also more generalizable • We provided more detailed operationalization of the cognitive presence coding scheme. Current work: – Analysis of discussion messages from “Functional Programming” MOOC by TU Delft – Investigation of the suitability of the same classification approach in various contexts (Essay analysis, Twitter messages) Kovanovic et al. (2016)
  • 17. 17 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Feature importance Phase # Variable Description MDG* Other TE Exp. Int. Res. 1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68 2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18 3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58 4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00 5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78 6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24 7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38 8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39 9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16 10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99 11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05 12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22 13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84 14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29 15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67 16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73 17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97 18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19 19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99 20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47 * MDG - Mean decrease Gini impurity index, FPS - first person singular Phase # Variable Description MDG* Other TE Exp. Int. Res. 1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68 2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18 3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58 4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00 5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78 6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24 7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38 8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39 9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16 10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99 11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05 12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22 13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84 14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29 15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67 16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73 17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97 18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19 19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99 20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47 Kovanovic et al. (2016)
  • 18. 18 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info The higher the cognitive presence… • The longer the message, the more paragraphs and sentences. • More concepts mentioned (more named entities). • The lower the lexical diversity (both at content level and in general). • The later its position in the thread. Except non- cognitive messages, they tend to occur closer to the end as well. • The fewer the question marks (except non-cognitive, they have fewest question marks) • The higher the average length of sentence and their similarity to each other. • The more money-related terms. Operationalization of cognitive presence • Exploring and testing solutions • Dilemma or problem identified • Synthesis of relevant information • Brainstorming • Exploring ideas 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection • Lowest message readability • Syllabi count inconsistent • Most replies • Low similarity with the next message • More replies than exploration and resolution • Least replies • Question marks more frequent than integration 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection Kovanovic et al. (2016)
  • 19. 19 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Technology use profiling system • Used 200,000 LMS trace data records of student learning activities • Hierarchical clustering (Ward method, Euclidean distance) • MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis • We identified six different technology use profiles. • Large number of students poorly regulated technology use. • Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We observed large effect size (partial eta2 = .54). • Quality of interactions is more important than the quantity for the development of cognitive presence. Kovanovic et al. (2014)
  • 20. 20 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info In progress: Communities of Inquiry for Massive Open Online Courses Used the data from five TU Delft MOOCs 2,446 survey responses 5-item Likert scale responses Originally, CoI studies found 3-factor structure to best describe CoI survey instrument Goal: can we replicate those findings in MOOCs?
  • 21. 21 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Factor analysis Original model holds in the MOOC context CP6: Online discussions were valuable in helping me appreciate different perspectives Affectivity Resolution Course organization & design Better fit: 6 factor model:
  • 22. 22 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info How are they interacting Structural equation model between three presences • Direct effect of Teaching Presence on Cognitive presence, and • Mediating effect of social presence on teaching presence and cognitive presence relationship Cognitive presence Social presence Teaching presence Cognitive presence Social presence Teaching presence Original SEM model MOOC model 0.52 0.51 0.40 0.60 0.22 0.35
  • 23. 23 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Technology-use in MOOCs • Adopted same methodology as in “traditional” online setting • Significant MANOVA, (p=0.038). • Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )
  • 24. 24 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Summary • Important to realize the tremendous potential educational data has for improving teaching and learning. • How this potential of the data fits the existing models of online and blended learning? • How can we design analytics that are flexible enough and provide meaningful insights about student learning? • Can we better support research in online education by automating existing approaches used by researchers? • Can we use the existing pedagogical models in MOOC setting? • How can we make student learning in MOOCs les solitary?
  • 25. 25 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Summary: cognitive presence assessment framework • We developed a framework for assessing the levels of cognitive presence • We focused on two types of insights: from trace data & discussion data and • We focused on two contexts: traditional online courses and MOOCs • We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines • Automated existing tool for measuring cognitive presence • Enables easier adoption of Community of inquiry model • Better operationalization of the phases of cognitive presence • We developed a student profiling method that can be used to identify different profiles of student technology use • Better understand online learning, especially related to student self-regulation of learning • Insights into their private solitary learning not displayed in online discussion transcripts
  • 26. 26 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Summary: Cognitive presence in MOOCs • Explored the use of CoI model in MOOC context and use existing pedagogies at scale • We conducted a factor analysis of Community of Inquiry model in the MOOC context: • Original CoI model holds but with certain factors being more emphasized than others • The role of social group is much weaker in MOOCs • We identified profiles of students based on their technology use • Identified profiles differ in the levels of their cognitive and social presence • The differences are much smaller, primarily due to the use of survey-based instrument
  • 27. 27 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info Current and future work • Enable provision of formative feedback based on their use of educational systems • Possible to support instructional interventions by embedding the developed analytics into the learning platform • We are currently working on a MOOC experimentation platform that will include cognitive presence analytics • At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context • Also, exploring the use of the same classification methodology for similar problems Student reflection in discussions (OU) Essay grading (USF) Twitter and YouTube comments (QUT)
  • 28. 28 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info The End THANK YOU FOR YOUR TIME Q/A
  • 29. 29 Vitomir Kovanovic (The University of Edinburgh) http://vitomir.kovanovic.info REFERENCES Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136. Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105. Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods (Analytics4Learning). SRI Education. Available at: http://vitomir.kovanovic.info/publications/ Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950 Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74– 89. https://doi.org/10.1016/j.iheduc.2015.06.002 Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics and Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/