SlideShare a Scribd company logo
1 of 31
Download to read offline
A Novel Model of Cognitive Presence
Assessment Using Automated Learning
Analytics Methods
Vitomir Kovanovic
University of South Australia
Teaching innovation Unit and School of Education
#vkovanovic
http://vitomir.kovanovic.info
Vitomir.Kovanovic@unisa.edu.au
2
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Global Grand Challenges
Shifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges
3
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Four challenges for next decade
1. Affordable, sustainable energy.
2. Cures for HIV and neurodegenerative diseases like Alzheimer's.
3. Protection from future health epidemics.
4. Tools to provide a world-class education to all students.
4
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
1 GLOBAL NEED
FOR EDUCATION
Significant Challenges in Higher Education
2 INCREASE STUDENT
ENGAGEMENT
3 DECREASING
UNIVERSITY
FUNDING
MORE AND MORE
UNIVERSITIES TURNING TO
TECHNOLOGY
MORE AND MORE
UNIVERSITIES TURNING TO
TECHNOLOGY
5
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
ONLINE &
BLENDED
Blended is the new norm.
Increasing interest in online learning.
MOOCs and new forms of delivery.
LEARNING
DATA
Large amounts of data collected.
Improve student experience.
Understand learning processes.
NEW MARKETS
AND MODELS
Workspace learning.
Lifelong learning.
Developing world.
Technology & Data Trends
Three important ways in which technology is shaping the future of education
6
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Learning analytics is the measurement, collection, analysis and reporting of data about learners and their
contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.
CFP for the First Learning Analytics and Knowledge Conference
https://tekri.athabascau.ca/analytics/
What is Learning Analytics?
Learning Analytics
Making sense of the available learning data
Measurement Collection Analysis Reporting
7
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
What can Learning Analytics bring
Improvement in teaching quality, student retention, learning outcomes, and understanding of learning processes
8
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Traditional Online Learning
How can we use the educational data to understand student
learning?
Massive Open Online Courses
How to adapt existing models of online learning in MOOC
settings?
What are the challenges of MOOC pedagogies?
Learning Analytics for Online Learning
9
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
1 Pedagogy
The use of Learning analytics must be aligned with the existing
pedagogical approaches used in particular learning context.
Driving Principles
Foundations for my learning analytics research
2 Data analysis
Use automated data mining techniques to process large amounts of
student-generated data to understand how student learn in online
setting
3 Tools
Develop automated learning analytics tools that can be used to assess
student learning and improve research in online and distance education.
10
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Community of Inquiry Framework
Dimensions of student online learning experience
Social-constructivist model of learning
- Students construct knowledge rather than receive it
- Discussion is an essential part of learning
- Inquiry basic learning activity
Widely used in “traditional” online
and distance education
- Strong teacher presence
- Up to ~ 30 students
Cognitive
Presence
Teaching
Presence
Social
Presence
Student
Experience
Development of critical
and deep thinking skills
Social climate in the course
1. Group cohesion
2. Interactivity
3. Group affectivity
Instructor’s role in the course:
1. Design & organization,
2. Facilitation,
3. Direct instruction
(Garrison, Anderson, and Archer, 1999)
11
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Cognitive Presence
Operationalization of critical and deep thinking skills
• Exploring and
testing
solutions
• Dilemma or
problem
identified
• Synthesis of
relevant
information
• Brainstorming
• Exploring
ideas
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of
reflection
“an extent to which the participants
in any particular configuration of a
community of inquiry are able to
construct meaning through
sustained Communication”
(Garrison, Anderson, and Archer,
1999, p. 89)
Definition
12
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
1 Content analysis
Coding scheme for each
presence
Assessment of three presences
How do we measure levels of cognitive, social and teaching presence?
2 Self-reported survey
34 items, 1-5 Likert scale
questions
• Experience with the coding scheme
• Inter-rater reliability issues
• Time-consuming
• Non-real time
• Primary use: research
• Fait accompli: results known after the
course is over
• Self-selection bias
• Invasive
(Arbaugh et al., 2008).
(Garrison, Anderson,
and Archer, 1999)
13
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
1 From the online
discussions
Assessment based on Learning Analytics
Automate as much as possible
2 Based on the use of
educational
technology
• Identifying different study
approaches
• How those approaches relate
to development of cognitive
presence?
• Clustering based on trace data
Shared world learning
Private world learning
• Automating CoI coding
scheme for cognitive presence
• Real-time analysis of student
cognitive presence
development
• Easier adoption and research
• Text mining of student online
discussion messages
1 Traditional online
courses
2 MOOCs
Athabasca University
fully online course
Delft University of
Technology MOOC
data
Types of data and analytics Contexts
14
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Cognitive presence assessment framework
Driven by Evidence-Centered Design (ECD) framework
ASSESSMENT IMPLEMENTATION
Automated cognitive presence classification system
Technology use profiling system
ASSESSMENT FRAMEWORK
Evidence-centered design (ECD) assessment
framework: Student model, Evidence model and Task
model
EDUCATIONAL TECHNOLOGY
Technology use trace data and online discussion
data from Learning Management Systems (LMS)
and MOOC platforms:
THEORY & PEDAGOGY
Social constructivist learning & Community of Inquiry
model
Kovanovic et al. (in-press)
15
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Cognitive presence classification system
Five-class text classification problem:
1 – Triggering event,
2 – Exploration,
3 – Integration,
4 – Resolution,
0 – Other (non cognitive).
Data from six offers of a fully online course
1,747 messages coded for cognitive presence
Developed random forest classifier
Extracted features:
Linguistic Inquiry Word Count features (LIWC):
93 different counts indicative of different psychological processes (e.g.,
affective, cognitive, social, perceptual)
Coh-Metrix features:
108 metrics of text cohesion
LSA coherence:
Average LSA similarity of message’s paragraphs to each other.
LSA space is built from Wikipedia articles related to concepts extracted
from the topic start message (using TAGME).
Named entity count:
Number of concepts related to DBPedia computer science category (using
DBPedia spotlight)
Context features:
Number of replies
Message depth
Cosine similarity to previous/next message
Thread start/end Boolean indicators
Kovanovic et al. (2016)
16
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Performance evaluation
• We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ.
• Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014) and
Waters et al. (2015) studies.
• The feature space ~ 100x smaller
• The feature space is also more generalizable
• We provided more detailed operationalization of the cognitive presence coding scheme.
Current work:
– Analysis of discussion messages from “Functional Programming” MOOC by TU Delft
– Investigation of the suitability of the same classification approach in various contexts
(Essay analysis, Twitter messages)
Kovanovic et al. (2016)
17
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Feature importance
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
* MDG - Mean decrease Gini impurity index, FPS - first person singular
Phase
# Variable Description MDG* Other TE Exp. Int. Res.
1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68
2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18
3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58
4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00
5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78
6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24
7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38
8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39
9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16
10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99
11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05
12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22
13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84
14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29
15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67
16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73
17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97
18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19
19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99
20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47
Kovanovic et al. (2016)
18
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
The higher the cognitive presence…
• The longer the message, the more paragraphs and
sentences.
• More concepts mentioned (more named entities).
• The lower the lexical diversity (both at content level
and in general).
• The later its position in the thread. Except non-
cognitive messages, they tend to occur closer to the
end as well.
• The fewer the question marks (except non-cognitive,
they have fewest question marks)
• The higher the average length of sentence and their
similarity to each other.
• The more money-related terms.
Operationalization of cognitive presence
• Exploring and
testing
solutions
• Dilemma or
problem
identified
• Synthesis of
relevant
information
• Brainstorming
• Exploring
ideas
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of
reflection
•Lowest message
readability
•Syllabi count
inconsistent
•Most replies
•Low similarity with
the next message
•More replies than
exploration and
resolution
•Least replies
•Question marks
more frequent
than integration
2.
Exploration
3.
Integration
4.
Resolution
1.
Triggering
Event
Shared world of discourse
Private world of
reflection
Kovanovic et al. (2016)
19
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Technology use profiling system
• Used 200,000 LMS trace data records of student learning activities
• Hierarchical clustering (Ward method, Euclidean distance)
• MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis
• We identified six different technology use profiles.
• Large number of students poorly regulated technology use.
• Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We
observed large effect size (partial eta2 = .54).
• Quality of interactions is more important than the quantity for the development of cognitive presence.
Kovanovic et al. (2014)
20
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
CoI within Massive Open Online Courses
Used the data from five TU Delft MOOCs
2,446 survey responses
5-item Likert scale responses
Originally, CoI studies found 3-factor structure to best describe CoI survey instrument
Goal: can we replicate those findings in MOOCs?
21
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Factor analysis
CoI within Massive Open Online Courses
Original model holds in the MOOC context
CP6: Online discussions were valuable in helping me
appreciate different perspectives
Affectivit
y
Resolution
Course organization
& design
Better fit: 6 factor
model:
22
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
The updated model
CoI within Massive Open Online Courses
23
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
How are they interacting
Structural equation model between three presences
• Direct effect of Teaching Presence on Cognitive presence, and
• Mediating effect of social presence on teaching presence and cognitive presence relationship
Cognitive
presence
Social
presence
Teaching
presence
Cognitive
presence
Social
presence
Teaching
presence
Original SEM model MOOC model
0.52
0.51 0.40 0.60 0.22
0.35
24
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Study strategies within MOOCs (submitted)
• Adopted same methodology as in “traditional” online setting
• Significant MANOVA, (p=0.038).
• Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )
25
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Summary
• Important to realize the tremendous potential educational data has for improving teaching and learning.
• How this potential of the data fits the existing models of online and blended learning?
• How can we design analytics that are flexible enough and provide meaningful insights about student learning?
• Can we better support research in online education by automating existing approaches used by researchers?
• Can we use the existing pedagogical models in MOOC setting?
• How can we make student learning in MOOCs les solitary?
26
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Summary: cognitive presence assessment framework
• We developed a framework for assessing the levels of cognitive presence
• We focused on two types of insights: from trace data & discussion data and
• We focused on two contexts: traditional online courses and MOOCs
• We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines
• Automated existing tool for measuring cognitive presence
• Enables easier adoption of Community of inquiry model
• Better operationalization of the phases of cognitive presence
• We developed a student profiling method that can be used to identify different profiles of student technology use
• Better understand online learning, especially related to student self-regulation of learning
• Insights into their private solitary learning not displayed in online discussion transcripts
27
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Summary: Cognitive presence in MOOCs
• Explored the use of CoI model in MOOC context and use existing pedagogies at scale
• We conducted a factor analysis of Community of Inquiry model in the MOOC context:
• Original CoI model holds but with certain factors being more emphasized than others
• The role of social group is much weaker in MOOCs
• We identified profiles of students based on their technology use
• Identified profiles differ in the levels of their cognitive and social presence
• The differences are much smaller, primarily due to the use of survey-based instrument
28
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
Current and future work
• Enable provision of formative feedback based on their use of educational systems
• Possible to support instructional interventions by embedding the developed analytics into the learning platform
• We are currently working on a MOOC experimentation platform that will include cognitive presence analytics
• At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context
• Also, exploring the use of the same classification methodology for similar problems
Student reflection in discussions (OU)
Essay grading (USF)
Twitter and YouTube comments (QUT)
29
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
The End
THANK YOU FOR YOUR TIME
Q/A
30
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
REFERENCES
Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry
framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136.
Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer
Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105.
Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using Automated
Learning Analytics Methods (Analytics4Learning). SRI Education. Available at: http://vitomir.kovanovic.info/publications/
Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content
analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning
Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950
Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of
learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89.
https://doi.org/10.1016/j.iheduc.2015.06.002
Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In
Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics and
Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/
31
Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info
REFERENCES
Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., Čukić, I., de Vries, P., … Gašević, D. (2018). Exploring communities of inquiry
in Massive Open Online Courses. Computers & Education, 119, 44–58. https://doi.org/10.1016/j.compedu.2017.11.010

More Related Content

What's hot

Salford uni pres 2011
Salford uni pres 2011Salford uni pres 2011
Salford uni pres 2011
oseamons
 
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
ijtsrd
 
03 20250 classifiers ensemble
03 20250 classifiers ensemble03 20250 classifiers ensemble
03 20250 classifiers ensemble
IAESIJEECS
 
Career prospect system for citizens of malaysia
Career prospect system for citizens of malaysiaCareer prospect system for citizens of malaysia
Career prospect system for citizens of malaysia
Conference Papers
 
Dmw uoc may 2013
Dmw uoc may 2013Dmw uoc may 2013
Dmw uoc may 2013
cgirona
 

What's hot (17)

Higher Education Study Comparing Clickers to Paper
Higher Education Study Comparing Clickers to PaperHigher Education Study Comparing Clickers to Paper
Higher Education Study Comparing Clickers to Paper
 
Predicting the Presence of Learning Motivation in Electronic Learning: A New ...
Predicting the Presence of Learning Motivation in Electronic Learning: A New ...Predicting the Presence of Learning Motivation in Electronic Learning: A New ...
Predicting the Presence of Learning Motivation in Electronic Learning: A New ...
 
Salford uni pres 2011
Salford uni pres 2011Salford uni pres 2011
Salford uni pres 2011
 
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...
Data-Driven Education 2020: Using Big Educational Data to Improve Teaching an...
 
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
Comparative Study of Different Approaches for Measuring Difficulty Level of Q...
 
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 TutorialTopic Modeling for Learning Analytics Researchers LAK15 Tutorial
Topic Modeling for Learning Analytics Researchers LAK15 Tutorial
 
Dit elearning summer school 2015 analytics
Dit elearning summer school 2015   analyticsDit elearning summer school 2015   analytics
Dit elearning summer school 2015 analytics
 
03 20250 classifiers ensemble
03 20250 classifiers ensemble03 20250 classifiers ensemble
03 20250 classifiers ensemble
 
Investigating learning strategies in a dispositional learning analytics conte...
Investigating learning strategies in a dispositional learning analytics conte...Investigating learning strategies in a dispositional learning analytics conte...
Investigating learning strategies in a dispositional learning analytics conte...
 
Career prospect system for citizens of malaysia
Career prospect system for citizens of malaysiaCareer prospect system for citizens of malaysia
Career prospect system for citizens of malaysia
 
CAPE Information Technology Unit 1 - Sample SBA
CAPE Information Technology Unit 1 - Sample SBACAPE Information Technology Unit 1 - Sample SBA
CAPE Information Technology Unit 1 - Sample SBA
 
Exploring Motivational Aspects and User Experience of Mobile Mathematics Lea...
Exploring Motivational Aspects and  User Experience of Mobile Mathematics Lea...Exploring Motivational Aspects and  User Experience of Mobile Mathematics Lea...
Exploring Motivational Aspects and User Experience of Mobile Mathematics Lea...
 
De carlo rizk 2010 icelw
De carlo rizk 2010 icelwDe carlo rizk 2010 icelw
De carlo rizk 2010 icelw
 
Learning Analytics and student feedback
Learning Analytics and student feedbackLearning Analytics and student feedback
Learning Analytics and student feedback
 
Dmw uoc may 2013
Dmw uoc may 2013Dmw uoc may 2013
Dmw uoc may 2013
 
A Nobel Approach On Educational Data Mining
A Nobel Approach On Educational Data MiningA Nobel Approach On Educational Data Mining
A Nobel Approach On Educational Data Mining
 
Data Mining Application in Advertisement Management of Higher Educational Ins...
Data Mining Application in Advertisement Management of Higher Educational Ins...Data Mining Application in Advertisement Management of Higher Educational Ins...
Data Mining Application in Advertisement Management of Higher Educational Ins...
 

Similar to A novel model of cognitive presence assessment using automated learning analytics methods

Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Vitomir Kovanovic
 
Web Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdfWeb Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdf
Stefanie Panke
 
Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018
Simon Buckingham Shum
 
Evaluation Methods For Mobile Learning
Evaluation Methods For Mobile LearningEvaluation Methods For Mobile Learning
Evaluation Methods For Mobile Learning
Mike Sharples
 
Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...
Simon Buckingham Shum
 
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous TransformationalKeynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
the Hartsook Letter
 

Similar to A novel model of cognitive presence assessment using automated learning analytics methods (20)

A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of Inquiry
 
Automated System for Cognitive Presence Coding
Automated System for Cognitive Presence CodingAutomated System for Cognitive Presence Coding
Automated System for Cognitive Presence Coding
 
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
 
MOOCs & Social Learning: Challenges and opportunities
MOOCs & Social Learning: Challenges and opportunitiesMOOCs & Social Learning: Challenges and opportunities
MOOCs & Social Learning: Challenges and opportunities
 
Web Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdfWeb Science Session 4 E-Learning.pdf
Web Science Session 4 E-Learning.pdf
 
Tcc keynote Hawaii april 2013
Tcc keynote Hawaii april 2013Tcc keynote Hawaii april 2013
Tcc keynote Hawaii april 2013
 
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
«Implicaciones de datos de aprendizaje procedentes de MOOCs», por Edmundo Tov...
 
Educational Innovation & Technology at MIT at Moodle Share Fair
Educational Innovation & Technology at MIT at Moodle Share FairEducational Innovation & Technology at MIT at Moodle Share Fair
Educational Innovation & Technology at MIT at Moodle Share Fair
 
Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...Define massive open online course: results from systematic review of 84 publi...
Define massive open online course: results from systematic review of 84 publi...
 
Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018Transitioning Education’s Knowledge Infrastructure ICLS 2018
Transitioning Education’s Knowledge Infrastructure ICLS 2018
 
EDR 8204 Week 3 Assignment: Analyze Action Research
EDR 8204 Week 3 Assignment: Analyze Action ResearchEDR 8204 Week 3 Assignment: Analyze Action Research
EDR 8204 Week 3 Assignment: Analyze Action Research
 
Evaluation Methods For Mobile Learning
Evaluation Methods For Mobile LearningEvaluation Methods For Mobile Learning
Evaluation Methods For Mobile Learning
 
Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...Deliberative Democracy as a strategy for co-designing university ethics aro...
Deliberative Democracy as a strategy for co-designing university ethics aro...
 
Facilitating Primary Student Teachers’ Development of Critical Thinking Throu...
Facilitating Primary Student Teachers’ Development of Critical Thinking Throu...Facilitating Primary Student Teachers’ Development of Critical Thinking Throu...
Facilitating Primary Student Teachers’ Development of Critical Thinking Throu...
 
Open Educational Resources: Experiences of use in a Latin-American context
Open Educational Resources: Experiences of use in a Latin-American contextOpen Educational Resources: Experiences of use in a Latin-American context
Open Educational Resources: Experiences of use in a Latin-American context
 
EMOOCs 2014 Proceedings of the European MOOC Stakeholder Summit 2014
EMOOCs 2014 Proceedings of  the European  MOOC Stakeholder  Summit 2014EMOOCs 2014 Proceedings of  the European  MOOC Stakeholder  Summit 2014
EMOOCs 2014 Proceedings of the European MOOC Stakeholder Summit 2014
 
Nov 27: LibreTexts Overview by Delmar Larsen
Nov 27: LibreTexts Overview by Delmar LarsenNov 27: LibreTexts Overview by Delmar Larsen
Nov 27: LibreTexts Overview by Delmar Larsen
 
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous TransformationalKeynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
Keynote Speech, Vijay Kumar: Learning OUTed -- Open Ubiquitous Transformational
 
1738
17381738
1738
 

More from Vitomir Kovanovic

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
Vitomir Kovanovic
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
Vitomir Kovanovic
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Vitomir Kovanovic
 

More from Vitomir Kovanovic (17)

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
 
What does effective online/blended teaching look like?
What does effective online/blended teaching look like?What does effective online/blended teaching look like?
What does effective online/blended teaching look like?
 
MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
 
Inquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & OpportunitiesInquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & Opportunities
 
Penetrating the black box of time-on-task estimation
Penetrating the black box of time-on-task estimationPenetrating the black box of time-on-task estimation
Penetrating the black box of time-on-task estimation
 

Recently uploaded

會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
中 央社
 
SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code Examples
Peter Brusilovsky
 

Recently uploaded (20)

Trauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical PrinciplesTrauma-Informed Leadership - Five Practical Principles
Trauma-Informed Leadership - Five Practical Principles
 
Đề tieng anh thpt 2024 danh cho cac ban hoc sinh
Đề tieng anh thpt 2024 danh cho cac ban hoc sinhĐề tieng anh thpt 2024 danh cho cac ban hoc sinh
Đề tieng anh thpt 2024 danh cho cac ban hoc sinh
 
Book Review of Run For Your Life Powerpoint
Book Review of Run For Your Life PowerpointBook Review of Run For Your Life Powerpoint
Book Review of Run For Your Life Powerpoint
 
male presentation...pdf.................
male presentation...pdf.................male presentation...pdf.................
male presentation...pdf.................
 
Supporting Newcomer Multilingual Learners
Supporting Newcomer  Multilingual LearnersSupporting Newcomer  Multilingual Learners
Supporting Newcomer Multilingual Learners
 
demyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptxdemyelinated disorder: multiple sclerosis.pptx
demyelinated disorder: multiple sclerosis.pptx
 
Mattingly "AI & Prompt Design: Named Entity Recognition"
Mattingly "AI & Prompt Design: Named Entity Recognition"Mattingly "AI & Prompt Design: Named Entity Recognition"
Mattingly "AI & Prompt Design: Named Entity Recognition"
 
OSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & SystemsOSCM Unit 2_Operations Processes & Systems
OSCM Unit 2_Operations Processes & Systems
 
Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"Mattingly "AI and Prompt Design: LLMs with NER"
Mattingly "AI and Prompt Design: LLMs with NER"
 
Improved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio AppImproved Approval Flow in Odoo 17 Studio App
Improved Approval Flow in Odoo 17 Studio App
 
Basic Civil Engineering notes on Transportation Engineering & Modes of Transport
Basic Civil Engineering notes on Transportation Engineering & Modes of TransportBasic Civil Engineering notes on Transportation Engineering & Modes of Transport
Basic Civil Engineering notes on Transportation Engineering & Modes of Transport
 
How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17How to Send Pro Forma Invoice to Your Customers in Odoo 17
How to Send Pro Forma Invoice to Your Customers in Odoo 17
 
PSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptxPSYPACT- Practicing Over State Lines May 2024.pptx
PSYPACT- Practicing Over State Lines May 2024.pptx
 
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...When Quality Assurance Meets Innovation in Higher Education - Report launch w...
When Quality Assurance Meets Innovation in Higher Education - Report launch w...
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
SPLICE Working Group: Reusable Code Examples
SPLICE Working Group:Reusable Code ExamplesSPLICE Working Group:Reusable Code Examples
SPLICE Working Group: Reusable Code Examples
 
8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management8 Tips for Effective Working Capital Management
8 Tips for Effective Working Capital Management
 
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
TỔNG HỢP HƠN 100 ĐỀ THI THỬ TỐT NGHIỆP THPT TOÁN 2024 - TỪ CÁC TRƯỜNG, TRƯỜNG...
 
Major project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategiesMajor project report on Tata Motors and its marketing strategies
Major project report on Tata Motors and its marketing strategies
 
VAMOS CUIDAR DO NOSSO PLANETA! .
VAMOS CUIDAR DO NOSSO PLANETA!                    .VAMOS CUIDAR DO NOSSO PLANETA!                    .
VAMOS CUIDAR DO NOSSO PLANETA! .
 

A novel model of cognitive presence assessment using automated learning analytics methods

  • 1. A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods Vitomir Kovanovic University of South Australia Teaching innovation Unit and School of Education #vkovanovic http://vitomir.kovanovic.info Vitomir.Kovanovic@unisa.edu.au
  • 2. 2 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Global Grand Challenges Shifting from an era of scarcity to abundance http://singularityu.org/global-grand-challenges
  • 3. 3 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Four challenges for next decade 1. Affordable, sustainable energy. 2. Cures for HIV and neurodegenerative diseases like Alzheimer's. 3. Protection from future health epidemics. 4. Tools to provide a world-class education to all students.
  • 4. 4 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info 1 GLOBAL NEED FOR EDUCATION Significant Challenges in Higher Education 2 INCREASE STUDENT ENGAGEMENT 3 DECREASING UNIVERSITY FUNDING MORE AND MORE UNIVERSITIES TURNING TO TECHNOLOGY MORE AND MORE UNIVERSITIES TURNING TO TECHNOLOGY
  • 5. 5 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info ONLINE & BLENDED Blended is the new norm. Increasing interest in online learning. MOOCs and new forms of delivery. LEARNING DATA Large amounts of data collected. Improve student experience. Understand learning processes. NEW MARKETS AND MODELS Workspace learning. Lifelong learning. Developing world. Technology & Data Trends Three important ways in which technology is shaping the future of education
  • 6. 6 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Learning analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs. CFP for the First Learning Analytics and Knowledge Conference https://tekri.athabascau.ca/analytics/ What is Learning Analytics? Learning Analytics Making sense of the available learning data Measurement Collection Analysis Reporting
  • 7. 7 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info What can Learning Analytics bring Improvement in teaching quality, student retention, learning outcomes, and understanding of learning processes
  • 8. 8 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Traditional Online Learning How can we use the educational data to understand student learning? Massive Open Online Courses How to adapt existing models of online learning in MOOC settings? What are the challenges of MOOC pedagogies? Learning Analytics for Online Learning
  • 9. 9 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info 1 Pedagogy The use of Learning analytics must be aligned with the existing pedagogical approaches used in particular learning context. Driving Principles Foundations for my learning analytics research 2 Data analysis Use automated data mining techniques to process large amounts of student-generated data to understand how student learn in online setting 3 Tools Develop automated learning analytics tools that can be used to assess student learning and improve research in online and distance education.
  • 10. 10 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Community of Inquiry Framework Dimensions of student online learning experience Social-constructivist model of learning - Students construct knowledge rather than receive it - Discussion is an essential part of learning - Inquiry basic learning activity Widely used in “traditional” online and distance education - Strong teacher presence - Up to ~ 30 students Cognitive Presence Teaching Presence Social Presence Student Experience Development of critical and deep thinking skills Social climate in the course 1. Group cohesion 2. Interactivity 3. Group affectivity Instructor’s role in the course: 1. Design & organization, 2. Facilitation, 3. Direct instruction (Garrison, Anderson, and Archer, 1999)
  • 11. 11 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Cognitive Presence Operationalization of critical and deep thinking skills • Exploring and testing solutions • Dilemma or problem identified • Synthesis of relevant information • Brainstorming • Exploring ideas 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection “an extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained Communication” (Garrison, Anderson, and Archer, 1999, p. 89) Definition
  • 12. 12 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info 1 Content analysis Coding scheme for each presence Assessment of three presences How do we measure levels of cognitive, social and teaching presence? 2 Self-reported survey 34 items, 1-5 Likert scale questions • Experience with the coding scheme • Inter-rater reliability issues • Time-consuming • Non-real time • Primary use: research • Fait accompli: results known after the course is over • Self-selection bias • Invasive (Arbaugh et al., 2008). (Garrison, Anderson, and Archer, 1999)
  • 13. 13 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info 1 From the online discussions Assessment based on Learning Analytics Automate as much as possible 2 Based on the use of educational technology • Identifying different study approaches • How those approaches relate to development of cognitive presence? • Clustering based on trace data Shared world learning Private world learning • Automating CoI coding scheme for cognitive presence • Real-time analysis of student cognitive presence development • Easier adoption and research • Text mining of student online discussion messages 1 Traditional online courses 2 MOOCs Athabasca University fully online course Delft University of Technology MOOC data Types of data and analytics Contexts
  • 14. 14 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Cognitive presence assessment framework Driven by Evidence-Centered Design (ECD) framework ASSESSMENT IMPLEMENTATION Automated cognitive presence classification system Technology use profiling system ASSESSMENT FRAMEWORK Evidence-centered design (ECD) assessment framework: Student model, Evidence model and Task model EDUCATIONAL TECHNOLOGY Technology use trace data and online discussion data from Learning Management Systems (LMS) and MOOC platforms: THEORY & PEDAGOGY Social constructivist learning & Community of Inquiry model Kovanovic et al. (in-press)
  • 15. 15 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Cognitive presence classification system Five-class text classification problem: 1 – Triggering event, 2 – Exploration, 3 – Integration, 4 – Resolution, 0 – Other (non cognitive). Data from six offers of a fully online course 1,747 messages coded for cognitive presence Developed random forest classifier Extracted features: Linguistic Inquiry Word Count features (LIWC): 93 different counts indicative of different psychological processes (e.g., affective, cognitive, social, perceptual) Coh-Metrix features: 108 metrics of text cohesion LSA coherence: Average LSA similarity of message’s paragraphs to each other. LSA space is built from Wikipedia articles related to concepts extracted from the topic start message (using TAGME). Named entity count: Number of concepts related to DBPedia computer science category (using DBPedia spotlight) Context features: Number of replies Message depth Cosine similarity to previous/next message Thread start/end Boolean indicators Kovanovic et al. (2016)
  • 16. 16 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Performance evaluation • We obtained 70.3% classification accuracy (95% CI[0.66, 0.75]) and 0.63 Cohen’s κ. • Significant improvements over Cohen’s κ of 0.41 and 0.48 reported in Kovanovic et al. (2014) and Waters et al. (2015) studies. • The feature space ~ 100x smaller • The feature space is also more generalizable • We provided more detailed operationalization of the cognitive presence coding scheme. Current work: – Analysis of discussion messages from “Functional Programming” MOOC by TU Delft – Investigation of the suitability of the same classification approach in various contexts (Essay analysis, Twitter messages) Kovanovic et al. (2016)
  • 17. 17 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Feature importance Phase # Variable Description MDG* Other TE Exp. Int. Res. 1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68 2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18 3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58 4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00 5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78 6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24 7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38 8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39 9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16 10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99 11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05 12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22 13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84 14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29 15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67 16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73 17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97 18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19 19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99 20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47 * MDG - Mean decrease Gini impurity index, FPS - first person singular Phase # Variable Description MDG* Other TE Exp. Int. Res. 1 cm.DESWC Number of words 32.91 55.41 80.91 117.71 183.30 280.68 2 ner.entity.cnt Number of named entities 26.41 13.44 21.67 28.84 44.75 64.18 3 cm.LDTTRa Lexical diversity, all words 21.98 0.85 0.77 0.71 0.65 0.58 4 message.depth Position within a discussion 19.09 2.39 1.00 1.84 1.87 2.00 5 cm.LDTTRc Lexical diversity, content words 17.12 0.95 0.90 0.86 0.82 0.78 6 cm.LSAGN Avg. givenness of each sentence 16.63 0.10 0.14 0.18 0.21 0.24 7 liwc.Qmark Number of question marks 16.59 0.27 1.84 0.92 0.58 0.38 8 message.sim.prev Similarity with previous message 16.41 0.20 0.06 0.22 0.30 0.39 9 cm.LDVOCD Lexical diversity, VOCD 15.43 12.92 28.99 53.57 83.47 97.16 10 liwc.money Number of money-related words 14.38 0.21 0.32 0.32 0.65 0.99 11 cm.DESPL Avg. number of paragraphs 12.47 4.26 6.37 7.49 10.17 14.05 12 Message.sim.next Similarity with next message 11.74 0.08 0.34 0.20 0.22 0.22 13 Message.reply.cnt Number of replies 11.67 0.42 1.44 0.82 1.10 0.84 14 cm.DESSC Sentence count 11.67 4.28 6.36 7.49 10.17 14.29 15 lsa.similarity Avg. LSA sim. between sentences 9.69 0.29 0.47 0.54 0.62 0.67 16 cm.DESSL Avg. sentence length 9.60 11.88 13.62 16.69 19.36 21.73 17 cm.DESWLsyd SD of word syllables count 8.92 0.98 1.33 0.98 0.97 0.97 18 liwc.i Number of FPS* pronouns 8.84 4.33 2.82 2.37 2.51 2.19 19 cm.RDFKGL Flesch-Kincaid Grade level 8.29 7.68 10.30 10.19 11.13 11.99 20 cm.SMCAUSwn WordNet overlap between verbs 8.14 0.38 0.48 0.51 0.50 0.47 Kovanovic et al. (2016)
  • 18. 18 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info The higher the cognitive presence… • The longer the message, the more paragraphs and sentences. • More concepts mentioned (more named entities). • The lower the lexical diversity (both at content level and in general). • The later its position in the thread. Except non- cognitive messages, they tend to occur closer to the end as well. • The fewer the question marks (except non-cognitive, they have fewest question marks) • The higher the average length of sentence and their similarity to each other. • The more money-related terms. Operationalization of cognitive presence • Exploring and testing solutions • Dilemma or problem identified • Synthesis of relevant information • Brainstorming • Exploring ideas 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection •Lowest message readability •Syllabi count inconsistent •Most replies •Low similarity with the next message •More replies than exploration and resolution •Least replies •Question marks more frequent than integration 2. Exploration 3. Integration 4. Resolution 1. Triggering Event Shared world of discourse Private world of reflection Kovanovic et al. (2016)
  • 19. 19 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Technology use profiling system • Used 200,000 LMS trace data records of student learning activities • Hierarchical clustering (Ward method, Euclidean distance) • MANOVA analysis of differences in cognitive presence, followed by discriminant factor analysis (DFA) and ANOVA analysis • We identified six different technology use profiles. • Large number of students poorly regulated technology use. • Students with different technology-use profiles had significant differences in the levels of their cognitive presence. We observed large effect size (partial eta2 = .54). • Quality of interactions is more important than the quantity for the development of cognitive presence. Kovanovic et al. (2014)
  • 20. 20 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info CoI within Massive Open Online Courses Used the data from five TU Delft MOOCs 2,446 survey responses 5-item Likert scale responses Originally, CoI studies found 3-factor structure to best describe CoI survey instrument Goal: can we replicate those findings in MOOCs?
  • 21. 21 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Factor analysis CoI within Massive Open Online Courses Original model holds in the MOOC context CP6: Online discussions were valuable in helping me appreciate different perspectives Affectivit y Resolution Course organization & design Better fit: 6 factor model:
  • 22. 22 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info The updated model CoI within Massive Open Online Courses
  • 23. 23 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info How are they interacting Structural equation model between three presences • Direct effect of Teaching Presence on Cognitive presence, and • Mediating effect of social presence on teaching presence and cognitive presence relationship Cognitive presence Social presence Teaching presence Cognitive presence Social presence Teaching presence Original SEM model MOOC model 0.52 0.51 0.40 0.60 0.22 0.35
  • 24. 24 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Study strategies within MOOCs (submitted) • Adopted same methodology as in “traditional” online setting • Significant MANOVA, (p=0.038). • Univariate differences in the cognitive presence levels for the resolution phase (C1 – C2 and C1 – C3 )
  • 25. 25 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Summary • Important to realize the tremendous potential educational data has for improving teaching and learning. • How this potential of the data fits the existing models of online and blended learning? • How can we design analytics that are flexible enough and provide meaningful insights about student learning? • Can we better support research in online education by automating existing approaches used by researchers? • Can we use the existing pedagogical models in MOOC setting? • How can we make student learning in MOOCs les solitary?
  • 26. 26 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Summary: cognitive presence assessment framework • We developed a framework for assessing the levels of cognitive presence • We focused on two types of insights: from trace data & discussion data and • We focused on two contexts: traditional online courses and MOOCs • We developed an assessment framework for cognitive presence using Evidence-centered design (ECD) guidelines • Automated existing tool for measuring cognitive presence • Enables easier adoption of Community of inquiry model • Better operationalization of the phases of cognitive presence • We developed a student profiling method that can be used to identify different profiles of student technology use • Better understand online learning, especially related to student self-regulation of learning • Insights into their private solitary learning not displayed in online discussion transcripts
  • 27. 27 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Summary: Cognitive presence in MOOCs • Explored the use of CoI model in MOOC context and use existing pedagogies at scale • We conducted a factor analysis of Community of Inquiry model in the MOOC context: • Original CoI model holds but with certain factors being more emphasized than others • The role of social group is much weaker in MOOCs • We identified profiles of students based on their technology use • Identified profiles differ in the levels of their cognitive and social presence • The differences are much smaller, primarily due to the use of survey-based instrument
  • 28. 28 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info Current and future work • Enable provision of formative feedback based on their use of educational systems • Possible to support instructional interventions by embedding the developed analytics into the learning platform • We are currently working on a MOOC experimentation platform that will include cognitive presence analytics • At present, working on coding of MOOC discussion messages to evaluate the use of cognitive presence classifier in the MOOC context • Also, exploring the use of the same classification methodology for similar problems Student reflection in discussions (OU) Essay grading (USF) Twitter and YouTube comments (QUT)
  • 29. 29 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info The End THANK YOU FOR YOUR TIME Q/A
  • 30. 30 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info REFERENCES Arbaugh, J.B. et al. (2008). “Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample”. In: The Internet and Higher Education 11.3–4, pp. 133–136. Garrison, D. Randy, Terry Anderson, and Walter Archer (1999). “Critical Inquiry in a Text-Based Environment: Computer Conferencing in Higher Education”. In: The Internet and Higher Education 2.2–3, pp. 87–105. Kovanović, V., Gašević, D., Hatala, M., & Siemens, G. (in-press). A Novel Model of Cognitive Presence Assessment Using Automated Learning Analytics Methods (Analytics4Learning). SRI Education. Available at: http://vitomir.kovanovic.info/publications/ Kovanović, V., Joksimović, S., Waters, Z., Gašević, D., Kitto, K., Hatala, M., & Siemens, G. (2016). Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 15–24). New York, NY, USA: ACM. https://doi.org/10.1145/2883851.2883950 Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002 Kovanović, V., Joksimović, S., Gašević, D., & Hatala, M. (2014). Automated content analysis of online discussion transcripts. In Proceedings of the Workshops at the LAK 2014 Conference co-located with 4th International Conference on Learning Analytics and Knowledge (LAK 2014). Indianapolis, IN. Retrieved from http://ceur-ws.org/Vol-1137/
  • 31. 31 Vitomir Kovanovic (University of South Australia) http://vitomir.kovanovic.info REFERENCES Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., Čukić, I., de Vries, P., … Gašević, D. (2018). Exploring communities of inquiry in Massive Open Online Courses. Computers & Education, 119, 44–58. https://doi.org/10.1016/j.compedu.2017.11.010