SlideShare a Scribd company logo
1 of 30
Download to read offline
1
Understand students’ self-reflections
through learning analytics
Vitomir Kovanović¹, Srećko Joksimović¹, Negin Mirriahi1
Ellen Blaine2, Dragan Gašević3,4, George Siemens5, and Shane Dawson1
¹ The University of South Australia, Australia
² Stanford University, USA
³ Monash University, Australia
⁴ University of Edinburgh, United Kingdom
5 University of Texas at Arlington, USA
March 9, 2018
Sydney, NSW, Australia
Vitomir.Kovanovic@unisa.edu.au
http://vitomir.kovanovic.info
@vkovanovic
2
Reflection and Self-regulated Learning
Student self-reflection:
• A key strategy for practicing critical thinking
• Develop autonomy, and confidence in their learning
• Common approach for development of self-regulation skills
Common self-reflection methods:
• Writing reflective journals
• Reflective video annotations
3
COPES model of Self-regulation
(Butler and Winnie, 1995)
4
Reflection and Self-regulated Learning
Depth of reflections matters:
• Providing observations & commentaries
• Describing motivations behind certain behaviors
• Setting goals for practice
Prior work:
• Different depth  Different learning outcomes
• Prior experience with reflective activities matter (Gašević et al., 2017)
• Scaffolding and support vital for student success and effectiveness of
reflective activities
5
Reflection and Self-regulated Learning
• Providing feedback & guidance essential but time consuming!
• Research on student-reflection is also complicated and expensive
(content analysis)
• Hard to provide real-time feedback & monitoring of student progress
Previous work focused on reflection in essays (e.g., Ullmann et al 2013;
Ullmann, 2015; Gibson and Kitto, 2015; Gibson et al. 2016)
Three key approaches:
1. Dictionary-based
2. Rule-based
3. Machine-learning-based
6
Overall goal
and why it matters
Automate the analysis of students’ self-reflective
video annotations
RQ1: What are the indicators of self-reflection?
RQ2: Can we do it (accurately enough)?
7
Overall goal
and why it matters
Automate the analysis of students’ self-reflective
video annotations
Use psychologically-sound and established
measures of psychological processes
8
Collaborative Lecture Annotation System (CLAS)
Video annotation software
9
Open Video Annotations for Learning (OVAL)
Essentially CLAS 2.0
10
Data: Courses
Four UG courses in performing arts offered in the 12-13 academic year at a
large research-intensive public university in Canada.
Course Recording
type
CLAS
required
Enrolled
students
Coded units of
analysis
Course 1 Group No 31 145 (3.27%)
Course 2 Individual Yes 40 1,393 (31.44%)
Course 3 Individual Yes 28 2,457 (55.46%)
Course 4 Individual No 20 435 (9.82%)
Total: 119 (771) 4,430 (100.00%
11
Reflection coding scheme
• A modified version of Hulsman et al. (2009) coding scheme
• Annotations split into units of analysis (utterances)
• Two coders codes subset of data until Cohen’s K of 0.75 reached
Category Definition Example
Observation Student indicates what they
observed about their own
behavior, but does not indicate
why the behavior occurred.
“I still continue to have
problems making eye
contact...”
Motive Student indicates what they
observed and why it occurred.
“...being up there made me
insecure and nervous, which
led to my eyes dropping
frequently...”
Goal Student indicates what they will
do next time or what they need
to work on.
“What I really want to avoid
is ending up just mirroring
everything.”
12
Text classification
Build a classifier for video self-reflection
By automating coding of video annotations, we can
overcome many challenges of video reflection assessment.
Builds on previous text-mining work in education
We build on the previous work of Kovanovic et al. (2016)
Combine N-grams with linguistic metrics
To provide both context-free and context-specific
indicators of reflection
Four-class text classifier
Classifier needs to categorize each message to:
1-Observation, 2 – Motive, 3 – Goal, 0 - Other.
13
Train-test split
• Data split into 75%–25% train and test datasets
Dataset
Category Train (75%) Test (25%) All
Observation 1,135 (34.17%) 382 (34.48%) 1,517 (34.24%)
Motive 1,848 (55.63%) 625 (56.41%) 2,473 (55.82%)
Goal 174 (5.24%) 56 (5.05%) 230 (5.19%)
Other 165 (4.97%) 45 (4.06%) 210 (4.74%)
Total: 3,322 (100%) 1,108 (100%) 4,430 (100%)
14
Extracted features
503 features in total extracted
1 3
2 4
LIWC features
93 measures of different psychological processes
N-grams
Top 100 unigrams, bigrams, and trigrams
Coh-Metrix features
109 metrics of text coherence (and related metrics)
Context features
First unit in sentence (binary feature)
15
Features: Top 10 N-grams
Unigram Freq. Bigram Freq. Trigram Freq.
need 383 left hand 112 practice front mirror 17
conducting 279 eye contact 71 use left hand 14
think 248 need work 55 third goal would 11
music 239 make sure 54 make eye contact 10
really 200 front mirror 36 second goal would 10
hand 182 goal would 32 first goal would 10
practice 181 feel like 30 three critical goals 8
ensemble 171 beat pattern 30 critical goals improvement 8
work 170 right hand 29 really need work 8
beat 161 also need 26 influence sound moment 7
16
SMOTE preprocessing
SMOTE preprocessing for class balancing. Dark blue
– original instances which are preserved, light blue – synthetic instances
We generate new data points in minority classes by “syntactic resampling”
using SMOTE technique.
To generate a new data point (Z) ∈ Rn:
• Pick a random data existing data point (X),
• Pick K (in our case 5) instances most similar to the given data point,
• Pick randomly one of the K neighbors (Y)
• Create a new data point Z as a linear combination: Z = X + rand(0,1)*Y
17
Random Forest classifier
A state-of-the-art ensemble learning method:
• Builds a large collection of decision trees (i.e.,
forest) using a subset of features (i.e., columns)
• Reduces the variance without increasing the bias
• Final class for a data point: a simple majority vote
across the forest.
Two parameters:
1. ntree – the number of trees built
2. mtry – number of features used
ntree = 6
18
Random Forest classifier
Individual tree
mtry=3
8 features
19
Implementation
Feature Extraction
• N-grams; NLTK Python library
• Coh-Metrix (McNamara, Graesser, McCarthy, & Cai, 2014)
• LIWC (Tausczik & Pennebaker, 2010)
Algorithm implementation
• Stratified sampling implemented using scikit-learn Python library
• SMOTE algorithm implemented using unbalanced R package
• Random Forest classifier using randomForest R package
• Repeated cross-validation using carret R package
20
Hyper-parameter tuning
1. ntree – the number of trees built
(we built 500)
2. mtry – number of features used
(evaluated using 10-fold CV)
Values examined:
{2, 3, 4, 6, 8, 11, 15, 20, 27, 36, 48,
65, 87, 156, 209, 279, 373, 500}
mtry Accuracy (SD) Kappa (SD)
Min 2 .81 (.01) .75 (.02)
Max 116 .89 (.01) .85 (.02)
Difference .07 .10
Two parameters optimized
Hyper parameter tuning results
Hyper parameter tuning
21
Performance evaluation
We obtained:
• 75% classification accuracy
95% CI[0.72, 0.77]
• 0.51 Cohen’s κ
Predicted
Other Observation Goal Motive Error rate
Actual
Other 9 9 27 0 .80
Observation 1 250 131 0 .34
Goal 1 59 564 1 .10
Motive 0 30 25 1 .98
Confusion matrix
Out-of-bag (OOB) error rate
22
Performance evaluation
• “Moderate” level of agreement (Cohen, 1988)
• Parameter optimization important (.1 Cohen’s κ difference, 7%
classification accuracy).
• Best-performance for two most common classes
• Features both encode general properties of self-reflection and particular
learning context
• “Theory-driven” context-free features
• Simple context-specific features
23
Feature Importance
• A side product of Random Forest algorithm
• Mean Decrease Gini (MDG) measure of
feature contribution to reducing decision
tree impurity
• A long tail of feature importance
• Few features very important, most not so
much
• Provides more detailed insights into what
characterizes different types of self-
reflection
Min Q1 Median Mean Q3 Max.
0.00 0.12 1.06 10.82 15.34 219.94
Summary of classification feature importance
24Feature importance
Category
# Variable Description MDG Other Obser. Goal Motive
1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55
2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37
3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66
4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80
5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194
6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39
7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91
8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00
9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47
10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45
11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314
12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01
13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66
14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98
15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28
16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50
17 cm.SYNLE Syntactic complexity: mean number of words before
the main verb in the main clause
58 2.36 2.65 1.78 3.53
18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04
19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36
20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04
25Feature importance
Category
# Variable Description MDG Other Obser. Goal Motive
1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55
2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37
3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66
4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80
5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194
6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39
7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91
8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00
9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47
10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45
11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314
12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01
13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66
14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98
15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28
16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50
17 cm.SYNLE Syntactic complexity: mean number of words before
the main verb in the main clause
58 2.36 2.65 1.78 3.53
18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04
19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36
20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04
26
Characteristics of different self-reflection statements
Unique characteristics
O
Observation
Use of perception words
Oriented towards past
Low causal connectiveness
Low levels of higher order critical thinking
M
Motive
High ratio of causal particles to verbs
Frequent use of passive voice (agentless)
Oriented towards past
Expressing causality
Complex verb phrases
G
Goal
Use of perception words
Low expression of insight
Simple verb phrases
High levels of higher order critical thinking
Frequent use of causal verbs
Non-reflective (other)
Low use of perception words
Express of personal matters
High use of punctuation
High use of noun phrases
27
Summary
Almost done
• We developed a classifier for automated coding of students’ in-video annotations with respect to
the expressed
• Our exploratory analysis is encouraging. We achieved 75% percent accuracy, Cohen’s k=.51
• The adopted features both capture context-free and also context-specific nature of self-reflection
• We provided more detailed operationalization of the different types of reflective statements
Future work:
• Replicate the study using the new data from other course (use existing model on the new course)
• Examine additional features that better capture goal reflective statements
28
The end
That is all folks
Thank you
29
References (I/II)
Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis.
Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245
Cohen, J. (1988). The Analysis of Variance. In Statistical power analysis for the behavioral sciences
(pp. 273--406). Hillsdale, N.J.: L. Erlbaum Associates.
Gašević, D., Mirriahi, N., Dawson, S., & Joksimović, S. (2017). Effects of instructional conditions and
experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220.
https://doi.org/10.1016/j.chb.2016.10.026
Gibson, A., & Kitto, K. (2015). Analysing Reflective Text for Learning Analytics: An Approach Using
Anomaly Recontextualisation. In Proceedings of the Fifth International Conference on Learning
Analytics and Knowledge (pp. 275–279). New York, NY, USA: ACM.
https://doi.org/10.1145/2723576.2723635
Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the Discovery of Learner Metacognition From
Reflective Writing. Journal of Learning Analytics, 3(2), 22–36.
https://doi.org/10.18608/jla.2016.32.3
30
References (II/II)
Hulsman, R. L., Harmsen, A. B., & Fabriek, M. (2009). Reflective teaching of medical communication
skills with DiViDU: Assessing the level of student reflection on recorded consultations with
simulated patients. Patient Education and Counseling, 74(2), 142–149.
https://doi.org/10.1016/j.pec.2008.10.009
McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation of Text
and Discourse with Coh-Metrix. Cambridge University Press.
Tausczik, Y. R., & Pennebaker, J. W. (2010). The Psychological Meaning of Words: LIWC and
Computerized Text Analysis Methods. Journal of Language and Social Psychology, 29(1), 24–54.
https://doi.org/10.1177/0261927X09351676
Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based
approach (PhD thesis). The Open University.
Ullmann, T. D., Wild, F., & Scott, P. (2013). Comparing automatically detected reflective texts with
human judgements. In 2nd Workshop on Awareness and Reflection in Technology- Enhanced
Learning (pp. 101–116). Retrieved from http://oro.open.ac.uk/id/eprint/37830

More Related Content

Similar to Understand students’ self-reflections through learning analytics

Topic_6
Topic_6Topic_6
Topic_6
butest
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
butest
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
butest
 
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
mingliu107
 
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Vitomir Kovanovic
 
Modeling Electronic Health Records with Recurrent Neural Networks
Modeling Electronic Health Records with Recurrent Neural NetworksModeling Electronic Health Records with Recurrent Neural Networks
Modeling Electronic Health Records with Recurrent Neural Networks
Josh Patterson
 

Similar to Understand students’ self-reflections through learning analytics (20)

Topic_6
Topic_6Topic_6
Topic_6
 
Rui Meng - 2017 - Deep Keyphrase Generation
Rui Meng - 2017 - Deep Keyphrase GenerationRui Meng - 2017 - Deep Keyphrase Generation
Rui Meng - 2017 - Deep Keyphrase Generation
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
 
Learning to Search Henry Kautz
Learning to Search Henry KautzLearning to Search Henry Kautz
Learning to Search Henry Kautz
 
Multimodal Learning Analytics
Multimodal Learning AnalyticsMultimodal Learning Analytics
Multimodal Learning Analytics
 
Deep Learning Models for Question Answering
Deep Learning Models for Question AnsweringDeep Learning Models for Question Answering
Deep Learning Models for Question Answering
 
Wearable Accelerometer Optimal Positions for Human Motion Recognition(LifeTec...
Wearable Accelerometer Optimal Positions for Human Motion Recognition(LifeTec...Wearable Accelerometer Optimal Positions for Human Motion Recognition(LifeTec...
Wearable Accelerometer Optimal Positions for Human Motion Recognition(LifeTec...
 
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
 
A Study of Wearable Accelerometers Layout for Human Activity Recognition(Asia...
A Study of Wearable Accelerometers Layout for Human Activity Recognition(Asia...A Study of Wearable Accelerometers Layout for Human Activity Recognition(Asia...
A Study of Wearable Accelerometers Layout for Human Activity Recognition(Asia...
 
Joseph Jay Williams - WESST - Bridging Research and Practice via MOOClets & C...
Joseph Jay Williams - WESST - Bridging Research and Practice via MOOClets & C...Joseph Jay Williams - WESST - Bridging Research and Practice via MOOClets & C...
Joseph Jay Williams - WESST - Bridging Research and Practice via MOOClets & C...
 
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflect...
 
Database Research Principles Revealed
Database Research Principles RevealedDatabase Research Principles Revealed
Database Research Principles Revealed
 
Machine Learning and Artificial Neural Networks.ppt
Machine Learning and Artificial Neural Networks.pptMachine Learning and Artificial Neural Networks.ppt
Machine Learning and Artificial Neural Networks.ppt
 
EssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdfEssentialsOfMachineLearning.pdf
EssentialsOfMachineLearning.pdf
 
Do you really need to test with only 5 users
Do you really need to test with only 5 usersDo you really need to test with only 5 users
Do you really need to test with only 5 users
 
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
 
WUD Slovakia 2015: Experiment v UX class / Prof. Ing. Mária Bieliková, PhD.
WUD Slovakia 2015: Experiment v UX class / Prof. Ing. Mária Bieliková, PhD.WUD Slovakia 2015: Experiment v UX class / Prof. Ing. Mária Bieliková, PhD.
WUD Slovakia 2015: Experiment v UX class / Prof. Ing. Mária Bieliková, PhD.
 
Modeling Electronic Health Records with Recurrent Neural Networks
Modeling Electronic Health Records with Recurrent Neural NetworksModeling Electronic Health Records with Recurrent Neural Networks
Modeling Electronic Health Records with Recurrent Neural Networks
 
Predicting Contradiction Intensity: Low, Strong or Very Strong?
Predicting Contradiction Intensity: Low, Strong or Very Strong?Predicting Contradiction Intensity: Low, Strong or Very Strong?
Predicting Contradiction Intensity: Low, Strong or Very Strong?
 
Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)Deep Learning: concepts and use cases (October 2018)
Deep Learning: concepts and use cases (October 2018)
 

More from Vitomir Kovanovic

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
Vitomir Kovanovic
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
Vitomir Kovanovic
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Vitomir Kovanovic
 

More from Vitomir Kovanovic (20)

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
 
Validating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analyticsValidating a theorized model of engagement in learning analytics
Validating a theorized model of engagement in learning analytics
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of Inquiry
 
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
 
What does effective online/blended teaching look like?
What does effective online/blended teaching look like?What does effective online/blended teaching look like?
What does effective online/blended teaching look like?
 
MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
 
Automated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion TranscriptsAutomated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion Transcripts
 
Automated System for Cognitive Presence Coding
Automated System for Cognitive Presence CodingAutomated System for Cognitive Presence Coding
Automated System for Cognitive Presence Coding
 
Inquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & OpportunitiesInquiry-based Learning & MOOCs: Challenges & Opportunities
Inquiry-based Learning & MOOCs: Challenges & Opportunities
 

Recently uploaded

The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
heathfieldcps1
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
中 央社
 
Neurulation and the formation of the neural tube
Neurulation and the formation of the neural tubeNeurulation and the formation of the neural tube
Neurulation and the formation of the neural tube
SaadHumayun7
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
中 央社
 

Recently uploaded (20)

Open Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPointOpen Educational Resources Primer PowerPoint
Open Educational Resources Primer PowerPoint
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽會考英聽
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
Incoming and Outgoing Shipments in 2 STEPS Using Odoo 17
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 
The Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. HenryThe Last Leaf, a short story by O. Henry
The Last Leaf, a short story by O. Henry
 
factors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxfactors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptx
 
Morse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptxMorse OER Some Benefits and Challenges.pptx
Morse OER Some Benefits and Challenges.pptx
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
 
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdfTelling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
Telling Your Story_ Simple Steps to Build Your Nonprofit's Brand Webinar.pdf
 
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdfPost Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
Post Exam Fun(da) Intra UEM General Quiz 2024 - Prelims q&a.pdf
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
Removal Strategy _ FEFO _ Working with Perishable Products in Odoo 17
 
Neurulation and the formation of the neural tube
Neurulation and the formation of the neural tubeNeurulation and the formation of the neural tube
Neurulation and the formation of the neural tube
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 

Understand students’ self-reflections through learning analytics

  • 1. 1 Understand students’ self-reflections through learning analytics Vitomir Kovanović¹, Srećko Joksimović¹, Negin Mirriahi1 Ellen Blaine2, Dragan Gašević3,4, George Siemens5, and Shane Dawson1 ¹ The University of South Australia, Australia ² Stanford University, USA ³ Monash University, Australia ⁴ University of Edinburgh, United Kingdom 5 University of Texas at Arlington, USA March 9, 2018 Sydney, NSW, Australia Vitomir.Kovanovic@unisa.edu.au http://vitomir.kovanovic.info @vkovanovic
  • 2. 2 Reflection and Self-regulated Learning Student self-reflection: • A key strategy for practicing critical thinking • Develop autonomy, and confidence in their learning • Common approach for development of self-regulation skills Common self-reflection methods: • Writing reflective journals • Reflective video annotations
  • 3. 3 COPES model of Self-regulation (Butler and Winnie, 1995)
  • 4. 4 Reflection and Self-regulated Learning Depth of reflections matters: • Providing observations & commentaries • Describing motivations behind certain behaviors • Setting goals for practice Prior work: • Different depth  Different learning outcomes • Prior experience with reflective activities matter (Gašević et al., 2017) • Scaffolding and support vital for student success and effectiveness of reflective activities
  • 5. 5 Reflection and Self-regulated Learning • Providing feedback & guidance essential but time consuming! • Research on student-reflection is also complicated and expensive (content analysis) • Hard to provide real-time feedback & monitoring of student progress Previous work focused on reflection in essays (e.g., Ullmann et al 2013; Ullmann, 2015; Gibson and Kitto, 2015; Gibson et al. 2016) Three key approaches: 1. Dictionary-based 2. Rule-based 3. Machine-learning-based
  • 6. 6 Overall goal and why it matters Automate the analysis of students’ self-reflective video annotations RQ1: What are the indicators of self-reflection? RQ2: Can we do it (accurately enough)?
  • 7. 7 Overall goal and why it matters Automate the analysis of students’ self-reflective video annotations Use psychologically-sound and established measures of psychological processes
  • 8. 8 Collaborative Lecture Annotation System (CLAS) Video annotation software
  • 9. 9 Open Video Annotations for Learning (OVAL) Essentially CLAS 2.0
  • 10. 10 Data: Courses Four UG courses in performing arts offered in the 12-13 academic year at a large research-intensive public university in Canada. Course Recording type CLAS required Enrolled students Coded units of analysis Course 1 Group No 31 145 (3.27%) Course 2 Individual Yes 40 1,393 (31.44%) Course 3 Individual Yes 28 2,457 (55.46%) Course 4 Individual No 20 435 (9.82%) Total: 119 (771) 4,430 (100.00%
  • 11. 11 Reflection coding scheme • A modified version of Hulsman et al. (2009) coding scheme • Annotations split into units of analysis (utterances) • Two coders codes subset of data until Cohen’s K of 0.75 reached Category Definition Example Observation Student indicates what they observed about their own behavior, but does not indicate why the behavior occurred. “I still continue to have problems making eye contact...” Motive Student indicates what they observed and why it occurred. “...being up there made me insecure and nervous, which led to my eyes dropping frequently...” Goal Student indicates what they will do next time or what they need to work on. “What I really want to avoid is ending up just mirroring everything.”
  • 12. 12 Text classification Build a classifier for video self-reflection By automating coding of video annotations, we can overcome many challenges of video reflection assessment. Builds on previous text-mining work in education We build on the previous work of Kovanovic et al. (2016) Combine N-grams with linguistic metrics To provide both context-free and context-specific indicators of reflection Four-class text classifier Classifier needs to categorize each message to: 1-Observation, 2 – Motive, 3 – Goal, 0 - Other.
  • 13. 13 Train-test split • Data split into 75%–25% train and test datasets Dataset Category Train (75%) Test (25%) All Observation 1,135 (34.17%) 382 (34.48%) 1,517 (34.24%) Motive 1,848 (55.63%) 625 (56.41%) 2,473 (55.82%) Goal 174 (5.24%) 56 (5.05%) 230 (5.19%) Other 165 (4.97%) 45 (4.06%) 210 (4.74%) Total: 3,322 (100%) 1,108 (100%) 4,430 (100%)
  • 14. 14 Extracted features 503 features in total extracted 1 3 2 4 LIWC features 93 measures of different psychological processes N-grams Top 100 unigrams, bigrams, and trigrams Coh-Metrix features 109 metrics of text coherence (and related metrics) Context features First unit in sentence (binary feature)
  • 15. 15 Features: Top 10 N-grams Unigram Freq. Bigram Freq. Trigram Freq. need 383 left hand 112 practice front mirror 17 conducting 279 eye contact 71 use left hand 14 think 248 need work 55 third goal would 11 music 239 make sure 54 make eye contact 10 really 200 front mirror 36 second goal would 10 hand 182 goal would 32 first goal would 10 practice 181 feel like 30 three critical goals 8 ensemble 171 beat pattern 30 critical goals improvement 8 work 170 right hand 29 really need work 8 beat 161 also need 26 influence sound moment 7
  • 16. 16 SMOTE preprocessing SMOTE preprocessing for class balancing. Dark blue – original instances which are preserved, light blue – synthetic instances We generate new data points in minority classes by “syntactic resampling” using SMOTE technique. To generate a new data point (Z) ∈ Rn: • Pick a random data existing data point (X), • Pick K (in our case 5) instances most similar to the given data point, • Pick randomly one of the K neighbors (Y) • Create a new data point Z as a linear combination: Z = X + rand(0,1)*Y
  • 17. 17 Random Forest classifier A state-of-the-art ensemble learning method: • Builds a large collection of decision trees (i.e., forest) using a subset of features (i.e., columns) • Reduces the variance without increasing the bias • Final class for a data point: a simple majority vote across the forest. Two parameters: 1. ntree – the number of trees built 2. mtry – number of features used ntree = 6
  • 18. 18 Random Forest classifier Individual tree mtry=3 8 features
  • 19. 19 Implementation Feature Extraction • N-grams; NLTK Python library • Coh-Metrix (McNamara, Graesser, McCarthy, & Cai, 2014) • LIWC (Tausczik & Pennebaker, 2010) Algorithm implementation • Stratified sampling implemented using scikit-learn Python library • SMOTE algorithm implemented using unbalanced R package • Random Forest classifier using randomForest R package • Repeated cross-validation using carret R package
  • 20. 20 Hyper-parameter tuning 1. ntree – the number of trees built (we built 500) 2. mtry – number of features used (evaluated using 10-fold CV) Values examined: {2, 3, 4, 6, 8, 11, 15, 20, 27, 36, 48, 65, 87, 156, 209, 279, 373, 500} mtry Accuracy (SD) Kappa (SD) Min 2 .81 (.01) .75 (.02) Max 116 .89 (.01) .85 (.02) Difference .07 .10 Two parameters optimized Hyper parameter tuning results Hyper parameter tuning
  • 21. 21 Performance evaluation We obtained: • 75% classification accuracy 95% CI[0.72, 0.77] • 0.51 Cohen’s κ Predicted Other Observation Goal Motive Error rate Actual Other 9 9 27 0 .80 Observation 1 250 131 0 .34 Goal 1 59 564 1 .10 Motive 0 30 25 1 .98 Confusion matrix Out-of-bag (OOB) error rate
  • 22. 22 Performance evaluation • “Moderate” level of agreement (Cohen, 1988) • Parameter optimization important (.1 Cohen’s κ difference, 7% classification accuracy). • Best-performance for two most common classes • Features both encode general properties of self-reflection and particular learning context • “Theory-driven” context-free features • Simple context-specific features
  • 23. 23 Feature Importance • A side product of Random Forest algorithm • Mean Decrease Gini (MDG) measure of feature contribution to reducing decision tree impurity • A long tail of feature importance • Few features very important, most not so much • Provides more detailed insights into what characterizes different types of self- reflection Min Q1 Median Mean Q3 Max. 0.00 0.12 1.06 10.82 15.34 219.94 Summary of classification feature importance
  • 24. 24Feature importance Category # Variable Description MDG Other Obser. Goal Motive 1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55 2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37 3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66 4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80 5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194 6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39 7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91 8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00 9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47 10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45 11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314 12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01 13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66 14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98 15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28 16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50 17 cm.SYNLE Syntactic complexity: mean number of words before the main verb in the main clause 58 2.36 2.65 1.78 3.53 18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04 19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36 20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04
  • 25. 25Feature importance Category # Variable Description MDG Other Obser. Goal Motive 1 liwc.see Perceptual processes: seeing 220 1.01 1.64 1.62 1.55 2 cm.SMCAUSr Situation model: ratio of casual particles to verbs 201 0.12 0.13 0.11 0.37 3 cm.DRPVAL Agentless passive voice density, incidence 184 2.46 3.02 2.03 4.66 4 liwc.focuspast Time orientation: focus towards past 152 1.46 4.57 0.80 4.80 5 cm.WRDNOUN Word information: noun incidence 121 252 186 208 194 6 liwc.ingest Biological processes: ingestion (e.g., dish, eat) 112 0.58 0.30 0.33 0.39 7 cm.CNCCaus Causal connectives, incidence 105 28.27 20.86 28.57 42.91 8 trust.ensemble Freq. of “trust ensemble” bigram 95 0.00 0.00 0.00 0.00 9 cm.SMINTEr Ratio: intentional particles to int. actions/events 88 0.30 0.23 0.36 0.47 10 liwc.Period Punctuation: use of full stop 75 9.17 6.38 7.14 5.45 11 cm.DRNP Syntactic pattern den: incidence of noun phrases 72 391 320 306 314 12 chamber.music Freq. of “chamber music” bigram 71 0.00 0.00 0.00 0.01 13 liwc.AllPunc Punctuation: all (e.g., periods, commas) 69 19.22 13.02 14.06 11.66 14 liwc.cause Cognitive processes: causality (e.g., because) 63 2.02 1.85 2.03 3.98 15 cm.SMCAUSv Situational model: incidence of causal verbs 61 33.60 46.08 65.65 44.28 16 liwc.insight Cognitive processes: insight (e.g., think, know) 60 2.98 3.64 2.36 3.50 17 cm.SYNLE Syntactic complexity: mean number of words before the main verb in the main clause 58 2.36 2.65 1.78 3.53 18 liwc.home Personal concerns: home (e.g., kitchen, landlord) 56 0.10 0.08 0.05 0.04 19 liwc.Analytic Summary measures: formal, logical, hier. thinking 52 60.02 54.60 70.94 57.36 20 liwc.percept Perceptual processes: all (e.g., look, heard) 52 4.20 4.85 4.69 5.04
  • 26. 26 Characteristics of different self-reflection statements Unique characteristics O Observation Use of perception words Oriented towards past Low causal connectiveness Low levels of higher order critical thinking M Motive High ratio of causal particles to verbs Frequent use of passive voice (agentless) Oriented towards past Expressing causality Complex verb phrases G Goal Use of perception words Low expression of insight Simple verb phrases High levels of higher order critical thinking Frequent use of causal verbs Non-reflective (other) Low use of perception words Express of personal matters High use of punctuation High use of noun phrases
  • 27. 27 Summary Almost done • We developed a classifier for automated coding of students’ in-video annotations with respect to the expressed • Our exploratory analysis is encouraging. We achieved 75% percent accuracy, Cohen’s k=.51 • The adopted features both capture context-free and also context-specific nature of self-reflection • We provided more detailed operationalization of the different types of reflective statements Future work: • Replicate the study using the new data from other course (use existing model on the new course) • Examine additional features that better capture goal reflective statements
  • 28. 28 The end That is all folks Thank you
  • 29. 29 References (I/II) Butler, D. L., & Winne, P. H. (1995). Feedback and Self-Regulated Learning: A Theoretical Synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245 Cohen, J. (1988). The Analysis of Variance. In Statistical power analysis for the behavioral sciences (pp. 273--406). Hillsdale, N.J.: L. Erlbaum Associates. Gašević, D., Mirriahi, N., Dawson, S., & Joksimović, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220. https://doi.org/10.1016/j.chb.2016.10.026 Gibson, A., & Kitto, K. (2015). Analysing Reflective Text for Learning Analytics: An Approach Using Anomaly Recontextualisation. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (pp. 275–279). New York, NY, USA: ACM. https://doi.org/10.1145/2723576.2723635 Gibson, A., Kitto, K., & Bruza, P. (2016). Towards the Discovery of Learner Metacognition From Reflective Writing. Journal of Learning Analytics, 3(2), 22–36. https://doi.org/10.18608/jla.2016.32.3
  • 30. 30 References (II/II) Hulsman, R. L., Harmsen, A. B., & Fabriek, M. (2009). Reflective teaching of medical communication skills with DiViDU: Assessing the level of student reflection on recorded consultations with simulated patients. Patient Education and Counseling, 74(2), 142–149. https://doi.org/10.1016/j.pec.2008.10.009 McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge University Press. Tausczik, Y. R., & Pennebaker, J. W. (2010). The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology, 29(1), 24–54. https://doi.org/10.1177/0261927X09351676 Ullmann, T. D. (2015). Automated detection of reflection in texts. A machine learning based approach (PhD thesis). The Open University. Ullmann, T. D., Wild, F., & Scott, P. (2013). Comparing automatically detected reflective texts with human judgements. In 2nd Workshop on Awareness and Reflection in Technology- Enhanced Learning (pp. 101–116). Retrieved from http://oro.open.ac.uk/id/eprint/37830