SlideShare a Scribd company logo
1 of 18
Download to read offline
Validating a Theorized Model of Engagement in
Learning Analytics
Ed Fincham 1 Alex Whitelock-Wainwright 2 Vitomir Kovanovi´c 3
Sre´cko Joksimovi´c 3 Jan-Paul van Staalduinen 4 Dragan Gaˇsevi´c 2
1
University of Edinburgh
2
Monash University
3
University of South Australia
4
Delft University of Technology
Fincham et al. Counting Clicks is Not Enough 1 / 18
Overview – Engagement within MOOCs
- Big deal. Who cares?
- What do we mean by “engagement”?
- Can we even measure it?
Fincham et al. Counting Clicks is Not Enough 2 / 18
Why Engagement Matters
- Low completion rates
- Causal factors promoting student learning (Reich, 2015)
- Frequently used but rarely defined
- Informing course design and targeted interventions
Fincham et al. Counting Clicks is Not Enough 3 / 18
Defining Engagement – “I know it when I see it”
Authors Number of Types Categories
Finn (1989) 2
Participation
Identification
Fredricks et al. (2004) 3
Participation
Affective
Cognitive
Reschly et al. (2012) 4
Academic
Behavioural
Affective
Cognitive
Fincham et al. Counting Clicks is Not Enough 4 / 18
Defining Engagement – From Classrooms...
The original model of the association between context, engagement, and learning outcomes,
developed by Reschly et al. (2012).
Fincham et al. Counting Clicks is Not Enough 5 / 18
Defining Engagement – ... to MOOCs
The adapted model of the association between context, engagement, and learning outcomes,
originally developed by Reschly et al. (2012) with indicators for digital learning environments
proposed by Joksimovi´c et al. (2018).
Fincham et al. Counting Clicks is Not Enough 6 / 18
Measuring Engagement – The Past
Uni-Dimensional Multi-Dimensional
Forum Participation (Wang et al., 2015) Behavioural, linguistic, and social cues
Video Lectures (Li et al., 2015) (Ramesh et al., 2014)
Assessments (Ye et al., 2015)
Fincham et al. Counting Clicks is Not Enough 7 / 18
Measuring Engagement – The Present
RQ1. Can we identify a latent variable model structure consistent with our
engagement framework?
RQ2. To what extent are these latent variables associated with learning
outcomes?
RQ3. Does our latent variable model structure generalize across different
subjects and pedagogies?
Fincham et al. Counting Clicks is Not Enough 8 / 18
Measuring Engagement – The Data
Highest degree
Course Weeks Certificate
threshold
Median
age
High
School
BA MA
PhD
Solving Complex Problems 5 60% 33 18.7% 38.9% 40.3%
Delft Design Approach 10 60% 31 19.0% 42.0% 36.8%
Technology for Biobased Products 7 55% 30 21.2% 36.8% 39.5%
Fincham et al. Counting Clicks is Not Enough 9 / 18
Measuring Engagement – The Metrics
Trace Logs
1 Days Active Number of days student logs in
2 Weeks Active Number of weeks student logs in
3 Forum Posts Number of forum posts by student
4 Videos Watched Number of unique videos watched
5 Problem Submissions Number of unique problem submissions
6 Time on Task Median time on problem submission task
7 Average Problem Score Average problem submission score (%)
IBM Tone Analyzer
8 Sentiment Student’s median post sentiment
9 Sadness Student’s median post sadness
10 Joy Student’s median post joy
11 Anger Student’s median post anger
12 Disgust Student’s median post disgust
13 Fear Student’s median post fear
Coh-Metrix
14 Narrativity Student’s median post narrativity
15 Syntactic Simplicity Student’s median post syntactic simplicity
16 Word Concreteness Student’s median post word concreteness
17 Referential Cohesion Student’s forum post referential cohesion
18 Deep Cohesion Student’s median post deep cohesion
Fincham et al. Counting Clicks is Not Enough 10 / 18
Measuring Engagement – The Model
EFA Standardized Loading CFA Standardized Loading
Metric Factor 1 Factor 2 Factor 3 Coef. R2 p
Sentiment 0.93 0.99 0.98
Joy 0.66 0.57 0.32 0.00
Sadness -0.64 -0.52 0.27 0.00
Problem Submissions 0.93 0.92 0.84
Videos Watched 0.78 0.60 0.37 0.00
Weeks Active 0.72 0.79 0.63 0.00
Syntactic Simplicity -0.86 0.29 0.42
Referential Cohesion 0.46 -0.93 0.87 0.39
Word Concreteness 0.42 -0.51 0.26 0.09
Fincham et al. Counting Clicks is Not Enough 11 / 18
Measuring Engagement – Learning Outcomes
SEM Path Coefficients
Path Std. Coef. S.E. p
Factor 1 → Grade 0.02 0.01 0.60
Factor 2 → Grade 0.89 0.08 0.00
Factor 3 → Grade -0.02 0.04 0.84
Factor 1 → Factor 2 0.01 0.04 0.90
Factor 1 → Factor 3 0.08 0.06 0.38
Factor 2 → Factor 1 0.01 0.19 0.90
Factor 2 → Factor 3 0.29 0.21 0.05
Factor 3 → Factor 1 0.09 0.16 0.41
Factor 3 → Factor 2 0.29 0.09 0.03
Fincham et al. Counting Clicks is Not Enough 12 / 18
Measuring Engagement – Model Generality
MIMIC Modeling Bootstrapped Results
Factor Covariate Partial Std. Coef. p 95% CI
1 Delft Design Approach 0.06 0.43 [-0.14 – 0.24]
1 Technology for Biobased Products -0.01 0.48 [-0.22 – 0.17]
2 Delft Design Approach -0.14 0.31 [-0.35 – 0.09]
2 Technology for Biobased Products -0.00 0.47 [-0.18 – 0.20]
3 Delft Design Approach 0.06 0.47 [-0.21 – 0.31]
3 Technology for Biobased Products -0.09 0.44 [-0.34 – 0.17]
Fincham et al. Counting Clicks is Not Enough 13 / 18
Measuring Engagement – Discussion
RQ1. 3 v 4 latent variables?
RQ2. A mechanism through which learning occurs?
RQ3. The spectre of validity looms large.
Fincham et al. Counting Clicks is Not Enough 14 / 18
Measuring Engagement – Summary
- Engagement is relevant for learning analytics.
- It is essential to ground computational analyses within educational
theory.
- We should work to validate theoretical constructs.
Fincham et al. Counting Clicks is Not Enough 15 / 18
Measuring Engagement – Limitations
- Granularity: where does engagement lie?
- Validity: structural and general.
- Controlling for context (is hard).
Fincham et al. Counting Clicks is Not Enough 16 / 18
Measuring Engagement – Next Steps
- Contextual variables may influence student engagement.
- Engagement occurs in real time.
- Above all, we need more validation.
Fincham et al. Counting Clicks is Not Enough 17 / 18
Validating a Theorized Model of Engagement in
Learning Analytics
Ed Fincham 1 Alex Whitelock-Wainwright 2 Vitomir Kovanovi´c 3
Sre´cko Joksimovi´c 3 Jan-Paul van Staalduinen 4 Dragan Gaˇsevi´c 2
1
University of Edinburgh
2
Monash University
3
University of South Australia
4
Delft University of Technology
Fincham et al. Counting Clicks is Not Enough 18 / 18

More Related Content

Similar to Validating a theorized model of engagement in learning analytics

Survey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationSurvey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationJaymineShah
 
Leadership in the Digital Age
Leadership in the Digital AgeLeadership in the Digital Age
Leadership in the Digital AgeJane McConnell
 
Itciloaudienceresponsesystems
ItciloaudienceresponsesystemsItciloaudienceresponsesystems
ItciloaudienceresponsesystemsITCILO
 
Survey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationSurvey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationJaymineShah
 
RStrachanOct23.pptx
RStrachanOct23.pptxRStrachanOct23.pptx
RStrachanOct23.pptxJisc
 
Presentació Investigació TIC i EF
Presentació Investigació TIC i EFPresentació Investigació TIC i EF
Presentació Investigació TIC i EFrvilah
 
A journey towards programmatic assessment
A journey towards programmatic assessmentA journey towards programmatic assessment
A journey towards programmatic assessmentMedCouncilCan
 
When Student Confidence Clicks - Using Student Response Systems
When Student Confidence Clicks - Using Student Response SystemsWhen Student Confidence Clicks - Using Student Response Systems
When Student Confidence Clicks - Using Student Response SystemsFabio R. Arico'
 
Mapping the landscape of learning technology
Mapping the landscape of learning technologyMapping the landscape of learning technology
Mapping the landscape of learning technologyBrightwave Group
 
Newcastle Anatomy and TEL Symposium 2018
Newcastle Anatomy and TEL Symposium 2018Newcastle Anatomy and TEL Symposium 2018
Newcastle Anatomy and TEL Symposium 2018JamesPickering11
 
Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Touchstone Institute
 
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docx
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docxRunning Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docx
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docxtoltonkendal
 
TESTA Interactive Masterclass
TESTA Interactive MasterclassTESTA Interactive Masterclass
TESTA Interactive MasterclassTansy Jessop
 
New clinician learning data what does it mean for your medical education pr...
New clinician learning data   what does it mean for your medical education pr...New clinician learning data   what does it mean for your medical education pr...
New clinician learning data what does it mean for your medical education pr...Brian S. McGowan, PhD, FACEhp
 
Supporting Lit and Studies.docx
Supporting Lit and Studies.docxSupporting Lit and Studies.docx
Supporting Lit and Studies.docxisabeltacliad1
 

Similar to Validating a theorized model of engagement in learning analytics (20)

Survey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationSurvey report how technology can help in personalization of education
Survey report how technology can help in personalization of education
 
RtI to MTSS
RtI to MTSSRtI to MTSS
RtI to MTSS
 
Leadership in the Digital Age
Leadership in the Digital AgeLeadership in the Digital Age
Leadership in the Digital Age
 
Itciloaudienceresponsesystems
ItciloaudienceresponsesystemsItciloaudienceresponsesystems
Itciloaudienceresponsesystems
 
Survey report how technology can help in personalization of education
Survey report how technology can help in personalization of educationSurvey report how technology can help in personalization of education
Survey report how technology can help in personalization of education
 
RStrachanOct23.pptx
RStrachanOct23.pptxRStrachanOct23.pptx
RStrachanOct23.pptx
 
Presentació Investigació TIC i EF
Presentació Investigació TIC i EFPresentació Investigació TIC i EF
Presentació Investigació TIC i EF
 
Impact of ict on learning
Impact of ict on learningImpact of ict on learning
Impact of ict on learning
 
Stakeholder engagement guidance note
Stakeholder engagement guidance noteStakeholder engagement guidance note
Stakeholder engagement guidance note
 
A journey towards programmatic assessment
A journey towards programmatic assessmentA journey towards programmatic assessment
A journey towards programmatic assessment
 
When Student Confidence Clicks - Using Student Response Systems
When Student Confidence Clicks - Using Student Response SystemsWhen Student Confidence Clicks - Using Student Response Systems
When Student Confidence Clicks - Using Student Response Systems
 
Mapping the landscape of learning technology
Mapping the landscape of learning technologyMapping the landscape of learning technology
Mapping the landscape of learning technology
 
Newcastle Anatomy and TEL Symposium 2018
Newcastle Anatomy and TEL Symposium 2018Newcastle Anatomy and TEL Symposium 2018
Newcastle Anatomy and TEL Symposium 2018
 
Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment Theoretical Constructs and Best Practice in Assessment
Theoretical Constructs and Best Practice in Assessment
 
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docx
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docxRunning Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docx
Running Head REPLY TO OPINION 5.1 FOR KIMBRILEE SCHMITZ 1REPLY.docx
 
TESTA Interactive Masterclass
TESTA Interactive MasterclassTESTA Interactive Masterclass
TESTA Interactive Masterclass
 
Slhi june 23 08 imp part 1
Slhi june 23 08 imp part 1Slhi june 23 08 imp part 1
Slhi june 23 08 imp part 1
 
Slhi june 23 08 imp part 1
Slhi june 23 08 imp part 1Slhi june 23 08 imp part 1
Slhi june 23 08 imp part 1
 
New clinician learning data what does it mean for your medical education pr...
New clinician learning data   what does it mean for your medical education pr...New clinician learning data   what does it mean for your medical education pr...
New clinician learning data what does it mean for your medical education pr...
 
Supporting Lit and Studies.docx
Supporting Lit and Studies.docxSupporting Lit and Studies.docx
Supporting Lit and Studies.docx
 

More from Vitomir Kovanovic

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersVitomir Kovanovic
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Vitomir Kovanovic
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Vitomir Kovanovic
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Vitomir Kovanovic
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Vitomir Kovanovic
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Vitomir Kovanovic
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersVitomir Kovanovic
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersVitomir Kovanovic
 
A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...Vitomir Kovanovic
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning AnalyticsVitomir Kovanovic
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisVitomir Kovanovic
 
Understand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsUnderstand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsVitomir Kovanovic
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsVitomir Kovanovic
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...Vitomir Kovanovic
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryVitomir Kovanovic
 
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...Vitomir Kovanovic
 
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Vitomir Kovanovic
 
What does effective online/blended teaching look like?
What does effective online/blended teaching look like?What does effective online/blended teaching look like?
What does effective online/blended teaching look like?Vitomir Kovanovic
 
MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveVitomir Kovanovic
 
Automated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion TranscriptsAutomated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion TranscriptsVitomir Kovanovic
 

More from Vitomir Kovanovic (20)

Introduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and ManagersIntroduction to Learning Analytics for High School Teachers and Managers
Introduction to Learning Analytics for High School Teachers and Managers
 
Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...Extending video interactions to support self-regulated learning in an online ...
Extending video interactions to support self-regulated learning in an online ...
 
Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...Analysing social presence in online discussions through network and text anal...
Analysing social presence in online discussions through network and text anal...
 
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
Automated Analysis of Cognitive Presence in Online Discussions Written in Por...
 
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
Examining the Value of Learning Analytics for Supporting Work-integrated Lear...
 
Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...Developing Self-regulated Learning in High-school Students: The Role of Learn...
Developing Self-regulated Learning in High-school Students: The Role of Learn...
 
Unsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics ResearchersUnsupervised Learning for Learning Analytics Researchers
Unsupervised Learning for Learning Analytics Researchers
 
Introduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics ResearchersIntroduction to R for Learning Analytics Researchers
Introduction to R for Learning Analytics Researchers
 
A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...A novel model of cognitive presence assessment using automated learning analy...
A novel model of cognitive presence assessment using automated learning analy...
 
Introduction to Learning Analytics
Introduction to Learning AnalyticsIntroduction to Learning Analytics
Introduction to Learning Analytics
 
Introduction to Epistemic Network Analysis
Introduction to Epistemic Network AnalysisIntroduction to Epistemic Network Analysis
Introduction to Epistemic Network Analysis
 
Understand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analyticsUnderstand students’ self-reflections through learning analytics
Understand students’ self-reflections through learning analytics
 
Assessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methodsAssessing cognitive presence using automated learning analytics methods
Assessing cognitive presence using automated learning analytics methods
 
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
Kovanović et al.   2017 - developing a mooc experimentation platform: insight...Kovanović et al.   2017 - developing a mooc experimentation platform: insight...
Kovanović et al. 2017 - developing a mooc experimentation platform: insight...
 
Learning Analytics for Communities of Inquiry
Learning Analytics for Communities of InquiryLearning Analytics for Communities of Inquiry
Learning Analytics for Communities of Inquiry
 
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
A Novel Model of Cognitive Presence Assessment Using Automated Learning Analy...
 
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
Towards Automated Classification of Discussion Transcripts: A Cognitive Prese...
 
What does effective online/blended teaching look like?
What does effective online/blended teaching look like?What does effective online/blended teaching look like?
What does effective online/blended teaching look like?
 
MOOCs in the news- A European perspective
MOOCs in the news- A European perspectiveMOOCs in the news- A European perspective
MOOCs in the news- A European perspective
 
Automated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion TranscriptsAutomated Content Analysis of Discussion Transcripts
Automated Content Analysis of Discussion Transcripts
 

Recently uploaded

Keeping Your Information Safe with Centralized Security Services
Keeping Your Information Safe with Centralized Security ServicesKeeping Your Information Safe with Centralized Security Services
Keeping Your Information Safe with Centralized Security ServicesTechSoup
 
factors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxfactors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxSanjay Shekar
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文中 央社
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticspragatimahajan3
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/siemaillard
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Mohamed Rizk Khodair
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptxmansk2
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxJenilouCasareno
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxCapitolTechU
 
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringBasic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringDenish Jangid
 
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...Nguyen Thanh Tu Collection
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...Sayali Powar
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxheathfieldcps1
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdfVikramadityaRaj
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfmstarkes24
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfQucHHunhnh
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesashishpaul799
 

Recently uploaded (20)

Keeping Your Information Safe with Centralized Security Services
Keeping Your Information Safe with Centralized Security ServicesKeeping Your Information Safe with Centralized Security Services
Keeping Your Information Safe with Centralized Security Services
 
factors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptxfactors influencing drug absorption-final-2.pptx
factors influencing drug absorption-final-2.pptx
 
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文會考英文
 
size separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceuticssize separation d pharm 1st year pharmaceutics
size separation d pharm 1st year pharmaceutics
 
Championnat de France de Tennis de table/
Championnat de France de Tennis de table/Championnat de France de Tennis de table/
Championnat de France de Tennis de table/
 
Word Stress rules esl .pptx
Word Stress rules esl               .pptxWord Stress rules esl               .pptx
Word Stress rules esl .pptx
 
Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).Dementia (Alzheimer & vasular dementia).
Dementia (Alzheimer & vasular dementia).
 
2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx2024_Student Session 2_ Set Plan Preparation.pptx
2024_Student Session 2_ Set Plan Preparation.pptx
 
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptxMatatag-Curriculum and the 21st Century Skills Presentation.pptx
Matatag-Curriculum and the 21st Century Skills Presentation.pptx
 
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptxslides CapTechTalks Webinar May 2024 Alexander Perry.pptx
slides CapTechTalks Webinar May 2024 Alexander Perry.pptx
 
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & EngineeringBasic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
Basic Civil Engg Notes_Chapter-6_Environment Pollution & Engineering
 
“O BEIJO” EM ARTE .
“O BEIJO” EM ARTE                       .“O BEIJO” EM ARTE                       .
“O BEIJO” EM ARTE .
 
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
50 ĐỀ LUYỆN THI IOE LỚP 9 - NĂM HỌC 2022-2023 (CÓ LINK HÌNH, FILE AUDIO VÀ ĐÁ...
 
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
UNIT – IV_PCI Complaints: Complaints and evaluation of complaints, Handling o...
 
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
Operations Management - Book1.p  - Dr. Abdulfatah A. SalemOperations Management - Book1.p  - Dr. Abdulfatah A. Salem
Operations Management - Book1.p - Dr. Abdulfatah A. Salem
 
The basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptxThe basics of sentences session 4pptx.pptx
The basics of sentences session 4pptx.pptx
 
....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf....................Muslim-Law notes.pdf
....................Muslim-Law notes.pdf
 
MichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdfMichaelStarkes_UncutGemsProjectSummary.pdf
MichaelStarkes_UncutGemsProjectSummary.pdf
 
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdfDanh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
Danh sách HSG Bộ môn cấp trường - Cấp THPT.pdf
 
ppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyesppt your views.ppt your views of your college in your eyes
ppt your views.ppt your views of your college in your eyes
 

Validating a theorized model of engagement in learning analytics

  • 1. Validating a Theorized Model of Engagement in Learning Analytics Ed Fincham 1 Alex Whitelock-Wainwright 2 Vitomir Kovanovi´c 3 Sre´cko Joksimovi´c 3 Jan-Paul van Staalduinen 4 Dragan Gaˇsevi´c 2 1 University of Edinburgh 2 Monash University 3 University of South Australia 4 Delft University of Technology Fincham et al. Counting Clicks is Not Enough 1 / 18
  • 2. Overview – Engagement within MOOCs - Big deal. Who cares? - What do we mean by “engagement”? - Can we even measure it? Fincham et al. Counting Clicks is Not Enough 2 / 18
  • 3. Why Engagement Matters - Low completion rates - Causal factors promoting student learning (Reich, 2015) - Frequently used but rarely defined - Informing course design and targeted interventions Fincham et al. Counting Clicks is Not Enough 3 / 18
  • 4. Defining Engagement – “I know it when I see it” Authors Number of Types Categories Finn (1989) 2 Participation Identification Fredricks et al. (2004) 3 Participation Affective Cognitive Reschly et al. (2012) 4 Academic Behavioural Affective Cognitive Fincham et al. Counting Clicks is Not Enough 4 / 18
  • 5. Defining Engagement – From Classrooms... The original model of the association between context, engagement, and learning outcomes, developed by Reschly et al. (2012). Fincham et al. Counting Clicks is Not Enough 5 / 18
  • 6. Defining Engagement – ... to MOOCs The adapted model of the association between context, engagement, and learning outcomes, originally developed by Reschly et al. (2012) with indicators for digital learning environments proposed by Joksimovi´c et al. (2018). Fincham et al. Counting Clicks is Not Enough 6 / 18
  • 7. Measuring Engagement – The Past Uni-Dimensional Multi-Dimensional Forum Participation (Wang et al., 2015) Behavioural, linguistic, and social cues Video Lectures (Li et al., 2015) (Ramesh et al., 2014) Assessments (Ye et al., 2015) Fincham et al. Counting Clicks is Not Enough 7 / 18
  • 8. Measuring Engagement – The Present RQ1. Can we identify a latent variable model structure consistent with our engagement framework? RQ2. To what extent are these latent variables associated with learning outcomes? RQ3. Does our latent variable model structure generalize across different subjects and pedagogies? Fincham et al. Counting Clicks is Not Enough 8 / 18
  • 9. Measuring Engagement – The Data Highest degree Course Weeks Certificate threshold Median age High School BA MA PhD Solving Complex Problems 5 60% 33 18.7% 38.9% 40.3% Delft Design Approach 10 60% 31 19.0% 42.0% 36.8% Technology for Biobased Products 7 55% 30 21.2% 36.8% 39.5% Fincham et al. Counting Clicks is Not Enough 9 / 18
  • 10. Measuring Engagement – The Metrics Trace Logs 1 Days Active Number of days student logs in 2 Weeks Active Number of weeks student logs in 3 Forum Posts Number of forum posts by student 4 Videos Watched Number of unique videos watched 5 Problem Submissions Number of unique problem submissions 6 Time on Task Median time on problem submission task 7 Average Problem Score Average problem submission score (%) IBM Tone Analyzer 8 Sentiment Student’s median post sentiment 9 Sadness Student’s median post sadness 10 Joy Student’s median post joy 11 Anger Student’s median post anger 12 Disgust Student’s median post disgust 13 Fear Student’s median post fear Coh-Metrix 14 Narrativity Student’s median post narrativity 15 Syntactic Simplicity Student’s median post syntactic simplicity 16 Word Concreteness Student’s median post word concreteness 17 Referential Cohesion Student’s forum post referential cohesion 18 Deep Cohesion Student’s median post deep cohesion Fincham et al. Counting Clicks is Not Enough 10 / 18
  • 11. Measuring Engagement – The Model EFA Standardized Loading CFA Standardized Loading Metric Factor 1 Factor 2 Factor 3 Coef. R2 p Sentiment 0.93 0.99 0.98 Joy 0.66 0.57 0.32 0.00 Sadness -0.64 -0.52 0.27 0.00 Problem Submissions 0.93 0.92 0.84 Videos Watched 0.78 0.60 0.37 0.00 Weeks Active 0.72 0.79 0.63 0.00 Syntactic Simplicity -0.86 0.29 0.42 Referential Cohesion 0.46 -0.93 0.87 0.39 Word Concreteness 0.42 -0.51 0.26 0.09 Fincham et al. Counting Clicks is Not Enough 11 / 18
  • 12. Measuring Engagement – Learning Outcomes SEM Path Coefficients Path Std. Coef. S.E. p Factor 1 → Grade 0.02 0.01 0.60 Factor 2 → Grade 0.89 0.08 0.00 Factor 3 → Grade -0.02 0.04 0.84 Factor 1 → Factor 2 0.01 0.04 0.90 Factor 1 → Factor 3 0.08 0.06 0.38 Factor 2 → Factor 1 0.01 0.19 0.90 Factor 2 → Factor 3 0.29 0.21 0.05 Factor 3 → Factor 1 0.09 0.16 0.41 Factor 3 → Factor 2 0.29 0.09 0.03 Fincham et al. Counting Clicks is Not Enough 12 / 18
  • 13. Measuring Engagement – Model Generality MIMIC Modeling Bootstrapped Results Factor Covariate Partial Std. Coef. p 95% CI 1 Delft Design Approach 0.06 0.43 [-0.14 – 0.24] 1 Technology for Biobased Products -0.01 0.48 [-0.22 – 0.17] 2 Delft Design Approach -0.14 0.31 [-0.35 – 0.09] 2 Technology for Biobased Products -0.00 0.47 [-0.18 – 0.20] 3 Delft Design Approach 0.06 0.47 [-0.21 – 0.31] 3 Technology for Biobased Products -0.09 0.44 [-0.34 – 0.17] Fincham et al. Counting Clicks is Not Enough 13 / 18
  • 14. Measuring Engagement – Discussion RQ1. 3 v 4 latent variables? RQ2. A mechanism through which learning occurs? RQ3. The spectre of validity looms large. Fincham et al. Counting Clicks is Not Enough 14 / 18
  • 15. Measuring Engagement – Summary - Engagement is relevant for learning analytics. - It is essential to ground computational analyses within educational theory. - We should work to validate theoretical constructs. Fincham et al. Counting Clicks is Not Enough 15 / 18
  • 16. Measuring Engagement – Limitations - Granularity: where does engagement lie? - Validity: structural and general. - Controlling for context (is hard). Fincham et al. Counting Clicks is Not Enough 16 / 18
  • 17. Measuring Engagement – Next Steps - Contextual variables may influence student engagement. - Engagement occurs in real time. - Above all, we need more validation. Fincham et al. Counting Clicks is Not Enough 17 / 18
  • 18. Validating a Theorized Model of Engagement in Learning Analytics Ed Fincham 1 Alex Whitelock-Wainwright 2 Vitomir Kovanovi´c 3 Sre´cko Joksimovi´c 3 Jan-Paul van Staalduinen 4 Dragan Gaˇsevi´c 2 1 University of Edinburgh 2 Monash University 3 University of South Australia 4 Delft University of Technology Fincham et al. Counting Clicks is Not Enough 18 / 18