SlideShare a Scribd company logo
Examining the Effect of a Real-Time
Student Dashboard on Student Behavior
and Student Achievement
Robert Bodily, Charles Graham,
Tarah Kerr, and Ben Mackley
Brigham Young University
Why do student dashboards matter?
● Most learning analytics systems are concerned with collecting data, but
then what?
● Provide concept, assignment, unit, or course level feedback to help
students identify knowledge gaps
● Help students develop metacognitive or self-regulation strategies
● Provide information that can be understood at a glance
Review of Research: Dashboard Effect on Behavior
1. 21% of students accepted the system recommendation to view additional content
2. Students participating in courses using the system were more likely to continue taking classes than those who did not
enroll in these courses
3. Students who enabled notifications (on 2 out of 3 systems) showed increased contributions in the social network space
4. Students visited the discussion space more frequently but did not post more frequently
5. The percentage of posts viewed increased for all students, but there were few sustained changes
6. The number of students completing assignments increased and LMS use increased
7. About 50% of students accepted recommendations from the system
8. There was an 83.3% student interaction increase after recommendations were given
9. Students completed assignments more quickly and were able to complete the entire course more quickly
10. *For two of the three visualizations, students post quantity increased; for one of three, student post quantity
decreased.
11. *Students logged in more frequently, completed their coursework more quickly, completed more questions, and
answered more questions correctly on assignments
12. *There were no significant differences between the treatment and control groups in terms of learning efficiency
*Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent group mean
difference testing method)
Review of Research: Dashboard Effect on Achievement
1. No significant achievement differences
2. Increased A’s and B’s. Decreased C’s and D’s.
3. No significant achievement differences
4. Students received more passing grades
5. Frequency and quality of posts was affected positively and negatively
6. Students performed significantly better on the evaluation task
7. Treatment group performed significantly better on final exam
8. *No significant differences between treatment and control
9. *No significant achievement differences
10. *No significant achievement differences, but one course had an effect with Pell eligible students
11. *Treatment group performed significantly better on final exam
*Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent
group mean difference testing method)
Review of Research: Dashboard Effect on Skills
1. Significant increase in student self-awareness accuracy
2. Female students had increased interest when they had a choice to use the
system; male students reported higher interest with mandatory
notifications
Context 1: Fall 2015, Design
● Instructor advocated for video use
● Quizzes were short (3-5 questions), easy, and based on the videos
● Dashboard was accessed in the LMS next to videos
● Access was given after
the first exam
● Design: content
recommender dashboard
Context 1: Fall 2015, Methods
● Students were randomly placed in dashboard treatment group or control
group
● T-tests were used to make sure groups were equivalent across all
covariates (Exam 1 score, quiz scores, video use)
Context 1: Fall 2015, Results
● Randomized control trial showed no significant differences between
treatment and control for student behavior or student achievement
● Low student use
○ 11.5 clicks per session, 2 sessions per student, 42% of students accessed the dashboard
○ Data quality of dashboard?
○ Not useful to students?
● Evaluation showed many students:
○ Did not know they had access to the dashboard
○ Did not know how to use the system
○ Did not think it would be useful
○ Did not have time
○ Had a lot of other resources
Context 2: Winter 2016, Design
● Put videos within a videos tab in the dashboard to increase use (help
them see data visualizations more frequently)
● Longer quizzes but still formative (unlimited attempts)
● Everyone had access
● Design: scatterplot
dashboard
Context 2: Winter 2016, Results
● Low student use
○ Data quality in dashboard?
Fall Winter
Percent Access 42% 48%
Sessions per student 1.98 1.89
Clicks per session 11.5 14.2
Context 3: Spring 2016, Design
● Everyone had access
● Demo the dashboard for everyone at the beginning of the semester
● Quizzes are graded and attempts are limited to 3 (high stakes)
● Teach the TAs about the dashboard and have them encourage it
● Have the instructor mention the benefits of the system
● Provide more resources (videos, practice quizzes, and web resources)
● Design: content recommender 2 dashboard
Context 3: Spring 2016, Design
Context 3: Spring 2016, Results
● Increased frequency of use
● Decreased clicks per session (students are more efficient)
● Perceptions of dashboard
Fall Winter Spring
Percent Access 42% 48% 80%
Sessions per student 1.98 1.89 3.28
Clicks per session 11.5 14.2 10.6
Principles Learned
● Students need to be aware of and trained in a new system
○ Send students reminders
○ Instructor and teaching assistants can discuss benefits of the system
○ Demo the system for the students
● Systems need to be directly related to helping students achieve their
goals
○ Unit-level feedback helped with the test review process
○ Help students identify knowledge gaps
○ Remediate knowledge gaps with videos, text resources, and practice questions
● Usability tests and system evaluations are necessary
○ We changed a lot after our usability and evaluation tests
Future Research
● How can we support students as they engage with online feedback?
● More rigorous evaluations and measured effects research (randomized
control trial, quasi-experimental methods)

More Related Content

What's hot

Institutional roll out of submission and marking ss
Institutional roll out of submission and marking ssInstitutional roll out of submission and marking ss
Institutional roll out of submission and marking ss
LTatUoN
 
Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...
Alison Graham
 
Learning analytics and the learning and teaching journey | Prof Deborah West ...
Learning analytics and the learning and teaching journey | Prof Deborah West ...Learning analytics and the learning and teaching journey | Prof Deborah West ...
Learning analytics and the learning and teaching journey | Prof Deborah West ...
Blackboard APAC
 
NIU Blackboard Portfolio Pilot Information
NIU Blackboard Portfolio Pilot InformationNIU Blackboard Portfolio Pilot Information
NIU Blackboard Portfolio Pilot Information
Jason Rhode
 
Preparing Students
Preparing StudentsPreparing Students
Preparing Students
Katie Turner
 
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Blackboard APAC
 
Supporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual ClassroomSupporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual Classroom
Melanie
 
Effective Teaching with Learn@UW
Effective Teaching with Learn@UWEffective Teaching with Learn@UW
Effective Teaching with Learn@UW
jbohrer
 
Supporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual ClassroomSupporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual Classroom
Melanie
 
iGeneration Conference
iGeneration ConferenceiGeneration Conference
iGeneration Conference
nix1
 
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Blackboard APAC
 
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong LearningFootholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
annielibrarian
 
Session 1 e-Learning
Session 1 e-LearningSession 1 e-Learning
Faculty Mentoring Faculty
Faculty Mentoring FacultyFaculty Mentoring Faculty
Faculty Mentoring Faculty
Staci Trekles
 
NIU Alumni Survey Results 2001-2005
NIU Alumni Survey Results 2001-2005NIU Alumni Survey Results 2001-2005
NIU Alumni Survey Results 2001-2005
Jason Rhode
 
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
E-Learning in Newborn HealthA paradigm shift in continuing professional deve...E-Learning in Newborn HealthA paradigm shift in continuing professional deve...
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
nisaiims
 
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
Blackboard APAC
 
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi NadarajahCoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
Association for Instructional Design Advancement (AIDA)
 
Calamity & Creativity in Chemistry
Calamity & Creativity in ChemistryCalamity & Creativity in Chemistry
Calamity & Creativity in Chemistry
Katherine Haxton
 
Developing Conceptual Understanding Through Alternative Assessment
Developing Conceptual Understanding Through Alternative AssessmentDeveloping Conceptual Understanding Through Alternative Assessment
Developing Conceptual Understanding Through Alternative Assessment
Katherine Haxton
 

What's hot (20)

Institutional roll out of submission and marking ss
Institutional roll out of submission and marking ssInstitutional roll out of submission and marking ss
Institutional roll out of submission and marking ss
 
Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...Improving student engagement with the assessment process in undergraduate mic...
Improving student engagement with the assessment process in undergraduate mic...
 
Learning analytics and the learning and teaching journey | Prof Deborah West ...
Learning analytics and the learning and teaching journey | Prof Deborah West ...Learning analytics and the learning and teaching journey | Prof Deborah West ...
Learning analytics and the learning and teaching journey | Prof Deborah West ...
 
NIU Blackboard Portfolio Pilot Information
NIU Blackboard Portfolio Pilot InformationNIU Blackboard Portfolio Pilot Information
NIU Blackboard Portfolio Pilot Information
 
Preparing Students
Preparing StudentsPreparing Students
Preparing Students
 
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
Efficiency in teaching using these 5 Moodlerooms tools and tips | Grant Beeve...
 
Supporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual ClassroomSupporting Faculty in the Virtual Classroom
Supporting Faculty in the Virtual Classroom
 
Effective Teaching with Learn@UW
Effective Teaching with Learn@UWEffective Teaching with Learn@UW
Effective Teaching with Learn@UW
 
Supporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual ClassroomSupporting Faculty In The Angel Virtual Classroom
Supporting Faculty In The Angel Virtual Classroom
 
iGeneration Conference
iGeneration ConferenceiGeneration Conference
iGeneration Conference
 
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
Using Blackboard Learn alongside Microsoft OneNote: the overlaps, the complem...
 
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong LearningFootholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
Footholds and Foundations: Setting Freshmen on the Path to Lifelong Learning
 
Session 1 e-Learning
Session 1 e-LearningSession 1 e-Learning
Session 1 e-Learning
 
Faculty Mentoring Faculty
Faculty Mentoring FacultyFaculty Mentoring Faculty
Faculty Mentoring Faculty
 
NIU Alumni Survey Results 2001-2005
NIU Alumni Survey Results 2001-2005NIU Alumni Survey Results 2001-2005
NIU Alumni Survey Results 2001-2005
 
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
E-Learning in Newborn HealthA paradigm shift in continuing professional deve...E-Learning in Newborn HealthA paradigm shift in continuing professional deve...
E-Learning in Newborn Health A paradigm shift in continuing professional deve...
 
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
Delivering Student Retention & Success with Predictive Analytics | Nicole Wal...
 
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi NadarajahCoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
CoLT 2017 - Prof. Dr. Vishna Devi Nadarajah
 
Calamity & Creativity in Chemistry
Calamity & Creativity in ChemistryCalamity & Creativity in Chemistry
Calamity & Creativity in Chemistry
 
Developing Conceptual Understanding Through Alternative Assessment
Developing Conceptual Understanding Through Alternative AssessmentDeveloping Conceptual Understanding Through Alternative Assessment
Developing Conceptual Understanding Through Alternative Assessment
 

Viewers also liked

Blended and Online Language Learning Concepts
Blended and Online Language Learning ConceptsBlended and Online Language Learning Concepts
Blended and Online Language Learning Concepts
Jennifer Dobberfuhl Quinlan
 
Do Student Minutes Online Affect Final Grades?
Do Student Minutes Online Affect Final Grades?Do Student Minutes Online Affect Final Grades?
Do Student Minutes Online Affect Final Grades?
Jennifer Dobberfuhl Quinlan
 
Finding What Works
Finding What WorksFinding What Works
Finding What Works
Jennifer Dobberfuhl Quinlan
 
Bonjour! Integrating Peer Collaboration into Asynchronous Language Learning
Bonjour! Integrating Peer Collaboration into Asynchronous Language LearningBonjour! Integrating Peer Collaboration into Asynchronous Language Learning
Bonjour! Integrating Peer Collaboration into Asynchronous Language Learning
Jennifer Dobberfuhl Quinlan
 
Utilization Focused Evaluation
Utilization Focused EvaluationUtilization Focused Evaluation
Utilization Focused Evaluation
Jennifer Dobberfuhl Quinlan
 
Assess ESL Teacher Attitudes To Integrating Technology in Teaching
Assess ESL Teacher Attitudes To Integrating Technology in TeachingAssess ESL Teacher Attitudes To Integrating Technology in Teaching
Assess ESL Teacher Attitudes To Integrating Technology in Teaching
Troy Cox
 
Ritt Deitz The Case for French keynote 1/21/12
Ritt Deitz The Case for French keynote 1/21/12Ritt Deitz The Case for French keynote 1/21/12
Ritt Deitz The Case for French keynote 1/21/12
mdeitz
 
Eurocall2011
Eurocall2011Eurocall2011
Eurocall2011
Sake Jager
 
Korea 101: A course development adventure
Korea 101: A course development adventureKorea 101: A course development adventure
Korea 101: A course development adventure
Jennifer Dobberfuhl Quinlan
 
Openness and the Disaggregated Future of Higher Education
Openness and the Disaggregated Future of Higher EducationOpenness and the Disaggregated Future of Higher Education
Openness and the Disaggregated Future of Higher Education
David Wiley
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research Update
John Whitmer, Ed.D.
 
cipp model
 cipp model cipp model
cipp model
Orly Abellanosa
 
Hype vs. Reality: The AI Explainer
Hype vs. Reality: The AI ExplainerHype vs. Reality: The AI Explainer
Hype vs. Reality: The AI Explainer
Luminary Labs
 

Viewers also liked (13)

Blended and Online Language Learning Concepts
Blended and Online Language Learning ConceptsBlended and Online Language Learning Concepts
Blended and Online Language Learning Concepts
 
Do Student Minutes Online Affect Final Grades?
Do Student Minutes Online Affect Final Grades?Do Student Minutes Online Affect Final Grades?
Do Student Minutes Online Affect Final Grades?
 
Finding What Works
Finding What WorksFinding What Works
Finding What Works
 
Bonjour! Integrating Peer Collaboration into Asynchronous Language Learning
Bonjour! Integrating Peer Collaboration into Asynchronous Language LearningBonjour! Integrating Peer Collaboration into Asynchronous Language Learning
Bonjour! Integrating Peer Collaboration into Asynchronous Language Learning
 
Utilization Focused Evaluation
Utilization Focused EvaluationUtilization Focused Evaluation
Utilization Focused Evaluation
 
Assess ESL Teacher Attitudes To Integrating Technology in Teaching
Assess ESL Teacher Attitudes To Integrating Technology in TeachingAssess ESL Teacher Attitudes To Integrating Technology in Teaching
Assess ESL Teacher Attitudes To Integrating Technology in Teaching
 
Ritt Deitz The Case for French keynote 1/21/12
Ritt Deitz The Case for French keynote 1/21/12Ritt Deitz The Case for French keynote 1/21/12
Ritt Deitz The Case for French keynote 1/21/12
 
Eurocall2011
Eurocall2011Eurocall2011
Eurocall2011
 
Korea 101: A course development adventure
Korea 101: A course development adventureKorea 101: A course development adventure
Korea 101: A course development adventure
 
Openness and the Disaggregated Future of Higher Education
Openness and the Disaggregated Future of Higher EducationOpenness and the Disaggregated Future of Higher Education
Openness and the Disaggregated Future of Higher Education
 
Blackboard Learning Analytics Research Update
Blackboard Learning Analytics Research UpdateBlackboard Learning Analytics Research Update
Blackboard Learning Analytics Research Update
 
cipp model
 cipp model cipp model
cipp model
 
Hype vs. Reality: The AI Explainer
Hype vs. Reality: The AI ExplainerHype vs. Reality: The AI Explainer
Hype vs. Reality: The AI Explainer
 

Similar to Examining the effect of a real time student dashboard on student behavior and student achievement

TESTA to FASTECH Presentation
TESTA to FASTECH PresentationTESTA to FASTECH Presentation
TESTA to FASTECH Presentation
Tansy_Jessop
 
Successful Statistics Course Redesign
Successful Statistics Course RedesignSuccessful Statistics Course Redesign
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
Austin Mpando
 
An evidence based model
An evidence based modelAn evidence based model
An evidence based model
Jisc
 
TESTA to FASTECH (November 2011)
 TESTA to FASTECH (November 2011) TESTA to FASTECH (November 2011)
TESTA to FASTECH (November 2011)
TESTA winch
 
ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015
Ed Foster
 
ABLE - NTU Eunis case study - Nov 2017
ABLE - NTU Eunis case study - Nov 2017ABLE - NTU Eunis case study - Nov 2017
ABLE - NTU Eunis case study - Nov 2017
Ed Foster
 
3 Things to Know About Midterm Evaluation
3 Things to Know About Midterm Evaluation3 Things to Know About Midterm Evaluation
3 Things to Know About Midterm Evaluation
LianaYeghiazaryan1
 
ABLE - Inside Government Nov 2017 - Rebecca Edwards
ABLE - Inside Government Nov 2017 - Rebecca EdwardsABLE - Inside Government Nov 2017 - Rebecca Edwards
ABLE - Inside Government Nov 2017 - Rebecca Edwards
Ed Foster
 
MET_Gathering_Feedback_Practioner_Brief
MET_Gathering_Feedback_Practioner_BriefMET_Gathering_Feedback_Practioner_Brief
MET_Gathering_Feedback_Practioner_Brief
Paul Fleischman
 
Evaluating practice workshop TELFest 2016
Evaluating practice workshop TELFest 2016Evaluating practice workshop TELFest 2016
Evaluating practice workshop TELFest 2016
telshef
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
Ed Foster
 
Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14
nickpaolini81
 
ABLE - EMFD presentation - NTU student dashboard stream
ABLE - EMFD presentation - NTU student dashboard streamABLE - EMFD presentation - NTU student dashboard stream
ABLE - EMFD presentation - NTU student dashboard stream
Ed Foster
 
Flipped classroom brief, NYMU
Flipped classroom brief, NYMUFlipped classroom brief, NYMU
Flipped classroom brief, NYMU
翰泓 李
 
Phase I: Overview
Phase I: OverviewPhase I: Overview
Phase I: Overview
shannonstaley70
 
Standardized Information Literacy Assessment
Standardized Information Literacy AssessmentStandardized Information Literacy Assessment
Standardized Information Literacy Assessment
NicoleBranch
 
Teaching with lecture capture
Teaching with lecture captureTeaching with lecture capture
Teaching with lecture capture
Leonie Sloman
 
Instructional Choice
Instructional Choice Instructional Choice
Instructional Choice
Cathryn Monroe
 
ABLE - Learner Analytics Symposium - Rebecca Edwards
ABLE - Learner Analytics Symposium - Rebecca EdwardsABLE - Learner Analytics Symposium - Rebecca Edwards
ABLE - Learner Analytics Symposium - Rebecca Edwards
Ed Foster
 

Similar to Examining the effect of a real time student dashboard on student behavior and student achievement (20)

TESTA to FASTECH Presentation
TESTA to FASTECH PresentationTESTA to FASTECH Presentation
TESTA to FASTECH Presentation
 
Successful Statistics Course Redesign
Successful Statistics Course RedesignSuccessful Statistics Course Redesign
Successful Statistics Course Redesign
 
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
Using SMS-Based Classroom Response System To Enhance Student’s Active Learning
 
An evidence based model
An evidence based modelAn evidence based model
An evidence based model
 
TESTA to FASTECH (November 2011)
 TESTA to FASTECH (November 2011) TESTA to FASTECH (November 2011)
TESTA to FASTECH (November 2011)
 
ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015ABLE - Inside Government E Foster 26th November 2015
ABLE - Inside Government E Foster 26th November 2015
 
ABLE - NTU Eunis case study - Nov 2017
ABLE - NTU Eunis case study - Nov 2017ABLE - NTU Eunis case study - Nov 2017
ABLE - NTU Eunis case study - Nov 2017
 
3 Things to Know About Midterm Evaluation
3 Things to Know About Midterm Evaluation3 Things to Know About Midterm Evaluation
3 Things to Know About Midterm Evaluation
 
ABLE - Inside Government Nov 2017 - Rebecca Edwards
ABLE - Inside Government Nov 2017 - Rebecca EdwardsABLE - Inside Government Nov 2017 - Rebecca Edwards
ABLE - Inside Government Nov 2017 - Rebecca Edwards
 
MET_Gathering_Feedback_Practioner_Brief
MET_Gathering_Feedback_Practioner_BriefMET_Gathering_Feedback_Practioner_Brief
MET_Gathering_Feedback_Practioner_Brief
 
Evaluating practice workshop TELFest 2016
Evaluating practice workshop TELFest 2016Evaluating practice workshop TELFest 2016
Evaluating practice workshop TELFest 2016
 
ABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of DerbyABLE - the NTU Student Dashboard - University of Derby
ABLE - the NTU Student Dashboard - University of Derby
 
Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14Classroom diagnostic tools training 9.23.14
Classroom diagnostic tools training 9.23.14
 
ABLE - EMFD presentation - NTU student dashboard stream
ABLE - EMFD presentation - NTU student dashboard streamABLE - EMFD presentation - NTU student dashboard stream
ABLE - EMFD presentation - NTU student dashboard stream
 
Flipped classroom brief, NYMU
Flipped classroom brief, NYMUFlipped classroom brief, NYMU
Flipped classroom brief, NYMU
 
Phase I: Overview
Phase I: OverviewPhase I: Overview
Phase I: Overview
 
Standardized Information Literacy Assessment
Standardized Information Literacy AssessmentStandardized Information Literacy Assessment
Standardized Information Literacy Assessment
 
Teaching with lecture capture
Teaching with lecture captureTeaching with lecture capture
Teaching with lecture capture
 
Instructional Choice
Instructional Choice Instructional Choice
Instructional Choice
 
ABLE - Learner Analytics Symposium - Rebecca Edwards
ABLE - Learner Analytics Symposium - Rebecca EdwardsABLE - Learner Analytics Symposium - Rebecca Edwards
ABLE - Learner Analytics Symposium - Rebecca Edwards
 

Recently uploaded

Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
jitskeb
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
74nqk8xf
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
rwarrenll
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
bopyb
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
g4dpvqap0
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Walaa Eldin Moustafa
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
Bill641377
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
kuntobimo2016
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
zsjl4mimo
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
roli9797
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Social Samosa
 

Recently uploaded (20)

Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
一比一原版(GWU,GW文凭证书)乔治·华盛顿大学毕业证如何办理
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
 

Examining the effect of a real time student dashboard on student behavior and student achievement

  • 1. Examining the Effect of a Real-Time Student Dashboard on Student Behavior and Student Achievement Robert Bodily, Charles Graham, Tarah Kerr, and Ben Mackley Brigham Young University
  • 2. Why do student dashboards matter? ● Most learning analytics systems are concerned with collecting data, but then what? ● Provide concept, assignment, unit, or course level feedback to help students identify knowledge gaps ● Help students develop metacognitive or self-regulation strategies ● Provide information that can be understood at a glance
  • 3. Review of Research: Dashboard Effect on Behavior 1. 21% of students accepted the system recommendation to view additional content 2. Students participating in courses using the system were more likely to continue taking classes than those who did not enroll in these courses 3. Students who enabled notifications (on 2 out of 3 systems) showed increased contributions in the social network space 4. Students visited the discussion space more frequently but did not post more frequently 5. The percentage of posts viewed increased for all students, but there were few sustained changes 6. The number of students completing assignments increased and LMS use increased 7. About 50% of students accepted recommendations from the system 8. There was an 83.3% student interaction increase after recommendations were given 9. Students completed assignments more quickly and were able to complete the entire course more quickly 10. *For two of the three visualizations, students post quantity increased; for one of three, student post quantity decreased. 11. *Students logged in more frequently, completed their coursework more quickly, completed more questions, and answered more questions correctly on assignments 12. *There were no significant differences between the treatment and control groups in terms of learning efficiency *Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent group mean difference testing method)
  • 4. Review of Research: Dashboard Effect on Achievement 1. No significant achievement differences 2. Increased A’s and B’s. Decreased C’s and D’s. 3. No significant achievement differences 4. Students received more passing grades 5. Frequency and quality of posts was affected positively and negatively 6. Students performed significantly better on the evaluation task 7. Treatment group performed significantly better on final exam 8. *No significant differences between treatment and control 9. *No significant achievement differences 10. *No significant achievement differences, but one course had an effect with Pell eligible students 11. *Treatment group performed significantly better on final exam *Sample size greater than 150 and conducted an actual experiment (randomized control trial or other equivalent group mean difference testing method)
  • 5. Review of Research: Dashboard Effect on Skills 1. Significant increase in student self-awareness accuracy 2. Female students had increased interest when they had a choice to use the system; male students reported higher interest with mandatory notifications
  • 6. Context 1: Fall 2015, Design ● Instructor advocated for video use ● Quizzes were short (3-5 questions), easy, and based on the videos ● Dashboard was accessed in the LMS next to videos ● Access was given after the first exam ● Design: content recommender dashboard
  • 7. Context 1: Fall 2015, Methods ● Students were randomly placed in dashboard treatment group or control group ● T-tests were used to make sure groups were equivalent across all covariates (Exam 1 score, quiz scores, video use)
  • 8. Context 1: Fall 2015, Results ● Randomized control trial showed no significant differences between treatment and control for student behavior or student achievement ● Low student use ○ 11.5 clicks per session, 2 sessions per student, 42% of students accessed the dashboard ○ Data quality of dashboard? ○ Not useful to students? ● Evaluation showed many students: ○ Did not know they had access to the dashboard ○ Did not know how to use the system ○ Did not think it would be useful ○ Did not have time ○ Had a lot of other resources
  • 9. Context 2: Winter 2016, Design ● Put videos within a videos tab in the dashboard to increase use (help them see data visualizations more frequently) ● Longer quizzes but still formative (unlimited attempts) ● Everyone had access ● Design: scatterplot dashboard
  • 10. Context 2: Winter 2016, Results ● Low student use ○ Data quality in dashboard? Fall Winter Percent Access 42% 48% Sessions per student 1.98 1.89 Clicks per session 11.5 14.2
  • 11. Context 3: Spring 2016, Design ● Everyone had access ● Demo the dashboard for everyone at the beginning of the semester ● Quizzes are graded and attempts are limited to 3 (high stakes) ● Teach the TAs about the dashboard and have them encourage it ● Have the instructor mention the benefits of the system ● Provide more resources (videos, practice quizzes, and web resources) ● Design: content recommender 2 dashboard
  • 12. Context 3: Spring 2016, Design
  • 13. Context 3: Spring 2016, Results ● Increased frequency of use ● Decreased clicks per session (students are more efficient) ● Perceptions of dashboard Fall Winter Spring Percent Access 42% 48% 80% Sessions per student 1.98 1.89 3.28 Clicks per session 11.5 14.2 10.6
  • 14. Principles Learned ● Students need to be aware of and trained in a new system ○ Send students reminders ○ Instructor and teaching assistants can discuss benefits of the system ○ Demo the system for the students ● Systems need to be directly related to helping students achieve their goals ○ Unit-level feedback helped with the test review process ○ Help students identify knowledge gaps ○ Remediate knowledge gaps with videos, text resources, and practice questions ● Usability tests and system evaluations are necessary ○ We changed a lot after our usability and evaluation tests
  • 15. Future Research ● How can we support students as they engage with online feedback? ● More rigorous evaluations and measured effects research (randomized control trial, quasi-experimental methods)