• Save
LAK 2013: Open academic analytics initiative - Initial research findings
Upcoming SlideShare
Loading in...5

Like this? Share it with your network


LAK 2013: Open academic analytics initiative - Initial research findings



Presented at the Third Learning Analytics and Knowledge conference proceedings at Leuven, Belgium. The presentation talks about an open learning ecosystem and Predictive analytics based Early alert ...

Presented at the Third Learning Analytics and Knowledge conference proceedings at Leuven, Belgium. The presentation talks about an open learning ecosystem and Predictive analytics based Early alert system developed at Marist College. It also researches into how the portable the predictive model can be when deployed in a different academic contexts(community colleges) and gives the results about the model performace.more about OAAI at


Conference website: http://lakconference2013.wordpress.com/



Total Views
Views on SlideShare
Embed Views



1 Embed 4

https://twitter.com 4



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Predictive Modeling Markup Language (PMML)
  • Please advise if I should use the latest Journal version for results or just stick to the LAK 2013 paper ??
  • Should we add a future research interest section too?.

LAK 2013: Open academic analytics initiative - Initial research findings Presentation Transcript

  • 1. LAK 2013 – 3rd Conference on Learning Analytics and KnowledgeOpen Academic Analytics Initiative: Initial Research FindingsEitel J.M. Lauría, Erik W. Moody , Sandeep M. Jayaprakash, Nagamani Jonnalagadda, Joshua D. Baron Marist College, Poughkeepsie, NY, USA Leuven, Belgium April 8 -12 2013
  • 2. Project Overview:Open Academic Analytics Initiative OAAIEitel J.M. LauríaSchool of Computer Science & MathematicsMarist College
  • 3. Open Academic Analytics Initiative• Supported by Next Generation Learning Challenges (NGLC) grant• Funded by Bill and Melinda Gates and Hewlett Foundations• 18 month period, $250,000• Began mid 2011, we have completed the project
  • 4. Open Academic Analytics Initiative Objectives• Create “early alert” system • Predict “at risk” students in initial weeks of a course • Deploy intervention to ensure student succeeds• Based on Open ecosystem for academic analytics • Sakai Collaboration and Learning Environment • Pentaho Business Intelligence Suite • OAAI Predictive Model released under OS license (PMML) • Collaboration with other vendors (SPSS Modeler)
  • 5. Research Questions• How good are predictive models ?• What are good model predictors ?  Student Attitude Data (SATs, current GPA, etc.)  Student Demographic Data (Age, gender, etc.)  Sakai Event Log Data  Sakai Gradebook• How “portable” are predictive models?• What intervention strategies are most effective?
  • 6. Predictive Modeling and Portability AnalysisSandeep M. JayaprakashAcademic Technology & eLearningMarist College
  • 7. OAAI Early Alert System Overview SIS or Banner Student Attitude Data(Static data) (SATs, current GPA, etc.) Student Demographic Data (Age, gender, etc.) Identified Predictive Students at Model “academic risk”(Dynamic Data) Scoring SAKAI CLE Sakai Event Log Data Sakai Gradebook Data Intervention (Awareness & Online Academic Support Environment)
  • 8. Predictive Modeling using Marist DataPentaho Kettle Data Integration• Training Dataset – Marist Fall 2010 & Spring 2011 (7344 records) Testing Dataset – Marist Fall 2011 (5101 records )• Extractions were joined, cleaned, recoded, and powerful predictors were derived to produce an input data file for each student- course combination. Feature Type Feature Name GENDER, SAT_VERBAL, SAT_M ATH, APTITUDE_SCORE, FTPT, CLASS, CUM _GPA, Predictors ENROLLM ENT, ACADEM IC _STANDING, RM N_SCORE, R_SESSIONS, R_CONTENT_READ ACADEM IC_RISK (1 = at risk; 0 Target student in good standing)
  • 9. Predictive Modeling using Marist DataPentaho WEKA 3.7 and IBM SPSS Modeler 14.2• Generate 10 different training datasets by varying random seeds• Balance each training dataset using sampling techniques.• Train a predictive model(Logistic Regression, SVM/SMO, J48 decision Trees) for each balanced training dataset  10 datasets x 3 algorithms = 30 models• Score the testing dataset(Marist Fall 2011) for each student- course combination• Measure predictive performance of classifiers  Accuracy, Recall, Specificity and Precision.• Produce summary measures (mean and standard error)
  • 10. Predictive Modeling using Marist Data
  • 11. Predictive Performance on Marist Data
  • 12. Running Pilots at Partner Institutions Student Aptitude and AAR transferred from Marist Demographic Data into a Project Site for faculty at Extract (SIS) each institutions Sakai system Pentaho AAR [data processing, Project Site scoring and reporting] Sakai Event Academic Log Data Extract Alert Report The Sakai (AAR) Dropbox tool is used to provide each Gradebook faculty with a Dropbox Tool Data Extract Open Academic Analytic Initiative private folder Workflow for Academic Alert Reports (AAR) and deployment of Online Academic intervention strategies Faculty Folder Support Environment (OASE) A sub-folder for each course/ section used to organize the Academic Student AAR and course SIK Alert Report Identification (AAR) Key (SIK) Faculty notified when Messages Tool new AA is posted Identified and access their Dropbox Student to review AAR Faculty message identified students through the Specific Sakai Awareness class Course Site Course Site Intervention
  • 13. Academic Alert Reports (AARs)
  • 14. Predictive Performance on Spring Pilots
  • 15. Portability Analysis• The models developed at one academic context are scalable to other academic contexts.• The evaluation accuracies start at 65 % at the first wave and the accuracies improves to 75% - 80% with more availability of data in the subsequent waves.• Pilot Evaluation results show that recall and specificity completion values are just around 10% lower when compared to Marist results.• Gradebook (CMS data) and CUM_GPA have been very important predictors.• Evidence of good portability in institutions collecting such data.
  • 16. Intervention AnalysisErik MoodySchool of Social & Behavioral SciencesMarist College
  • 17. Intervention Strategies at Partner InstitutionsOnce “at risk” students had been identified this information could beused to alert them they are at risk of failing the course.Last spring three institutions (Cerritos College, College of theRedwoods and Savannah State University) participated in a pilot studydesigned to explore the effectiveness of the predictive model and twodifferent interventions.A total of 1,379 students were assigned to one of three groups: OASE Control Awareness (Online Academic Support Environment) Alerted of Risk of Failure No Intervention Alerted of Risk of Failure Access to Academic Support Services
  • 18. Intervention Strategies at Partner InstitutionsAt three different points during the semester Academic Alerts wereautomatically sent to the instructors.Instructors forwarded the Academic Alerts to the students they felt werestruggling in their course.Student in the Awareness group were sent emails with messages like:“Based on your performance on recent graded assignments and exams, as well asother factors that tend to predict academic success, I am becoming worried aboutyour ability to successfully complete this class.I am reaching out to offer some assistance and to encourage you to consider takingsteps to improve your performance. Doing so early in the semester will increase thelikelihood of you successfully completing the class and avoid negatively impacting onyour academic standing.”
  • 19. Intervention Strategies at Partner InstitutionsAdditionally Instructors were encouraged to recommend, the following:• Ask the student visit you during office hours.• Set up an appointment with a tutor, academic support person or consider participating in a study group.• Access web-based resources such as online tutoring tools.• Take practices exams, complete additional & homework questions.Students in the OASE group received the same messages plus links toAcademic Support Services like The Kahn Academy, Flat World Knowledgetextbooks, etc… as well access to mentoring from peers and professionalsupport staff.At the end of the semester we collected data on a number of measures includingcourse grade, content mastery and course withdrawal.
  • 20. Intervention Analysis (Spring 2012) Mean Final Grade for "at Risk" Students 100 Final Grade (%) 90 80 70 60 50 Awareness OASE ControlOne-way ANOVA analysis revealed statistical significancedifferences between the control group and the twotreatment groups. F (2,448) = 8.484, p = .000*
  • 21. Intervention Analysis (Spring 2012) Content Mastery for "at Risk" Students 500 Frequency 400 300 200 100 0 Yes No Yes No Control InterventionX2 analysis reveled a significant difference in contentmastery (C or better) between the control group and thecollapsed treatment groups (X2(1) = 8.913, p = .003*).
  • 22. Intervention Analysis (Spring 2012) Withdrawal rates for "at Risk" Students 500 400 Frequency 300 200 100 0 Yes No Yes No Control InterventionX2 analysis reveled significantly different withdrawal ratesbetween the control group and the collapsed treatmentgroups (X2 (1)=7.097, p = .008*).
  • 23. Intervention Analysis (Spring & Fall 2012) Mean Final Grade for "at Risk" Students Final Grade (%) 100 90 80 70 60 50 Awareness OASE ControlOne-way ANOVA analysis revealed statistical significancedifferences between the control group and the twotreatment groups. F (2, 714) = 7.076, p = .001*
  • 24. ConclusionsBoth Treatment groups performed significantly better onmeasures of final grade and content mastery than controls.Both Treatment groups had higher rates of course withdrawalthan controls.The first of three Academic Alerts were the most effective.Why do Academic Alerts Help?• Early feedback is important• Despite poor grade students may not believe they are at risk• In large classes students don’t receive the attention they do in smaller classes
  • 25. Questions
  • 26. ReferenceOAAI Sakai confluence Wiki pagehttps://confluence.sakaiproject.org/pages/viewpage.action?pageId=75671025 ContactJosh Baron - Senior Academic Technology Officer josh.baron@Marist.eduEitel Lauría - School of Computer Science & Mathematics eitel.lauria@Marist.eduErik Moody - School of Social & Behavioral Sciences erik.moody@Marist.eduSandeep Jayaprakash - Learning Analytics Specialist sandeep.jayaprakash1@Marist.edu