Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Predictive Analytics: Turning Insights Into Action to Improve Student Success


Published on

UMUC is committed to providing high quality education that is accessible to all as an open access institution while lowering costs. Using predictive analytics and effective learner interventions are critical to our strategy. UMUC developed a Student Success Initiative to evaluate the potential for predictive analytics to improve student outcomes and selected Civitas Learning as a key strategic partner. Since Spring 2013, the UMUC has engaged in three pilots, utilizing Civitas Learning's predictive analytics platform and Student Success Application, to apply targeted interventions and improve course completion rates. UMUC will share empirical results, insights regarding predictive variables and student risk factors, as well as lessons learned.
Presentation at Sloan-C: 7th Annual International Symposium, Emerging Technologies for Online Learning.
Darren Catalano, Vice President of Analytics, UMUC
Karen Vignare, Ph.D., Associate Provost, Center for Innovation and Learning, UMUC
Laura Malcolm, Vice President of Product, Civitas Learning

Published in: Education
  • Be the first to comment

Predictive Analytics: Turning Insights Into Action to Improve Student Success

  1. 1. Predictive Analytics Turning Insights into Action to Improve Student Success Darren Catalano – Vice President of Analytics, UMUC Karen Vignare, Ph.D. - Associate Provost, Center for Innovation in Learning, UMUC Laura Malcolm - Vice President of Product, Civitas Learning Sloan C Emerging Technologies for Online Learning April 2014
  2. 2. • Pioneer in adult and distance education since 1947 • One of 11 accredited, degree-granting institutions in the University System of Maryland • Focus on the unique educational and professional development needs of adult students • More than 90,000 students enrolled worldwide About UMUC
  3. 3. About Civitas Learning • Founded in 2011 by Charles Thornburgh and Mark Milliron • Provides cloud-based predictive analytics applications for administrators, faculty, students, and advisors • Helps answer the question of what’s working, what’s not working, for which students, at each point in their learning journeys Organize Institutional Data Unlock Foundational Insights Enable Targeted Action
  4. 4. Your Analytics Story • Are you pulling all your data? Some of your data? None of your data? • If you have data, how are you analyzing it? • Do you measure student interventions? • Do you see analytics driven interventions as strategic? 4
  5. 5. Descriptive Analytics • What has happened? • Informative • Reports & dashboards • Describing information or data Predictive Analytics • What is most likely to happen? • Potentially transformative • Targeted alerts • Delivering actionable, real- time insights 5 MORE ACTIONABLE DATA What is Predictive Analytics?
  6. 6. Asking the Right Questions to Prompt Action 6 Predictive Analytics Which applicants are most likely to succeed at this institution? Which students are most likely to not complete a particular course? Which students are at the highest risk of withdrawing? Which students and faculty have low online engagement within a course? Which students are more likely to persist in their program of study? Which students are least likely to complete a particular course?
  7. 7. Predictive Analytics @ UMUC Opportunity • UMUC has a wealth of historical data that can be used to understand student risk profiles • Predictive modeling of UMUC data shows there is a complex set of variables contributing to student success or failure Solution • Partner with a vendor for advanced modeling expertise and rapid application development • A Student Success System provides a daily, individualized risk prediction for every enrollment • Pilot programs underway to determine best ways to leverage and act upon student risk predictions 7
  8. 8. Predictive Modeling in Action Data is gathered and normalized for analysis. Predictive factors are identified and custom models are created. Personalized, real-time recommendations are delivered via web-based apps. 8
  9. 9. Model Development 1. Create features based on available datasets 2. Segment students based on data availability 3. Select top variables through model development (allow the data to tell the story) 4. Create models and test predictions 5. Cluster students based on top feature value similarities (Repeat steps 3 and 4) 6. Choose model through extensive model competition 7. Iterate models as data refreshes, new variables become available, and methodologies improve This approach uses the same predictive methodologies as consumer industries such as equity trading, internet search, book recommendation (Amazon).
  10. 10. Overall Model Accuracy (for Predicting Course Completion)  84% at day 0  88% at day 6 (10% into course) 10 Five types of factors measured:  Student Incoming Profile: Factors present at the time the student enrolls in the course and do not typically change during the course of the term  Student Attendance: Frequency and patterns of online course attendance  Student Activity: Student behavior (tool usage, interaction, etc.) in the LMS classroom  Faculty Activity: Faculty behaviors in the LMS online classroom  Grades: Assignment submissions and grades from throughout the course Risk Factors and Model Results
  11. 11. Student Success Application • Early and through-out term, identify enrollments at risk of not successfully completing the course • Identify the courses and faculty with the highest number of at- risk enrollments • Provide a student-level summary of risk factors to better understand why a student is at risk and inform necessary action • Facilitate intervention to impact predicted outcome
  12. 12. DEMO
  13. 13. Cross Functional Intervention Team Interventions Team Advising Academic Departments Provost Office Analytics Inst. Research Student Services 13
  14. 14. Academic vs. Advising Interventions Academic • Email only • Includes subject or course- specific content • Promotes value of course— what they will learn and why its important • Highlights key milestones or challenges in the course • Provides specific tips for success Advising • Email followed-up by a phone call • Generic talking points used for all courses, such as… • Are they prepared and in the right class? • Do they have any issues and are they getting the help they need? • Do they have a clear plan for success? 14
  15. 15. Pilot Results • Currently in 4th iteration of pilot • Fall 2013 results are statistically significant – Successful course completion for undergraduate courses was 3 percentage points higher – Graduate courses did not show meaningful change 15
  16. 16. Current Pilot Activities Based on Analytics • Re-envisioned Onboarding (Jumpstart) • Continuation of Nudge Interventions • OERs/Eresources • Adaptive Learning • Competency Based Education
  17. 17. Can We Change the Equation? 17 Learner Characteristics Academic Integration Course & Program characteristics Instructor behaviors Learner Behaviors Social-Psychological integration Completion, Reenrollment, completion Predictive Analytics Adapted from PAR Predictive Analytics & Retention Model
  18. 18. Next Steps • Building UMUC Roadmap with Civitas Learning – Illume – ISSM (Incoming Student Success Model) – Inspire for Advisors • Learn from Civitas Community • Learn from PAR and Kresge grant on transfer students 18
  19. 19. 19 Predictive Analytics Discussion
  20. 20. Thank you!