Predictive Analytics: Turning Insights Into Action to Improve Student Success
Upcoming SlideShare
Loading in...5
×
 

Predictive Analytics: Turning Insights Into Action to Improve Student Success

on

  • 355 views

UMUC is committed to providing high quality education that is accessible to all as an open access institution while lowering costs. Using predictive analytics and effective learner interventions are ...

UMUC is committed to providing high quality education that is accessible to all as an open access institution while lowering costs. Using predictive analytics and effective learner interventions are critical to our strategy. UMUC developed a Student Success Initiative to evaluate the potential for predictive analytics to improve student outcomes and selected Civitas Learning as a key strategic partner. Since Spring 2013, the UMUC has engaged in three pilots, utilizing Civitas Learning's predictive analytics platform and Student Success Application, to apply targeted interventions and improve course completion rates. UMUC will share empirical results, insights regarding predictive variables and student risk factors, as well as lessons learned.
Presentation at Sloan-C: 7th Annual International Symposium, Emerging Technologies for Online Learning.
Presenters:
Darren Catalano, Vice President of Analytics, UMUC
Karen Vignare, Ph.D., Associate Provost, Center for Innovation and Learning, UMUC
Laura Malcolm, Vice President of Product, Civitas Learning

Statistics

Views

Total Views
355
Views on SlideShare
341
Embed Views
14

Actions

Likes
1
Downloads
7
Comments
0

2 Embeds 14

http://innovationinlearning.wordpress.com 11
http://www.slideee.com 3

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • -The University has partnered with Civitas Learning to evaluate the potential for predictive analytics to improve student outcomes.-UMUC has a wealth of historical data that can be used to understand student risk profiles.-Predictive modeling of UMUC data shows there is a complex set of variables contributing to student success or failure -Using this predictive model, we can provide an individualized risk prediction by student before the term starts and throughout the semester.-We have created a Student Success Application to visually present the resulting information from these models that will be used to test student interventions.-We are running pilot programs to understand our ability to leverage these models to influence successful course completion.
  • Would deleteBring data in from UMUC Take data from different systems and collect it into one dataset (matching ids)Use “big data” technologies to manage enormous datasetsUnderstand and prepare the data for modeling Explore data to ensure precise understanding of what is unique, in addition, identify the "core data" we expect from all implementationsMap UMUC data to normalized model (i.e. perform quantile normalization)Take special care to assess "epochal" shifts in data definitionQuality check processVerify mappings and normalizations with UMUCWork w/ UMUC system owners and experts to vet mappings“Sanity check" our transformed data against UMUC BIe.g. We show that in Spring 2012, x dropped, y passed w/ C or better, z failed; does this match UMUC records?
  • Let the data tell the story. Through model iteration, reduce variables to only those that are informativeCreate models and test predictions. Build models using training datasets and run the models against test data set to determine accuracyChoose model through extensive model competition. Test multiple models against the dataset; only the most accurate survivesIterate models as data refreshes, new variables become available, and methodologies improve. This is very important. Our day zero model is very different from our day 10 model because other variables such as learning activities and grade data are available.Production model to use:Support vector machines for prediction, an opaque machine learning algorithm which implicitly maps input variables into a high dimensional space in order to find an optimal decision boundaryLogistic regression to expose and rank variables most highly correlated to successful course completion
  • Our models considers 5 different types of variables.Before the class starts our risk prediction for each student is 84% accurate.As the term continues and the students / faculty generate more data, the models become more and more accurate.

Predictive Analytics: Turning Insights Into Action to Improve Student Success Predictive Analytics: Turning Insights Into Action to Improve Student Success Presentation Transcript

  • Predictive Analytics Turning Insights into Action to Improve Student Success Darren Catalano – Vice President of Analytics, UMUC Karen Vignare, Ph.D. - Associate Provost, Center for Innovation in Learning, UMUC Laura Malcolm - Vice President of Product, Civitas Learning Sloan C Emerging Technologies for Online Learning April 2014
  • • Pioneer in adult and distance education since 1947 • One of 11 accredited, degree-granting institutions in the University System of Maryland • Focus on the unique educational and professional development needs of adult students • More than 90,000 students enrolled worldwide About UMUC
  • About Civitas Learning • Founded in 2011 by Charles Thornburgh and Mark Milliron • Provides cloud-based predictive analytics applications for administrators, faculty, students, and advisors • Helps answer the question of what’s working, what’s not working, for which students, at each point in their learning journeys Organize Institutional Data Unlock Foundational Insights Enable Targeted Action
  • Your Analytics Story • Are you pulling all your data? Some of your data? None of your data? • If you have data, how are you analyzing it? • Do you measure student interventions? • Do you see analytics driven interventions as strategic? 4
  • Descriptive Analytics • What has happened? • Informative • Reports & dashboards • Describing information or data Predictive Analytics • What is most likely to happen? • Potentially transformative • Targeted alerts • Delivering actionable, real- time insights 5 MORE ACTIONABLE DATA What is Predictive Analytics?
  • Asking the Right Questions to Prompt Action 6 Predictive Analytics Which applicants are most likely to succeed at this institution? Which students are most likely to not complete a particular course? Which students are at the highest risk of withdrawing? Which students and faculty have low online engagement within a course? Which students are more likely to persist in their program of study? Which students are least likely to complete a particular course?
  • Predictive Analytics @ UMUC Opportunity • UMUC has a wealth of historical data that can be used to understand student risk profiles • Predictive modeling of UMUC data shows there is a complex set of variables contributing to student success or failure Solution • Partner with a vendor for advanced modeling expertise and rapid application development • A Student Success System provides a daily, individualized risk prediction for every enrollment • Pilot programs underway to determine best ways to leverage and act upon student risk predictions 7
  • Predictive Modeling in Action Data is gathered and normalized for analysis. Predictive factors are identified and custom models are created. Personalized, real-time recommendations are delivered via web-based apps. 8
  • Model Development 1. Create features based on available datasets 2. Segment students based on data availability 3. Select top variables through model development (allow the data to tell the story) 4. Create models and test predictions 5. Cluster students based on top feature value similarities (Repeat steps 3 and 4) 6. Choose model through extensive model competition 7. Iterate models as data refreshes, new variables become available, and methodologies improve This approach uses the same predictive methodologies as consumer industries such as equity trading, internet search, book recommendation (Amazon).
  • Overall Model Accuracy (for Predicting Course Completion)  84% at day 0  88% at day 6 (10% into course) 10 Five types of factors measured:  Student Incoming Profile: Factors present at the time the student enrolls in the course and do not typically change during the course of the term  Student Attendance: Frequency and patterns of online course attendance  Student Activity: Student behavior (tool usage, interaction, etc.) in the LMS classroom  Faculty Activity: Faculty behaviors in the LMS online classroom  Grades: Assignment submissions and grades from throughout the course Risk Factors and Model Results
  • Student Success Application • Early and through-out term, identify enrollments at risk of not successfully completing the course • Identify the courses and faculty with the highest number of at- risk enrollments • Provide a student-level summary of risk factors to better understand why a student is at risk and inform necessary action • Facilitate intervention to impact predicted outcome
  • DEMO
  • Cross Functional Intervention Team Interventions Team Advising Academic Departments Provost Office Analytics Inst. Research Student Services 13
  • Academic vs. Advising Interventions Academic • Email only • Includes subject or course- specific content • Promotes value of course— what they will learn and why its important • Highlights key milestones or challenges in the course • Provides specific tips for success Advising • Email followed-up by a phone call • Generic talking points used for all courses, such as… • Are they prepared and in the right class? • Do they have any issues and are they getting the help they need? • Do they have a clear plan for success? 14
  • Pilot Results • Currently in 4th iteration of pilot • Fall 2013 results are statistically significant – Successful course completion for undergraduate courses was 3 percentage points higher – Graduate courses did not show meaningful change 15
  • Current Pilot Activities Based on Analytics • Re-envisioned Onboarding (Jumpstart) • Continuation of Nudge Interventions • OERs/Eresources • Adaptive Learning • Competency Based Education
  • Can We Change the Equation? 17 Learner Characteristics Academic Integration Course & Program characteristics Instructor behaviors Learner Behaviors Social-Psychological integration Completion, Reenrollment, completion Predictive Analytics Adapted from PAR Predictive Analytics & Retention Model
  • Next Steps • Building UMUC Roadmap with Civitas Learning – Illume – ISSM (Incoming Student Success Model) – Inspire for Advisors • Learn from Civitas Community • Learn from PAR and Kresge grant on transfer students 18
  • 19 Predictive Analytics Discussion
  • Thank you!