The document describes an open-source academic early alert and risk assessment application programming interface (API) called the Open Academic Alert Initiative (OAAI). It was developed using big data concepts to create a predictive early alert system for higher education. The system collects student data from learning management systems and other sources, analyzes it using predictive models, and identifies at-risk students. Researchers tested interventions for at-risk students and found they had higher final grades and content mastery compared to control groups. The project aims to develop the technology into an open learning analytics platform consisting of modular components for data collection, storage, analysis and reporting.

1. P r e s e n t e d b y
Members of the Apereo LAI Community
March 18, 2015
Sandeep Jayaprakash, Marist
Gary Gilbert, Unicon
OPEN-SOURCE ACADEMIC EARLY ALERT &
RISK ASSESSMENT API

2. Presenters
Sandeep Jayaprakash
Learning Analytics Specialist,
Marist College
Gary Gilbert
Software Architect, Unicon
Integrations & Analytics

3. Agenda
 Marist early Alert framework
 Open Learning Analytics vision
 Learning Analytics Processor
 Demo
 Discussion

4. OAAI: Overview and Impact
 EDUCAUSE Next
Generation Learning
Challenges (NGLC)
 Funded by Bill and
Melinda Gates Foundations
 $250,000 over a 15 month period
 Goal: Leverage Big Data concepts to create an
open-source academic early alert system and
research “scaling factors”

5. OAAI: Overview and Impact
 Build learning analytics-based early alert system
 Sakai Collaboration and Learning Environment
 Secure data capture process for extracting LMS data
 Pentaho Business Intelligence Suite
 Open-source data mining, integration, analysis & reporting
 OAAI Predictive Model released under open license
 Predictive Modeling Markup Language
 Researching learning analytics scaling factors
 How “portable” are predictive models?
 What intervention strategies are most effective?

6. Student Aptitude Data
(SATs, current GPA, etc.)
Student Demographic
Data (Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive
Model
Scoring
Identifies
students
“at risk” to
not
complete
course
SISDataLMSData
OAAI Early Alert System Overview
Intervention Deployed
“Awareness” or Online
Academic Support
Environment (OASE)
“Creating an Open Academic
Early Alert System”
Model Developed
Using Historical Data
Step #1: Developed
model using historical
data
Academic Alert
Report (AAR)

7. Predictors of
Student Risk
Some predictors
were discarded if
not enough data
was available.
LMS predictors were
measured relative
to course averages.

8. OAAI Predictive Process

9. Research Design
 Deployed OAAI system to 2200 students across four
institutions
 Two Community Colleges
 Two Historically Black Colleges and Universities
 Design > One instructor teaching 3 sections
 One section was control, other 2 were treatment groups
 Each instructor received an AAR three times during
the semester:
 Intervals were 25%, 50% and 75% into the semester

10. Prediction Results

11. Spring ’12 Portability Findings

12. Fall ’12 Portability Findings
Conclusion
1. Predictive models
are more “portable”
than anticipated.
2. It is possible to
create generic
models that are
then “tuned” for use
at specific types of
institutions.

13. Intervention Research Findings
Final Course Grades
 Analysis showed a
statistically significant
positive impact on final
course grades
 No difference between
treatment groups
 Saw larger impact in
spring than fall
 Similar trend amount
low income students
50
60
70
80
90
100
Awareness OASE Control
FinalGrade(%)
Mean Final Grade for "at Risk" Students

14. Intervention Research Findings
Content Mastery
 Student in intervention
groups were statistically
more likely to “master
the content” than those
in controls.
 Content Mastery = Grade
of C or better
 Similar for low income
students.
0
200
400
600
800
1000
Yes No Yes No
Content Mastery for "at Risk" Students
Control Intervention
Frequency

15. Instructor Feedback
"Not only did this project directly assist my students by guiding
students to resources to help them succeed, but as an instructor,
it changed my pedagogy; I became more vigilant about
reaching out to individual students and providing them with
outlets to master necessary skills.
P.S. I have to say that this semester, I received the highest
volume of unsolicited positive feedback from students, who
reported that they felt I provided them exceptional individual
attention!

16. JAYAPRAKASH, S. M., MOODY, E. W., LAURÍA, E. J.,
REGAN, J. R., & BARON, J. D. (2014). EARLY ALERT OF
ACADEMICALLY AT-RISK STUDENTS: AN OPEN SOURCE
ANALYTICS INITIATIVE. JOURNAL OF LEARNING
ANALYTICS, 1(1), 6-47.
More Research Findings…

17. Strategic Vision: Open Learning
Analytics Platform
Collection
Standards-based data
capture from any
potential source using
Experience API and/or IMS
Caliper/Senor API
Storage
Single repository for all
learning-related data
using Learning Record
Store (LRS) standard.
Analysis
Flexible Learning Analytics
Processor (LAP) that can
handle data mining, data
processing (ETL), predictive
model scoring and
reporting.
Communication
Dashboard technology for
displaying LAP output.
Action
LAP output can be fed
into other systems to
trigger alerts, etc.

18. Technology Stack
 Learning Analytics Processor (LAP)
 JAVA-based web application
 Maven for builds
 Temporary Storage - H2 in-memory database
 Persistence Storage - MySQL
 Predictive Model Mark-up Language (PMML)
 OAAI Early Alert Pipeline
 Pentaho Kettle – Data Integration & ETL
 Pentaho WEKA – Data Mining & Predictive Modelling

19. High-Level Workflow
Sakai
Admin
tool
activities.csv
grades.csv
Learning Analytics Processor (LAP)
Student ID,
Course ID,
Risk Rating
Demographics
from SIS
Go!
grabs files
OAAI XML
Kettle pipeline
applies model
outputs results
..
.
.
------------------ EXTRACT -------- TRANSFORM ------- LOAD ---------
RESTful API

20. LAP Pipeline Architecture

21. Features
Key pieces of the LAP architecture
 Input source management
 Data storage – temporary & persistent
 Configuration manager
 Pipeline processor
 Output results management
 Extensibility
 Supports multiple pipelines
 Supports varied pipeline platforms

22. Demo Overview
● Three core components of a
collection of open source
applications and services that
represent the “Analytics Diamond”
● Can be used individually or
collectively
● Work with a shared infrastructure
and data model
Technologies:
• AngularJS
• Spring-Boot
• Pluggable Datastores
(redis, elasticsearch, mongodb)
OpenLRS
Learning
Analytics
Processor
Sakai
Open
Dashboard
xAPI
LTI
API
API

23. Demo

24. Early Alert Insights – Open Dash

25. Questions?
APEREO LEARNING ANALYTICS INITIATIVE COMMUNITY
• Accelerate the operationalization of Learning
Analytics software and frameworks
• Support the validation of analytics pilots across
institutions
• Work together so as to avoid duplication
analytics-coordinator@apereo.org
Josh Baron
josh.baron@marist.edu
Sandeep Jayaprakash
sandeep.jayaprakash1@
marist.edu
Gary Gilbert
ggilbert@unicon.net

26. Appendix

27. Early Alert - Kettle ETL Flows

28. WEKA Predictive Modelling Flows

29. Learning Analytics Processor