This document summarizes an open academic analytics initiative that aimed to create an open-source early alert system. Key points: - The initiative developed predictive models using historical student data to identify students at risk of not completing courses. - The early alert system was deployed across four institutions, and research found it had a statistically significant positive impact on final course grades, particularly for low-income students. - Instructors reported the system changed their pedagogy by making them more proactive in reaching out to struggling students. - The document advocates for open learning analytics and describes Apereo's open source learning analytics platform that incorporates open standards, a learning record store, analytics processor and dashboards.

1. Open Learning
Analytics (OLA)
THE 5TH INTERNATIONAL LEARNING ANALYTICS & KNOWLEDGE
CONFERENCE

2. Josh Baron
ASSISTANT VICE PRESIDENT
INFORMATION TECHNOLOGY FOR DIGITAL EDUCATION
MARIST COLLEGE

3. Alan Berg
COMMUNITY OFFICER
APEREO LEARNING ANALYTICS INITIATIVE

4. Sandeep Jayaprakash
LEARNING ANALYTICS SPECIALIST
MARIST COLLEGE

5. LESSONS LEARNED – VALUE OF AN OPEN PLATFORM
Open Academic
Analytics Initiative

6. OAAI: Overview and Impact
EDUCAUSE Next Generation Learning
Challenges (NGLC)
Funded by Bill and Melinda Gates Foundations
$250,000 over a 15 month period
Goal: Leverage Big Data concepts to create an
open-source academic early alert system and
research “scaling factors”

7. Student Aptitude Data
(SATs, current GPA, etc.)
Student Demographic Data
(Age, gender, etc.)
Sakai Event Log Data
Sakai Gradebook Data
Predictive
Model
Scoring
Identifies
students “at
risk” to not
complete
course
SISDataLMSData
OAAI Early Alert System Overview
Intervention Deployed
“Awareness” or Online
Academic Support
Environment (OASE)
“Creating an Open Academic
Early Alert System”
Model Developed
Using Historical Data
Step #1: Developed model
using historical data
Academic Alert
Report (AAR)

8. Research Design
Deployed OAAI system to 2200 students across four institutions
• Two Community Colleges
• Two Historically Black Colleges and Universities
Design > One instructor teaching 3 sections
• One section was control, other 2 were treatment groups
Each instructor received an AAR three times during the
semester:
• Intervals were 25%, 50% and 75% into the semester

9. Institutional Profiles

10. Spring ’12 Portability Findings

11. Fall ’12 Portability Findings
Conclusion
1. Predictive models are more
“portable” then anticipated.
2. It is possible to create generic
models that are then “tuned” for
use at specific types of
institutions.
3. It is possible to create a library of
open predictive models that
could be shared globally.

12. Intervention Research Findings
Final Course Grades
Analysis showed a statistically significant
positive impact on final course grades
• No difference between treatment groups
Saw larger impact in spring then fall
Similar trend amount low income students
50
60
70
80
90
100
Awareness OASE Control
FinalGrade(%)
Mean Final Grade for "at Risk" Students

13. Instructor Feedback
"Not only did this project directly assist my students by guiding students to
resources to help them succeed, but as an instructor, it changed my pedagogy;
I became more vigilant about reaching out to individual students and
providing them with outlets to master necessary skills.
P.S. I have to say that this semester, I received the highest volume of
unsolicited positive feedback from students, who reported that they felt I
provided them exceptional individual attention!

14. JAYAPRAKASH, S. M., MOODY, E. W., LAURÍA, E. J., REGAN, J. R.,
& BARON, J. D. (2014). EARLY ALERT OF ACADEMICALLY AT-RISK
STUDENTS: AN OPEN SOURCE ANALYTICS INITIATIVE. JOURNAL
OF LEARNING ANALYTICS, 1(1), 6-47.
More Research Findings…

15. Lesson Learned #1
Openness will play a critical
role in the future of
Learning Analytics

16. Intersections between
openness and Learning Analytics
Open Source Learning Analytics Software
• Weka, Kettle, Pentaho, R, Python etc.
Open Standards and APIs for Learning Analytics
• Experience API, IMS Caliper/Sensor API
Open Models - Predictive models, knowledge maps, PMML etc.
Open Content/Access – Journals, whitepapers, policies documents
Openness or Transparency with regards to Ethics/Privacy
NOT anti-commercial – Commercial ecosystems help sustain OSS

17. Lesson Learned #2
Software Silos Limit
Learning Analytics

18. Software Silos vs. Platforms
Many learning analytics solutions today are
“tool” or “software-centric”
• Analytics tools are built into existing software such as the
Learning Management System (LMS)
Can make it harder to capture data and
integrate across systems (limits Big Data)
A platform solution would allow institutions
to collect data from across many systems
• A “modularized platform” approach allows institutions to use all or just some components
• Integration points allow data to “flow” in for processing and results to flow out

19. Apereo Learning Analytics Initiative (LAI)
Goal: Operationalize outcomes from Learning Analytics research as means to
develop, maintain and sustain an open platform for Learning Analytics
Current Proof-of-Concept Projects
◦ University of Amsterdam – Larrisa (open-source Learning Record Store)
◦ Marist College – Learning Analytics Processor (LAP)
◦ Uniformed Services University – OpenDashboard
◦ Sinclair Community College – Student Success Plan
◦ Unicon – OpenLRS and commercial support services
Contact: Alan Berg, Community Officer
Email: analytics-coordinator@apereo.org,
Wiki Page: https://confluence.sakaiproject.org/x/rIB_BQ
GitHub: https://github.com/Apereo-Learning-Analytics-Initiative

20. Strategic Vision: Open Learning
Analytics Platform
Collection – Standards-
based data capture from any
potential source using Experience
API and/or IMS Caliper/Senor API
Storage– Single repository
for all learning-related data using
Learning Record Store (LRS)
standard.
Analysis– Flexible Learning
Analytics Processor (LAP) that can
handle data mining, data
processing (ETL), predictive model
scoring and reporting.
Communication–
Dashboard technology for
displaying LAP output.
Action– LAP output can be fed
into other systems to trigger alerts,
etc.
Library of
Open Models

21. Learning Record Store & Data Collection
• OpenLRS is a secure, standards-based,
standalone Learning Record Store built to fill
the need for a high i/o storage mechanism for
an open learning analytics environment
• Technical Stack
• Spring-Boot
• Pluggable Datastores (redis, elasticsearch, mongodb)
• xAPI integrations to get activity streams
• Roadmap
• Integration & Support for IMS Caliper

22. Learning Analytics Processor Architecture

23. OAAI Early Alert & Risk Assessment API
Sakai
Admin tool
activities.csv
grades.csv
Learning Analytics Processor (LAP)
Student ID, Course ID,
Risk Rating
Demographics from SIS
Go!
grabs files
OAAI XML
Kettle pipeline applies
model
outputs results
.
.
.
.
------------------ EXTRACT -------- TRANSFORM ------- LOAD ---------
RESTful API

24. Student Success Plan
• Case Management Software
• Academic Advising Tools (MAP)
• Early Alert / Faculty Access (EAL)
• Student Interface (Tasks, MAP, Self
Help)
• Student Information System (SIS)
Integration
• Reporting Tools / Data Collection

25. Student Success Plan

26. Open Dashboard
• Web application that provides a framework
for displaying analytics visualizations and
data views called “cards”.
• Cards represent a single discrete
visualization or data view but share an API
and data model
• LTI compliant
• Widget(Card) library for Learning Analytics

27. LAK15 Hackathon - Open Dashboards
Early Alert Insights Chart
Course Engagement Pathways – Resource &
Content Access

28. Demo

29. Demo Overview
• Three core components of a collection of
open source applications and services that
represent the “Analytics Diamond”
• Can be used individually or collectively
• Work with a shared infrastructure and data
model
Technologies:
• AngularJS
• Spring-Boot
• Pluggable Datastores
(redis, elasticsearch, mongodb)
Sakai
OpenLRS
Learning
Analytics
Processor
Open
Dashboard
xAPI
LTI
API
API
AWS
Local

30. Engaging with the
Community

31. Engaging with Apereo Learning Analytics
Initiative (LAI)
We believe in Do-ocracy.If you see an opportunity or area of
enrichment then you should take leadership and the community will support
you. Bear this in mind as you ask me questions
Examples
Alan - community officer, organizes hackathons & workshops
Sandeep - warding incubation process, analytics
Patrick - communications officer, student requirements
Kate - marketing/communications, Evangelist
Josh - many roles (not even going to start)
Gary - builds the living daylights out of LAI. Sanity check, etc.

32. Engaging with Apereo Learning Analytics
Initiative (LAI)
Where to start? LEVEL 1 NINJA (in no particular order):
• Review the homepage
https://confluence.Sakaiproject.Org/display/LAI/learning+analytics+initiative
• Read the notes from the regular meetings
• Join the mailing list: analytics@apereo.org
(subscribe by sending a message to analytics+subscribe@apereo.Org)
• Join the calls (every other Wednesday) :
https://confluence.Sakaiproject.Org/display/LAI/community+hangouts
• Review github: https://github.Com/apereo-learning-analytics-initiative
• Meet us at a BOF or online.
• Take on a role on a subject you care about

33. Engaging with Apereo Learning Analytics
Initiative (LAI)
Where to start? LEVEL 2 NINJA (in no particular order):
• Buy us/ME beer
• Host a hackathon or workshop
• Present at a conference
• New project / consortium building / grant proposal
• Enrich a current product
• Add parts to Apereo LAI
• Consider co-developing
• Act as a communication channel between organizations
• Surf, JISC , Apereo
• SoLAR, LACE
• Unicon,uva,marist,hull,oxford, <<your name here>>

34. Discussion and Q&A
JOSH: JOSH.BARON@MARIST.EDU
ALAN: A.M.BERG@UVA.NL
SANDEEP: SANDEEP.JAYAPRAKASH1@MARIST.EDU