Improving Student Achievement with New Approaches to Data

Improving Student Achievement with
New Approaches to Data:
Learning Analytics &
the CSU Data Dashboard

John Whitmer, Ed.D.
Academic Technology Services
California State University, Office of the Chancellor

WASC ARC Conference
April 11, 2013
slides @ slideshare.net/JohnWhitmer/

Outline
1. Context: California State University &
Graduation Initiative

2. Chico State Learning Analytics Case Study

3. CSU Data Dashboard Project

4. Next Steps

5. Discussion


1. CONTEXT


California State University
http://calstate.edu
 23 campuses
 437,000 FTE students
 44,000 faculty and staff
 Largest, most diverse, &
one of the most
affordable university
systems in the country
 Play a vital role in the
growth & development of
California's communities
and economy

CSU Achievement Gap


.
– Baseline 6-Year Graduation Rate: 46%
– Target 6-Year Graduation Rate: 54%

– Baseline Achievement Gap: 11%
– Target Achievement Gap: 5.5%

2

New Approaches to Using Data
Enable data-driven decision making for
interventions earlier in the student experience by

1. Integrate new data sources & variables

2. Disseminate findings to a broader audience

3. Provide ability to interact with data analysis,
conduct ad-hoc and custom reporting


2. CHICO STATE LEARNING
ANALYTICS CASE STUDY


200MB of data emissions annually!
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy.
The Economist.

Logged into course within 24
hours

Interacts frequently in
discussion boards

Failed first exam

Hasn’t taken college-level
math
Source: jisc_infonet @ Flickr.com
No declared major

Source: jisc_infonet @ Flickr.com slides @ slideshare.net/JohnWhitmer/

Case Study: Intro to Religious Studies
• Undergraduate, introductory, high
demand
54 F’s
• Redesigned to hybrid delivery format
through “academy eLearning program”

• Enrollment: 373 students
(54% increase on largest section)

• Highest LMS (Vista) usage
entire campus Fall 2010
(>250k hits)

• Bimodal outcomes compared to
traditional course
• 10% increase on final exam
• 7% & 11% increase in DWF

• Why? Can’t tell with aggregated data

Learner Analytics

“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)


Pervasive Adoption of Learning Management Systems

Institution-Supported IT
Resources and Tools. Reprinted
from “The ECAR Study of
Undergraduate Students and
Information Technology,” Eden
Dahlstrom, 2012 by the
EDUCAUSE Center for Applied
Research.


Guiding Questions
1. How is student LMS use related to academic
achievement in a single course section?

2. How does that finding compare to the relationship of
achievement with traditional student characteristic
variables?

3. How are these relationships different for
“at-risk” students (URM & Pell-eligible)?

4. What data sources, variables and methods are most
useful to answer these questions?


LMS Use Variables Student Char. Variables
1. Administrative Activities 1. Enrollment Status
(calendar, announcements) 2. First in Family to Attend
2. Assessment Activities College
(quiz, homework, assignments, 3. Gender
grade center) 4. HS GPA
3. Content Activities 5. Major-College
(web hits, PDF, content pages)
6. Pell Eligible
4. Engagement Activities
(discussion, mail) 7. URM and Pell-Eligibility
Interaction
8. Under-Represented
Minority
9. URM and Gender
Interaction

Correlation: Student Char. w/Final Grade

Scatterplot of
HS GPA vs. Course
Grade

Predict the trend
 LMS use and final grade is _______ compared to
student characteristics and final grade:

a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger


Correlation LMS Use w/Final Grade

Scatterplot of
Assessment Activity
Hits vs. Course
Grade


Chart: LMS & Student Characteristics


Combined Variables Regression Final Grade by
LMS Use & Student Characteristic Variables

LMS Student
Use Characteristic
Variables Variables

25%
(r2=0.25)
> +10%
(r2=0.35)

Explanation of change Explanation of change
in final grade in final grade


Predict the trend
 LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger

Relationship indicates how strongly LMS use is correlated
with final grade; lower value equals less impact
*at-risk = BOTH under-represented minority and Pell-eligible

Predict the trend
 LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger

*at-risk = BOTH under-represented minority and Pell-eligible


Question 3 Results:
Regression by “At Risk” Population Subsamples


At-Risk Students: “Over-Working Gap”

27


Activities by Pell and Grade

Extra effort
in content-
related
activities


Conclusions
1. LMS use is a better predictor of academic
achievement than student characteristics.
– LMS use frequency is a proxy for effort.

2. LMS data requires extensive filtering to be useful;
student variables need pre-screening for missing
data.

3. LMS effectiveness for at-risk students may be
caused by non-technical barriers.

4. Small strength magnitude suggests that better
methods could produce stronger results.


Next Generation Learning Analytics

Graphic Courtesy Sasha Dietrichson, X-Ray Research SRL

3. DATA DASHBOARD PROJECT


THE FRAMEWORK
Advancing by Degrees: A
Framework for Increasing
College Completion by
Offenstein, Moore &
Schulock

Institute for Higher
Education Leadership and
Policy and The Education
Trust (http://bit.ly/10QtMXC)


This research describes
academic patterns (or leading
indicators) that occur early in
the pipeline that can be tracked
and monitored in real time
against milestones on the
graduation route.


Milestones Leading Indicators
Year-to-year Retention Remediation
Transition to college level coursework Begin remedial coursework in the first term, if
(English and Math) needed.
Earn one year of college level credits Complete needed remediation
Complete General Education
Complete degree Gateway Courses
Complete college-level math and/or English in
the first or second year
Complete a college-success course or other
first-year experience program

Credit Accumulation and Related Academic
Behaviors
Complete high percentage of courses
attempted (low rate of course dropping and/or
failure)
Complete 20-30 credits in the first year
Earn summer credits
Enroll full time
Enroll continuously, without stop-outs
Register on-time for courses
Maintain adequate academic progress

Driving Questions for Dashboard
1. What percentage of students reach each of the
leading indicators?
2. What is the impact of reaching each of the
leading indicators on success rate?
3. Does meeting any of the indicators reduce or
eliminate gaps between student demographic
groups?


PROOF OF CONCEPT


Purpose
 Demonstrate potential value of combined
reporting and statistics

 Evaluate availability and integration of data

 Pilot potential tools in real-world scenario

 NOTE: production system may be dramatically
different from POC, given lessons learned and
scalability


1. Report
Parameters


2. Retention Rates


3. Retention Rates by URM Status


4. Data Export Options

Concern: Male, 2nd
Year Persistence


4. NEXT STEPS


What’s Now … And Next
 Conducting 3 multi-campus pilots
1. mCURL: Moodle Common Usage Reporting & Learning
Analytics: (8 CSU & 2 UC campuses)
2. Blackboard Analytics for Learn (3 campuses)
3. LMS-agnostic campus surveys

 Investigating additional pilot with LMS-agnostic tool to
move beyond “clickometry” into social network analysis,
discourse analysis, etc.

 Raises question for MOOC research: relationship between
student intent/motivation, student characteristics/leading
indicators, MOOC use, and achievement


Data Dashboard

CCA LMS
Data Data

Other
ERS
Data
Data
Data Sources
Dashboard


Feedback? Questions?

John Whitmer
jwhitmer@calstate.edu

 Monograph @
http:www.johnwhitmer.net
 Twitter: johncwhitmer

Desdemona Cardoza
dcardoza@calstate.edu


Improving Student Achievement with New Approaches to Data

More Related Content

What's hot

Viewers also liked

Similar to Improving Student Achievement with New Approaches to Data

Recently uploaded

Improving Student Achievement with New Approaches to Data

Editor's Notes