Presented at the 2014 Blackboard Institutional Performance Conference (30-31 October 2014). ABSTRACT: Passing grades and retention through to degree are essential to success in higher education, but these factors are too often mistaken for ends in themselves. A high-performing student environment has provided teachers and researchers at Emory University with a space to think critically about what success means, and about the extent to which data might inform the design of successful learning environments. This presentation will (1) discuss some of the unique challenges encountered by Emory University during its 2013-2014 Blackboard Analytics pilot, (2) describe several provisional insights gained from exploratory data mining, and (3) outline how Emory’s pilot experience has informed support of learning analytics on campus. What we have learned at Emory University has both broad and deep implications for how institutions use data in support of student success, but these insights could only have been achieved in an environment where grade-performance and retention are not significant issues.

1. Rethinking Student Success
Analytics in Support of Teaching and Learning
Timothy D. Harfield, Scholar in Residence (Learning Analytics)
Emory University Libraries and Information Technology Services
@tdharfield

2. 2
Why Learning Analytics?
@tdharfield

3.  Founded in 1836
 11 Schools
(4 Undergraduate, 7 Graduate)
 14,513 Students
(7,836 Undergraduate, 6,677 Graduate &
Professional)
 Average Freshman Retention Rate:
95.3%
 4-year Graduation Rate: 84%
 94% College Success Rate (<>
DFW)
 US News National University
Ranking #21
Fact & Figures as of Fall 2013
(http://www.emory.edu/home/about/factsfigures/)
@tdharfield
Why Learning Analytics?
Emory University

4. • “The most dramatic factor shaping the future of higher education”
• Drop in U.S. News & World Report Rankings, from 20th to 21st
• Existing rates of student success makes at-risk students difficult to identify
Why Blackboard Analytics for Learn?
• Learning Management Gap in Oracle BI Solution
• Fast, lightweight, inexpensive
@tdhar4field
Why Learning Analytics?
Emory University

5. 5
Blackboard Analytics Pilot
@tdharfield

6. @tdhar6field
Blackboard Analytics Pilot
Methodology
Blackboard Analytics for Learn™ Adds to Emory’s existing analytical capacity
In three areas:
I. Learning Analytics
Integrated Blackboard Reports for teachers and students
 Survey Instrument based on Learning Analytics Acceptance Model
II. Academic Analytics
Online Analytical Processing for instructional designers and university leadership
 Focus Group
III. Educational Data Mining / Predictive Analytics
Direct database queries and table exports for longitudinal modeling ( >180 days) and analysis using
statistical software packages and machine learning algorithms
 Exploratory Data Mining

7. Overall
Ease of Use
@tdhar7field
Blackboard Analytics Pilot
Integrated Reports
Would You Like to Have Access
Yes
51%
in Other Courses?
No
9%
Maybe
40%
6%3%
27%
21%
43%
3%
23%
47%
15%
12%
Overall
Usefulness
Would You Recommend It
Yes
36%
to Others?
No
17%
Maybe
47%
Strongly Disagree Disagree Neither Agree nor Disagree Agree Strongly Agree

8. @tdhar8field
Blackboard Analytics Pilot
Integrated Reports | Student Feedback
“I believe it's important for professors to have this tool available to them, so they can see early if we're not turning up for
class enough and intervention might be warranted. I do know that I need to get in a habit of logging onto BB at least twice a
day. I need to make it part of my daily life”
“I liked being able to compare my activity to others, but I don't think it changed my behavior. It may be useful to others.”
“I think its useful if someone is concerned about how they are doing in the class or if they are uncertain about how much
time they should be committing to the class. I personally did not find it useful but I can see how someone may use it as a
tool to determine if they are meeting expectations”
“I think this system also puts undue pressure on students to log in or click on specific items in blackboard more, without
taking into consideration factors such as work done offline in the class or via email. This is especially true for distance
learners. I personally cut and paste many of my assignments in Word so they are easier to view and manipulate. I also like
to keep files from each semester so I do a considerable amount of work offline and I am doing very well in my classes so
far. When I logged in to check my activity, the data I saw pretty much showed the opposite-it looked like I wasn't very
engaged at all.”
“If this is a tool that will be used by Professors to grade participation I would have likely used it more; however, if this tool
were indeed used by Professors it may cause students to post unnecessarily.”
“You should be a big button at the front of the page with an icon to remind people to use it.”

9. @tdhar9field
Blackboard Analytics Pilot
Integrated Reports | Faculty Feedback
• Anecdotally, faculty report that students like the new analytics feature, although it is a
distraction for some high performing students who become overly driven by metrics at
the expense of actual course objectives
Takeaway: Use of Course Analytics should be accompanied by clear, continuous
communication about the tool’s use and limitations, and the ways in
which data are being used for assessment. There is much to be
gained from the use of embedded analytics, but also risk if not
implemented in manner that is pedagogically sound.

10. @tdha1r0field
Blackboard Analytics Pilot
Online Analytical Processing
• Delivered reports for the Pyramid BI reporting environment are of little value to
instructional designers.
a. Data dictionary is out of date, inaccurate, confusing, and inconsistent
b. It is unclear what various reports are meant to accomplish, and how they might help generate
actionable insights
Takeaways: Reporting using the Pyramid BI reporting environment is more useful
to the Office of Institutional Research than to instructional designers
Instructional designers are interested in analyzing data using more
inferential and predictive techniques. Program improvement should
start with questions rather than with data.

11. 11
62%
38%
Course Items Added No Course Items Added
DFW Rate by Blackboard Use
5% 5%
100%
75%
50%
25%
0%
Course Items Added No Course Items Added
Course Items Added No Course Items Added
@tdharfield
Blackboard Analytics Pilot
Exploratory Data Mining

12. @tdha1r2field
Blackboard Analytics Pilot
Exploratory Data Mining
DFW Rate by Design Quartile
4.8%
5.2%
5.8%
4.0%
5.1%
10%
5%
0%
First Quartile Second Quartile Third Quartile Fourth Quartile Non-Blackboard

13. @tdha1r3field
Blackboard Analytics Pilot
Exploratory Data Mining
DFW Rate by Class
3.3%
4.1%
7.3%
5.5%
10%
5%
0%
Freshman Sophomore Junior Senior

14. DFW Rate by Class
(Oxford College)
10.1% 9.8%
Junior Senior
@tdha1r4field
Blackboard Analytics Pilot
Exploratory Data Mining
DFW Rate by Class
(Excluding Oxford College)
3.3%
4.1%
20%
10%
0%
6.4%
4.4%
10%
5%
0%
Freshman Sophomore Junior Senior

15. 15
5.3%
6.1%
4.8% 4.8%
10%
5%
0%
Files Items
DFW Rate by Item Type
DFW Rate by Link Type
5.3% 5.4%
1.0%
4.0%
10%
5%
0%
Course Links External URLs
Not Used Used
@tdharfield
Blackboard Analytics Pilot
Exploratory Data Mining

16. 16
DFW Rate by Grade Center Use
4.1%
5.8%
10%
5%
0%
No Yes
@tdharfield
Blackboard Analytics Pilot
Exploratory Data Mining

17. 17
DFW Rate by Minutes Z-Score
2.9%
4.3%
5.8% 6.5%
20%
10%
0%
>= 1 >= 0 & < 1 > -1 & > 0 <= -1
DFW Rate by Interactions Z-Score
2.8% 3.5%
6.5%
8.6%
20%
10%
0%
>= 1 >= 0 & < 1 > -1 & > 0 <= -1
DFW Rate by Access Z-Score
2.5%
3.4%
6.7%
10.3%
20%
10%
0%
>= 1 >= 0 & < 1 > -1 & > 0 <= -1
@tdharfield
Blackboard Analytics Pilot
Exploratory Data Mining

18. 18
Next Steps
@tdharfield

19. @tdha1r9field
Regional Community of Practice
“Data Wrangling”
Local Community of Practice
Next Steps

20. @tdha2r0field
Next Steps
Data Wrangling
Predicting Students at Risk
 Data Validation
 Data Dictionary
 Predictive Modeling

21. @tdha2r1field
Next Steps
Data Wrangling
Online Course Evaluation Framework
Roxanne Russell, PhD
 Are logged course behaviors predictive of
student satisfaction with course design,
implementation, and interaction dynamics?
 Can we develop algorithms for identifying
pain points, and effective interventions to
mitigate dissatisfaction and other barriers to
learning?

22. @tdha2r2field
Next Steps
Data Wrangling
Getting a Leg Up at Emory (GLUE)
Drew Kohlhorst, PhD
 Does participation in the GLUE program contribute
significantly to student success in STEM courses at
Emory?
 Does GLUE function to attract and retain more
STEM majors?
 Are there significant differences in the success
rates of students participating in the online versus
residential versions of the program?
 Does participation see changes in
attitudes/confidence in scientific skills, and/or
changes in on-campus resource utilization?

23. @tdha2r3field
Next Steps
Data Wrangling
Emory Coursera Initiative
Stephanie Parisi
 Who are our learners?
 Why do learners take a particular
course?
 What factors contribute most to
learner satisfaction?

24. @tdha2r4field
Next Steps
Local Community of Practice
Analytics for Learning at Emory (ALE)
 Receive inspiration and assistance in the
formation of questions that might guide the use
of data to optimize learning and learning
environments
 Share current interests and projects, in order to
receive feedback and encouragement, and to
develop possibilities for future collaboration
 Gain exposure to the field of learning analytics
(language, key concepts, methodologies, etc)
through access to experts

25. @tdha2r5field
Next Steps
Regional Community of Practice
https://scholarblogs.emory.edu/ale

26. THANKS!
Timothy D. Harfield
Scholar in Residence (Learning Analytics)
timothy.harfield@emory.edu
@tdharfield