The document discusses findings from analyzing data of 3 million learners to optimize course design and the use of learning analytics. It emphasizes the relationship between tool usage and student achievement, revealing that tool usage has a diminishing return on grades and identifies factors influencing student performance. Next steps include refining analytics triggers for achievement and exploring alternative measures of success beyond just final grades.

1. What data from 3
million learners can
tell us about effective
course design
John Whitmer, Ed.D.
Director, Analytics & Research
john.whitmer@blackboard.com | @johncwhitmer

2. 22
Meta- questions driving our Learning Analytics research @ Blackboard
1. How do students & teachers use our platforms? How is this use
related to student achievement? [or satisfaction, or risk, or …]
3. How can we integrate these findings into features/functionality
that apply to the broad spectrum of ways people use our
platforms?
2. Do these findings apply equally to students ‘at promise’ due to
their academic achievement or background characteristics? (e.g.
race, class, family education, geography)

3. 33
Outline
1. Positioning Learning Analytics
2. Research Findings
3. Q & A

4. 4
1. Positioning Learning Analytics

5. 55

6. 66
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
200MBof data emissions annually

7. 77
Logged into course within 24
hours
Failed first exam
Interacts frequently in
discussion boards
No declared majorHasn’t taken college-level math

8. 8
“ ...measurement, collection, analysis and reporting of data about learners and
their contexts, for purposes of understanding and optimizing learning
and the environments in which it occurs.”
Learning and Knowledge Analytics Conference, 2011
What is learning analytics

9. 9
Strong interest by faculty and students

10. 1010
Blackboard’s Learning Data Footprint (2015 #’s)
1.6M Unique Courses
40M Course Content Items
4M Unique Students
Blackboard Learn =
¼ total data
775M LMS Sessions

11. 1111
Commitment to Privacy & Openness
• Analyze data records
that are not only
removed of PII, but
de-personalized
(individual &
institutional levels)
• Share results and open
discussion procedures
for analysis to inform
broader educational
community
• Respect territorial
jurisdictions and safe
harbor provisions

12. 12
2. Research Findings

13. 13
Why this is hard?
• LMS data is messy
• Adoption is hugely varied

14. 14
Same model doesn't work for all courses
• 1.2M students
• 34,519 courses
• 788 institutions
• Overall effect size < 1%

15. 15
But strong effect in some courses
(n=7,648, 22%)

16. 16
The Effectiveness of Blackboard Learn Tool
Usage

17. 17
Purpose of Investigation
To determine the most important tools in Bb Learn, by observing:
– Tools that are used the most (in minutes, for instance)
– Tools that have strongest relationship with final grade
– Tools that are ‘underused’ the most (by learners & instructors); tools that have the greatest
potential to improve learning outcomes
Allows us to see which tools educate students, and are therefore useful
Reinforce the educational impact of the Blackboard Learn platform

18. 18
Data Filtering
Filters decreased the number of students analyzed from 3.37 million users in 70,000 courses from 927 institutions
to 601,544 users (17% of total) in 18,810 courses (26.8% of total) from 663 institutions (71.5% of total)
Class Size
between 10 and 500 students
Activity Rates
over 1 hour online as a course average
Grade Distribution
average grade between 40% & 95%

19. 19
Finding: Tool Use & Grade
Tool use and Final Grade do not have a linear relationship;
there is a diminishing marginal effect of tool use on Final Grade
Interpretations
• Students absent from course activity are at
greatest risk of low achievement.
• The first time you read/see a PowerPoint
presentation, you learn a lot, but the
second time you read/see it, you learn
less.
• Getting from a 90% to a 95% requires
more effort than getting from a 60% to a
65%.

20. 20
Finding: Tool Use & Grade
Tool use and Final Grade do not have a linear relationship;
there is a diminishing marginal effect of tool use on Final Grade
Interpretations
• Students absent from course activity are at
greatest risk of low achievement.
• The first time you read/see a PowerPoint
presentation, you learn a lot, but the
second time you read/see it, you learn
less.
• Getting from a 90% to a 95% requires
more effort than getting from a 60% to a
65%.
Log transformation shows
stronger trend

21. 21
Investigation Achievement by Specific Tools Used
Analysis Steps
• Identify most frequently used tools
• Separate tool use into no use + quartiles
• Divide students into 3 groups by course grade
• High (80+)
• Passing (60-79)
• Low/Failing (0-59)

22. 22
Finding: MyGrades
At every level, probability of higher grade increases with increased use.
Causal? Probably not. Good indicator? Absolutely.

23. 23
Finding: Course contents
More is not always better. Large jump none to some; then no relationship

24. 24
Finding: Assessments/Assignments
Students above mean have lower likelihood of achieving a high grade than students below the mean

25. 25
Next Steps & Discussion

26. 26
Next Steps in Product & Research
• Refine Ultra “Learning Analytics” triggers for low/high achievement; focus on
getting started, not achieving “top of class” in activity.
• Explore data points beyond time on task; semantic analysis, writing level analysis,
other more rich data points
• Investigate course design structures and patterns in how teachers create course
experiences using Learn
• Collaborate with institutions on research to consider alternative measures of
success besides course final grade (course evaluations, grades in subsequent
courses)

27. Questions?
John Whitmer, Ed.D.
john.whitmer@blackboard.com
@johncwhitmer