The online nature of Harvard DCE's classroom content puts us in an ideal position to record, analyze and act on data related to the behavior and outcomes of our learners. Join us for an overview of how we are combining information from Canvas, Banner, and Opencast Matterhorn to gain insight into the best ways to shape DCE's learning environments and advance the quality and effectiveness of our course materials through the application of Learning Engineering principles.

1. Learning Engineering Initiatives
at Harvard DCE
Jay Luker, Senior Software Engineer Phoebe Miller, Senior Software Engineer
Harvard IT Summit, 2017

2. What is Learning Engineering?
"Using learning analytics data to help faculty and course designers tailor DCE's
learning environments and improve student outcomes."
— Phoebe & Jay, 2017
"[To achieve] the professional design of learning environments and learning
experiences, [...] the most important step is to find a place on the campus for a
team of individuals who are professionals in the design of learning
environments—learning engineers."
— Herbert A. Simon, The Job of the College President, 1967

3. Who & What
● DCE's Online Teaching & Learning Development Group
○ 4 Software Engineers*
○ 1 QA Engineer
○ 1 A/V Engineer
○ 1 Benevolent Overlord
● Building Software to benefit students, faculty and producers
● Lecture capture, lecture delivery, Canvas LTI tools, deployment, testing &
monitoring tools
● Java, Python, JavaScript, Ruby, R
● Fully AWS-ified since January 2016
* we're hiring!

4. Collectable class attendance data
● Online (live) web conference
○ Zoom web conference classes
● Lecture Videos
○ Recorded live lectures with live streaming
○ Republished lectures from past semesters

5. ● Open source video capture and distribution system
● Developed by a robust community, mostly higher-ed institutions
● Viewer Annotation, automatic closed captioning
● v1.0 released August, 2010
● v3.0 to be released this summer!
● Multistream video player designed specifically for lectures
● Supports all major browsers, mobile, live streaming
● Themeable & customizable via plugins
● Extensive user-tracking functionality, i.e., "useraction" events

6. Paella records useraction events: PLAY, SEEK, HEARTBEAT, etc
Opencast API: IDs user, maintains session, stores in MySQL
Python harvester runs at 2m intervals, adds lecture metadata
SQS used as a buffer queue
Logstash adds geolocation info, parses user-agent data
Elasticsearch: daily rolling indexes; search & aggregation
Apps & APIs, e.g. Kibana dashboards
Useraction Event Pipeline

7. Example
Useraction
Event
Record

8. User Activity Dashboard
● Powered by Kibana, an open source
visualization tool for Elasticsearch data
● Time-series graphs of active user sessions
● Filter by:
○ live viewers vs on-demand
○ identified vs anonymous
○ browser/OS, course, geolocation
● Top terms tables for any event field,
geolocation heat map
● Very useful for fixing a few data integrity
issues

9. Phase One: Proof of Concept
● Attempting to use the data on hand to answer
basic, high-level questions:
○ What is being watched?
○ Who is watching?
○ How is it being watched?
● 3-week "sprint" in late 2016
● Django + Elasticsearch + d3.js + DataTables.js
● Useful!
○ stakeholder feedback
○ refining the questions
○ highlighted gaps and integrity issues with the data

10. More Data!
● Canvas data from VPAL
○ Office of the Vice Provost for Advances in Learning
○ Research group focused on learning science
○ DCE contributes useraction event data & code
○ VPAL provides access to "data products" generated from Canvas activity data
● Auto-generated lecture transcripts
○ IBM Watson speech-to-text service
● Opencast Social
○ Video annotation text

11. Data Inventory
● Useraction event data
○ Lecture viewing behavior, timestamps
● Zoom web conferencing data
○ Meetings, Participants, connection data, timestamps
● Banner student information data
○ Course rolls, registration status
● Canvas activity data
○ Discussions, quizzes, time-on-task
● Transcript data
○ Searchable transcript text, auto-transcription confidence scores
● Annotation text
○ Text, timestamps

12. Phase Two: Answering Questions
● Shifting our development approach
○ from "Hey, look what we can do!"
○ to "What questions do you want us to answer?"
● Switching to R + Shiny
○ Powerful data manipulation
○ Awesome graphs
○ Designed specifically for interactive visualization UIs
○ Allows for more rapid generation of rich graphs and interactive visualizations
○ Smaller, stand-alone apps vs monolithic UI
○ Faster iterations to answer experimental questions

13. How many students watched the lecture videos
for a course?

14. How many students watched the lecture or the section videos in a series?

15. Which students are (and are not)
keeping up with the online lectures?

17. What parts of the lectures are most watched?

18. What parts of the lectures are most watched?

19. What parts of the lectures are most watched?

20. What topics were discussed by students during a
lecture?

21. What topics were discussed by students during a
lecture?
Word cloud generated from viewer annotations
during lecture 6 of the course “Organizational
Behavior”

22. Which students attended a lecture or meeting and were they
there the whole time?

23. Q: Which students attended a lecture or meeting and were they there the whole
time?
Opencast Lectures
Using the heartbeats
from opencast/paella,
one can see student’s
coverage of one video

24. Which students attended a lecture or meeting and were they there the whole time?
Zoom Web Conferences
Zoom provides a REST API to
access meetings, reports and
other data.
Participant are identified by
user name and ip address, as
well as connection type, city,
country, duration and other
information

25. Details of each student’s connection experience can be displayed on one graph

26. Thanks! Questions?
● These slides: http://bit.ly/2sHcWyo
● Github: https://github.com/harvard-dce
● Opencast: http://www.opencast.org/
● VPAL: https://vpal.harvard.edu/