Presentation on learning analytics made to the UM Honors Advisory Council in 2012.

1. Students and learning: a complex system
Mysteries of teaching and learning
What do students know when they come?
What do they do while in class?
What do they learn and remember?
Can we predict outcomes? When? For whom?
Can we change outcomes? What should we do?
When? How? For whom?
How learning analytics and
computer tailored
communications enable us to
provide individualized
feedback, encouragement, and
advice to thousands of
students
6/7/2012 Honors Advisory Council
Tim McKay, University of Michigan Departments of Physics and Astronomy, LSA Honors Program

2. Learning Analytics and Knowledge
• Education systems generate • Learning Analytics: the
a rich stream of increasingly Better-Than-Expected
accessible data which can project for introductory
inform teaching and learning physics
• I’m from big data cosmology, – Generates actionable
how 100s of millions of intelligence
galaxies get to be the way • An intervention: adaptation
they are. We’re exploring of PH computer tailored
what data tells us about how communication in E2Coach
students get to be the way – Acts on the intelligence
they are provided by LA
Our E2Coach application – grown from analytics, aware of general
information like identity and goals, reaching into real time, content
6/7/2012 specific student performance data
Honors Advisory Council

3. Better-than-expected in Physics
• LA investigation of two • Admissions information
year-long intro physics – High school GPA
sequences – SAT and ACT
– State and Country of origin
• 48,579 students over 14 – First generation and SES
years – Gender
• Institutional data => • Internal UM information
construct predictions of – Cumulative GPA
student outcomes – Number of credits: UM and
transfer
• Identify those who do – Exam scores
better (and worse) than – Homework grades
expected: Find out why – Final grade in this course
Much more data is available…
6/7/2012 Honors Advisory Council

4. Essential findings from BTE
• Student grades can be • Significant performance
predicted with half disparities are apparent
letter grade accuracy – Gender: especially
– Incoming UM GPA the strong in courses where
most powerful predictor female students are
– Weak additional seriously
information in SAT/ACT underrepresented
Math – First generation college
students
• There is real dispersion:
– Students from low socio-
students do better (and economic status
worse) than expected households
6/7/2012 Honors Advisory Council

5. One-to-one line…
One sigma dispersion
around the mean for
each bin
Mean and error
on the mean for
each bin
6/7/2012 Honors Advisory Council

6. Exploring BTE/WTE
Learning analytics make all of
Male students Non first-gen students
these explorations possible, even
the qualitative ones. They
Female students First-gen students
provide actionable intelligence.
Gendered performance
Performance disparity seen for
first-generation college
disparity seen in intro students, also for low SES
physics courses nationwide students…
6/7/2012 Honors Advisory Council

7. What to do with actionable
intelligence?
• We now have John • Options for response:
Campbell’s ‘obligation – Tell someone and let
of knowing’: how we them act:
expect them to student, instructor, advis
ors: scale remains a
perform, and what
challenge
leads to success
– Develop tools which act
• How can we tailor our directly in response to
approaches and student state:
interactions to optimize • Intelligent tutors etc.
the success of all? • Computer tailored
communication systems
6/7/2012 Honors Advisory Council

8. Tailoring is well established and tested in public health, and has seen major
commercial application. An extensive body of peer-reviewed research reports on
tests of efficacy in design across interventions ranging from smoking cessation and
diabetes control to cancer treatment decision making and depression. This research
provides a strong base for the design of new computer tailored interventions
6/7/2012 Honors Advisory Council

9. MTS was built by the University of Michigan’s Center for
Health Communications Research, an established leader in
computer tailored public health interventions. MTS is a
mature, fully open-source software system for computer
tailored communication.
6/7/2012 Honors Advisory Council

10. E2Coach:
tailored support for • Three groups of players:
physics students – Department of Physics
– CHCR leadership and staff
• Used LA and MTS to – Consultants from across
construct “E2Coach”: an the campus
Electronic Expert • Project goals:
coaching system for intro – Improved performance and
physics courses affect for all students
• You can find a basic – Reduced disparities
information about the The E2Coach team:
project online: Tim McKay, Kate Miller, Jared Tritz, Gus
Evrard, Dave Gerdes in Physics
http://sitemaker.umich.edu/ecoach Vic Strecher, Ed Saunders, Holly
6/7/2012 Honors Advisory Council Derry, Mike Nowak at CHCR

11. How does E2Coach work?
Where the real Expertise of hundreds of
effort lies students, dozens of instructors
and behavior change experts Individually
Detailed personalized
information messages:
about what we all
thousands
of students
and their
MTS agree we
would say to
each
current student, if
The Michigan Tailoring System: a mature
status open-source software system for only we
creating content designed specifically could…
for an individual based on data about
that individual
6/7/2012 Honors Advisory Council

12. Expertise and Information
• Structured interviews • Knowledge of each
with faculty course and its structure
• Survey of 70+ student • Real-time input from
study group leaders the course gradebook
• Better-than-expected • Input from the student
interviews – Background, goals and
• Input from students interests, planned
effort, desired and
with different expected grades, self-
backgrounds has efficacy, confidence in
extreme relevance! physics
• Opt-In: 54% (953 total)
6/7/2012 Honors Advisory Council

13. What we provide
• Tailored advice on all aspects of the
course, including testimonials from relevant
peers
6/7/2012 Honors Advisory Council

14. Performance
feedback
6/7/2012 Honors Advisory Council

15. First measures of impact
• First term ended ten • Testing in fall using
days ago: final scores fractional factorial design
for enrolled students
2.3% (4 ) higher
Actual
• Currently examining: Score differences
observed in 104 measured
– effects vs. usage random samples score
difference
– Disparities on
gender, SES, first-gen
status
• This is a complex
intervention, with many
parts: which are key?
6/7/2012 Honors Advisory Council

16. E2Coach in the LA landscape…
Computer
tailored
communication
SOLAR: Open
Learning
Analytics: an BTE project
integrated & and other
modularized analytics
platform
Siemans, G., et al. July 2011
http://solaresearch.org
6/7/2012 Honors Advisory Council

17. Where is tailoring headed?
• Redesigning for Physics • Ultimately, the interface
in the fall between the student
– Full enrollment and the University’s
– Much tighter approach information systems
• Expanding to situations should be tailored
w/diverse student – Advising
bodies – Registration
– Intro Stats 250 – Feedback
– Epidemiology: Masters – Research and Study
Abroad
– Other STEM disciplines
– Careers
6/7/2012 Honors Advisory Council

18. Larger picture: Learning Analytics
• We live at the dawn of
big data: information
recorded intentionally
and inadvertently
• Analytics reduce big
data to actionable
intelligence
• Explosion in the use of
data to personalize
technology
interactions
6/7/2012

19. Learning Analytics at Michigan
• Provost Phil Hanlon has • Some strong heritage:
empowered a Learning the ART system (2003)
Analytics Task Force • UM Data Warehouse:
• Charge: big, rich set of data
– Improve information providing many
environment for LA opportunities for
– Support LA projects with research
funding cycles • Some examples of
– Review institutional potential projects
metrics used for
teaching and learning follow
Part of the University of Michigan Third Century Initiative
6/7/2012 Honors Advisory Council

20. Tim McKay
The seminars Steve Lonn
Stephanie Teasley
• Brought together people http://sitemaker.umich.edu/slam
interested in understanding • Seminars will continue in fall
our academic mission under the auspices of the
through analysis of data Learning Analytics Task Force
6/7/2012 Honors Advisory Council

21. Extensions to ART: Honors/CSP
student life course project
• Starting to build tools to • Selected groups could be:
provide access to – Students in a program like
information about Honors, the Residential
cohorts of students College, or CSP
– Students entering with an
• Useful for program interest
evaluation of many kinds – Students departing with a
• Will include multiple concentration
methods for identifying • Outcomes: anything
comparison groups measured in the data –
matched on different GPA, courses, majors, pro
criteria gress
Joint project of the Honors and Comprehensive Studies
Programs (McKay, Noori, Green, & Williamson)
6/7/2012 Honors Advisory Council

22. Testing the impact of Honors
Zac Nichol: joint
employee of Honors
and CSP, coding the
database applications
and web interface for
the PART project.
Funding comes from a
proposal we wrote to
the LSA IT committee.
6/7/2012 Honors Advisory Council

23. Testing the impact of Honors
6/7/2012 Honors Advisory Council

24. Testing the impact of Honors
6/7/2012 Honors Advisory Council

25. Testing the impact of Honors
6 Honors, 1 LSA
2 Honors, 21 LSA
12 Honors, 22 LSA
89 Honors, 60 LSA 12 Honors, 3 LSA
12 Honors, 13 LSA
6/7/2012 Honors Advisory Council

26. Many, many other measurements
• Incoming GPA/Grade • Teacher and technique
correlation across impact: we can test
campus whether changes in
• Influence of association: method lead to
between improved performance
roommates, among • Secular changes:
halls, in particular technology has altered
dorms the classroom, how are
• Admissions efficacy: student outcomes
how are criteria related evolving?
to outcomes?
Three years from now: anyone with a question about teaching and learning at UM that
can be addressed with University data should be able to do this in a prompt and accurate
6/7/2012 way.
Honors Advisory Council