Slides from Daltai-supported seminar to explore the role of continuous professional development in supporting staff & student engagement with learning analytics.

1. Evidence Based Decision Making in the Classroom
Evolution of Our Understanding of
How it Happens, the Challenges Involved
& Approaches to Design and Support
Alyssa Wise
Data Informed

2. Trajectory of Our Work
Teaching and Student Facing Learning Analytics
van Leeuwen, Teasley & Wise (Handbook of Learning Analytics 2nd ed, forthcoming)
Engaging Students as Co-Designers of Learning Analytics
Sarmiento, Campos & Wise (LAK Conference 2020)
Teaching with Analytics
Wise & Jung (Journal of Learning Analytics 2019)
Designing Actionable Educational Data Visualizations
Wise & Jung (LAK Conference 2019)
Using Data-Informed Decision-Making to Improve Teaching and Learning
Wise (Contemporary Technologies in Education 2019)
Learning Analytics Implementation Design
Wise & Vytasek (Handbook of Learning Analytics 1st ed, 2017)
The Student Tuning Model and Align Design Framework for Learning Analytics Use
Wise, Vytasek, Hausknecht & Zhao (Online Learning, 2016)
Designing Pedagogical Interventions to Support Student Use of Learning Analytics
Wise (LAK Conference 2014)
2020
2019
2017
2016
2014
Discussion Analytics
Dashboard 2.0

3. Who?

4. Who?

5. Who?

6. Who?

7. How it Happens
Interpret Data
Sense-Making Pedagogical Response
Get Oriented/
Focused Attention
Find Absolute & Relative
Reference Points
Read Data
Triangulate
Contextualize
Make Attribution
Explain Pattern
AFFECTIVE PROCESSES
Area of
Curiosity
Question
Generation Wait-and-See
Reflect on
Pedagogy
Check
Impact
Take Action
Whole-Class Scaffolding
Targeted Scaffolding
Revise Course Design
Sense-Making Pedagogical Response
Model of Instructor Analytics Use
(based on data from 2017-2018)

8. How it Happens
Evolution of Instructor Analytics Use
(based on data from 2019-2020)
Data Conceptualized Broadly, Always Situated in Context of Students, Design, Etc.
Increase in Confidence, but Continued Expert-Novice Tensions
Range of Driving Pedagogical Goals
• Understand if and how students are engaging
• Identify students struggling or w/ bad habits (e.g., procrastination and cramming)
• Look for areas which students focus on in the content
• Inform future iterations of the course
• Prioritize time and effort
Range of Use Approaches / Strategies
• Narrow and focused vs. Broad and exploratory
• Comparisons: w/ hypothesis or desired behaviors, across time, materials, students
• Timing: Check just before class, when assignments due, end of semester debrief

9. Challenges Involved
Getting Started
• Finding the impetus (and time) to look at the data
• Navigating to find the information wanted
• Finding out the course structure doesn’t give the right data
Taking Action
• Connecting abstract data forward
to concrete activity
• When do we know enough to act
• The big brother problem
(sometimes)
• How do we know what works
Interpretation
• Connecting data to questions they
care about
• What good looks like
• Trust and validity
• Mismatches between data and needs
• One size does not fit all
• Changing needs over time
• The snapshot problem
• The bookmark problem

10. Challenges Involved
Getting Started
• Finding the impetus (and time) to look at the data
• Navigating to find the information wanted
• Finding out the course structure doesn’t give the right data
Taking Action
• Connecting abstract data forward
to concrete activity
• When do we know enough to act
• The big brother problem
(sometimes)
• How do we know what works
• The “helicopter analytics”
problem
Interpretation
• Connecting data to questions they
care about
• What good looks like
• Trust and validity
• Mismatches between data and needs
• One size does not fit all
• Changing needs over time
• The snapshot problem
• The bookmark problem
• Knowing pedagogical intent (what is
expected)

11. (New) Approaches To Design
Process Design
• Visual aids for finding entry points
• Support for working with reference points
• Features for question generation & maintenance
• Flags for later interpretation and action
Pedagogical Design
• End (or start) of semester reviews
• Just-in-time pre-class checks
• Pro-active reminders
• Mid-year checkpoints
Can also be
supported
with PD
outside of
the tool

12. (New) Approaches To Design
Process Design
• Visual aids for finding entry points
• Support for working with reference points
• Features for question generation & maintenance
• Flags for later interpretation and action
Pedagogical Design
• End (or start) of semester reviews
• Just-in-time pre-class checks
• Pro-active reminders
• Mid-year checkpoints
Holistic Design
• Distribution: What kind of information to give to whom when?
• Symmetry: Do teachers and students get parallel information?
• Transparency: Do teachers and students know what the other sees?
• Mediational: How to position analytics as a third “voice” (de-identified views)
Participatory Design
• Involve instructors and students meaningfully in the creation process
Can also be
supported
with PD
outside of
the tool

13. Approaches To Support
Iterative Approach
• Introduction to tools
• Use case demonstrations
• Opportunity to explore with help ready at hand
• Explore on own, but come back with questions
• Often need a full cycle to see things play out
Develop Routines
• Find times to incorporate data use into daily or weekly routines
• Identify primary purpose(s) of use
• Create clear chains of question, answers and actions
• Develop straightforward processes to get need data needed quickly
Build Community
• Collaborative interpretation is powerful
• Social support for increased need but reduced time with remote learning

14. ALYSSA WISE
alyssa.wise@nyu.edu
@alywise
Thank You
@NYU_LEARN
https://nyu.edu/learn-analytics
Contributing LEARN
Team Members
Yeonji Jung
PhD student
Fabio Campos
PhD student
JP Sarmiento
PhD student
Qiujie Li
Post-Doctoral Researcher