Keynote Talk at #SALTISE2019 on conference theme of "Promoting Deeper Learning: From Analytics to New Strategies"

1. learn.center.nyu@gmail.com@NYU_LEARN
The Analytic Future
of (Science) Education
Alyssa Friend Wise, PhD
Associate Professor of Educational Technology
Director, Learning Analytics Research Network

2. LEARNING
ANALYTICS

3. 0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
0
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
1
1
1
0
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
1
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
0
0
1
1
0
1
0
1
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
0
1
1
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
1
0
1
0
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
1
0
0
1
0
0
1
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
0
1
1
0
0
1
1
1
0
1
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
0
1
0
1
0
1
0
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
0
1
0
0
1
0
1
1
1
0
1
0
0
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
0
1
0
1
1
0
0
1
1
0
1
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
0
1
1
0
1
0
1
0
0
0
1
1
0
0
1
0
1
0
1
0
1
0
0
1
0
1
1
0
1
1
1
1
0
1
1
0
0
1
1
0
1
1
0
1
0
0
1
0
1
0
1
1

4. Sub UniquePostsRead()
For k = 1 To MaxUser Step 1
RowCount = Range("A1").CurrentRegion.Rows.Count
For w = 1 to MaxWeek Step 1
StartTime = Sheets("Week").Cells(w + 1, 2)
EndTime = Sheets("Week").Cells(w + 1, 3)
PostNum = 0
PostsIndex = 0
Do While Cells(i, datestamp) <= EndTime And i <= RowCount
If Cells(i, Source) = “Read" Then
If Cells(i, Message_Author) <> Val(ActiveSheet.Name)
And Cells(i, Scan) <> "X" Then
flag = 0
For j = 1 To PostsIndex Step 1
If Posts(j) = Cells(i, Message_Id) Then
flag = 1
j = PostsIndex
End If
Next j
If flag = 0 Then
PostsIndex = PostsIndex + 1
Posts(PostsIndex) = Cells(i, Message_Id)
End If
End If
End If
Sheets(“Stats").Cells(Line, 22) = PostsIndex
Next w
Next k
End Sub
PercentPostsRead =SUniquePostsRead
TotalPostNumber

6. Image Credit: Lumen Learning

7. Image Credit: Spear Education

8. CC BY 2.0 Adapted from Black The Fall Review - A Bleak, Dystopian Future by Bago Games on Flickr

9. CC BY-SA 2.0 Adapted from At the Computer by Lars Plougmann on Flickr

10. ©Slezak Courtesy of NYU Photo Bureau

11. ©Slezak Courtesy of NYU Photo Bureau

12. ©Slezak Courtesy of NYU Photo Bureau

13. Learning Analytics
is the development and application
of data science methods to the
distinct characteristics, needs, and
concerns of educational contexts
for the purpose of
better understanding &
supporting learning

14. Learning Analytics
is a critical emerging technology of
the 21st century, with high
expectations of short- and long-
cycle positive impacts on learning
and teaching

15. NUMBER OF
INDIVIDUALS
WHAT’S
NEW?
IMMEDIACY
OF ACCESS
Image Credits: anthony l, andrew revkin, parameter_bond, nullfy and hjl via Flickr

16. Instructors
Design
Learners
Engage
Data
Generated
Analytics
Produced
Human
Insight
Learning
Analytics
Cycle

17. Instructors
Design
Learners
Engage
Data
Generated
Analytics
Produced
Human
Insight
Learning
Analytics
Cycle

18. Image Credit: UGA College of Agriculture and Environmental Sciences via Flickr (CC BY 2.0), adapted

19. Image Credit: UGA College of Agriculture and Environmental Sciences via Flickr (CC BY 2.0), adapted

24. Instructors
Design
Learners
Engage
Data
Generated
Analytics
Produced
Human
Insight
Learning
Analytics
Cycle

25. DEMOGRAPHICS ( W H O P E O P L E A R E )
PERFORMANCE ( H O W T H E Y ’ V E D O N E )
ACTIVITY ( T H I N G S P E O P L E D O )
LO G - F I L E S , P H Y S I C A L T R A C K S , S E L F - R E P O R T
ARTIFACT ( T H I N G S P E O P L E C R E AT E )
P R O B L E M A N S W E R S , W R I T T E N E X P L A N AT I O N S
ASSOCIATION ( C O N N E C T I O N S P E O P L E M A K E )
W H O A N D W H AT T H E Y I N T E R A C T W I T H
DATA
3A’S
WISE (2019)
HOPPE (2015)

26. FOR LEARNERS’
EFFORT
ENGAGEMENT
INTERACTION
KNOWLEDGE
SKILL
EMOTION
VALUES
DATA
IS A
PROXY
WISE (2017)

27. S I N H A E T A L . ( 2 0 1 4 )
THE KEY IS
TO CONNECT
LOW-LEVEL
BEHAVIORS
WITH
HIGH-LEVEL
CONSTRUCTS
F L I P P E D C L A S S V I D E O V I E W I N G

28. S I N H A E T A L . ( 2 0 1 4 )
THE KEY IS
TO CONNECT
LOW-LEVEL
BEHAVIORS
WITH
HIGH-LEVEL
CONSTRUCTS
Raw
Clicks
Aggregate
Features
Critical
Concept
Info
Process.
Play
SeekFwd
ScrollFwd
RateFast
Skipping Disengaged Low
Play
Pause
SeekBw
SeekFwd
Checkback Searching
for specific
info
Med
Play
Pause
SeekBw
Rewatch Reviewing
content
High
Play
Pause
SeekBw
ScrollBw
Clarify
Idea
Tussling
with
content
Very High
F L I P P E D C L A S S V I D E O V I E W I N G

29. Instructors
Design
Learners
Engage
Data
Generated
Analytics
Constructed
Human
Insight
Learning
Analytics
Cycle
Teaching Actions

30. DATA MINING
Image Credit: Scott Clark via Flickr (CC BY 2.0), adapted

31. DATA GEOLOGY
Image Credit: APS Museum via Flickr (CC BY 2.0), adapted
( S H A F F E R , 2 0 1 3 )

32. DATA ARCHEOLOGY
Image Credit: Pedro Szekely via Flickr (CC BY 2.0), adapted
( W I S E , 2 0 1 4 )

33. MEANINGFUL VARIABLES TO
CONSTRUCT + INCLUDE
POTENTIAL CONFOUNDS,
SUBGROUPS, OR COVARIATES
WHICH RESULTS TO ATTEND TO,
WHAT THEY MAY MEAN, WHERE
THEY MAY GENERALIZE TO
HOW TO TAKE ACTION BASED
ON OUTCOMES
W I S E & S H A F F E R ( 2 0 1 5 )
THEORY
GIVES
GUIDANCE

34. ONLINE DISCUSSION
LEARNING MODEL
Externalizing one’s
ideas by contributing
posts to an online
discussion
Taking in the
externalizations of
others by accessing
existing posts
• Social constructivist perspective - online discussions as a forum
for learning through conversation
• Students learn as they articulate their ideas, are exposed to the
ideas of others, and negotiate differences in perspective
• Focus on how students contribute comments (“speak”) and
attend to other’s messages (“listen”)

35. Criteria Metric Definition
Listening
Breadth
Percent of posts viewed
Number of unique posts that a student viewed divided
by the total number of posts in the discussion
Percent of posts read
Number of unique posts that a student read divided by
the total number of posts in the discussion
Listening Depth Percent of real reads
Number of times that a student read a post divided by
the total number of times they read and viewed posts
Listening
Reflectivity
Number of reviews of
own / others’ posts
Number of times a student revisited posts that they had
made / viewed previously in the discussion
Conversational
Distribution
Posts made / viewed
throughout discussion
Dispersion or concentration of posts made / viewed by a
student in the discussion space
Speaking
Quantity
Number of posts
Total number of posts a student contributed to the
discussion
Average post length
Total number of words posted by a student divided by
the number of posts they made to the discussion
ONLINE DISCUSSION
LEARNING MODEL ANALYTICS

36. Listening Reflectivity
• Reviewing others’ posts multiple times predicts greater
responsiveness
Listening Depth
• A greater % of real reads predicts richer argumentation
(Informed) Listening Breadth
• Reading a greater % of posts and viewing a greater % of posts
than those read predicts richer argumentation
ONLINE DISCUSSION
LEARNING INSIGHTS

37. EMBEDDED LISTENING ANALYTICS

38. Metric Your Data
(Week X)
Class Average
(Week X)
Observations
Range of participation 4 days 5 days
# of sessions 6 13
Average session length 33 min 48 min
% of sessions with posts 67% 49%
# of posts made 8 12
Average post length 386 words 125 words
% of posts read 42% 87%
#of reviews of own posts 22 13
#of reviews of others’ posts 8 112
EXTRACTED LISTENING ANALYTICS

39. CHANGES OVER TIME
“I found that I wanted the challenge of trying to up the % of overall posts
that I reviewed each week. This also meant slowing down my reading since
the data would not record a quick read of the information. The overall result
was that I think I learned more and was able to get a broader sense of
opinion concerning the readings.”

40. Adapted from Galaxy by Image by Gerd Altmann from Pixabay
professional
clinical
becoming
dentistry
Multidimensional
Word Embedding
Space of
Dental Student
Reflective Writing

42. best
possible
needs
patient
patients
treatment
procedurestreat
care
provide
need continue
learn
improve
ability Patient Care
Continued Growth

43. competent
goal
dentist
become
dentists
us
must
responsibility
respect
trust
professionals
Collective Identity
Personal Goal

44. Instructors
Design
Learners
Engage
Data
Generated
Analytics
Constructed
Human
Insight
Learning
Analytics
Cycle
Humans
Machines
Teaching ActionsGoal & Strategy Setting

45. Interpret Data
Sense-Making Pedagogical Response
Get Oriented/
Focused Attention
Find Absolute & Relative
Reference Points
Read Data
Triangulate
Contextualize
Make Attribution
Explain Pattern
AFFECTIVE PROCESSES
Area of
Curiosity
Question
Generation Wait-and-See
Reflect on
Pedagogy
Check
Impact
Take Action
Whole-Class Scaffolding
Targeted Scaffolding
Revise Course Design
Teaching with Analytics

46. Pedagogical
Questions
Are my students
preparing?
Which of my
students need
help?
How are my
students thinking
about STEM?
Data-Based
Answers
Student interaction
grid
Predictive
modelling
Concept network
examination
Educational
Action
Emphasize
important
resources
Offer help in a
targeted manner
Evaluate / update
curricular design

47. Are My Students Preparing?

48. When Are My Students Preparing?

49. Are My Students Engaging?
Student Wk2 Wk3 Wk4 Wk5 Wk6
95551286 0 0 3 2 1
67532604 3 4 1 6 8
21510565 3 12 6 0 9
54578847 0 4 0 6 4
35178942 4 3 7 13 6
24782487 1 13 13 10 8
76003679 4 11 6 16 9
81153801 7 9 7 6 10
24349293 13 10 12 14 8
96009678 12 24 4 6 7
13983875 12 18 24 26 14
14183090 13 25 14 23 15

50. Who Are My Students Engaging With?
Always Same People Always Different People
My Actual Class
Avg Degree = 3
Modularity = .81 Avg Degree = 10
Modularity = .14
Avg Degree = 9
Modularity = .27

51. How Are My Students Engaging?

52. StatMed’13
StatMed’14
StatLearn
YBW
PSY
Course Subject Learning Process Question Words Connectors Existence/Condition Course TasksQuality/Quantity Effort / Action People Appreciation/Greeting
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Content - Related
NonContent - Related
Numberoffeatures Content-Related
Discussion Posts
Non-Content-Related
Discussion Posts
Question Words + Connectors
e.g. “can” “does” “why” “how”
“which “and” “of” “than” “is”
Course Tasks + People
+ Appreciation / Greetings
e.g. “answer” “exam” “course”
“lecture” “thank” “good” “I” “my”
Top Feature Distribution by Category

53. How am I Facilitating Interaction?
U1
• Responses at all levels
• Coaching and supporting
• Social presence cues
“That is correct -
Nice! So how
would you use
this to solve the
question?”
“A bell shape is
not necessary.
You could have
a bimodal
distribution”
U417
• Responses only to top level
• Straight forward answers
• Little social presence
Degree = 4.4
Weight = 2.2
Degree = 2.8
Weight = 1.8

54. SAT Math
Score
Diagnostic
Test
High School
GPA
Race /
Gender Drop, Fail
Withdraw
Grade < C+
Drop, Fail
Withdraw
Grade < C+
Which of My Students Will Need Help?
Introductory Calculus as a Challenging Gateway Course
Beyond identifying who is likely to need help, the model
indicates that the kind of help is support for algebra skills

55. How Are My Students Thinking About…?
D1 D4
Becoming a Dentist
Start D1 End D4Start D3

56. How Are My Students Thinking About…?
D1 D4
Becoming a Dentist
Start D1 End D4Start D3
“I started dental school with the focus of
wanting to become an excellent healthcare
provider for my patients. I did not fully
understand what that meant until D3 and
D4 year, when I had my patients' oral
health needs in my hands.”

57. How Are My Students Reflecting?

58. The Six Elements of Reflection (Wise, Cui & Allen, 2019)
Reflection
Element
Definition
Description Student indicates what they observed about a learning event
Analysis
Student makes sense of the learning event and/or why something
occurred
Feeling Student describes affective reaction to the learning event
Perspective
Student demonstrates change in their own perspective or consideration
of other perspectives
Evaluation Student describes what was good and bad about the learning event
Outcome Student describes lessons learned and future intentions / plans

59. D1 D2 D3 D4
Start End Start End Start End Start End
Description 22% 72% 56% 77% 68% 70% 76% 81%
Analysis 9% 43% 39% 41% 43% 42% 43% 60%
Feeling 12% 30% 28% 44% 33% 40% 35% 44%
Perspective 62% 31% 67% 49% 67% 65% 65% 72%
Evaluation 13% 58% 39% 55% 57% 58% 60% 71%
Outcome 88% 91% 83% 96% 93% 94% 92% 95%
Depth 2% 5% 6% 8% 14% 11% 16% 26%

60. D1 D2 D3 D4
Start End Start End Start End Start End
Description 22% 72% 56% 77% 68% 70% 76% 81%
Analysis 9% 43% 39% 41% 43% 42% 43% 60%
Feeling 12% 30% 28% 44% 33% 40% 35% 44%
Perspective 62% 31% 67% 49% 67% 65% 65% 72%
Evaluation 13% 58% 39% 55% 57% 58% 60% 71%
Outcome 88% 91% 83% 96% 93% 94% 92% 95%
Depth 2% 5% 6% 8% 14% 11% 16% 26%

61. D1 D2 D3 D4
Start End Start End Start End Start End
Description 22% 72% 56% 77% 68% 70% 76% 81%
Analysis 9% 43% 39% 41% 43% 42% 43% 60%
Feeling 12% 30% 28% 44% 33% 40% 35% 44%
Perspective 62% 31% 67% 49% 67% 65% 65% 72%
Evaluation 13% 58% 39% 55% 57% 58% 60% 71%
Outcome 88% 91% 83% 96% 93% 94% 92% 95%
Depth 2% 5% 6% 8% 14% 11% 16% 26%

62. Personal
Public
Personal
Private
Professional
Public
Professional
Private
Progress Ethics
Description 54% 68% 60% 74% 92% 69%
Analysis 37% 48% 37% 49% 63% 35%
Feeling 35% 46% 22% 31% 57% 22%
Perspective 78% 61% 67% 64% 45% 74%
Evaluation 42% 56% 49% 57% 86% 51%
Outcome 99% 98% 85% 94% 96% 85%
Depth 13% 17% 9% 15% 26% 15%

63. Personal
Public
Personal
Private
Professional
Public
Professional
Private
Progress Ethics
Description 54% 68% 60% 74% 92% 69%
Analysis 37% 48% 37% 49% 63% 35%
Feeling 35% 46% 22% 31% 57% 22%
Perspective 78% 61% 67% 64% 45% 74%
Evaluation 42% 56% 49% 57% 86% 51%
Outcome 99% 98% 85% 94% 96% 85%
Depth 13% 17% 9% 15% 26% 15%

64. Personal
Public
Personal
Private
Professional
Public
Professional
Private
Progress Ethics
Description 54% 68% 60% 74% 92% 69%
Analysis 37% 48% 37% 49% 63% 35%
Feeling 35% 46% 22% 31% 57% 22%
Perspective 78% 61% 67% 64% 45% 74%
Evaluation 42% 56% 49% 57% 86% 51%
Outcome 99% 98% 85% 94% 96% 85%
Depth 13% 17% 9% 15% 26% 15%

65. Personal
Public
Personal
Private
Professional
Public
Professional
Private
Progress Ethics
Description 54% 68% 60% 74% 92% 69%
Analysis 37% 48% 37% 49% 63% 35%
Feeling 35% 46% 22% 31% 57% 22%
Perspective 78% 61% 67% 64% 45% 74%
Evaluation 42% 56% 49% 57% 86% 51%
Outcome 99% 98% 85% 94% 96% 85%
Depth 13% 17% 9% 15% 26% 15%

66. Developing a Model to Automatically Assess Reflective Qualities
The Six Elements of Reflection
Reflection
Element
Linguistic Features (using LIWC)
Model
Accuracy
Description I, we, see, hear, feel, past focus 77%
Analysis
Analytic index, insight, cause, discrepancy,
tentative, certainty, difference
64%
Feeling
I, we, affect, positive emotion, anxiety, anger,
sadness
67%
Perspective She, he, they 60%
Evaluation - N/A
Outcome Analytic index, I, future focus 93%

67. How Are My Students Thinking About…?
Failing an Competency Assessment
Top Words
Overall

68. Analytics in Physical Spaces (by Xavier Ochoa @ NYU)
• Provides an automated
report to students
about their basic oral
presentations skills
• Such as: looking at the
audience, body
posture, speaking
volume and pauses
• Tool for preparing for
class presentations (not
automated assessment)

69. Image Credit: Dakotilla via Flickr (CC BY 2.0)
The clock is a powerful machinery that created
the product of “seconds” and “minutes”
and thus changed our relationship to time
(Mumford, 1934)

70. Image Credit: Dakotilla via Flickr (CC BY 2.0)
How will analytics change our
relationship with our students and
their relationships to (active)
learning process?
How can data inform (w/o dictating)
our pedagogical decisions?
How can our pedagogical decisions
generate better data?
What dangers must we be on the
watch for?
Learning analytics is a powerful machinery that
creates new products of data that change our
relationship to teaching and learning

71. Image Credit: Dakotilla via Flickr (CC BY 2.0)
Ownership, Consent & Choice
Agency & Context
Transparency & Accountability
Privacy & Surveillance
The Right to Be Forgotten
The Creation of New Labels
Learning analytics is a powerful machinery that
creates new products of data that change our
relationship to teaching and learning

72. Learning Analytics at NYU
NYU Learning Analytics is a collaborative effort, focused on
community change that puts people, not data, first
Key Characteristics
• Partnerships between IT, faculty, administrators and students
• User-Centered Design involving stakeholders from the start
• Scalability to serve a large university with 10+ global
campuses and a diverse international student body
• Research to innovate and build a knowledge base for data-
informed teaching and learning in higher education

73. What is NYU-LEARN?
• The Learning Analytics Research Network -> An innovative
new model for synergistic learning analytics research and practice in
the university.
• A supportive and inclusive university wide community
of faculty, instructors, technologists -> United with the
common goal of building the future of data-informed teaching and
learning research at NYU
• A collaborative partner -> For those who want to incorporate
learning analytics into existing research or develop new efforts

74. Predicting
Calculus Success
• Can detect “students
who will struggle”
• Working on effective
intervention strategies
Instrumented
Learning Spaces
• NSF CIRCL subaward to
hold working group
• Generate shareable
multistream data corpus
Many Local Projects
NYU Learning Analytics
Instructor’s Use
of Dashboards
• Documenting practices
of use -> support model
• Fed back to influence
tool design + support
Presentation
Feedback Tool
• Automatic feedback to
student or oral
presentation skills
• Introduce to pedagogy
Dental Reflection
Analytics
• Detecting rise of
patient-centeredness
• In conversation about
impact and action
Interpret Data
Sense-Making Pedagogical Response
Get Oriented/
Focused Attention
Find Absolute & Relative
Reference Points
Read Data
Triangulate
Contextualize
Make Attribution
Explain Pattern
AFFECTIVE PROCESSES
Area of
Curiosity
Question
Generation Wait-and-See
Reflect on
Pedagogy
Check
Impact
Take Action
Whole-Class Scaffolding
Targeted Scaffolding
Revise Course Design
Sense-Making Response
D1 D4

75. NYU Learning Analytics
Three Global Services
Images used under CC0 1.0 and CC BY-NC-ND 2.0

76. What information
will learning
analytics
provide?
NYU Learning Analytics
Ongoing Community Conversation
©Slezak Courtesy of NYU Photo Bureau
What will learning analytics
be able to do?
How will learning
analytics play a role
in students’ learning?How will
learning
analytics play
a role in
instructor’s
teaching?
What is our shared
vision for learning
analytics at NYU?

77. TO LEARN MORE
https://learning-analytics.info
2019 - 6(2) – Human Centered Learning Analytics
2018 - 5(1) – Temporal Learning Analytics
2017 - 4(2) – The Shape of Educational Data
2016 - 3(2) – Multimodal Learning Analytics
2016 - 3(1) – Ethics & Privacy in Learning Analytics
2015 - 2(2) – Learning Analytics & Learning Theory
2015 - 2(1) – Self-Regulated Learning & Analytics
Handbook of Learning Analytics
https://solaresearch.org/hla-17/
Proceedings of LAK’11-LAK’19
https://dl.acm.org
Publications at tinyurl.com/wise-scholar

78. learn.center.nyu@gmail.com@NYU_LEARN
The Analytic Future
of (Science) Education
Alyssa Friend Wise, PhD
Associate Professor of Educational Technology
Director, Learning Analytics Research Network