The document discusses the role of educational technologies like learning analytics and artificial intelligence. It provides examples of how learning analytics can be used to analyze academic data to gain insights about difficult courses, dropout paths, and the relationship between courses. This allows universities to identify issues and redesign programs. It also discusses using learning analytics to build tools like academic advising dashboards that provide personalized recommendations to students about course loads. While artificial intelligence can provide automated feedback at scale, the quality of feedback is still limited and human judgment remains important.

1. Educational Technologies:
Learning Analytics
and Artificial Intelligence
Xavier Ochoa
Assistant Professor of Learning Analytics
Learning Analytics Research Network (LEARN)
School of Culture, Education and Human Development
New York University (NYU)

2. http://www.slideshare.net/xaoch

3. The role of Educational
Technologies

5. Profession
Isaac Asimov (1957)

7. Augmenting human capabilities
in educational contexts

8. Freeman, A., Adams Becker, S., Cummins, M., Davis, A., and Hall Giesinger, C. (2017). NMC/CoSN Horizon Report: 2017 K–12 Edition.
Austin, Texas: The New Media Consortium.
Bryan Alexander, Kevin Ashford-Rowe, Noreen Barajas-Murphy, Gregory Dobbin, Jessica Knott, Mark McCormack, Jeffery Pomerantz, Ryan
Seilhamer, and Nicole Weber, EDUCAUSE Horizon Report: 2019 Higher Education Edition (Louisville, CO: EDUCAUSE, 2019).

9. Freeman, A., Adams Becker, S., Cummins, M., Davis, A., and Hall Giesinger, C. (2017). NMC/CoSN Horizon Report: 2017 K–12 Edition.
Austin, Texas: The New Media Consortium.
Bryan Alexander, Kevin Ashford-Rowe, Noreen Barajas-Murphy, Gregory Dobbin, Jessica Knott, Mark McCormack, Jeffery Pomerantz, Ryan
Seilhamer, and Nicole Weber, EDUCAUSE Horizon Report: 2019 Higher Education Edition (Louisville, CO: EDUCAUSE, 2019).

10. Learning Analytics
Learning analytics is the measurement,
collection, analysis and reporting of data
about learners and their contexts, for
purposes of understanding and
optimizing learning and the environments in
which it occurs.
Society for Learning Analytics Research (SoLAR)

12. Sensemaking
“Sensemaking is a
motivated, continuous
effort to understand
connections . . . in order
to anticipate their
trajectories and act
effectively”
Klein, Gary, Brian Moon, and Robert R. Hoffman. "Making sense of
sensemaking 1: Alternative perspectives." IEEE intelligent systems 21.4
(2006): 70-73.

13. Learning Analytics
=
Augmenting Sensemaking

14. Maier, Karin, Philipp Leitner, and Martin Ebner. "Learning Analytics Cockpit for MOOC Platforms." Emerging Trends in Learning Analytics. Brill Sense, 2019.
271-285.

15. Maier, Karin, Philipp Leitner, and Martin Ebner. "Learning Analytics Cockpit for MOOC Platforms." Emerging Trends in Learning Analytics. Brill Sense, 2019.
271-285.

16. Maier, Karin, Philipp Leitner, and Martin Ebner. "Learning Analytics Cockpit for MOOC Platforms." Emerging Trends in Learning Analytics. Brill Sense, 2019.
271-285.

17. Siemens, George, and Phil Long. "Penetrating the fog: Analytics in learning
and education." Educause Review 46.5 (2011): 30-32.

18. An (simple) example:
Computer Science Program Redesign
Mendez, G., Ochoa, X., Chiluiza, K., & De Wever, B. (2014). Curricular design analysis: a data-driven perspective. Journal
of Learning Analytics, 1(3), 84-119.

19. Computer Science Program Redesign

20. Which are the hardest/more difficult courses?
What lead our students to success/failure?
How courses are related?
Are there courses that could be eliminated?
Is the work-load adequate for our students?

21. Good Old Academic Data
Student Course Section Semester Grade
9093233 HCD001 1 2005-1S 85
9093233 LMS003 2 2005-1S 97
9088442 HCD001 2 2005-2S 100
… … … … …

22. Difficulty Estimation
How difficult a course is?

23. Course Difficulty Estimation
grade > GPA
grade < GPA
0
grade = GPA
Three scenarios:
Differences between
GPA and grade
> 0< 0

24. But…
Why do these distributions look like this?
They are skewed!

25. Real examples

26. Difficult Courses (Top 10)
Perceived Estimated
Algorithms Analysis
Operating Systems
Physics A
Differential Equations
Linear Algebra
Programming Fundamentals
Object-Oriented Programming
Differential Calculus
Data Structures
Statistics
Operating Systems
Statistics
Differential Equations
Linear Algebra
Programming Languages
Electrical Networks I
Artificial Intelligence
Programming Fundamentals
Data Structures
Hardware Architecture and Organization

27. Difficult Courses (Top 10)
Perceived Estimated
Algorithms Analysis
Operating Systems
Physics A
Differential Equations
Linear Algebra
Programming Fundamentals
Object-Oriented Programming
Differential Calculus
Data Structures
Statistics
Operating Systems
Statistics
Differential Equations
Linear Algebra
Programming Languages
Electrical Networks I
Artificial Intelligence
Programming Fundamentals
Data Structures
Hardware Architecture and Organization

28. Perception != Estimation
That lead to a deeper conversation
among Faculty

29. Curriculum Coherence
How courses group together

30. CORE - CS CURRICULUM
Basic Physics
Integral Calculus
Multivariate Calculus
Electrical Networks
Digital Systems I
Hardware Architectures
Operative Systems
General Chemistry
Programming
Fundamentals
Object-oriented
Programming
Data Structures
Programming
Languages
Database Systems I
Software Engineering I
Software Engineering II
Oral and Written
Communication Techniques
Computing and Society
Discrete Mathematics
Algorithms Analysis
Human-computer
Interaction
Differential Calculus
Linear Algebra
Differential Equations
Ecology and
Environmental Education
Statistics
Economic Engineering I
Artificial Intelligence
PROFESSIONAL TRAINING HUMANITIES BASIC SCIENCE

31. Principal Factor Analysis

32. UNDERLYING STRUCTURE
Electrical
Networks
Differential
Equations
Software Engineering II
Software Engineering I
HCI
Oral and Written
Communication Techniques
General Chemistry
Programming
Languages
Object-Oriented
Programming
Data Structures
Artificial Intelligence
Operative Systems
Software Engineering
Object-Oriented Programming
Economic Engineering
Hardware Architectures
Database Systems
Digital Systems I
HCI
Differential and Integral Calculus
Linear Algebra
Multivariate Calculus
Digital Systems I
Basic Physics
Programming Fundamentals
Discrete Mathematics
General Chemistry
Statistics
Data Structures
Computing and Society
Algorithms Analysis
Differential Equations
Ecology and Environmental Education
Object-Oriented Programming
FACTOR 1: The basic training factor
FACTOR 2: The advanced
CS topics factor
FACTOR 3: The client
interaction factor
FACTOR 4: The
programming factor
FACTOR 5: The ? factor

33. Grouping is off
Fundamental Programming is not in the Programming factor?
What to do with Electrical Networks and Differential
Equations?

34. Drop-out Paths
What courses lead the students to
drop-out

35. DROPOUT PATHS
Sequence Support
<Physics A, Dropout> 0.608196721
<Differential Calculus , Dropout> 0.570491803
<Programming Fundamentals , Dropout> 0.532786885
<Integral Calculus , Dropout> 0.496721311
<Physics A, Differential Calculus , Dropout> 0.43442623
<Linear Algebra , Dropout> 0.432786885
<Differential Calculus, Integral Calculus , Dropout> 0.385245902
<Physics C , Dropout> 0.347540984
<Physics A, Integral Calculus , Dropout> 0.327868852
<General Chemistry , Dropout> 0.319672131
<Differential Equations , Dropout> 0.31147541
Sequence Mining
(Sequential PAttern
Discovery using
Equivalence classes -
SPADE)

36. Most drop-outs fail basic courses
Should students start with CS topics?
Too much pressure in engineering courses?

37. Load/Performance Graph
What students think they can
manage vs. what they can actually
manage

39. LOAD/PERFORMANCE GRAPH

40. LOAD/PERFORMANCE GRAPH

41. LOAD/PERFORMANCE GRAPH

42. Unrealistic Suggested Load
How to the present the Curriculum in a better way?
How we can recommend students the right load?

43. Which are the hardest/more difficult courses?
What lead our students to success/failure?
How courses are related?
Are there courses that could be eliminated?
Is the work-load adequate for our
students?

44. What makes a course difficult then?
Why Programming Fundamentals does not correlate?
Why Computers and Society correlates with a lot of courses?
Fundamental Programming is not in the Programming
factor?
Should students start with CS topics?
Too much pressure in engineering courses?
How to the present the Curriculum in a better way?
How we can recommend students the right load?
What to do with Electrical Networks and Differential
Equations?

45. Good Old Academic Data
Student Course Section Semester Grade
9093233 HCD001 1 2005-1S 85
9093233 LMS003 2 2005-1S 97
9088442 HCD001 2 2005-2S 100
… … … … …

46. Another example:
Augmenting Academic Advising
Gutiérrez, F., Seipp, K., Ochoa, X., Chiluiza, K., De Laet, T., & Verbert, K. (2018). LADA: A learning analytics dashboard for
academic advising. Computers in Human Behavior

47. How to better recommend
academic load to different
students
In 15 minutes or less!

48. Basic
Advising
Dashboard

49. Add a student-dependent
load-recommender system
Based on CS-program
data analysis

51. Prediction + Uncertainty

52. Results from Evaluation

53. Results from Evaluation
Amount of explored scenarios increased in
experts, especially for hard-cases.
Detrimental effect on laymen (non-
advisors)

54. What about Artificial
Intelligence?

55. Artificial Intelligence
The theory and development of computer
systems able to perform tasks normally requiring
human intelligence, such as visual perception,
speech recognition, decision-making, and
translation between languages.

56. Not so much….

57. Providing feedback on student writing

58. Feedback is valuable… but costly

59. ACA Writer: Automatic Feedback on Written Essays
Gibson, A., Aitken, A., Sándor, Á., Buckingham Shum, S., Tsingos-Lucas, C., & Knight, S. (2017). Reflective
writing analytics for actionable feedback.

60. ETS Writing Mentor: Automatic Feedback on Written Essays
Madnani, N., Burstein, J., Elliot, N., Klebanov, B. B., Napolitano, D., Andreyev, S., & Schwartz, M. (2018, August). Writing
mentor: Self-regulated writing feedback for struggling writers. In Proceedings of the 27th International Conference on
Computational Linguistics: System Demonstrations (pp. 113-117).

61. Oral presentation feedback

62. Again… feedback is very valuable… but costly

63. Automatic Oral Presentation Feedback System
Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018, March). The rap system:
automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In Proceedings
of the 8th international conference on learning analytics and knowledge (pp. 360-364). ACM.

65. (Multimodal) Formative Feedback Report

66. (Multimodal) Formative Feedback Report

72. Experiment
•Used by 1140 students
•2 times during the semester
•Real classroom presentations
•Students used the system on their own

73. Effect of the System
• Looking at the public: +++
• Presentation posture: +
• Filled pauses: = +++ (For students with low scores)
• Slides: = ++

76. Can we mix LA with AI?

77. Medical Collaboration Feedback
Martinez-Maldonado, R., Power, T., Hayes, C., Abdiprano, A., Vo, T., Axisa, C., & Buckingham Shum, S. (2017, March). Analytics meet
patient manikins: challenges in an authentic small-group healthcare simulation classroom. In Proceedings of the seventh
international learning analytics & knowledge conference (pp. 90-94). ACM.

79. Multimodal Transcript
Ochoa, X. et al. Multimodal Transcript of Face-to-Face Group-Work Activity Around Interactive Tabletops
CrossMMLA Workshop, Learning Analytics and Knowledge conference 2018, In Print

80. Multimodal Transcript

82. Conclusions
and a word of caution

83. LA & AI will
not replace the
Teacher
They will enable the Augmented Teacher

84. LA & AI will not limit the
options of students
They will empower Augmented Students

85. Responsible LA & AI

87. Want to know more…
https://steinhardt.nyu.edu/learn/connectwithus
Xavier Ochoa
xavier.ochoa@nyu.edu
http://wp.nyu.edu/xavier_ochoa
Twitter: @xaoch