This document discusses using text analysis to enhance conversations between students and teachers. It describes text analysis techniques like word frequency, collocations, and statistical/rule-based approaches. Case studies are presented on using text analysis of student work, discussion posts, and evaluations to provide pedagogical insights. The tool Quantext is introduced, which analyzes short student texts to evaluate understanding over time and extract insights. Partnerships are encouraged to further develop these applications of text analysis for teaching and learning.

1. Text analysis in context: enriching the conversation
between students and teachers
JENNY MCDONALD
Honorary Academic, Centre for Learning and Research in Higher Education, University of Auckland
Director, Quantext

2. SESSION GOALS
UNDERSTAND THE PURPOSES OF TEXT ANALYTICS
DESCRIBE AND DEMONSTRATE SOME BASIC TEXT ANALYTIC TECHNIQUES
FORMULATE QUESTIONS TO EXPLORE USING TEXT ANALYTICS

3. SESSION OUTLINE
WHAT IS TEXT ANALYSIS/ANALYTICS? – A TALE OF TWO HISTORIES
SOURCES OF VARIATION IN STUDENT LEARNING
REFOCUSING FEEDBACK
TEXT ANALYTIC CASE STUDIES - QUANTEXT
PARTNERSHIP APPROACH TO DEVELOPMENT
WHERE TO FROM HERE

4. 1250

5. 1860

6. HUMANS REPLACED …. 1950

7. 1951

8. 1980
2018

9. 1980
2018

10. CONCORDANCING
WORD FREQUENCY
COLLOCATIONS
2018

11. 2018

12. 2018HUMANS REQUIRED ….

13. 1950

14. 1966

15. 2008

16. 2012

17. 2018
STATISTICAL AND RULE-BASED APPROACHES
MODEL AND SIMULATE NATURAL LANGUAGE

18. 2018

19. 2018HUMANS REQUIRED ?

22. TEACHERS REQUIRED?
2018
YES

23. “depends on the fragile interplay between
the teacher and the student”
Biesta, 2012
The gift of teaching …

24. Sources of variation
Learning
Sources
of
variation

25. Sources of variation
One source of variation that is 100% within
the control of the teacher…

26. Refocus feedback
Learning Pedagogenic ‘error’
(20-25% of variance in
learning within teacher
control – Hattie(2015)

27. Refocus feedback
Analyse student writing, work, interactions
Informal ‘noticing’
Formal feedback e.g.SET

28. Refocus feedback
Text analytics – derive pedagogic
insights to enhance student learning

29. What is Quantext
A text analytic tool designed to
foreground the fragile interplay
between student and teacher…
https://quantext.org
https://github.com/quantext/quantext

30. Quantext uses cases

31. What is Quantext?
A work in progress:
Explore short-form student text from any source …
Evaluate text at scale, over time and in real-time
Extract pedagogic insights
Explore the interplay between student and teacher

32. Case study: Undergraduate teaching

33. Case study: Undergraduate teaching
Identify student conceptions
Inspect top keywords, entities or themes in student work
Explore keywords etc. in context
Compare with teaching materials
Remediate

36. https://www.researchgate.net/publication/324417971_Analysis_of_
student_discussion_posts_in_a_MOOC_Proof_of_concept
Case study: MOOC discussion posts
Analysis of student discussion posts in a MOOC:
proof of concept
Irina Elgort, Karsten Lundqvist, Jenny McDonald &
Adon Moskal

37. Case study: MOOC discussion posts
Evaluate posts
Sort and filter posts by length,
Identify common content words and word groups
Relate comments to participation
Relate discussion points to interests?

38. Results (RQ1)
Points of interest identified in the
analysis of introductory discussion posts
•climate change - third most frequently-
used bigram
•geology – in the first 20 most frequently-
used words
•earth science/s and interest in geology -
other frequently-used language chunks
•less so in history (about 11% of the
students mentioned history)

39. Case study: MOOC discussion posts
0.24
0.10 0.11 0.12 0.11
0.14
0.17
0.10 0.11
0.14
0.12
0.14
0.12
0.11
0.07
0.08
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Q12 Q13 Q14 Q15 Q16
pre week 1 week 1 week 2 week 2 week 2 week 3 week 3 week 3 week 4 week 4 week 4 week 4 week 5 week 5 week 5
Proportion of participants in the forums
Q7:
deep time -
geology &
history
Q10 & 12:
climate
change

40. Case study: Student evaluations of teaching

41. Case study: Student evaluations of teaching
Explore student perceptions of SET
Sort and filter responses by length,
Identify common terms from keywords, ngrams, entities
Identify common themes from random sample
Do common terms reflect themes?
Automatic classification of responses into themes?

42. Case study: Student evaluations of teaching
What were your reasons for not
completing surveys?

43. Case study: Student evaluations of teaching
Qualitative review
Time/timing
Forgot
Disinterest
No point
Top 3-word groups
Couldn’t be bothered
Time to complete
Lack of time
Anything to say

44. Case study: Student evaluations of teaching
Automatic labelling
based on human-
marked sample

45. Case study: Student evaluations of teaching
Automatic labelling
Inter-rater reliability
scores (Cohen’s
kappa) N=25
Researcher Cohen’s Kappa
R1 and R2 0.81
R2 and R3 0.77
R3 and R1 0.79
Machine R1 R2 R3
Machine 1.0 0.77 0.88 0.54
R1 0.77 1.0 0.88 0.76
R2 0.88 0.88 1.0 0.65
R3 0.54 0.76 0.65 1.0
Human Inter-rater
reliability scores
(Cohen’s kappa)
N=594

46. Case study: Student evaluations of teaching
All labels assigned
by machine classifier
and compared to
relative proportions
assigned by three
researchers
(brackets).
Institution Polytechnic University %Total (N=648)
Label
disinterest 24 121 22% (24%)
forgot 15 109 19% (16%)
lack-of-time 67 162 35% (37%)
no-point 28 122 23% (22%)
Total 134 514

47. Case study: Student evaluations of teaching
“…if we are going to ask students to take the time to respond to open
questions, we owe it to them to take the time to analyse their
responses. …even for large classes and student cohorts, advances in
text analytics make this quite feasible.”
McDonald, Moskal, Goodchild, Stein & Terry (Submitted)

48. Quantext uses cases

49. Quantext Live
WHAT DO YOU SEE AS THE BIGGEST CHALLENGES FACING TERTIARY TEACHERS
IN THE NEXT DECADE?
https://bit.ly/2wnkGYQ
IS THERE A PLACE FOR TEXT ANALYTICS TO ENHANCE TEACHING AND LEARNING?
https://bit.ly/2Ntg4rO
DO YOU THINK YOU WOULD USE TEXT ANALYSIS IN YOUR TEACHING?
CAN YOU SAY WHY OR WHY NOT?
https://bit.ly/2LxyyWi

50. Partnership approach to development
Open source – setup your own Quantext or use
our service
New case studies welcome

51. Where to from here?
Contact
jenny@quantext.co.nz
or
adon@quantext.co.nz
Thank-you!

52. References
Biesta, G.J. (2013) Giving Teaching Back to Education: Responding to the Disappearance of the Teacher.
Phenomenology & Practice, 6(2), 35-49.
Elgort, I., Lundqvist, K., McDonald, J., & Moskal, A. C. M. (2018). Analysis of student discussion posts in a
MOOC: Proof of concept. In Companion Proceedings 8th International Conference on Learning Analytics &
Knowledge (LAK18).
Hattie, J. (2015). The applicability of Visible Learning to higher education. Scholarship of Teaching and Learning
in Psychology, 1(1), 79-91.
McDonald, J., Bird, R. J., Zouaq, A., & Moskal, A. C. M. (2017). Short answers to deep questions: supporting
teachers in large-class settings. Journal of Computer Assisted Learning, 33(4), 306-319.
McDonald, J., Moskal, A.C.M., Goodchild, A., Stein, S & Terry, S (Submitted) Using Quantext to tap into the
student voice: a proof-of-concept study
Stokes, J. & McDonald, J (Submitted) Learning analytics to promote deep learning: framing information literacy
assessment around student interests in enabling education