Ascilite 2017 Short Paper
•Download as PPTX, PDF•
0 likes•140 views
We introduce Quantext, a web-based tool for teachers to support the rapid analysis of student responses to short answer or mini-essay questions. Designed to support teaching in large-class settings, it aims to bring to practicing teachers analytic tools that can reveal insights in their student text data. In this presentation we provide the pedagogical rationale for development of Quantext, briefly describe its features, and and introduce a pilot study to evaluate the tool in classrooms at three NZ universities and one polytechnic. We conclude with options for accessing the tool and outline plans for ongoing development.
Recommended
More Related Content
Similar to Ascilite 2017 Short Paper
Similar to Ascilite 2017 Short Paper (20)
Recently uploaded
Recently uploaded (20)
Ascilite 2017 Short Paper
- 1. Quantext Analysing student responses to short-answer questions JENNY MCDONALD & ADON C. M. MOSKAL
- 2. SESSION OUTLINE MOTIVATION FOR QUANTEXT APPROACH TO DEVELOPMENT & PROTOTYPING KEY FEATURES & WORKFLOW BENCH-TESTING PILOT STUDY WHERE TO FROM HERE?
- 3. MOTIVATION Laurillard - Conversational Framework (simplified)
- 6. MOTIVATION Probe what is happening at the site of learning¹ through the systematic analysis of teacher and student text. 1. Knight & Littleton, 2015
- 7. MOTIVATION Blood flow in the systemic circulation is arranged in parallel. Please explain why. Question
- 8. MOTIVATION Student responses ...all organs...receives fresh blood through their own blood supplies. Student C ...Every organ receives a fresh supply of oxygenated blood ... Student A ...so that each organ receives a fresh supply of oxygenated blood... Student B
- 9. MOTIVATION Lecture ...So blood flow within the pulmonary system or within the systemic system flows in parallel. This means to say that every organ gets its own fresh supply of oxygenated blood…. ...This means to say that because we’re getting this branching going on, every organ’s getting its own fresh supply of oxygenated blood….
- 10. MOTIVATION McDonald, J., Bird, R. J., Zouaq, A., & Moskal, A.C.M. (2017). Short answers to deep questions: supporting teachers in large‐class settings. Journal of Computer Assisted Learning. 33(4), pp. 306–319 Gunn, C., McDonald, J., Donald, C., Milne J., & Blumenstein, M. (2017). Building an evidence base for teaching and learning design using learning analytics. Research Report. Ako Aotearoa National Centre for Tertiary Teaching Excellence, Wellington, New Zealand.
- 11. MOTIVATION
- 12. MOTIVATION A tool designed for teachers and teaching context with relevant functionality and easy to use.
- 14. Import questions and student short answers/mini essays Quick comparison within and between cohorts Analyse in context with reference to teaching materials Label/categorise responses through iterative similarity (keyword/phrase & semantic) and sorting - multiple labels possible. Identify teaching insights Export for subsequent reporting/grading KEY FEATURES & WORKFLOW
- 15. Measuring semantic similarity BENCH-TESTING pump heart toe Words represented by vectors which capture meaning in use - word embeddings pump 𝜃toe 𝜃 Cosine of angle 𝜽 between vectors => similarity -1 to +1
- 16. Measuring semantic similarity BENCH-TESTING For very short responses (a few words) - average word vectors for each response and for a reference response and then compare each response to the reference response. => reasonable results compared to human evaluation (refer paper). For longer responses - hopeless! (which makes sense...) Plan to extend and develop this feature
- 17. 8 teachers from six different courses, spanning the humanities, sciences, health sciences and commerce, and with cohorts of under 100 students to well over 1000 and including 2 MOOCs PILOT STUDY Feedback from participants to date: Informed interface development Demonstrated the need to cater for mini-essays (4/6 courses) in addition to SAQs Clear requirement for visualisation options Range of data management and interface usability improvements Integration with LMS
- 18. WHERE TO FROM HERE? Elgort, I., Lundqvist, K., McDonald, J., & Moskal, A.C.M. (2018). Analysis of student discussion posts in a MOOC: Proof of concept. LAK18, Sydney. 5TH-9TH MARCH McDonald, J., Moskal, A.C.M., Elgort, I., & Gunn, C. (2018). Analysing student responses: early lessons from a pilot study. LAK18, Sydney. Moskal, A.C.M. & McDonald, J. (2018). Quantext Demonstration. SoTEL, AUT, Auckland. 15TH & 16TH FEBRUARY
Editor's Notes
- Introduce Quantext – a new text analysis tool for teachers. Will spend most of the time for this session explaining the motivation for developing Quantext – why would teachers need it and what practical issues can it address I’ll briefly talk about our approach to development and prototyping and introduce the key features and workflow. We’ll conclude the session with some remarks about how we’ve evaluated Quantext text similarity feature on the workbench and explain where we are up to with a pilot study currently underway with teachers in the NZ tertiary sector.
- Starting point - Laurillard’s framework - nice since it accommodates a wide range of educational theories and focuses squarely on what happens in the classroom. Here we have the teacher creating some context for the student to think about or act on. The student may ask questions back to the teacher and engage in the activity or context set by the teacher. They also present something back for the teacher to review. The teacher reflect on this and then may represent or answer questions, revise activities or present the next context and so on. Should be a familiar idea? This version is simplified or compressed but captures the key idea which is a central dialogue between teacher-student (can also be between students etc) Nice too because it sits comfortably with some key lingustic theories – Wittgenstein – language is meaning in use and Bakhtin – the dialogic – what we say or write or speak about is always related in a way back to that which we have previously been exposed to..
- I’d like to present an illustration or metaphor if you like for analysing what happens in the classroom. Here’s a wizzydizzy device to measure learning Typically, ‘ground truth’ is provided by whatever construct we want to measure – grades, engagement, self-regulation.... Then we stick a probe across whatever the student does – click on a link, submit a quiz, download something…if we’re lucky we might stick a probe across what they produce – questions, answers, reflective writing, experimental design etc.. Lo and behold we measure something – cool eh? But what are we measuring and what does it mean? A wee peek at the literature and we can see – all our ‘ground truth’ constructs are strongly context dependent – assessment, engagement ... So what exactly do our measurements mean – what in the classroom context affects our measurements? So what if we move our ground truth from out here to the context we set – we’re still measuring something but now at least our measurement is stable! Are there issues with generalising – context is ‘queen’ – probably but ...
- What this means is that we now have the opportunity to explain a grade or level of engagement or etc etc rather than trying to infer what was going on in the classroom.. Sounds good but how does this work in practice?
- I’ve got time for one concrete example to show you what I mean....
- More detailed examples and a complete case study analysiing student responses from a human biology class of 1500+ students - largely qualittive => very time consuming. Text analytics to bring the power of this kind of analysis to teachers... Cathy, Claire and myself will be talking a bit more about this at a lightening round session this afternon.. Ako project led to developing a text analytics for teachers workshop using freely available tools
- Lots of interest and good feedback but beyong simple word cloud or as part of a reseacrh study – barriers to tool use just too high.
- Here are the key features – you can explore yourself by going to Quantext.org – url at the end of these slides. Quantext does simple and obvious things like counting key words and identifying key word groups/phrases – also does semantic similarity using some fancier techniques – called words embeddings. Explain if time...
- What have we found from pilot thus far?
- Here’s our development roadmap and presentations and reporting on outcome of pilot schedule for Feb March next year..