Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Qualitative approaches to learning analytics

Tutorial on qualitative approaches to learning analytics given by Rebecca Ferguson of The Open University UK at the Learning Analytics Summer Institute (LASI) run by the Society for Learning Analytics Research (SoLAR) at the University of British Columbia (UBC) in Vancouver, Canada, on 17 June 2019

  • Login to see the comments

  • Be the first to like this

Qualitative approaches to learning analytics

  1. 1. Qualitative approaches to learning analytics Rebecca Ferguson The Open University, UK Learning Analytics Summer Institute Vancouver, June 2019
  2. 2. Why focus on qualitative?
  3. 3. Learning analytics help us to identify and make sense of patterns in the data to improve our teaching, our learning and our learning environments
  4. 4. Why do learners and teachers act as they do? What do we know about the contexts in which learning analytics are employed? How can we increase the value of learning analytics tools and methods? What do learners and teachers want from learning analytics?
  5. 5. Latent Dirichlet Allocation Word six Word five Word four Word seven Qualitative Analysis Used top words to identify initial themes Looked at top 10 responses Then updated themes, adding new aspects Document One (D1) Word three Word two Word one Document Two (D2) Used this method to identify underlying topics and themes from the responses Document Three (D3) Word ten Word nine Word eight D1 Topic 1 Word 1 Word 2 Word 3Topic 2 D2 Topic 3 Topic 4 Word 4 Word 5 Word 6 Word 7 D3 Topic 5 Topic 6 Word 8 Word 9 Word 10 Word four Word five Word six Word seven We conducted quantitative and qualitative studies to examine systematically the algorithms’ acceptance, accuracy and impacts. The feedback was positive. Here are two examples. Transcripts from all the focus groups were then created and common themes were identified Summaries of responses are given below
  6. 6. ACTIVITY • Think of a time when you [could] have used a qualitative approach as part of your work on learning analytics. • Share your example with those on your table. • Do these examples have any common features? When aren’t numbers enough?
  7. 7. What is a qualitative approach? There exists a fundamental distinction between two types of data: qualitative and quantitative. The way we typically define them, we call data ‘quantitative’ if it is in numerical form and ‘qualitative’ if it is not. https://www.simplypsychology.org/qualitative-quantitative.html
  8. 8. What is a qualitative approach? The question is not whether you are using a mixture of numerical and non-numerical data, but how that data is being viewed. Within a qualitative methodology both numerical and non-numerical data are viewed in the same way; all data is a symbolic representation, which needs to be interpreted and thus its meaning is subjective and context dependent. Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C.-C. (2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1-A9.
  9. 9. ACTIVITY • When are these statements correct? • What do you need to know about context in order to decide whether these statements are correct? Putting figures in context (a) l + l = l0 (b) l + l = ll (c) l + l = l (d) l + l = 2
  10. 10. Nomothetic = relating to the study or discovery of general scientific laws Hermeneutic = concerned with interpretation Twining, P., Heller, R. S., Nussbaum, M., & Tsai, C.-C. (2017). Some guidance on conducting and reporting qualitative studies. Computers & Education, 106, A1-A9.
  11. 11. A qualitative approach in LA (1) RQ1: What are the individual perceptions of 20 educational managers within a distance learning HE institution about the adoption of predictive learning analytics in the organisation? RQ2: What are the challenges that may inhibit wider adoption of PLA across the institution? We need more information to know how and when to use it best The tools need to be able to sell themselves Using it on a voluntary basis does not work... we need to be clear on how much easier we can make life for teachers with it Plethora of data and sometimes contradictory outputs Herodotou, C; Rienties, B; Verdin, B; and Boroowa, A.(2019). Predictive learning analytics ‘at scale’: Guidelines to successful implementation in Higher Education based on the case of The Open University UK. Journal of Learning Analytics, 6(1) pp. 85–95.
  12. 12. Methods • Research questions identified • Method of data gathering identified and justified • Sample described and justified • Broad topics identified • Sample questions provided • Data collection clarified • Method of data analysis identified: thematic analysis • Themes presented clearly • Explained how interpretations were checked Herodotou, C; Rienties, B; Verdin, B; and Boroowa, A.(2019). Predictive learning analytics ‘at scale’: Guidelines to successful implementation in Higher Education based on the case of The Open University UK. Journal of Learning Analytics, 6(1) pp. 85–95.
  13. 13. What would your superpower be? Holstein et al, Journal of Learning Analytics, in press
  14. 14. A qualitative approach in LA (2) Initial needs analysis and concept generation, including design activities such as generative card sorting exercises, directed storytelling, semi-structured interviews with teachers, and field observations Initial concept validation via speed dating sessions with teachers Iterative lower-fidelity prototyping, gradually increasing the fidelity of both prototypes and simulated use contexts, using methods such as role-playing and bodystorming exercises, participatory sketching and comicboarding, and behavioral mapping Iterative higher-fidelity prototyping with replay-based simulation exercises, using Replay Enactments Iterative classroom piloting and experimental evaluation, using field observations, pre-post assessments of student learning, semi- structured interviews with teachers and students, and behavioral mapping. Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics.
  15. 15. Methods used Generative card sorting exercises Come up with superpower ideas, sort by subjective priority, think aloud about reasoning. Can add or align with cards from other teachers Directed storytelling Talk through specific recent events. Refflect on how challenges could be overcome Speed dating sessions Rapidly explore possible futures to probe the boundaries of what individuals will find acceptable Role playing Bringing future scenarios to life Bodystorming exercises Design questions are asked in relevant locations where activities can be acted out Participatory sketching Sketching out new ideas Comicboarding Leaving the final panel(s) of a story blank Behavioral mapping Observing and recording activity in a certain place or over a certain period of time Replay Enactments Use authentic sets of data and algorithms to prototype a scenario
  16. 16. Detailed account of analysis Two researchers then worked though transcriptions of approximately 5 hours of video and audio recorded interviews, to synthesize design findings using two standard techniques from Contextual Design: interpretation sessions and affinity diagramming (Beyer & Holtzblatt, 1997; Hanington & Martin, 2012). Interpretation sessions are aimed at helping design teams develop a shared understanding of collected interview and think-aloud data, by collaboratively extracting quotes representing key issues. Affinity diagramming is a widely used, bottom-up synthesis method, aimed at summarising qualitative patterns across study participants’ responses, by iteratively clustering participant quotes into successively higher-level themes (Beyer & Holtzblatt, 1997; Hanington & Martin, 2012). Following several interpretation sessions, the resulting 301 extracted quotes were iteratively synthesized into 40 level-1 themes, 10 level-2 themes, and 4 level-3 themes. Size and scope of the dataset Methods of analysis explained, referenced and justified Broad overview of the outcomes Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics.
  17. 17. Making themes clear to readers • Help me to intervene where, when, and with what I am most needed. • Make sure the technology does not draw my attention away from my students! • How can I know whether what I’m doing is actually working? • Help me understand the “why”, not just the “what”. • I’m just one person: help ease my load. • But how do I judge whether my students are really doing well? • Help me monitor and manage student motivation. • What can you tell me about my students that I do not already know? • Allow me to customize the technology to meet my needs. • Allow me to override the technology. Holstein, K; McLaren, B M; and Aleven, V. (in press). Co-designing a real-time classroom orchestration tool to support teacher-AI complementarity. Journal of Learning Analytics. Teachers want to stay in control and provide value. They want analytics that provide information they don’t already know. They are concerned that analytics could do more harm than good
  18. 18. Trying a qualitative approach Note: Qualitative data analysis is typically a lengthy process. The approach used here has been selected as a structured approach that can be trialled quickly in the time available using data generated by participants. Scupin, R., 1997. The KJ method: A technique for analyzing data derived from Japanese ethnology. Human Organization, pp.233-237. (Method developed by Jiro Kawakita). The KJ Method (similar to affinity diagramming) • Identify the problem and state it clearly to the group • Create labels – one idea per post-it note • Randomise the ideas (shuffle them) • Work together to group the ideas into teams that fit together • Give each team a title • Group the teams into families (no more than ten) • Give each family a title • Pattern them into a chart, indicating relationships between them • Explain the chart in words.
  19. 19. ACTIVITY What are the barriers to the use of qualitative approaches in learning analytics and how can they be overcome? Trying a qualitative approach (1) Use a thick pen in a dark colour, so your idea is visible https://medium.com/design-research-methods/how-to-use-post-it-notes-9ca0904a03d1 One idea, concept or question per Post-It note Be concise The more ideas the better
  20. 20. ACTIVITY Building the teams Trying a qualitative approach (2) Group the ideas that go together in ‘teams’ Shuffle the notes Give each team a label Discuss your decisions and share responsibility
  21. 21. ACTIVITY Building the families; creating a chart Trying a qualitative approach (3) All your ideas need to be associated with a family Group the teams into bigger families No more than ten families Represent the relationships between the families If there is time, explain your joint interpretation of the data
  22. 22. Trying the KJ Method https://uxdict.io/design-thinking-methods-affinity-diagrams-357bd8671ad4 Note: Several steps are missing here. No consideration of sample size and selection, briefing, ethics, or original choice of question. • Provides a structured way of dealing with disparate data • The route to analysis can be explained clearly • Data are originally given equal weightings. No voice is privileged • Rapid means of organising ideas and information • Enables analysts to work together • Does not automatically privilege interpretation of one analyst • Makes analysis visible to others • Analysts can compare interpretations • Analysts can relate interpretations to each other Affinity diagramming typically applies this approach to a much broader set of data, first breaking it down into separate concepts.
  23. 23. Carrying out qualitative analysis
  24. 24. Case study is a method. A case is not automatically a case study. Photo by Marvin Esteve on Unsplash Grounded theory is a complex set of methods. It isn’t simply a way of saying your codes are grounded in your data Themes do not emerge from the data. They are generated by the researcher(s) Don’t reference Glaser, Corbin, Charmaz, or Strauss unless you have read them. Their views change over time and they don’t agree.
  25. 25. First, transcribe your data Audibly breathes in T1 Liz: Anyway, I er was hopin’ I could pop by tomorrow (.) if possible T2 (2.0) T3 Liz: Probably not, hey T4 George: .hh. = T5 Liz: =That's OK? T6 George: Yeah T7 Liz: ?I won’t stay long? Very short gap Non-word Representation of accent 0.1 sec gap 2 sec gap At the same time In a lower tone Direction of gaze Facial expression Gesture People don’t naturally speak in punctuated sentences. Words only make up part of our communication. Acknowledge the decisions you have made
  26. 26. Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY Thematic analysis Thematic analysis is a method for identifying, analysing and reporting patterns (themes) within data. It minimally organizes and describes your data set in (rich) detail. However, frequently if goes further than this, and interprets various aspects of the research topic Inductive (working from the data up) Deductive (theory led) Experiential (how people report their lives) Critical (what is going on in the data) [Critical] realist (Insight to the truth) Constructionist How topic is framed and understood You need to make decisions and justify them
  27. 27. Thematic analysis Straightforward analysis • Describes • Summarises • Gives voice Sophisticated analysis • Tells a story • Interprets • Makes an argument • Locates data and participants within wider – social – cultural – historical – political – ideological context Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY
  28. 28. Reflexivity As a researcher, you engage with the data You make conscious decisions Reflexivity is an ongoing process • What are you doing? • Why are you doing it? • What are your assumptions? Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY Ask people to check, and query, your interpretations
  29. 29. Immersion Immerse yourself in your data Read • actively • analytically • critically Identify things of interest Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY Immersion Reflect on what you bring to your interpretation
  30. 30. Generate codes A code is a label that captures something of interest • It should include a few words • It should be able to stand alone Code your data comprehensively and systematically Then repeat the process End this phase with a list of codes plus the data connected with them Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY Semantic codes capture surface meaning Latent codes capture implicit meaning and assumptions
  31. 31. Generate themes Another active stage • Organise the codes into themes • What are the bigger patterns of meaning that cut across the dataset? • Cluster similar codes together • Go from the codes to the coded data and check • Don’t just summarise • Good themes are distinctive and part of a larger whole
  32. 32. Review, define, name themes What is the name of this theme? What is the quality of this theme? What are the boundaries of this theme? Are there enough data to support this theme? Are the data too diverse? Does the theme work in relation to the extracts? Does it work in relation to the whole dataset? What is the overall story of the analysis? Thematic analysis: an introduction https://www.youtube.com/watch?v=5zFcC10vOVY Six themes are probably enough Be prepared to let things go!
  33. 33. Produce the report Analysis should include: • commentary • data extracts • themes Look for vivid and compelling examples for each theme – not all from the same eloquent person! Relate analysis to • research questions • wider literature • wider context Thematic analysis: an introduction (June 2018) https://www.youtube.com/watch?v=5zFcC10vOVY
  34. 34. ACTIVITY Identify some semantic codes (based on what the students say) that could be applied to these data Identify some latent codes (based on what the students assume or imply) that could be applied to these data Semantic and latent themes Comments from students studying Creative Writing Looking for ideas about analytics to support forum use What they liked or didn’t like about the course / forums Data are fake but based on real data
  35. 35. Trustworthy research Quantitative: Is this generalisable? Empirical: this setting is typical Explanatory: this explains what happens in cases with similar characteristics Theoretical: there are necessary relationships between sets of phenomena Qualitative: Is this transferable? Are these insights helpful in other contexts? Does this rich account add to our understanding of a broader picture?
  36. 36. Trustworthy research Quantitative: Is this reliable? Would these measures give the same results in the same circumstances? Qualitative: Is this credible? Is your line of reasoning clear and transparent? Do your conclusions make sense to participants? Qualitative: Is this plausible? Have you shown clear and multiple connections with the data? Have you made a persuasive case for the ways in which you engaged with the data? Qualitative (more positivist stance): Is this reliable? Can you demonstrate strong inter-rater reliability or intra-rater reliability?
  37. 37. Trustworthy research Quantitative: Is this valid? How well does this piece of research reflect the reality that it claims to represent? Qualitative: Is this trustworthy? Have you handled the data carefully? How can you make it clear you have done a good job of capturing the views of others? Qualitative: Is this dependable? Does the research account for the setting in which it takes place? Qualitative: Is this confirmable? Can your results be confirmed or corroborated by others?
  38. 38. Why do learners and teachers act as they do? What do we know about the contexts in which learning analytics are employed? How can we increase the value of learning analytics tools and methods? What do learners and teachers want from learning analytics?
  39. 39. slideshare.net/R3beccaF r3beccaf.wordpress.com/ twitter.com/R3beccaF

×