Can you tell if they're learning? ICALT 7 July 2015
•Download as PPTX, PDF•
0 likes•906 views
Paper (http://eprints.soton.ac.uk/376609/) presented at ICALT2015 at Hualien, Taiwan.
![Can you tell if they’re learning?
• Problems finding/evaluating Web resources1.
• Ineffective tagging2.
• Resources unrecognised as useful for learning.
• Potential for comments.
Motivations
[1] M. B. Eisenberg, “Information literacy: essential skills for the information
age,” DESIDOC Journal of Library and Information Technology, vol. 28, no.
2, 2010, pp. 39-47.
[2] M. Bienkowski & J. Klo, “The Learning Registry: applying social metadata
for learning resource recommendations,”, Springer, 2014, pp. 77-95.](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
Viewers also liked
Viewers also liked (15)
Similar to Can you tell if they're learning? ICALT 7 July 2015
Similar to Can you tell if they're learning? ICALT 7 July 2015 (20)
More from studywbv
Recently uploaded
Recently uploaded (20)
Can you tell if they're learning? ICALT 7 July 2015
- 1. Can you tell if they’re learning? Tim O’Riordan Using a pedagogical framework to measure pedagogical activity Web and Internet Science (WAIS) www.ecs.soton.ac.uk/people/tor1w07
- 2. Can you tell if they’re learning? • Problems finding/evaluating Web resources1. • Ineffective tagging2. • Resources unrecognised as useful for learning. • Potential for comments. Motivations [1] M. B. Eisenberg, “Information literacy: essential skills for the information age,” DESIDOC Journal of Library and Information Technology, vol. 28, no. 2, 2010, pp. 39-47. [2] M. Bienkowski & J. Klo, “The Learning Registry: applying social metadata for learning resource recommendations,”, Springer, 2014, pp. 77-95.
- 3. Can you tell if they’re learning? • ‘Trace data’ proxy for learning. • SNA, discourse, predictive modelling3. • Methods are inaccurate4. Learning analytics [3] S. Dawson, D. Gašević, G. Siemens & S. Joksimovic, “Current state and future trends: a citation network analysis of the learning analytics field,” ACM International Conference Proceeding Series, 2014, pp. 231-240. [4] Gašević, D. 2015. Comments during plenary session at LAK '15 the 5th International Learning Analytics and Knowledge Conference. Poughkeepsie, NY, USA — March 16 - 20, 2015
- 4. …typical measurements include time spent, number of logins, number of mouse clicks, number of accessed resources, number of artifacts produced, number of finished assignments, etc. But is this really getting to the heart of the matter? Duval, E. (2011). Attention please! Learning Analytics for Visualization and Recommendation. In Proceedings of LAK11: 1st International Conference on Learning Analytics and Knowledge, 9–17. February 27-March 1, 2011, Banff, Alberta. Eric Duval, 2011 Can you tell if they’re learning?
- 5. Can you tell if they’re learning? • Develop effective interventions. • Constructivism, experiential & reflection. • Pragmatic methods. Pedagogical frameworks
- 6. Can you tell if they’re learning? What is DiAL-e? [5] S. Atkinson, “What is the DiAL-e Framework?,” 2009; http://dial- e.net/what-is-the-dial-e/. • Familiarity. • Non-hierarchical5. • Facilitates social interaction5. • Suitable for evaluating comments
- 7. Can you tell if they’re learning? What is DiAL-e?
- 8. Can you tell if they’re learning? • Word pattern and content analysis6. • Coding patterns in dialogue7. • Identify pedagogically meaningful dialogue8. Language analysis [6] J. W. Pennebaker, M. R. Mehl, & K. G. Niederhoffer, “Psychological aspects of natural language use: our words, our selves,” Annual Review of Psychology, 54(1), 2003, pp. 547-577. [7] M. A. Khawaja, F. Chen, & N. Marcus, “Using language complexity to measure cognitive load for adaptive interaction design,” Proc.15th International Conference on Intelligent User Interfaces, 2010, pp. 333-336. [8] A. De Liddo et al., “Discourse-centric learning analytic,” Proc. of the 1st International Conference on Learning Analytics and Knowledge, 2011, 23–33
- 9. Data University of Southampton (2014) Archaeology of Portus MOOC. FutureLearn Limited. https://www.futurelearn.com/courses/portus/details Can you tell if they’re learning?
- 10. Can you tell if they’re learning? Method • Code 525 comments from 12 steps (2.4%). • Categorised and scored learning activity by DiAL-e. • Calculate average Dial-e values per step (PV).
- 11. Can you tell if they’re learning? Examples
- 12. Can you tell if they’re learning? Method [10] J. W. Pennebaker et al., “The development and psychometric properties of LIWC2007,” University of Texas, 2007. [11] D. W. Wortham, “Nodal and matrix analyses of communication patterns in small groups,” Proc. Computer Support for Collaborative Learning Conference (CSCL 1999), 1999, pp. 681-686. • Linguistic Inquiry and Word Count10. • Graph density11.
- 13. Can you tell if they’re learning? Results
- 14. Can you tell if they’re learning? Results Correlation between 2nd person pronoun and PV Correlation between Positive emotion words and PV
- 15. Can you tell if they’re learning? Results [12] Y. R. Tausczik & J. W. Pennebaker, “The psychological meaning of words: LIWC and computerized text analysis methods,” Journal of Language and Social Psychology, vol. 29, no. 1, 2010, pp. 24-54. Correlation between prepositions and PV
- 16. Can you tell if they’re learning? Results [13] A. F. Hadwin et al., “Examining trace data to explore self-regulated learning,” Metacognition and Learning, vol. 2 no. 2-3, 2007, pp. 107-124.
- 17. Can you tell if they’re learning? Conclusions • no correlation between pedagogic activity and language complexity and graph density. • correlation with linguistic markers of socially engaging language (-ve) and informational language (+ve).
- 18. Can you tell if they’re learning? Recent work • Coded a larger sample. • Applied more, different frameworks. • Undertaken IRR.
- 19. Can you tell if they’re learning? Future work G. Siemens, D. Gasevic, C. Haythornthwaite, S. Dawson, S. B. Shum, and R. Ferguson (2011). “Open Learning Analytics : an integrated & modularized platform Proposal” Knowl. Creat. Diffus. Util. • Diverse subjects. • Machine learning. • Interviews. • Visualisation.
- 20. @studywbv timswww.wordpress.com tim.oriordan@soton.ac.uk Can you tell if they’re learning? Thank you… eprints.soton.ac.uk/376609/
Editor's Notes
- Hi, I’m Tim O’Riordan, and I’m presenting the outputs from a preliminary study I undertook at the end of last year. I coded MOOC comment data to a pedagogic framework and looked for correlations with typical learning analysis methods and linguistic markers.
- So why am I interested in this? Well, I’m a web scientist with a background in education, so I’m interested in how we learn on the web and how the web is changing the way we learn. We know that finding and evaluating online learning resources is a significant hurdle to overcome. Web objects are rarely effectively tagged, and ensuring that tags are sufficient, relevant and kept up to date is a problem. Also, some people don’t know what they have - a significant amount of content on the Web that may have value for supporting informal learning, isn’t annotated to emphasise its pedagogical usefulness.
- Analysing and making visible the trace data that people leave behind when they traverse the web can be useful in helping us understand how people learn – a key assumption underpinning Learning Analytics. Web-based proxies for behaviour have value as evidence of knowledge, competence and learning – and current research focuses on social network analysis, discourse, and predictive modelling methods. For example, providing evidence of links between positive sentiment , ‘creative capacity’ and betweeness centrality, and the co-construction of knowledge in discussion forums [18]. But it’s recognised that methods are inaccurate and need refinement.
- Duvall (2011) asserts that “one of the big problems around learning analytics is the lack of clarity about what exactly should be measured” (2011:15) and suggests that typical measurements aren’t adequate for finding out how learning being accomplished. Maybe there’s something related to how we practice teaching and learning that can help? Like pedagogical frameworks.
- These help instructors develop effective interventions. Most emphasise learning built on prior knowledge, learning by doing, and encouraging metacognitive activities. A number of generic frameworks have been developed – like ‘affinity spaces’, DialogPlus, and DiAL-e, and they use pragmatic methods to map out theoretically consistent learning designs.
- The Digital Artefacts for Learning Engagement (DiAL-e) Framework was developed about 10 years ago to support the use of digitized news films in education – which is where I became familiar with it. I thought it would be useful for this study because – well - I know it. It also has a distinct non-hierarchical structure. It suggests approaches to using digital objects to support learning, and supports interventions that involve social interaction. All of which could be appropriate for assessing the social and situated nature of online comments.
- DiAL-e consists of 10 learning design categories. Only nine were selected for this study, because, as all contributors had engaged in a discussion in some way the category referring to “inspiring learner engagement” (‘stimulation’) could be assumed to have occurred. Additional categories (‘technical’ and ‘non-relevant’) were included in the scheme in order to measure the incidence of off topic comments. So - I used 11 categories to code key aspects of comments, and measure the extent to which engagement, knowledge construction and reflection could be inferred.
- This is a highly subjective approach – so to bring a bit of objectivity into the process, I also adopted language analysis methods. Language analysis explores the content and style of language used in everyday communications and can indicate psychological and social meaning [14]. Analysis involves identifying, coding and interpreting similar patterns – and these are supported by statistical tests of significance [15]. For this study - evidence that pedagogically meaningful dialogue in Web-based environments can be automatically identified was important [16].
- 6 week Massive Open Online Course 20 steps per week > 7,000 registered learners > 1,800 ‘social’ learners > 20,000 comments containing 1,000,000 words were made available on an anonymised excel spreadsheet
- I went through and hand coded 525 comments from 12 steps in alignment with each DiAL-e category. I then calculated the average score for each step to reach a Pedagogical Value – or PV – for each step.
- In practice what I did was look at parts of each comment and coded accordingly. Most categories could be applied more than once to some comments. For example an individual comment that presented three distinct questions would be given three ‘Inquiry’ counts.
- The linguistic properties of the comments were analysed using a well-known text analysis software program, and I extracted social network graphs from the interactions within the comment streams. Essentially - whenever a contributor responded to a message from another contributor I created directed edges between the two of them – and the graph density for each graph was calculated using NodeXL. By analysing this data in this way I aimed to compare correlations between inferred pedagogical activity (the PV scores), with methods used in Learning Analytics (sentiment, learner engagement) and Language Analysis (language complexity, sentiment, significant word categories).
- And here are the results. LIWC analysis indicated three word types that had significant relationships with PV – 2nd person pronouns, positive emotion words and prepositions. Whereas graph density and complexity in writing were shown not to be significant indicators on on-topic activity.
- The most striking outcome from regression analysis is the clear, statistically significant, correlation between those words identified in the literature as being associated with the affective domain - expressions of empathy, value judgments or “involved” writing - and learning objects with low PV scores. So we see the higher PV value – the less use of pronouns and positive emotion words. Also, through keywords in context analysis these words were seen to be more connected with course ‘housekeeping’ issues and expressions of gratitude, than with on-topic activity.
- The correlation between prepositions (e.g. to, with, above) and PV scores, suggest that the DiAL-e coding schema was successful in identifying comments that reveal depth of thinking. The higher the PV – the greater use of prepostions. Tausczik & Pennebaker report a high incidence of prepositions associated with attention to reflective behaviour and complex writing [23:35].
- So we have this picture emerging of a lack of correlation between higher levels of engagement with learning activities and typical interactional measures like graph density and wps – and negative relationship between positive sentiment and the DiAL-e based measure on-topic behaviour. Looking at graph density - Hadwin et al. speculate that lower density networks like we have here - are indicative of learners engaging in distinct, regular study patterns [30]. Key Word in Context analysis suggests that with these comments, connections tended to occur in distinct patterns that were unrelated to my measure of pedagogical activity. So connections between contributors were observed to have occurred in a variety of contexts – not specifically topic related.
- I set out to compare standard Learning Analytics and Language Analysis measures with alignment to an appropriate pedagogical framework, and applied my coding schema to comment data associated with an online learning environment. Results show that the approach produces results that are distinct from typical interactional measures. I identified approximately negative and statistically insignificant correlations with established Learning Analytic methods, but have found strong correlation with linguistic indicators of pedagogical activity. While my approach may indicate the usefulness of close attention to the variety of pedagogic activity that occurs online, I don’t claim that it represents a full account of learners’ activity, but adds a degree of nuance to established measures.
- I concluded this highly subjective stage of my work in January. Since then I have: Coded a larger sample, Applied a variety of frameworks, Undertaken IRR I’m currently writing this stage up – but with some nuance, I’m seeing a similar picture.
- Future work involves coding a new training sample, running comments through a machine learning process and evaluating results. It’s very easy in this type of analysis to get lost in the process and lose track of the fact this this involves real people who are really trying to learn something – so I also think it’s important to talk to real people about what this means – does any of this relate to people actual experience of engaging in web based environments? At some point in the not too distant future I will need to look into how to go about making this data readily and meaningfully understandable to users – learners, instructors, as well as institutional administrators.
- http://eprints.soton.ac.uk/376609/