Recommended
Recommended
More Related Content
Similar to Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing
Similar to Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing (20)
More from Hassan Khosravi
More from Hassan Khosravi (7)
Recently uploaded
Recently uploaded (20)
Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing
- 1. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 1 Hatim Lahza h.lahza@uq.edu.au A/prof. Hassan Khosravi h.khosravi@uq.edu.au A/prof. Gianluca Demartini g.demartini@uq.edu.au Prof. Dragan Gasevic Dragan.Gasevic@monash.edu
- 2. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 2 Introduction Results Discussion Questions Method
- 3. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 3 Motivation Self-regulation scaffolds are challenging in large classes Computer supported learning environments help scaling scaffolding • How should SRL scaffolds be implemented? • Does the context or task matter? • What SRL phases are most important than others?
- 4. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 4 Our study Objectives • Designing interfaces (interventions) for an online learning environment based on self- regulated learning (SRL) strategies to help learners create high-quality resources • Conducting an in-the-field experiment to evaluate our design
- 5. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 5 Our study Self-regulated learning (SRL) • Essential learning skill • Most common SRL models are cyclic processes and have preparatory, performance and appraisal phases [1] • Learner as an agent • Evidence of the effectiveness in online learning environments is still needed [1] Ernesto Panadero. 2017. A review of self-regulated learning: Six models and four directions for research. Frontiers in psychology 8 (Apr 2017), 422. [2] Puustinen, M., & Pulkkinen, L. (2001). Models of self-regulated learning: A review. Scandinavian journal of educational research, 45(3), 269-286.
- 6. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 6 Our study Learnersourcing • Learner as a partner in creating learning resources [1][2] • Benefits for learner and instructor [3] • High-quality resources • Quality of resources varies • The creation interface might be part of the issue [1] HassanKhosravi,GianlucaDemartini,ShaziaSadiq,andDraganGasevic.2021. Charting the Design and Analytics Agenda of Learnersourcing Systems. In 11thInternational Learning Analytics and Knowledge Conference. ACM, 32–42. [2] JuhoKim.2015.Learnersourcing:improvinglearningwithcollectivelearneractivity. Thesis. Massachusetts Institute of Technology. [3] AaronMcDonald,HeathMcGowan,MollieDollinger,RyanNaylor,andHassan Khosravi. 2021. Repositioning students as co-creators of curriculum for online learning resources. Australasian Journal of Educational Technology (2021), 102– 118.
- 7. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 7 Introduction Results Discussion Questions Method
- 8. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 8 RiPPLE • Adaptive educational system • Enables learnersourcing • Three main activities: creation, evaluation and practice Process overview Overview Main creation interface
- 9. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 9 Interventions for SRL Strategies • SRL strategies: • Planning then creating • Externally-facilitated monitoring • Self-assessing after creating • All the above strategies combined together • An interface was created for each strategy Interventions
- 10. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 10 Planning Then Creating • Six main elements • We hypothesis that the planner can benefit both students who have a clear idea of what to create and those looking for new ideas Overview Interface - Piazza - Tutorials - Lecture Slides - Table - Video
- 11. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 11 Planning Then Creating • The presentation includes: • Definitions • Examples • Tips • Students could view it at any time Overview Bloom’s taxonomy
- 12. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 12 Externally-facilitated monitoring • Aims for providing monitoring elements • Three main elements • Best practices for high- quality MCQs from the literature • Presented as tick boxes Overview Interface
- 13. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 13 Self-assessing after creating • Added as a last step • Prompt students to review and revise their resources • Two main elements: • Self-assessment according to the peer assessment set of criteria • Reflection Overview Interface
- 14. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 14 Experimental Design RQ1. How do the proposed interventions change the process and effort required for creating a learning resource? RQ2. How beneficial do students find the proposed interventions? RQ3. What is the impact of the proposed interventions on the quality of the resources created by students? • In-the-field randomised control experiment • Five conditions: • Control (A) • planning then creating (B) • monitoring while creating (C) • self-assessing after creating (D) • All (E) Settings Research questions
- 15. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 15 Experimental Design • Location: The University of Queensland • Course: Neuroscience & Information systems • Level: Undergraduate • Number of students: 797 • Consent required: Yes • Purpose of use: Assessment, but engagement with the intervention was optional • Duration: Multiple rounds over 6-7 weeks • Contribution to grade: 10% (e.g., create, moderate and practice) • Average resource (μ ± σ): • Both courses: 2.39 ± 1.22 • Considered engaged: If all the intervention components used Context & Participants
- 16. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 16 Experimental Design
- 17. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 17 Experimental Design Metrics and Analysis (Combined data- both courses) • Process mining using trace data • Quantitative analysis using duration (min) on the elements of the interfaces RQ1. Impact on the Process and Effort: • Effectiveness rating (i.e., 5-point scale) • Written feedback • Quantitative analysis • Qualitative: general inductive approach [1] & sentiment analysis RQ2. Impact on Student Perceptions: RQ3. Impact on Performance: • Quantitative analysis using resource quality rating provided by the system RQ1. How do the proposed interventions change the process and effort required for creating a learning resource? RQ2. How beneficial do students find the proposed interventions? RQ3. What is the impact of the proposed interventions on the quality of the resources created by students? [1] David R. Thomas. 2006. A General Inductive Approach for Analyzing Qualitative Evaluation Data. Am J Eval 27, 2 (Jun 2006), 237–246.
- 18. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 18 Introduction Results Discussion Questions Method
- 19. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 19 Impact on the Process and Effort Creation processes Time spent by the experimental groups vs time spent by the control group Treatment group (vs control) Mann-Whitney U test Duration- Mdn (min) Effect Condition B p < .05 * 15.73 Condition C Condition D Condition E p = .38 p = .18 p < .01 * 11.21 13.31 19.59 (Control = 12.23 min) [1] David R. Thomas. 2006. A General Inductive Approach for Analyzing Qualitative Evaluation Data. Am J Eval 27, 2 (Jun 2006), 237–246. 1 RQ1. How do the proposed interventions change the process and effort required for creating a learning resource?
- 20. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 20 Impact on Student Perceptions • Most effective interface based on the rating: (μ ± σ) 1. Control (4.62 ± .61) 2. Condition C (4.45 ± .74) 3. Condition D (4.43 ± .80) 4. Condition B (4.16 ± .99) 5. Condition E (4.25 ± 0.98) Quantitative RQ2. How beneficial do students find the proposed interventions?
- 21. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 21 Impact on Student Perceptions • Th control group found the creation interface more effective than the treatment groups • The treatment groups’ comments were mostly about technology • Conditions B and E encountered technical issues • Some students questioned the pedagogical benefits Qualitative RQ2. How beneficial do students find the proposed interventions?
- 22. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 22 Impact on Performance Figure 3: Overview of the quality rating of resources for all students and engaged students across control and treatment conditions. RQ3. What is the impact of the proposed interventions on the quality of the resources created by students? All students Engaged • Quality rating order (median): 1. Conditions B and C (Mdn = 3.9) 2. Condition D (Mdn = 3.85) 3. Conditions A and E (Mdn = 3.8) • Not statistically significant • Quality rating order (median): 1. Conditions B, C and D (Mdn = 3.9) 2. Condition A and E (Mdn = 3.80) • Statistically significant between conditions A and C, with small effect (U = 25719, r = .11, p < .01).
- 23. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 23 Introduction Results Discussion Questions Method
- 24. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 24 • Scaffolds increased task complexity and time with no significant improvements • Similar results observed in previous research [1] Effect on process & performance • Interventions were not perceived as effective • Can be explained by Winne [2] four conditions 1. learners are aware that the scaffold is useful to them, 2. learners know that the scaffold can be useful for the task at hand, 3. learners have enough skills to use the scaffold and 4. learners are motivated to use it. Learners’ perceptions Discussion & Conclusion [2] Philip H Winne. 2006. How software technologies can improve research on learning and bolster school reform. Educ. Psychol. 41, 1 (Mar 2006), 5–17. [1] Jaclyn Broadbent, Ernesto Panadero, Jason M Lodge, and Paula de Barba. 2020. Technologies to enhance self-regulated learning in online and computer-mediated learning environments. In Handbook of research in educational communications and technology. Springer, Cham, 37–52.
- 25. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 25 • Analytics-based environments that enhance learning • Evaluation of the environments using LA Contribution • Training and feedback using LA • Empirical research for evaluating interventions • Involving stockholders Implications Discussion & Conclusion
- 26. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 26 Discussion & Conclusion • Duration • No ground truth • Does not consider learning gain • Data from only two course • Longer duration • Obtaining ground truth • Pre- and post-tests • Data from several courses (>10) Limitations Future work
- 27. Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing 27 Introduction Results Discussion Questions Method
- 28. 28 Thank you! Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing Hatim Lahza h.lahza@uq.edu.au A/prof. Hassan Khosravi h.khosravi@uq.edu.au A/prof. Gianluca Demartini g.demartini@uq.edu.au Prof. Dragan Gasevic Dragan.Gasevic@monash.edu
- 29. 29
Editor's Notes
- Hi I’m Hatim Lahza, a PhD candidate at the University of Queensland. Today, I’m going to present our work that has been done collaboratively with my great supervisory team, Professor Hassan Khosravi and Professor Gianluca Demartini from the University of Queenslad, and Professor Dragan Gasevic from Monash University. Our paper title is “Effects of Technological Interventions for Self-regulation: A Control Experiment in Learnersourcing”.
- I’m going to start with a brief introduction and abit of a background, then methods, then results, and then Discussion and conclusion.
- As we all know, a part of instructors' responsibilities is supporting students in need. But, when class size increases, it's challenging for them to meet each individual student needs. This issue is mitigated with modern computer-supported learning environments as they can scale instructional solutions or scaffolds in large classes. However, there is no clear guidance on how self-regulation strategies can be integrated with them -------------------------------------------------------------------------------- Here I'm going to share the motivation of this study. As we all know, a part of instructors' responsibilities is supporting students in need. But, when the class size increases, it's challenging for them to assist students. Fortunately, computer-supported learning environments can scale instructional solutions very well in large classes. But, there is no clear guidance on how they can be integrated with self-regulation strategies
- In our study we have two main objectives: Designing an online educational environment interfaces based on self-regulated learning strategies Conducting an in-the-field experiment to evaluate our design ==================================================================================================== We choose self-regulated learning as it’s an essential lifelong learning skill that can help students to take hold of their learning tasks and reduce their dependency on teachers. As a background, there are many different models of self-regulated learning from different psychological and educational perspectives. But, in this study we choose to look at their commonalities. In Panadero’s work, we could see that most of the common SRL models are cyclic processes and have preparatory, performance and evaluation phases. Moreover, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of self-regulated learning effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice. In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor. ----------------------------------------------------------------------------- So, why self-regulation strategies and what do they consist of? Self-regulated learning is considered an essential competence and lifelong learning skill. The most common SRL models are cyclic and have three common phases: task analysis in which the plans and goals are set; followed by Performance in which tactics and strategies are implemented with monitoring And the third phase is “evaluation”, which involves reflection, self-assessment, regulation and adaptation In addition, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of SRL effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice. In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor.
- Why self-regulated learning? We choose self-regulated learning as it’s an essential lifelong learning skill that can help students to take hold of their learning tasks and reduce their dependency on teachers. There are many different models of self-regulated learning from different psychological and educational perspectives. But, in this study we choose to look at their commonalities. In the literature, we could see that most of the common SRL models are cyclic processes and have preparatory, performance and evaluation phases. Moreover, a self-regulated learner should practice agency by choosing, adapting and developing strategies and instructions to reach their goals. On the other hand, a poor self-regulated learner can be supported by scaffolds within educational technologies. However, evidence of self-regulated learning effectiveness in online learning environments are still non-conclusive and insufficient to rigorously guide the practice.
- In our study, we use a learnersourcing environment: Which refers to the activity of engaging students in creating learning resources. This type of activity can benefit learners in many areas such as: Promoting active learning Supporting evaluative judgment skill Reducing Instructor load And more Past studies suggest that learner has the ability to create high-quality resources. However, not all contents created by students are high-quality. We assume that the creation interface might be part of the issue as they are usually designed for the instructor.
- In our study we used RiPPLE which is an adaptive learning environments that enables learnersourcing of different types of learning resources such as MCQs and worked examples. It also supports learner models so that students can understand and monitor their performance in the platform. The process graph here is a high level overview of the process showing the three main types of activities, creating learning resources, moderating learning resources and practicing resources tailored to students abilities. And here is the original interface for creating a learning resource which is in this case a multiple choice question. So, the students first write the stem of the question, then choose the associated topic, then create the correct response and the distractors, and lastly write an explanation for the question.
- To reach or objectives, we first developed interventions for SRL strategies In our study, we considered three main interventions, namely planning then creating externally-facilitated monitoring And, self-assessing after creating Additionally, we considered a fourth intervention that combine all of the strategies Then we created an interface corresponding to each of the interventions as I'm going to show in the next slides
- In our study, the first self-reglation strategy is the planning then creating which includes six main elements. We hypothesis that the planner can benefit both students who have a clear idea of what to create and those looking for new ideas Here in the figure, the first step is: Inspiration source of the resource ides The target concept or skill The learning level according to Bloom’s taxonomy. The students were also provided with a short presentation that explains how to choose the appropriate level which we will see in the next slide Level of difficulty of the item Mistakes or misconception related to the question Type of media to be used It’s important to mention that here we focus only on multiple choice questions And also these components are optional
- This slide shows the presentation provided to students on learning levels. The presentation includes the definitions of the learning levels and the associated verbs. Also, exemplars and general tips were provided
- The second intervention is the externally-facilitated monitoring that should support the SRL “performance” phase. We aim to provide monitoring elements while creating a resource These elements support: Best practices in writing the body of the question Best practices in writing plausible distractors And best practices in writing explanation of the correct answer We presented these elements as tick boxes
- The self-assessing after creation is the third intervention and it aims to support the SRL evaluation phase Our interface consists of self-assessment and reflection. The self-assessment was built according to the rubric with a set of criteria used in the moderation activity. In the reflection component, the students were provided with an open-ended question to write their reflection on the resource they created.
- We used the following research questions to guide our study To reach our second objective, we conducted an in-the-field randomised control experiment that considers a control group A and four other treatment groups: planning then creating (B) monitoring while creating (C) self-assessing after creating (D) All (E)
- The study was conducted at the university of Queensland. We used data from two courses, Neuroscience & Information systems. The participants were 797 undergraduate students who provided their consent on the platform. RiPPLE was used for assessment over multiple rounds and contributed 10% to the final grade. But engagement with the intervention was optional. So, the creation task along with other tasks such as the moderation or practices were required to gain the 10%, but not the intervention. The resources created by the students for both courses were about 2-3 per student. We considered a student to be engaged if they at least completed all the components of an intervention. So, an engaged student is the student how at least completed all the intervention components at one session
- Table 1 provides an overview of the experimental groups. In total, the students created 1907 resources. The students in each group created from 300 to 400 resources. The engaged students were more than 60% in both courses.
- Here are the RQs again. In our analysis, we used the combined data. First, we measured the impact of the interventions on the creation process and effort. To do so, we modelled the creation processes using the trace data and First Order Markuv Model. We also conducted a qualitative analysis using the time spent by students in min. Second, we measured the impact on student perception. Students rated t he effectiveness of the interfaces using a scale of 5 after each creation. Also, we used 137 responses to an open-ended question provided by 103 students. Third, we used descriptive and inferential statistics on the effectiveness rating, and we used a general inductive approach and sentiment analysis on the written feedback. We measured the impact on performance by using qualitative analysis on the resource quality rating provided by the system.
- Here we present the result of RQ1 in which we investigate the impact of the interventions on the process and effort The visualisation of the process consists of connected oval shapes representing the type of activity and arrows showing the direction of the connection. The strength of the connection is indicated by the thickness of the arrows and the transition probability next to it. Inside the oval shapes, we added the fractional time the students spent on the activities. Key observations In general the interventions changed the process in which the students create learning resources, but with additional complexity. Condition B: The students used the planner and there was a strong interaction between the planning and creation components However, the students sometimes skipped the planning component (28%). This might mean many different things, such as the students knew what to create. We can see that the panner required extra time to complete the creation task and this was statistically different than the control group. Condition C: Similar to the planner, the students used the monitoring components. There was large probability of ending the process after writing the resource (40%). I expected this to be lower. The students spent only 4% of the total time on the scaffolding. This is similar to previous findings [1]. Condition D: strong interactions between the self-assessment and creation (Cr –> Sa: 87% and Sa –> Cr: 32%). Condition E: Combination of the processes observed in the other groups
- In RQ2, we investigate the impact of the interventions on Student Perceptions Table 2 summarises the results And here, we order the conditions based on the average of the student rating It seems that the students liked the control interface more than the treatment interfaces Next, the qualitative analysis will reveal more information
- Here we present the qualitative analysis results. We identified three main themes, technology, pedagogy and generic In table 3 we summarise the results with some examples of students’ responses In summary,
- Here we investigate the impact on the performance. It’s important to recall that the performance here is measured by quality rating that the resource receive from the moderation process. Figure three summarises the results of RQ3. We can see that for both “all students” and “engaged students”, condition “B” and “C” had the higher median followed by condition D. The control group and condition E had the least median. The difference between condition C and A was statistically significant for the engaged students with small effect size.
- Our results indicate that: Scaffolds increased task complexity and time with no significant improvements Similar results observed in previous research [1] Many factors can help explain why the SRL strategies might have a small or no impact on content creation. The students could have different regulation levels Some students could low prior knowledge Technical difficulties that we found in the qualitative analysis Also, our results indicate that the Interventions were not perceived as effective Which means that the metacognitive prompts were not assisted by the learners positively: This can be explained by Winne [2] four conditions According to Winne [35], for learners to use a learning tool that introduces metacognitive scaffolds, four conditions need to be met: - learners are aware that the scaffold is useful to them, learners know that the scaffold can be useful for the task at hand, learners have enough skills to use the scaffold and learners are motivated to use it.
- Our work contribute to the growing interest in creating analytics-based environments that enhance learning and using learning analytics to evaluate such environments We highlight the need for combining the newly introduced scaffolds with rationale for their value along with training to facilitate student acceptance and adoption. The training can be delivered with LA personalised feedbacks We emphasize the importance of empirical research in measuring the effectiveness of theory-based technological interventions and the involvement of stockholders in assessing interventions
- We acknowledge some limitation of our work that we plan to overcome in our future work. The duration was relatively short which may not have allowed the benefits of the interventions to be fully achieved. We only used the rating provided by the system which might not be as accurate as instructors rating We also did not consider the impact of the interventions on students learning Finally, we did not use data from multiple courses