Recommended
Recommended
More Related Content
Viewers also liked
Similar to LAK13 Supporting Action Research with Learning Analytics
Similar to LAK13 Supporting Action Research with Learning Analytics (20)
Recently uploaded
Recently uploaded (20)
LAK13 Supporting Action Research with Learning Analytics
- 1. 3rd Conference on Learning Analytics and Knowledge (LAK’13) Leuven, Belgium, 8-12 April, 2013 Anna Lea Dyckhoff Vlatko Lukarov Arham Muslim Mohamed Amine Chatti Ulrik Schroeder
- 2. LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov 2
- 3. LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov 3
- 4. Students Teachers Requirements Useful Tool Action Research Diverse Scenarios Non-experts Different Questions University Related Indicators 4 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 5. Action Research and Learning Analytics Summary of Prior Research Research Questions and Method Discussion of Findings Recap 5 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 6. Active improvement of teaching Feedback for awareness & reflection Human-driven Data-driven Manually Automatically Individual, perfectly fitting Standardized, approved by experts Limited by time-constraints ... Limited by missing data ... 6 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 7. Develop Question Share Formulate Research Plan Action Research Record in Collect Data Writing Develop Action Analyze & Reflect Learning Analytics Plan 7 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 8. Do native speakers How do students Are there specific have fewer problems value specific learning learning offerings NOT with learning offerings offerings? used at all? than non-native speakers? Are students using specific learning Which didactical Is the performance in materials (e.g., lecture acitivities, e.g. email e-tests somehow recordings) in addition notifications, facilitate related to exam to attendance or continous learning? grades? alternatively? Dyckhoff, 2010 8 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 9. Q1 - Indicator-Question-Mapping: Which questions cannot be mapped to the available (sets of) indicators? Which indicators could deliver what kind of enlightenment? Q2 - Teacher-Data-Indicators: Are there tools that explicitly correlate teacher data with student data? How should teacher data be correlated? Q3 - Missing Impact Analysis: How could learning analytics impact teaching? How could this impact be evaluated? 9 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 10. Study is based on results of a qualitative meta-analysis (Dyckhoff, 2011) collection of publications on 27 learning analytics tools Collected Question Analysis 198 Categorized Indicator and Indicators Mapping Discussion 10 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 11. 11 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 12. 12
- 13. Questions that can be answered EASY NOT so EASY Qualitative measure Effects on performance Quantitative measure Complex correlation Different data sources Missing: Teacher Data Indicators 13 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 14. • Current situation • Possible Scenarios Sociogram (Dawson, 2010) - Forum Participation - Correspondence Indicator - Average assignment grading time 14 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 15. Tool Classroom Impact Exit Conclusions Availability State Prediction Interviews 15 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 16. International Context Set of all indicators German International teachers’ indicators of 27 questions available tool 16 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 17. Supporting Action Research with Learning Analytics Several Questions NOT yet answered by indicators Teacher Data Indicators should be researched What’s the Impact of Learning Analytics? Participatory Design for LA Tools/Indicators Inform Tool Design/Learning Technologies Design 17 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 18. Dyckhoff, A.L. 2011. Implications for Learning Analytics Tools: A Meta-Analysis of Applied Research Questions. IJCISIM. 3, (2011), 594–601. Dawson, S. 2010. “Seeing” the learning community: An exploration of the development of a resource for monitoring online student networking. BRIT J EDUC TECHNOL. 41, 5 (Sep. 2010), 736–752. Dyckhoff, A.L., Lukarov V., Muslim A., Chatti M.A. and Schroeder, U. 2013. Supporting Action Research with Learning Analytics. Proceedings of LAK’13, April 08- 12 2013, Leuven, Belgium, 220-229. 18 LAK13 "Learning Analytics and Action Research" Dyckhoff, Lukarov
- 19. Indicator Documentation Name and Snapshot Description/Use Case Data Sources Limitations 19
Editor's Notes
- Version 1:In yesterday’s workshop on teacher’s analytics, during the discussion of one talk the question was raised, what’s the impact on Learning Analytics?Version 2: So what’s the impact of Learning analytics? This questions was actually raised at yesterday’s workshop on teaching analytics.
- When you read the literature, I extracted goals that were mentioned in the paper. I sorted them by goals concerning tools, what tools should do. For example track users, track interactons, etc. There are also goals for educators…And also there are goals for students…From this table, you can find out how Learning Analytics should/supposed to impact the teaching and learning processes. This is how it is supposed to be, but in the papers there is no information how the actual situation..there are the goals that come in literature (a collection of goals) about how Learning analytics should impact the learning/teaching processes
- We had problems identifying what questions exactly these indicators answer/represent. Only a few indicators were well documented.For us, it was unclear what data exactly these indicators need, in order to calculate and present information to the teachers. Also, only a few of them provided information about the limitations concerning usage/interpretation. This led us thinking that we need to figure out a way how to present,and structure these pieces of information, which is crucial and necessary. If we, as researchers (and experts in the field) have problems identifying these things (issues), then I am positively inclined to say that educators, teachers will definitely have problems in understanding the purpose of the indicators.
- This is a sample snapshot of our mapping of indicators with questions
- Quantitative measureDiverse indicators can answer these questions, because they are concerned with quantitative measures on the basis of student generated usage data.Qualitative measures which are concerned with the satisfaction and preferences of students, still cannot be answered sufficiently. Performance indicators are also present in the research communities, but not in the tools we analyzed. There are questions that cannot be mapped, or answered by single indicators, or from single data source. So this raises the question of combining different data sources and making complex correlations. Don’t get me wrong, there are indicators that calculate complex correlations (especially in EDM), but we have seen only few indicators in the tools we analyzed. Furthermore, our analysis and discussion revealed that we have something missing… We don’t really have Indicators that represent Teacher data.
- The closest thing that we have to using teacher data, or having teachers’ indicators are sociograms from Social Network analysis which correlate Student and Teacher activities. They correlate data of the course phase, course events, and show the interaction between teachers and students.Of course we need to collect teacher data in order to build these indicators. There already exists activity data from teachers available from log files. But, we also need to think about ways how to collect the missing data such as data about the quality of the learning materials, metadata of information about course events, and activities outside the classroom/lecture.Possible scenarios:We think that it would be good for the teachers to see the patterns of their participation and their online presence in the forum discussions. Also, it would be interesting to see whether in the periods when the teacher was more active and more present on the forum, students’ online presence and activities also increased. For the other two, you are more than welcomed to read our paper. :-) Missing indicators does not mean that they don’t exist in the research world.There isEDM research that does this.There are tools/indicators that exist but they might be not usable for the normal user. And these ideas led us to our thrid research questions, about how we can analyze the impact of indicators and the LA tools.
- This is not THE method how to measure impact. This is just a method that we want to discuss, and we want to know what do you think how this method can be improved to measure the impact of the tool. This is very similar to design based research, but here we try to evaluate, measure, and describe more closely the impact of these tools. Make the tool available to diverse group of users (Non experts) for use during an extensive amount of time. We ourselves have a LA tool, and we started impact analysis and evaluation of it last semester. It was available to the teachers for about 10 weeks, and it turned out that’s not enough. It shows some impact, but we need longer studies for at least a whole semester. In our method we suggest to discern and understand the current state and of the classroom, so we have a reference point for comparison and measurement later in the study. In our study, we did this with questionnaires. And this questionnaire should not be revealing or providing information that might bias the activities that come after. This is why we try to find out what are they doing now. (preferably not related to LA, to see whether they are already reflecting on their teaching, and what they think about the students?) Once the current state is well established, we need to identify which activities are likely to improve by the usage of the LA tool/indicators. This is the place where we pose hypotheses how the usage of the tool will improve (have impact on) the behavior of students and teachers, their activities, and the learning process. With out tool, we have the goal to initialize action research, or initiate awareness on a lower level, or reflection on a higher level. Conduct interviews, ask questions, give questionnaires to be filled in, evaluation forms for feedback with both teachers and students. These can help in capturing their personal feelings, and opinions after using the tool in the given time period. Did a specific indicator puzzled them? Did they reflect upon the data presented to them? Did they revise the action plan? In the exit interviews, we are trying to find any signs for action research initiations. The final step is to compare the results in a “before/after” fashion to draw conclusions on how the hypothesized impact relates to the actual impact of the learning analytics tool.
- The point in every context you have specific questions. If the international tools and indicators don’t fit into our specific context, we need to create our own tool. The idea behind is to acknowledge that there exist questions in the context that cannot be answered, and these questions should be addressed by the LA tool. We are aware of the limitations, and you should also be aware of these limitations. There will be new questions, new data.
- Put reference from an exact location, reference it.