Successfully reported this slideshow.
Your SlideShare is downloading. ×

VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Loading in …3
×

Check these out next

1 of 31 Ad

VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.

Download to read offline

VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.

VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.

Advertisement
Advertisement

More Related Content

Slideshows for you (20)

Similar to VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017. (20)

Advertisement

More from eMadrid network (20)

Advertisement

VII Jornadas eMadrid "Education in exponential times"."Maturing the learning analytics framework for applied learning analytics". Hendrik Drachsler. OUNL, Países Bajos. 03/06/2017.

  1. 1. Fighting level 3: From the LA framework to LA practice on the micro-level Hendrik Drachsler, @hdrachsler LASI Spain, JULY 4-5, 2017 CC-by: https://www.flickr.com/photos/juandelpozo/6110435588/
  2. 2. 3 WhoAmI Full Professor of Educational Technologies and Learning Analytics School of Computer Science Goethe University Frankfurt German Institute for international Educational Research (DIPF) Frankfurt am Main, Germany Welten Institute Research Centre for Learning, Teaching and Technology Open University, Heerlen, The Netherlands
  3. 3. @HDrachsler, # _NL, Zeist, Netherlands Slide 3 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD?
  4. 4. 4 Greller, W. & Drachsler, H. (2012). Turning Learning into Numbers. Toward a Generic Framework for Learning Analytics. Journal of Educational Technology & Society. http://ifets.info/journals/15_3/4.pdf Learning Analytics Framework
  5. 5. Let‘s not forget, LA is for teachers and learners Learner’s and teachers are most of the located on the micro-level with their needs and skills
  6. 6. @HDrachsler, # _NL, Zeist, Netherlands Slide 7 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD?
  7. 7. Sophistican model Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Productivity of the Higher Education Sector – Policy and Strategy for Systems-Level Deployment of Learning Analytics. Canberra, Australia: Office of Learning and Teaching, Australian Government. Retrieved from http://solaresearch.org/Policy_Strategy_Analytics.pdf LA Sophistication Model 1 2 3 4 5 Who of you arrived at level 3 with their organisation? How long you are already in level 1 and 2?
  8. 8. Maturing, Roll-Out, Uptake? VERY URGENT for uptake of LA • Ethics & Privacy (Dragan keynote) • Stakeholder needs & Competences (Hendrik) • LA supported learning design (Hendrik) LESS URGENT for uptake of LA (but lots of fun) • Infrastructure • Data (Katrien keynote) • Dashboards (Katrien keynote)
  9. 9. @HDrachsler, # _NL, Zeist, Netherlands Slide 10 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD?
  10. 10. STOTA Dashboards 1. Verbert, et al., (2013) Learning analytics dashboard applications. American Behavioral Scientist 57(10), 1500 2. Verbert, et al. (2014) Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 18(6), 1499 3. Yoo, Y., Lee, H., Jo, I.H., Park, Y.: Educational dashboards for smart learning: Review of case studies. In: Emerging Issues in Smart Learning, pp. 145. Springer (2015) 4. Schwendimann, et al., Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies (2016) 5. Bodily, R., Verbert, K.: Trends and issues in student-facing learning analytics reporting systems research. In: Proc. of LAK'17. pp. 309{318. ACM (2017) For the question of uptake we have been investigating 26 out of 95 papers that have grounding in learning sciences: According to which educational concepts are learning analytics dashboards designed?
  11. 11. Educational concepts mentioned in LA dashboard papers Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practise. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate. 18 10 1 7 5 12
  12. 12. Competences mentioned in LA dashboard papers Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practise. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate. 34 8 12 5 13
  13. 13. Relation between educational concepts and competencies
  14. 14. Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practise. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate. Reference frames mentioned in LA dashboard papers 24 16 10
  15. 15. • Design of LA Dashboard are technology driven (only 26 out of 95 papers use edu. concepts) • Mostly to support awareness and reflection • According to McAlpine & Weston, refection is a mechanism to facilitate learning & teaching not an goal in itself. • More than half of the dashboards rely on SRL but none of the tool really support the cycle: fore-thought -> planning -> monitoring -> self-evaluation. • Social framing is more common than achievement framing According to which educational concepts are learning analytics dashboards designed? Jivet, I., Scheffel, M., Drachsler, H., and Specht, M. (2017). Awareness is not enough. Pitfalls of learning analytics dashboards in the educational practise. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017. Best paper candidate.
  16. 16. @HDrachsler, # _NL, Zeist, Netherlands Slide 18 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD?
  17. 17. Learning Analytics supported Learning Design Three main opportunities: 1. Using LA indicators for on demand measures of the learning design 2. Intervening during the runtime of a course due to LA indicators 3. Increasing students learning outcomes and interventions through earlier interventions with LA indicators
  18. 18. Opportunities and challenges for using LA in LD Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and challenges in using Learning Analytics in Learning Design. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
  19. 19. Opportunities and challenges for using LA in LD Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., Drachsler., H. (2017). Opportunities and challenges in using Learning Analytics in Learning Design. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
  20. 20. @HDrachsler, # _NL, Zeist, Netherlands Slide 22 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD?
  21. 21. Learning Analytics Readiness Instrument (LARI) Oster, M., Lonn, S., D. Pistilli, M., and Brown., M., G. (2016). The learning analytics readiness instrument. LAK '16. ACM, DOI: https://doi.org/10.1145/2883851.2883925A
  22. 22. Evaluation of the effect of LA • If you want to measure uptake of LA you need an instrument to explore progress and adoption • It needs to be a light way instrument as students and teachers don’t fill overwhelming long surveys again and again.
  23. 23. We were looking for something quick and robust like ... ... the SUS scale. But than deidicated for LA.
  24. 24. Scheffel, M., Drachsler, H., Stoyanov, S., and Specht, M. (2014). Quality Indicators for Learning Analytics. Educational Technology & Society, 17 (4):117–132. Scheffel, M., Drachlser, H., Specht, M. (2015). Developing an evaluation framework of quality indicators for learning analytics. LAK ’15, NY, USA. ACM. Scheffel, M., Drachsler, H., de Kraker, J., Kreijns, K., Slootmaker, A., Specht, M. (2017). Widget, widget on the wall, am I performing well at all? IEEE Transactions on Learning Technologies, 10 (1):42–52. Evaluation Framework for Learning Analytics (EFLA)
  25. 25. Evaluation Framework for Learning Analytics (EFLA) Scheffel, M., Drachsler, H., Kreijns, K., de Kraker, J., Specht, M. (2017). Widget, widget as you lead, I am performing well indeed!: Using results from an exploratory offline study to inform an empirical online study about a learning analytics widget in a collaborative learning environment, LAK ’17, NY, USA. ACM. Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
  26. 26. Evaluation Framework for Learning Analytics (EFLA) Quick evaluation of LA tools in micro-settings. We are looking for partners to try EFLA and further develop it. A great outcome would be N numbers of LA case studies that applied EFLA and share their experiences. Can we work towards an EFLA score like the SUS score?http://www.laceproject.eu/evaluation- framework-for-la/
  27. 27. @HDrachsler, # _NL, Zeist, Netherlands Slide 29 / 29 June 2014 1. LA Framework in a nutshell 2. Fighting level 3 Lecture structure 5. How to evaluate LA on the work floor? 3. Why are our dashboards not used? 4. How to connect LA and LD? CONCLUSIONS for Uptake of LA LA is about learning! LA is about learning!
  28. 28. The problem for uptake of LA and mastering level 3 is ... Educational Science Computer Science Learning Analytics MIDDLE SPACE
  29. 29. The solution, LA as a real middle space Educational Science Computer Science Learning Analytics MIDDLE SPACE
  30. 30. 32 @hdrachsler drachsler@dipf.de | hendrik.drachsler@ou.nl linkedin.com/in/hendrikdrachsler Many thanks for the invitation! Questions now or later Looking for PhD or PostDoc positions in computer science on Learning Analytics? Get in contact!

Editor's Notes

  • Who of you has a computer science and who has a more educational background?
    => HUH, this is great than we can nicely penetrated LA in the middle space between Informatics, Analytics and Educational Science


    Who of you is new (so less than a year) to LA research field?

    a. To bad you will not understand to much of this talk.

    b. Very good so all of you know what I‘m referring to in my title?


    The picture of the road as symbol to getting to Level 3 and finally rollout and use what we are working on since almost 7 years or longer.




  • Who of you know the LA framework?
  • Although we have implemented various dashboards, policies, tools, have a national SIG in NL and plenty of attention we are still more prototyping than transforming the university into a data driven organisation -> WHY? Any ideas from the audience?
  • Maybe this is the answer? Maybe LA is not that much the middle space between Educational technologies and Computer Science that it should be.
  • What needs to be done to get more uptake, roll-out, maturing of LA in the educational everyday life?

    What have you done to achieve the uptake?
  • Check programme again !!!!
  • Not taking away Katrien‘s keynote on Dashboards, so just the quick overview

    Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics dashboard applications. American Behavioral Scientist 57(10), 1500{1509 (2013)
    Verbert, K., Govaerts, S., Duval, E., Santos, J.L., Van Assche, F., Parra, G., Klerkx, J.: Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 18(6), 1499{1514 (2014)
    Yoo, Y., Lee, H., Jo, I.H., Park, Y.: Educational dashboards for smart learning: Review of case studies. In: Emerging Issues in Smart Learning, pp. 145{155. Springer (2015)
    Bodily, R., Verbert, K.: Trends and issues in student-facing learning analytics
    reporting systems research. In: Proc. of LAK'17. pp. 309{318. ACM (2017) 3 .Design and evluation of 10 edu dashboards
    Schwendimann, B., Rodriguez-Triana, M., Vozniuk, A., Prieto, L., Boroujeni, M., Holzer, A., Gillet, D., Dillenbourg, P.: Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies (2016) 55 dashboards (context, purpose, indicators & technologies used, evaluation, futuer work)
    Very little research about the effects of LA Dashboards, none of them investigated the connection to learning sciences or use of educational concepts.

    For the question of uptake we have been investigating 26 out of 95 papers that have grounding in learning sciences: According to which educational concepts are learning analytics dashboards designed?
  • Six clusters presentation of 17 educational theories, models, concepts identified in the LA Dashboard papers
  • Competencies and the goal that are inteded to affect each competence
  • Competences target by the dashboards in relation to educational concepts that were used as theoretical basis for their development.

    Strong emphasis on Self regulated learning, through Cognitivism. SRL is a concept that requires forethought, performance, and self-reflection. Less used are instructional design, and for formative assessment as means of reflection.
    Influence learners metacognitive skills. Surprisingly, very few dashboards aim to support learners on the cognitive level i.e. acquiring knowledge and improve performance
  • Wise, A., Learners need a references frame to interpret their data. We analysed this in the studied papers by looking at how information was contextualised on the dashboard.

    Two types:
    Social Comparison (most papers)
    Goal Achievement
  • The competence level target by the dashbaord in realtion to the three reference frames identified: social (S: red), achievement (A: blue), and progress (P: yellow)
  • With constantly being exposed to motivational triggers that rely on social comparison ‚being better than your peers‘ becomes the norm in what defines an successful leaner? We argue that learning and education should be about mastering knowledge and therefore social comparison should be used carefully in the design of dashboards.

    Attention should be given to different needs of learners with different performance levels and motivating factors.
  • These findings of LA Dashboards emphasis the need of desining LA dashboards as a tool embedded into learning design, and might be a reason for low uptake of LA in stakeholder groups.

  • Scheffel, M., Drachsler, H., Toisoul, C., Ternier, S., Specht, M. (2017). The Proof of the Pudding: Examining Validity and Reliability of the Evaluation Framework for Learning Analytics. 12th European Conference on Technology-Enhanced Learning. Tallinn, Estonia, 12-15 September 2017.
  • Does any one know what this is ?
  • Average scores of EFLA for students (left) and teachers (right)
  • Average scores of EFLA for students (left) and teachers (right)
  • LA is about learning and teaching

    Our dashboards are not that much designed for learning theories and models

    There are challenges for LA in LD that need to be addressed in each organisation as learning is strongly context based.

    In order to measure Uptake we need a light evaluation instrument like EFLA
  • If we really want to achieve level 3 of the LA sophistication model, LA needs to ‚really‘ become the middle layer between educational science and computer science. So far on educational dashboard side as well as connection of LA to Learning Design, LA is still to far away from educational practice and even promoting in some papers contra productive incentive to promote ‚better‘ learning like focusing on social comparison rather than goal achievement.

    We need to work closer with educational practise, take into account innovative learning and teaching concepts rather than visualising and comparing what is easy to do by the data we have.
  • If we really want to achieve level 3 of the LA sophistication model, LA needs to ‚really‘ become the middle layer between educational science and computer science. So far on educational dashboard side as well as connection of LA to Learning Design, LA is still to far away from educational practice and even promoting in some papers contra productive incentive to promote ‚better‘ learning like focusing on social comparison rather than goal achievement.

    We need to work closer with educational practise, take into account innovative learning and teaching concepts rather than visualising and comparing what is easy to do by the data we have.

×