Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SHEILA project LAK17 workshop slides

711 views

Published on

Slides from the LAK17 SHEILA workshop "LA Policy: Developing an Institutional Policy for Learning Analytics using the RAPID Outcome Mapping Approach"

Published in: Education

SHEILA project LAK17 workshop slides

  1. 1. Supporting Higher Education to Integrate Learning Analytics http://sheilaproject.eu/ LA Policy: Developing an Ins4tu4onal Policy for Learning Analy4cs using the RAPID Outcome Mapping Approach LAK’17 Workshop 13th March 2017 Yi-Shan Tsai, Dragan Gašević, Pedro J. Muñoz-Merino, Shane Dawson, Maren Scheffel, Alexander Whitelock-Wainwright
  2. 2. Workshop schedule Time Schedule 13.00-13.05 Welcome & introduc5on 13.05-14.30 Five presenta5ons & discussions 1.  Overview & LA in Australia: Prof Dragan Gasevic (University of Edinburgh) 2.  LA in Europe (SHEILA Project): Prof Pedro J. Muñoz-Merino (Universidad Carlos III de Madrid) 3.  SHEILA – group concept mapping: Maren Scheffel (Open University of the Netherlands) 4.  SHEILA – ins5tu5onal interviews & survey: Dr Yi-Shan Tsai (University of Edinburgh) 5.  SHEILA – student survey: Alexander Whitelock-Wainwright (University of Liverpool) 14.30-15.00 A^ernoon Tea 15.00-16.00 Group work on the assessment of ins5tu5onal readiness 16.00-17.00 LA Policy dra^ing
  3. 3. National project to benchmark LA status, policy and practices for Australian Universities Dragan Gasevic (thanks Shane Dawson for the slides!)
  4. 4. Introduction Massive interest and investment in data and analy5cs •  Academic performance •  Student reten5on •  Pastoral care •  Academic literacies •  Social networks – collabora5ons
  5. 5. Introduction Yet in terms of wide-scale ins5tu5onal adop5on there are few examples Why?
  6. 6. Aims   understand current LA practice in Australia   unpack the challenges to institutional adoption   identify practices that can aid the implementation of LA
  7. 7. Approach 2 complementary but separate studies   Study 1 – interviews with senior institutional leaders   Study 2 – concept mapping with LA expert panel
  8. 8. Study 1 First study Interviews with 32 Universities:   Identification of current practice, methods and approaches   Identification of key drivers for institutions, stage of development, process for implementation, project leads
  9. 9. Study 1   Much interest in LA   Stated organisational priority   LA projects were in the early phases of implementation and small scale (at time of interview July 2014)   2 distinct clusters across variables such as: implementation, conceptualisation, readiness Cluster 1 (n=15) – Solutions focused Cluster 2 (n=17) – Process focused
  10. 10. Strategic capability
  11. 11. Strategic capability 1.  Solutions focused ›  LA to address a pressing need ›  Time sensitive 2.  Process focused ›  Networked and integrated model ›  Minimal time pressures ›  Innovation and experimentation
  12. 12. Study 2 What are the ideal dimensions for long term sustainable uptake of LA?   Invite to Australian and international LA experts   28 completed the entire concept mapping phases   Prompt: ‘for LA to make a continued impact on learning and teaching it would need to…’   3 phases – brainstorming; sorting and ranking of statements
  13. 13. Study 2
  14. 14. Bringing it together   Study 1 – 2 clusters   Study 2 – 7 clusters Essentially – how an organisation approaches its conceptualisation of LA underpins (2 clusters) the method for deployment and adoption (7 clusters)
  15. 15. Systems Model
  16. 16. Systems Model
  17. 17. Systems Model
  18. 18. Bringing it together Challenges to be addressed:   Leadership awareness   Teams are seldom interdisciplinary   IT driven and system focused   Scale versus understanding   Capabilities and skills deficit.   Over reliance on current research – requires further validation across different contexts to demonstrate transportability of models
  19. 19. Complexity Leveraging the outcomes of short term goals for long term gain   How do we merge both models to gain both short and long term impact?
  20. 20. Conclusion   LA requires alternate models for implementation and leadership ›  Enabling leadership ›  Whole of organisation ›  Models that are agile and research informed   Working in complexity creates friction ›  Embrace the friction – generates innovation
  21. 21. Conclusion   A solutions based model can drive change – but need to be mindful of responding to changing organsational needs   Process based model can drive innovation and interest – but need to be mindful of how to scale
  22. 22. Conclusion   Combined model framed in the organisational context ›  Small, diffuse pockets of innovation to build capacity and build interest ›  View to scale adoption – demonstration of impact (technical, pedagogical) ›  Distributed enabling leadership (complexity leadership)
  23. 23. Conclusion Any “successful” adoption of LA will be dependent on an institution’s ability to rapidly recognise and respond to the organisational culture and the concerns of all stakeholders.
  24. 24. Thank you
  25. 25. SHEILA Project - Overview - Dragan Gašević, Yi-Shan Tsai, Maren Scheffel, Shane Dawson, Pedro J. Muñoz-Merino March, 13, 2017 LAK SHEILA Workshop Vancouver, Canada hgp://sheilaproject.eu/
  26. 26. SHEILA Partners •  University of Edinburgh (UE) (Coordinator) •  Brussels Educa5on Services (BES) •  Open University Netherlands (OUNL) •  Tallinn University (TLU) •  Universidad Carlos III Madrid (UC3M) •  European Associa5on for Quality Assurance in Higher Educa5on (ENQA) •  Erasmus Student Network (ESN)
  27. 27. Policy development framework Used by different higher educa5on ins5tu5ons Recognizes ins5tu5on specific drivers Promotes learning analy5cs for forma5ve assessment hgp://sheilaproject.eu/
  28. 28. hgp://sheilaproject.eu/
  29. 29. Project approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP2
  30. 30. Project approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP1 and WP3
  31. 31. Project approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP3
  32. 32. Project approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP3
  33. 33. Project approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP4
  34. 34. SHEILA approach Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analy5cs impera5ve and the sociotechnical challenge: Policy for complex systems. Research & Prac+ce in Assessment, 9(Winter 2014), 17-28. WP4
  35. 35. Project ac5vi5es Literature review hgp://sheilaproject.eu/
  36. 36. Tsai, Y-S., Gašević, D. (2016). Adop+on of Learning Analy+cs in Higher Educa+on: State, Challenges, and Policies (Execu+ve summary). SHEILA Project Report, hgp://bit.ly/sheila_lr_es
  37. 37. Project ac5vi5es Literature review Interviews with decision makers hgp://sheilaproject.eu/
  38. 38. Project ac5vi5es Literature review Interviews with decision makers Group concept mapping with experts hgp://sheilaproject.eu/
  39. 39. Project ac5vi5es Literature review Interviews with decision makers Group concept mapping with experts Surveys and focus groups with students and teaching staff hgp://sheilaproject.eu/
  40. 40. Expected 5meline Policy development framework (v1) – June 2017 Four ins5tu5onal policies (v1) – June 2017 hgp://sheilaproject.eu/
  41. 41. Expected 5meline Policy development framework (v1) – June 2017 Four ins5tu5onal policies (v1) – June 2017 Policy development framework (v2) – March 2018 Four ins5tu5onal policies (v2) – March 2018 hgp://sheilaproject.eu/
  42. 42. Interested in joining? Associate partners as early adopters of the policy development framework are welcome hgp://sheilaproject.eu/
  43. 43. Many thanks! hgp://sheilaproject.eu/
  44. 44. Group Concept Mapping Maren Scheffel Open Universiteit Nederland maren.scheffel@ou.nl @m_a_s_c
  45. 45. Group Concept Mapping Phase 1: Brainstorming Phase 2: Sor5ng Phase 3: Ra5ng An essen5al feature of a higher educa5on ins5tu5on's learning analy5cs policy should be ...
  46. 46. Point Map 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
  47. 47. Cluster Replay Map
  48. 48. Cluster Replay Map
  49. 49. Cluster Replay Map
  50. 50. Cluster Map 1. privacy & transparency 2. roles & responsibili4es (of all stakeholders) 3. objec4ves of LA (learner and teacher support) 4. risks & challenges 5. data management 6. research & data analysis
  51. 51. Ra5ng Map – Importance 1. privacy & transparency 2. roles & responsibili4es (of all stakeholders) 3. objec4ves of LA (learner and teacher support) 4. risks & challenges 5. data management 6. research & data analysis Cluster Legend Layer Value 1 5.08 to 5.27 2 5.27 to 5.46 3 5.46 to 5.65 4 5.65 to 5.84 5 5.84 to 6.03
  52. 52. Ra5ng Map – Ease 1. privacy & transparency 2. roles & responsibili4es (of all stakeholders) 3. objec4ves of LA (learner and teacher support) 4. risks & challenges 5. data management 6. research & data analysis Cluster Legend Layer Value 1 3.79 to 4.12 2 4.12 to 4.45 3 4.45 to 4.78 4 4.78 to 5.11 5 5.11 to 5.44
  53. 53. Ra5ng Ladder Graph importance ease privacy & transparency privacy & transparency risks & challenges risks & challenges roles & responsibili5es (of all stakeholders) roles & responsibili5es (of all stakeholders) objec5ves of LA (learner and teacher support) objec5ves of LA (learner and teacher support) data management data management research & data analysis research & data analysis 3.79 3.79 6.03 6.03 r = 0.66
  54. 54. Go Zone – Privacy & Transparency 1 7 20 31 43 86 2 10 17 24 45 64 65 88 92 9 15 56 60 74 87 34 69 96 6.08 5.44 3.12 ease 3.83 6.03 6.59 importance r = 0.45
  55. 55. Go Zone – Privacy & Transparency 1 7 20 31 43 86 2 10 17 24 45 64 65 88 92 9 15 56 60 74 87 34 69 96 6.08 5.44 3.12 ease 3.83 6.03 6.59 importance r = 0.45 2. transparency, i.e. clearly informing students of how their data is collected, used and protected 88. a clear descrip5on of data protec5on measures taken 10. a clear descrip5on of data usage 17. being clear about the purpose for collec5on certain types of data 24. aligned with data protec5on regula5ons (ins5tu5onal, na5onal, interna5onal) 34. to assure that the collected data is used only for the purpose of improving learning and instruc5on 96. an agreement between learners, teachers and policy makers on regula5ng a proper use of data
  56. 56. Go Zone – Roles & Responsibili5es 5 38 62 11 19 22 33 39 48 70 91 25 28 37 40 55 61 66 27 47 49 6.08 4.72 3.12 ease 3.83 5.48 6.59 importance r = 0.26
  57. 57. Go Zone – Roles & Responsibili5es 5 38 62 11 19 22 33 39 48 70 91 25 28 37 40 55 61 66 27 47 49 6.08 4.72 3.12 ease 3.83 5.48 6.59 importance r = 0.26 55. being clear about the purpose of learning analy5cs 61. a clear ar5cula5on of roles and responsibili5es when it comes to the use of ins5tu5onal data 39. to promote broad adop5on of learning analy5cs by specifying suppor5ve regula5ons and case law
  58. 58. Go Zone – Objec5ves of LA 3 80 84 99 12 78 98 35 50 93 42 44 58 63 76 79 94 6.08 3.79 3.12 ease 3.83 5.44 6.59 importance r = -0.12
  59. 59. Go Zone – Objec5ves of LA 3 80 84 99 12 78 98 35 50 93 42 44 58 63 76 79 94 6.08 3.79 3.12 ease 3.83 5.44 6.59 importance r = -0.12 63. to use learning analy5cs for improving the quality of teaching 58. to allow for proac5ve behavior of students towards their educa5on, with sevng personal goals 44. to support learners learning to learn and improve skills 98. to encourage meaningful rela5onships between students
  60. 60. Go Zone – Risks & Challenges 18 41 83 26 46 97 53 59 90 95 6.08 4.16 3.12 ease 3.83 5.52 6.59 importance r = 0.52
  61. 61. Go Zone – Risks & Challenges 18 41 83 26 46 97 53 59 90 95 6.08 4.16 3.12 ease 3.83 5.52 6.59 importance r = 0.52 90. to draw agen5on to what the "big picture" end goal of the data use is 59. to demarcate clearly between different uses of analy5cs in ins5tu5onal sevngs 95. to ensure the benefits to students outweigh the risks 53. to ensure adequate representa5on during high-stakes decision processes 18. integrated with a reward system for the use of evidence-based improvements to teaching and learning 41. to discourage students and faculty from gaming the system
  62. 62. Go Zone – Data Management 4 14 89 23 67 71 85 6.08 4.5 3.12 ease 3.83 5.16 6.59 importance r = -0.54
  63. 63. Go Zone – Data Management 4 14 89 23 67 71 85 6.08 4.5 3.12 ease 3.83 5.16 6.59 importance r = -0.54 23. to define clear rules for collabora5on with other researchers
  64. 64. Go Zone – Research & Data Analysis 6 30 57 68 72 75 77 81 8 32 52 73 82 13 51 16 21 29 36 54 6.08 4.24 3.12 ease 3.83 5.08 6.59 importance r = 0.27
  65. 65. Go Zone – Research & Data Analysis 6 30 57 68 72 75 77 81 8 32 52 73 82 13 51 16 21 29 36 54 6.08 4.24 3.12 ease 3.83 5.08 6.59 importance r = 0.27 57. to assure that instruc5onal interven5ons are based on well-studied and empirically validated analy5cal methods and algorithms 75. to encourage the development of dashboards with meaningful and understandable outcomes 68. to recognise the limita5ons of learning analy5cs 72. to provide clear examples of how to interpret data, especially when there may be interac5ons 54. to enhance the connec5on between the academic and the society and industry, showing what is being done at the ins5tu5ons is related to the educa5on and training 21. to make explicit maths from innova5ve development / use of analy5cs in individual classrooms to scaling up
  66. 66. Thank You! Maren Scheffel Open Universiteit Nederland maren.scheffel@ou.nl @m_a_s_c
  67. 67. Institutional interviews •  Methodology •  Topics of the ques5ons •  Analysis •  Preliminary findings Researchers: Yi-Shan Tsai (University of Edinburgh) Ioana Jivet (Open University of the Netherlands) Pedro Manuel Moreno Marcos(Universidad Carlos III de Madrid) Kairit Tammets (Tallinn University)
  68. 68. Interviews – methodology •  Purpose: to understand the adop5on of Learning Analy5cs in higher educa5on ins5tu5ons through direct conversa5ons with decision makers. •  Sampling: deans, vice-deans, vice-rectors, vice principals, heads of IT or eLearning, and senior researchers or project managers of learning analy5cs. •  Length of interviews: 25 ~ 60 mins •  Period of data collec5on: August 2016 ~ January 2017
  69. 69. Interviews – methodology •  64 interviews •  51 HEIs •  16 countries * Two addi5onal interviews were carried out with a na5onal collabora5ve ICT organisa5on in Netherlands and the Ministry of Educa5on and Science in Estonia.
  70. 70. Interviews – topics •  LA projects, scope, mo5va5ons, and goals •  LA strategy •  Progress and achieved goals •  Challenges •  Ethics and privacy
  71. 71. Interviews – analysis Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9(Winter 2014), 17-28.
  72. 72. Interviews – preliminary findings •  Current state of adop4on NO PLANS IN PREPARATION IMPLEMENTED 9 7 5 12 18 THE ADOPTION OF LEARNING ANALYTICS Ins5tu5on-wide Par5al/ Pilots Data explora5on/cleaning N/A
  73. 73. Interviews – preliminary findings •  Step 1 – Map poli4cal context LA Learner Teaching Ins5tu5onal External q Data protec5on regula5ons •  2018 GDPR •  The strictness of exis5ng DP regula5ons à A stopper q Pressure to adopt LA q Exis5ng solu5ons focus on addressing reten5on problems. q No one-size-fits-all solu5ons
  74. 74. Interviews – preliminary findings •  Step 2 – Iden4fy key stakeholders Managers Students Teachers q  25 ins5tu5ons have established formal working groups. q  Not all ins5tu5ons have planned to provide analy5cs data to students. •  Concerns about demo5va5on. Professional Supports
  75. 75. Interviews – preliminary findings •  Step 3 – Iden4fy desired behaviour changes LA Learner Teaching Ins5tu5onal q Interven5ons •  E-mail alerts and personal contacts with students (7 HEIs) •  Teaching reports (1 UK HEI) q Feedback •  3 HEIs received posi5ve feedback •  Low response rates How can ins5tu5ons engage students with LA?
  76. 76. No defined strategy Interviews – preliminary findings •  Step 4 – Develop engagement strategy LA Digitalisa5on strategies Teaching & learning strategies q  Consulta4ons with primary stakeholders •  23 HEIs •  Surveys •  focus groups •  Workshops •  Annual staff development conference •  Self-help toolkit q  Visualised dashboard •  24 HEIs Danger of being data driven
  77. 77. Interviews – preliminary findings •  Step 5 – Analyse internal capacity to effect change Managers Students Teachers Professional Supports Ins5tu5onal context q  Technological resources q  Human resources q  Funding q  Ins5tu5onal culture •  Awareness •  Buy-in •  Common understanding •  Analy5cs skills Ø  Clear value and relevance Ø  Ins5tu5onal priori5es Ø  Workload Ø  Teaching styles Ø  Monitoring Ø  Confidence in handling new technology & analy5cs data Ø  Passive engagement with studies Ø  Lack of interest in certain subjects Ethics & Privacy Trust in LA
  78. 78. Interviews – preliminary findings •  Step 6 – Establish monitoring and learning frameworks •  Few were able to talk about plans for evalua5on. •  It is difficult to isolate and define the success of a learning analy5cs project that is implemented alongside other projects with the same goals to enhance learning and teaching. •  12 ins5tu5ons have developed or planned to develop a policy to accommodate the use of learning analy5cs.
  79. 79. Conclusion •  Success to date: •  Scaled up the ins5tu5onal capacity (step 5) •  Beger understanding of challenges •  Gained experience Siemens, G., Dawson, S., & Lynch, G. (2014). Improving the Quality and Produc5vity of the Higher Educa5on Sector - Policy and Strategy for Systems-Level Deployment of Learning Analy5cs. Canberra, Australia: Office of Learning and Teaching, Australian Government.
  80. 80. Institutional survey •  Topics inves4gated The development of LA •  Current adop5on •  Ins5tu5onal infrastructure and capacity •  Strategy and policy •  Legal and ethical considera5ons •  Evalua5on Self-evalua5on of LA maturity and Ins5tu5onal readiness •  LA maturity •  LA success •  Culture •  Data and research capabili5es •  Legal and ethical considera5ons •  Training and communica5on
  81. 81. Institutional survey •  Distribu4on: six consor5ums in Europe – EADTU (European Associa5on of Distance Teaching Universi5es), EUA (European University Associa5on), HeLF (Heads of e-Learning Forum in the UK), EUNIS (European University Informa5on Systems), SNOLA (Spanish Network of Learning Analy5cs) and the eMadrid Network. •  Survey 4me: September 2016 ~ February 2017 •  Returns: 46 ins5tu5ons (response rate: 15%) European HEIs in general were new to LA and hence did not feel equipped with sufficient experience or knowledge to answer the survey??
  82. 82. Institutional survey •  22 countries NO PLANS IN PREPARATION IMPLEMENTED 2 13 15 16 The adop4on of LA Ins5tu5on-wide Small scale N/A
  83. 83. Institutional survey •  Mo4va4ons for adop4ng LA Among the 11 op5ons for mo5va5ons specific to learning and teaching, the top 5 drivers are: § To improve student learning performance (16%) § To improve student sa5sfac5on (13%) § To improve teaching excellence (13%) § To improve student reten5on (11%) § To explore what learning analy5cs can do for our ins5tu5on/ staff/ students (10%)
  84. 84. Institutional survey •  Strategy and evalua5on •  Among ins5tu5ons that have implemented LA: § There is a strategy: 20% § In the process of developing a strategy: 20% § No clear strategy: 46.7% § Have developed success criteria: 27%
  85. 85. Conclusion •  ~ Half of the European countries inves5gated have HEIs that are either observing the development of learning analy5cs or have engaged with it prac5cally. •  The early adopters are likely to scale up the culture for learning analy5cs in Europe.
  86. 86. •  SHEILA will assist with the development of learning analy5cs policies in four ins5tu5ons drawing upon the output of the project. •  SHEILA will push the development of a long term learning analy5cs policy agenda and community among higher educa5on ins5tu5ons. What next?
  87. 87. •  Become an associate partner of the SHEILA project? •  Visit: hgp://sheilaproject.eu/ Yi-Shan Tsai Research associate University of Edinburgh @yi_shan_tsai yi-shan.tsai@ed.ac.uk
  88. 88. What do students want? Towards an instrument for students’ evalua6on of quality of learning analy6cs services Alexander Whitelock-Wainwright, Dragan Gašević, & Ricardo Tejeiro A.Wainwright@Liverpool.ac.uk
  89. 89. Aims •  Highlight the importance of service quality in learning analy5cs. •  Develop an instrument to explore student expecta5ons towards learning analy5cs. •  Steps towards a model of learning analy5cs use.
  90. 90. What is service quality? •  Subjec5ve assessment of the degree to which a service user’s needs or expecta5ons were met (Parasuraman, Zeithaml, & Malhotra , 2005). Expecta5 ons Service Usage/ Exposure Percep5 ons Avtu de
  91. 91. What is service quality? (Contd.) •  Encourages users to use own service over compe5tors (Parasuraman, Zeithaml, & Berry, 1988). •  Service quality in higher educa5on (Spooren, Brockx, & Mortelmans, 2013). •  Ideological gap (Ng & Forbes, 2009).
  92. 92. Service Quality in Learning Analy6cs •  Learning analy5cs services designed to support learning. •  Various stakeholder groups within learning analy5cs (e.g., students, teachers, managers) (Clow, 2012). •  Quality indicators of learning analy5cs tools (Scheffel, Drachsler, Stoyanov, & Specht, 2014).
  93. 93. Ques6onnaire Development •  Measuring student expecta5ons of learning analy5cs. •  Development of ins5tu5onal policies (SHEILA). •  Theories of human behaviour.
  94. 94. Ques6onnaire Development (Contd.) •  Iden5fying themes within past literature (Ifenthaler & Schumacher, 2016; Sclater, 2016; West, Heath, & Huijser, 2016): •  Ethics and Privacy •  Meaningfulness •  Agency •  Interven5ons
  95. 95. Ques6onnaire Development (Contd.) Expecta5 ons Predic 5ve Desir ed Swan & Trawick (1980)
  96. 96. Ques6onnaire Development (Contd.) Created 79 Items Peer Review Pilot Study with 37 Items Exploratory Factor Analysis Redistribute the Ques5onnaire
  97. 97. Pilot Study Results •  Instrument reduced to 19 items. •  Two factor solu5on for both scales: •  Service expecta5ons: o  Desires scale – 0.88 alpha. o  Predic5ve scale – 0.88 alpha. •  Ethical expecta5ons: o  Desires scale – 0.82 alpha. o  Predic5ve scale – 0.86 alpha.
  98. 98. Pilot Study Results (Contd.)
  99. 99. Pilot Study Results (Contd.)
  100. 100. Pilot Study Results (Contd.)
  101. 101. Pilot Study Results (Contd.)
  102. 102. Pilot Study Results (Contd.)
  103. 103. Future Direc6ons •  Develop the corresponding percep5ons scale. •  Model inten5ons towards using learning analy5cs. Avtu des Social Norm s Inten5ons to Use Learning Analy5cs Perceive d Behaviou ral Control
  104. 104. Ques6ons?
  105. 105. Workshop schedule Time Schedule 13.00-13.05 Welcome & introduc5on 13.05-14.30 Five presenta5ons & discussions 1.  Overview & LA in Australia: Prof Dragan Gasevic (University of Edinburgh) 2.  LA in Europe (SHEILA Project): Prof Pedro J. Muñoz-Merino (Universidad Carlos III de Madrid) 3.  SHEILA – group concept mapping: Maren Scheffel (Open University of the Netherlands) 4.  SHEILA – ins5tu5onal interviews & survey: Dr Yi-Shan Tsai (University of Edinburgh) 5.  SHEILA – student survey: Alexander Whitelock-Wainwright (University of Liverpool) 14.30-15.00 A^ernoon Tea 15.00-16.00 Group work on the assessment of ins5tu5onal readiness 16.00-17.00 LA Policy dra^ing
  106. 106. •  Name •  Organisa5on •  Your experience with LA •  Expecta5ons of the workshop Tell us about you
  107. 107. Assessment of institutional readiness •  Use the ROMA framework to map your progress towards deploying learning analy5cs. Macfadyen, L., Dawson, S., Pardo, A., Gašević, D., (2014). The learning analytics imperative and the sociotechnical challenge: Policy for complex systems. Research & Practice in Assessment, 9 (Winter 2014), 17-28. 1)  Map poli4cal context 2)  Iden4fy key stakeholders 3)  Iden4fy desired behaviour changes 4)  Develop engagement strategy 5)  Analyse internal capacity to effect changes 6)  Establish monitoring and learning frameworks
  108. 108. •  Discuss in your group what LA projects and developments have taken / are taking/ will take place in your organisa5ons LA projects in your organisations ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  109. 109. •  Is there government/teacher/student/ administrator pressure to deploy LA in your organisa5on? Or pressure not to deploy it? 1) Map political context ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  110. 110. •  Who / which people inside and outside your organisa5on would be / are the key stakeholders for the use of LA? •  Who will benefit from the use of LA? 2) Identify key stakeholders ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  111. 111. •  What are your reasons for adop5ng LA? •  What benefits is it expected to bring / have brought? 3) Identify desired behaviour changes ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  112. 112. •  How well prepared do you think your organisa5on is to deploy LA? •  What needs to be done to achieve the desired changes iden5fied previously? Areas to consider are ethical/ legal, technical, pedagogical, and cultural. 4) Develop engagement strategy ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  113. 113. •  Does your organisa5on has the capacity to implement the planned strategy? Do you need to acquire any skill-sets, funding, or technological systems? 5) Analyse internal capacity to effect changes ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  114. 114. •  How will you track and evaluate the progress? How will this process inform the next step of implementa5on? Will there be any success criteria? 6) Establish monitoring and learning frameworks ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  115. 115. •  Review the 6 ROMA steps. Can you iden5fy any challenges that may come up in any steps? (Put your notes in the same boxes.) Challenges ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  116. 116. •  Choose one case to work on as a group or work individually and share ideas in your group. Draft a LA policy 1.  Audience 2.  Purpose 3.  Process §  Data collec5on §  Data management §  Stakeholder engagement §  Evalua5on 4. Policy evalua5on Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  117. 117. •  1. Audience •  Who is this policy for? Consider whose working ac5vi5es the policy will shape. •  2. Purpose •  What are your policy objec5ves? Consider your objec5ves for LA and the changes you seek to achieve. The policy will set out as the reason for introducing LA. Audience & Purpose ~5mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  118. 118. •  3. Process •  a) Data collec5on •  b) Data management •  c) Stakeholder engagement •  d) Evalua5on Process ~15mins Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  119. 119. •  4. Policy management Policy management ~10mins Final check-ups •  Will the policy need to state anything that should NOT be done in rela5on to LA? •  Are there any limita5ons or poten5al challenges and risks about LA that should be addressed in the policy? •  Can you narrow the policy down to a short list of principles? Keep your group notes here: Group 1: hgps://goo.gl/CcGlTk Group 2: hgps://goo.gl/zrMsyi Group 3: hgps://goo.gl/3WNr62 Group 4: hgps://goo.gl/WzLmQc
  120. 120. •  Become an associate partner of the SHEILA project? •  Visit: hgp://sheilaproject.eu/

×