SlideShare a Scribd company logo
1 of 30
Download to read offline
jobknowledge.eu
facebook.com/jobknowledge
@Jobknowledge
Barriers to Adoption for Learning
Analytics at a Dutch University
Dr. Stefan T. Mol, s.t.mol@uva.nl
Dr. Gábor Kismihók, g.kismihok@uva.nl
jobknowledge.eu
facebook.com/jobknowledge
@Jobknowledge
Overcoming
Barriers to Adoption for Learning
Analytics at a Dutch University
Dr. Stefan T. Mol, s.t.mol@uva.nl
Dr. Gábor Kismihók, g.kismihok@uva.nl
Who are we?
The Center of Job Knowledge Research is a
multidisciplinary thinktank and research
incubator that aims to engage stakeholders
from academia, industry, and government to
elucidate how job knowledge may be used to
facilitate individual, organization, and nation
level person-education-workplace matching
processes.
6/30/2014 LASI-Utrecht 2014 3
Contents
• UvaInform History
• Process and Pilots
• Organization and Project Management
• Lessons Learnt
6/30/2014 LASI-Utrecht 2014 4
The UvAInform Project - History
• Initiated as a proposal from the ICTS Department
• Expertise group Education ICT (EGO-ICT) Reserved budget of 150K
• EGO-ICT dislike of (bottom-up) tender procedure with limited
strategic vision
• Focus Group Learning Analytics Established Late 2012
• UvAInform proposal approved in June 2013
• Central infrastructure (LRS)
• (De?)centralized pilots
6/30/2014 LASI-Utrecht 2014 5
Responsibilities of the LA Focus Group
• Create vision on LA
• Avoid duplication of effort
• Starter engine for ethics board
• Source of advice for LA (related) projects
• Communication channel with the VU and HvA, Surf, &
Internationally
6/30/2014 LASI-Utrecht 2014 6
UvAInform Process
6/30/2014 LASI-Utrecht 2014 7
UvAInform Process
The process
Focus Group: evaluation criteria
6/30/2014 LASI-Utrecht 2014 8
Rated Evaluation Criteria
(1: Not at all important-7 - Extremely important)
1. Feedback for learning (yes/no) 6.33
2. Scalability of the project (Course / Faculty / Institution / National /
World-wide / National / International) 5.89
3. Feedback for teaching (yes/no) 5.86
4. Feedback for curriculum (yes/no) 5.75
5. Ethical concerns associated with the project (free text) 5.53
6. Use of the common infrastructure (multiple-select, none, dashboard,
Learning Record Store, Sakai CLE xAPI enabled) 4.94
7. Estimated Cost (Low < 10K, Medium 10K-20K, High > 20K)" 4.92
8. Production of open source software (yes / no) 4.83
6/30/2014 LASI-Utrecht 2014 9
Rated Evaluation Criteria
(1: Not at all important-7 - Extremely important)
9. Sustainability (No chance of adoption, planned as a local service, planned
as an UvA wide service) 4.83
10. Project risk (such as bottleneck with resources, project deadline (low,
medium, high) 4.78
11. Dissemination of knowledge (whitepaper, scientific publication) 4.78
12. Reusability from external projects (none, some, totally) 4.69
13. Product Risk such as nonsense feedback being provided to students (low,
medium, high) 4.61
14. Complexity (low, medium, high) 3.56
15. Novel research? (yes/no) 3.33
6/30/2014 LASI-Utrecht 2014 10
UvAInform Process
The process
Focus Group: evaluation criteria
Definition of initial pilots
Discussion with an external expert (Erik Duval)
Dilemma – how to agree on the separate pilots
Dilemma – To LRS or not to LRS?
6/30/2014 LASI-Utrecht 2014 11
6/30/2014 LASI-Utrecht 2014 12
Learning Record Store
• Community sourced, secure, scalable repository/infrastructure
• Store and retrieve statement data reliably and ensures a good
scalable storage layer for various types of data and data streams
• Scales above 100 billion records.
• These data can be made available in a secure and consistent way for
further analysis.
• Upon this LRS infrastructure dashboards can be built or developed for
the delivery of (analysed) data to students, educators, and
researchers
http://tincanapi.com
6/30/2014 LASI-Utrecht 2014 13
UvAInform Process
The process
Focus Group: evaluation criteria
Definition of initial pilots
Discussion with an external expert (Erik Duval)
Dilemma – how to agree on the separate pilots
Dilemma – To LRS or not to LRS
Definition of LRS as separate project
All faculties present their research ideas (Pecha Kucha Format)
Dilemma – How to agree on pilot requirements and associated budgets
Result: 7 small pilots in 3 pilot clusters
6/30/2014 LASI-Utrecht 2014 14
Cluster 1: Mirroring of traditional and non-
traditional study performance to students
• UvA Mirror (COACH)
Visualize the position of individuals in the context of the group
• Cluster Exam Feedback (qDNA)
More fine grained mirroring of exam results to provide students and
teachers with insight in the development on four competencies and
knowledge goals
6/30/2014 LASI-Utrecht 2014 15
Cluster 1: Goal Setting in Education
• Building on functionality of schedules (www.roosters.uva.nl) to
include goal setting and goal tracking.
• Students will be instructed/taught to formulate goals in a concrete
manner such that they are specific, measurable, attainable, relevant
and time-bound (SMART).
• Dashboard will facilitate individual students to choose from, and set
their own goals (and deadlines) against specific course events
deadlines based on mirrored data.
• Dashboard shows them in a glance how they are scoring/succeeding
in attaining the goals compared to their fellows
6/30/2014 LASI-Utrecht 2014 16
Cluster 1 Side project: Labour market
oriented learning trajectories
Match learners’ pathways to
those of alumni
Help incoming students
find a long term focus
during their university
education
Mirror alumni data
to current students
based on
desired/acquired job
roles
6/30/2014 LASI-Utrecht 2014 17
Cluster 2: Using specific student data to
provide feedback for teachers
• Assignments’ criteria validation
Visualization of Turnitin scores for students and teachers over (partial)
assignments or courses
• Web lecture statistics
Visualization of use of video’s/weblectures to find potential problems,
difficult material to improve teaching, provide better remedial material
6/30/2014 LASI-Utrecht 2014 18
Cluster 3: Using other people’s data to
provide recommendation system to students
• Reduce dropouts in Bachelor programs
Get existing and new data in gathering as much demographic personal
info on learning styles, motivation, personality to optimize prediction of
study success
Aim: First find predictors, then start developing interventions
• Still under negotiation
6/30/2014 LASI-Utrecht 2014 19
UvAInform Process
The process
Focus Group: evaluation criteria
Definition of initial pilots
Discussion with an external expert (Erik Duval)
Dilemma – How to agree on the separate pilots
Dilemma – To LRS or not to LRS
Definition of LRS as separate project
All faculties present their research ideas (Pecha Kucha Format)
Dilemma – How to agree on pilot requirements and associated budgets
Result: 7 small pilots in 3 pilot clusters
Challenge: Clarifying requirements and exploiting synergies
6/30/2014 LASI-Utrecht 2014 20
Organization and Project management
6/30/2014 LASI-Utrecht 2014 21
Stakeholder analysis
(Szörényi, 2014)6/30/2014 LASI-Utrecht 2014 22
Engaging critical stakeholders
• Potential users
• Students
• Teachers
• Legal/Ethical Committees
• University Administration
• Researchers
• Communities outside of the organisation
6/30/2014 LASI-Utrecht 2014 23
Data owners
• Who is owning what data?
• Highly political issue
• Organizational resistance
• Gatekeepers resistance
• Complex infrastructure
6/30/2014 LASI-Utrecht 2014 24
Lessons Learnt
6/30/2014 LASI-Utrecht 2014 25
Positioning a learning analytics project
• Centralized vs. decentralized
• Research vs. practice
• Imposed vs. desired
• Technology vs. Pedagogy
• Make or buy (resource oriented)
• Generic infrastructure vs. pilot specific infrastructure in relation to
UvA learning analytics lifecycle
• Adoption of best practices or local identification thereof
6/30/2014 LASI-Utrecht 2014 26
Lessons Learnt
• Organizational awareness is growing
• LA is on the agenda in many stakeholder groups
• Data is everywhere, but accessibility is an issue
• Need to document failures and agreements properly
• Legal and ethical concerns are high on the agenda
6/30/2014 LASI-Utrecht 2014 27
Towards a Center of Data Governance and
Innovation
• How do researchers/project members involved in data science
projects ensure full compliance with the UvA’s ethical standards?
• How do these people, whose expertise is seldom or never in the legal
domain, ensure that their work does not infringe on the law?
• Threat: bad publicity, legal costs due to litigation caused by
uninformed data management, efficiency losses due to different
parties having to reinvent the wheel (left hand does not know what
the right hand is doing)
6/30/2014 LASI-Utrecht 2014 28
Organizational Lessons
• Responsibilities are not clear – not connected to the budget
• Decisions are hard to make
• Internal policies and processes should be clearer
• Keep the university’s vision in focus
6/30/2014 LASI-Utrecht 2014 29
6/30/2014 LASI-Utrecht 2014 30

More Related Content

What's hot

Innovating Open Education: Critical Pathways and Communities of Practice
Innovating Open Education: Critical Pathways and Communities of PracticeInnovating Open Education: Critical Pathways and Communities of Practice
Innovating Open Education: Critical Pathways and Communities of PracticeRobert Farrow
 
Using learning analytics to support applied research and innovation in higher...
Using learning analytics to support applied research and innovation in higher...Using learning analytics to support applied research and innovation in higher...
Using learning analytics to support applied research and innovation in higher...Julià Minguillón
 
MOOC: Challenges for HE Institutions
MOOC: Challenges for HE InstitutionsMOOC: Challenges for HE Institutions
MOOC: Challenges for HE InstitutionsRobert Schuwer
 
Dutch acceleration plan, the butterfly effect 2.0
Dutch acceleration plan, the butterfly effect 2.0Dutch acceleration plan, the butterfly effect 2.0
Dutch acceleration plan, the butterfly effect 2.0Robert Schuwer
 
OER and MOOCs in the Netherlands: State of Affairs
OER and MOOCs in the Netherlands: State of AffairsOER and MOOCs in the Netherlands: State of Affairs
OER and MOOCs in the Netherlands: State of AffairsRobert Schuwer
 
Building an Open Operations Room for the OER Community #opened16
Building an Open Operations Room for the OER Community #opened16Building an Open Operations Room for the OER Community #opened16
Building an Open Operations Room for the OER Community #opened16Robert Farrow
 
Openness of education: Urgent need for recognizing OER-based learning outcome...
Openness of education: Urgent need for recognizing OER-based learning outcome...Openness of education: Urgent need for recognizing OER-based learning outcome...
Openness of education: Urgent need for recognizing OER-based learning outcome...EDEN Digital Learning Europe
 
Linq 2013 session_green_2_establish
Linq 2013 session_green_2_establishLinq 2013 session_green_2_establish
Linq 2013 session_green_2_establishLINQ_Conference
 
Data sharing and analytics in research and learning
Data sharing and analytics in research and learningData sharing and analytics in research and learning
Data sharing and analytics in research and learningJisc
 
Lessons learned_FINAL_07112014pptx
Lessons learned_FINAL_07112014pptxLessons learned_FINAL_07112014pptx
Lessons learned_FINAL_07112014pptxJuanita Fernando
 
160407 eua open education
160407 eua open education160407 eua open education
160407 eua open educationYves Punie
 
Closing the loop between learning and employability with OER: Impact of Brin...
Closing the loop between learning and employability with OER: Impact of Brin...Closing the loop between learning and employability with OER: Impact of Brin...
Closing the loop between learning and employability with OER: Impact of Brin...Robert Farrow
 
Opening up Higher Education in Europe
Opening up Higher Education in EuropeOpening up Higher Education in Europe
Opening up Higher Education in Europeiptsedu
 
Technology acceptance of augmented reality and wearable technologies ilrn 201...
Technology acceptance of augmented reality and wearable technologies ilrn 201...Technology acceptance of augmented reality and wearable technologies ilrn 201...
Technology acceptance of augmented reality and wearable technologies ilrn 201...Mikhail Fominykh
 
Characterising Professional Higher Education in Europe
Characterising Professional Higher Education in EuropeCharacterising Professional Higher Education in Europe
Characterising Professional Higher Education in EuropeAnthony Fisher Camilleri
 
Learning Analytics – Research challenges arising from a current review of LA use
Learning Analytics – Research challenges arising from a current review of LA useLearning Analytics – Research challenges arising from a current review of LA use
Learning Analytics – Research challenges arising from a current review of LA useRiina Vuorikari
 
Oer panel
Oer panelOer panel
Oer panelpmundin
 
Interim report and main findings, by Lasakova, Bajzikova & Sasso
Interim report and main findings, by Lasakova, Bajzikova & SassoInterim report and main findings, by Lasakova, Bajzikova & Sasso
Interim report and main findings, by Lasakova, Bajzikova & Sassogaihe
 

What's hot (20)

Innovating Open Education: Critical Pathways and Communities of Practice
Innovating Open Education: Critical Pathways and Communities of PracticeInnovating Open Education: Critical Pathways and Communities of Practice
Innovating Open Education: Critical Pathways and Communities of Practice
 
Using learning analytics to support applied research and innovation in higher...
Using learning analytics to support applied research and innovation in higher...Using learning analytics to support applied research and innovation in higher...
Using learning analytics to support applied research and innovation in higher...
 
MOOC: Challenges for HE Institutions
MOOC: Challenges for HE InstitutionsMOOC: Challenges for HE Institutions
MOOC: Challenges for HE Institutions
 
Dutch acceleration plan, the butterfly effect 2.0
Dutch acceleration plan, the butterfly effect 2.0Dutch acceleration plan, the butterfly effect 2.0
Dutch acceleration plan, the butterfly effect 2.0
 
OER and MOOCs in the Netherlands: State of Affairs
OER and MOOCs in the Netherlands: State of AffairsOER and MOOCs in the Netherlands: State of Affairs
OER and MOOCs in the Netherlands: State of Affairs
 
Building an Open Operations Room for the OER Community #opened16
Building an Open Operations Room for the OER Community #opened16Building an Open Operations Room for the OER Community #opened16
Building an Open Operations Room for the OER Community #opened16
 
Openness of education: Urgent need for recognizing OER-based learning outcome...
Openness of education: Urgent need for recognizing OER-based learning outcome...Openness of education: Urgent need for recognizing OER-based learning outcome...
Openness of education: Urgent need for recognizing OER-based learning outcome...
 
Linq 2013 session_green_2_establish
Linq 2013 session_green_2_establishLinq 2013 session_green_2_establish
Linq 2013 session_green_2_establish
 
Data sharing and analytics in research and learning
Data sharing and analytics in research and learningData sharing and analytics in research and learning
Data sharing and analytics in research and learning
 
open_school
open_schoolopen_school
open_school
 
Lessons learned_FINAL_07112014pptx
Lessons learned_FINAL_07112014pptxLessons learned_FINAL_07112014pptx
Lessons learned_FINAL_07112014pptx
 
160407 eua open education
160407 eua open education160407 eua open education
160407 eua open education
 
Closing the loop between learning and employability with OER: Impact of Brin...
Closing the loop between learning and employability with OER: Impact of Brin...Closing the loop between learning and employability with OER: Impact of Brin...
Closing the loop between learning and employability with OER: Impact of Brin...
 
Opening EDEN Oldenrbug RW9
Opening EDEN Oldenrbug RW9Opening EDEN Oldenrbug RW9
Opening EDEN Oldenrbug RW9
 
Opening up Higher Education in Europe
Opening up Higher Education in EuropeOpening up Higher Education in Europe
Opening up Higher Education in Europe
 
Technology acceptance of augmented reality and wearable technologies ilrn 201...
Technology acceptance of augmented reality and wearable technologies ilrn 201...Technology acceptance of augmented reality and wearable technologies ilrn 201...
Technology acceptance of augmented reality and wearable technologies ilrn 201...
 
Characterising Professional Higher Education in Europe
Characterising Professional Higher Education in EuropeCharacterising Professional Higher Education in Europe
Characterising Professional Higher Education in Europe
 
Learning Analytics – Research challenges arising from a current review of LA use
Learning Analytics – Research challenges arising from a current review of LA useLearning Analytics – Research challenges arising from a current review of LA use
Learning Analytics – Research challenges arising from a current review of LA use
 
Oer panel
Oer panelOer panel
Oer panel
 
Interim report and main findings, by Lasakova, Bajzikova & Sasso
Interim report and main findings, by Lasakova, Bajzikova & SassoInterim report and main findings, by Lasakova, Bajzikova & Sasso
Interim report and main findings, by Lasakova, Bajzikova & Sasso
 

Viewers also liked

Pass along Plants in West Texas
Pass along Plants in West TexasPass along Plants in West Texas
Pass along Plants in West TexasFiona9864
 
Sustainable Composting in the Vineyard
Sustainable Composting in the VineyardSustainable Composting in the Vineyard
Sustainable Composting in the VineyardFiona9864
 
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...job_knowledge_research
 
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...Fiona9864
 
Manure Composting Manual
Manure Composting ManualManure Composting Manual
Manure Composting ManualFiona9864
 
Metzler Ranch Xeriscape Information and Guidelines
Metzler Ranch Xeriscape Information and GuidelinesMetzler Ranch Xeriscape Information and Guidelines
Metzler Ranch Xeriscape Information and GuidelinesFiona9864
 
A Level Geography at Abbeyfield 2009
A Level Geography at Abbeyfield 2009A Level Geography at Abbeyfield 2009
A Level Geography at Abbeyfield 2009David Drake
 

Viewers also liked (9)

Pass along Plants in West Texas
Pass along Plants in West TexasPass along Plants in West Texas
Pass along Plants in West Texas
 
Instructional
InstructionalInstructional
Instructional
 
Angelica
AngelicaAngelica
Angelica
 
Sustainable Composting in the Vineyard
Sustainable Composting in the VineyardSustainable Composting in the Vineyard
Sustainable Composting in the Vineyard
 
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...
Mol, S.T. (2013, October). Job Knowledge Based Personnel Selection & The Cent...
 
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...
Salt Management Guide for Landscape Irrigation with Recycled Water in Coastal...
 
Manure Composting Manual
Manure Composting ManualManure Composting Manual
Manure Composting Manual
 
Metzler Ranch Xeriscape Information and Guidelines
Metzler Ranch Xeriscape Information and GuidelinesMetzler Ranch Xeriscape Information and Guidelines
Metzler Ranch Xeriscape Information and Guidelines
 
A Level Geography at Abbeyfield 2009
A Level Geography at Abbeyfield 2009A Level Geography at Abbeyfield 2009
A Level Geography at Abbeyfield 2009
 

Similar to Overcoming barriers to adoption for Learning Analytics in a Dutch University

Learning analytics the good the bad & the ugly
Learning analytics the good the bad & the uglyLearning analytics the good the bad & the ugly
Learning analytics the good the bad & the uglyEduworks Network
 
Learning analytics bij de Universiteit van Amsterdam en ROC Noorderpoort
Learning analytics bij de Universiteit van Amsterdam en ROC NoorderpoortLearning analytics bij de Universiteit van Amsterdam en ROC Noorderpoort
Learning analytics bij de Universiteit van Amsterdam en ROC NoorderpoortSURF Events
 
Investigación en REA: un recorrido hasta el presente y la agenda pendiente
Investigación en REA: un recorrido hasta el presente y la agenda pendienteInvestigación en REA: un recorrido hasta el presente y la agenda pendiente
Investigación en REA: un recorrido hasta el presente y la agenda pendienteMarcelo Maina
 
Open Education Research : Overview, Benefits and Challenges
Open Education Research : Overview, Benefits and Challenges Open Education Research : Overview, Benefits and Challenges
Open Education Research : Overview, Benefits and Challenges Robert Farrow
 
Evaluating an open research project: Benefits and challenges from the ROER4D ...
Evaluating an open research project: Benefits and challenges from the ROER4D ...Evaluating an open research project: Benefits and challenges from the ROER4D ...
Evaluating an open research project: Benefits and challenges from the ROER4D ...SarahG_SS
 
Evaluating an open research project: Some practical lessons from the ROER4D p...
Evaluating an open research project: Some practical lessons from the ROER4D p...Evaluating an open research project: Some practical lessons from the ROER4D p...
Evaluating an open research project: Some practical lessons from the ROER4D p...Open Education Consortium
 
Tlt for it summit 2015 final
Tlt for it summit 2015 finalTlt for it summit 2015 final
Tlt for it summit 2015 finalkevin_donovan
 
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...PPGE UFF
 
The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...ROER4D
 
The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...SarahG_SS
 
Effective Creation, Mediation and Use of Knowledge in and about Education.
Effective Creation, Mediation and Use of Knowledge in and about Education.Effective Creation, Mediation and Use of Knowledge in and about Education.
Effective Creation, Mediation and Use of Knowledge in and about Education.EduSkills OECD
 
Open Education Licensing
Open Education LicensingOpen Education Licensing
Open Education LicensingNicolas Suzor
 
TU Delft Open Science Strategic Plan #oeglobal19
TU Delft Open Science Strategic Plan #oeglobal19TU Delft Open Science Strategic Plan #oeglobal19
TU Delft Open Science Strategic Plan #oeglobal19Willem van Valkenburg
 
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...EADTU
 
Open learning in higher education an institutional approach
Open learning in higher education  an institutional approachOpen learning in higher education  an institutional approach
Open learning in higher education an institutional approachBrian Murphy
 
OER Hub: An Overview
OER Hub: An OverviewOER Hub: An Overview
OER Hub: An OverviewOER Hub
 
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...Jisc
 
Learning Analytics Community Exchange
Learning Analytics Community ExchangeLearning Analytics Community Exchange
Learning Analytics Community ExchangeDoug Clow
 

Similar to Overcoming barriers to adoption for Learning Analytics in a Dutch University (20)

Learning analytics the good the bad & the ugly
Learning analytics the good the bad & the uglyLearning analytics the good the bad & the ugly
Learning analytics the good the bad & the ugly
 
Learning analytics bij de Universiteit van Amsterdam en ROC Noorderpoort
Learning analytics bij de Universiteit van Amsterdam en ROC NoorderpoortLearning analytics bij de Universiteit van Amsterdam en ROC Noorderpoort
Learning analytics bij de Universiteit van Amsterdam en ROC Noorderpoort
 
Investigación en REA: un recorrido hasta el presente y la agenda pendiente
Investigación en REA: un recorrido hasta el presente y la agenda pendienteInvestigación en REA: un recorrido hasta el presente y la agenda pendiente
Investigación en REA: un recorrido hasta el presente y la agenda pendiente
 
Open Education Research : Overview, Benefits and Challenges
Open Education Research : Overview, Benefits and Challenges Open Education Research : Overview, Benefits and Challenges
Open Education Research : Overview, Benefits and Challenges
 
Evaluating an open research project: Benefits and challenges from the ROER4D ...
Evaluating an open research project: Benefits and challenges from the ROER4D ...Evaluating an open research project: Benefits and challenges from the ROER4D ...
Evaluating an open research project: Benefits and challenges from the ROER4D ...
 
Evaluating an open research project: Some practical lessons from the ROER4D p...
Evaluating an open research project: Some practical lessons from the ROER4D p...Evaluating an open research project: Some practical lessons from the ROER4D p...
Evaluating an open research project: Some practical lessons from the ROER4D p...
 
Tlt for it summit 2015 final
Tlt for it summit 2015 finalTlt for it summit 2015 final
Tlt for it summit 2015 final
 
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...
Seminário Recursos e Práticas Educacionais Abertas no Ensino Superior: desafi...
 
The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...
 
The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...The iterative engagement between curation and evaluation in an open research ...
The iterative engagement between curation and evaluation in an open research ...
 
Effective Creation, Mediation and Use of Knowledge in and about Education.
Effective Creation, Mediation and Use of Knowledge in and about Education.Effective Creation, Mediation and Use of Knowledge in and about Education.
Effective Creation, Mediation and Use of Knowledge in and about Education.
 
OEL Toolkit
OEL Toolkit OEL Toolkit
OEL Toolkit
 
Open Education Licensing
Open Education LicensingOpen Education Licensing
Open Education Licensing
 
TTTNet - Teamwork, Training and Technology Network
TTTNet - Teamwork, Training and Technology NetworkTTTNet - Teamwork, Training and Technology Network
TTTNet - Teamwork, Training and Technology Network
 
TU Delft Open Science Strategic Plan #oeglobal19
TU Delft Open Science Strategic Plan #oeglobal19TU Delft Open Science Strategic Plan #oeglobal19
TU Delft Open Science Strategic Plan #oeglobal19
 
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...
[OOFHEC2018] Kate Lister: Embedding and sustaining inclusive practices in onl...
 
Open learning in higher education an institutional approach
Open learning in higher education  an institutional approachOpen learning in higher education  an institutional approach
Open learning in higher education an institutional approach
 
OER Hub: An Overview
OER Hub: An OverviewOER Hub: An Overview
OER Hub: An Overview
 
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...
Crossing the threshold: moving e-portfolios into the mainstream - Jisc Digita...
 
Learning Analytics Community Exchange
Learning Analytics Community ExchangeLearning Analytics Community Exchange
Learning Analytics Community Exchange
 

Recently uploaded

In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxAditiChauhan701637
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...raviapr7
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfMohonDas
 
How to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingHow to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingCeline George
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxEduSkills OECD
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17Celine George
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesMohammad Hassany
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.raviapr7
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxraviapr7
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphNetziValdelomar1
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptxSandy Millin
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfTechSoup
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxKatherine Villaluna
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxDr. Santhosh Kumar. N
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17Celine George
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 

Recently uploaded (20)

In - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptxIn - Vivo and In - Vitro Correlation.pptx
In - Vivo and In - Vitro Correlation.pptx
 
Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...Patient Counselling. Definition of patient counseling; steps involved in pati...
Patient Counselling. Definition of patient counseling; steps involved in pati...
 
Diploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdfDiploma in Nursing Admission Test Question Solution 2023.pdf
Diploma in Nursing Admission Test Question Solution 2023.pdf
 
How to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 AccountingHow to Filter Blank Lines in Odoo 17 Accounting
How to Filter Blank Lines in Odoo 17 Accounting
 
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptxPISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
PISA-VET launch_El Iza Mohamedou_19 March 2024.pptx
 
How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17How to Add Existing Field in One2Many Tree View in Odoo 17
How to Add Existing Field in One2Many Tree View in Odoo 17
 
Human-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming ClassesHuman-AI Co-Creation of Worked Examples for Programming Classes
Human-AI Co-Creation of Worked Examples for Programming Classes
 
Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.Drug Information Services- DIC and Sources.
Drug Information Services- DIC and Sources.
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptx
 
Presentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a ParagraphPresentation on the Basics of Writing. Writing a Paragraph
Presentation on the Basics of Writing. Writing a Paragraph
 
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
2024.03.23 What do successful readers do - Sandy Millin for PARK.pptx
 
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdfMaximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
Maximizing Impact_ Nonprofit Website Planning, Budgeting, and Design.pdf
 
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptxPractical Research 1: Lesson 8 Writing the Thesis Statement.pptx
Practical Research 1: Lesson 8 Writing the Thesis Statement.pptx
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
M-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptxM-2- General Reactions of amino acids.pptx
M-2- General Reactions of amino acids.pptx
 
How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17How to Show Error_Warning Messages in Odoo 17
How to Show Error_Warning Messages in Odoo 17
 
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdfPersonal Resilience in Project Management 2 - TV Edit 1a.pdf
Personal Resilience in Project Management 2 - TV Edit 1a.pdf
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Finals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quizFinals of Kant get Marx 2.0 : a general politics quiz
Finals of Kant get Marx 2.0 : a general politics quiz
 

Overcoming barriers to adoption for Learning Analytics in a Dutch University

  • 1. jobknowledge.eu facebook.com/jobknowledge @Jobknowledge Barriers to Adoption for Learning Analytics at a Dutch University Dr. Stefan T. Mol, s.t.mol@uva.nl Dr. Gábor Kismihók, g.kismihok@uva.nl
  • 2. jobknowledge.eu facebook.com/jobknowledge @Jobknowledge Overcoming Barriers to Adoption for Learning Analytics at a Dutch University Dr. Stefan T. Mol, s.t.mol@uva.nl Dr. Gábor Kismihók, g.kismihok@uva.nl
  • 3. Who are we? The Center of Job Knowledge Research is a multidisciplinary thinktank and research incubator that aims to engage stakeholders from academia, industry, and government to elucidate how job knowledge may be used to facilitate individual, organization, and nation level person-education-workplace matching processes. 6/30/2014 LASI-Utrecht 2014 3
  • 4. Contents • UvaInform History • Process and Pilots • Organization and Project Management • Lessons Learnt 6/30/2014 LASI-Utrecht 2014 4
  • 5. The UvAInform Project - History • Initiated as a proposal from the ICTS Department • Expertise group Education ICT (EGO-ICT) Reserved budget of 150K • EGO-ICT dislike of (bottom-up) tender procedure with limited strategic vision • Focus Group Learning Analytics Established Late 2012 • UvAInform proposal approved in June 2013 • Central infrastructure (LRS) • (De?)centralized pilots 6/30/2014 LASI-Utrecht 2014 5
  • 6. Responsibilities of the LA Focus Group • Create vision on LA • Avoid duplication of effort • Starter engine for ethics board • Source of advice for LA (related) projects • Communication channel with the VU and HvA, Surf, & Internationally 6/30/2014 LASI-Utrecht 2014 6
  • 8. UvAInform Process The process Focus Group: evaluation criteria 6/30/2014 LASI-Utrecht 2014 8
  • 9. Rated Evaluation Criteria (1: Not at all important-7 - Extremely important) 1. Feedback for learning (yes/no) 6.33 2. Scalability of the project (Course / Faculty / Institution / National / World-wide / National / International) 5.89 3. Feedback for teaching (yes/no) 5.86 4. Feedback for curriculum (yes/no) 5.75 5. Ethical concerns associated with the project (free text) 5.53 6. Use of the common infrastructure (multiple-select, none, dashboard, Learning Record Store, Sakai CLE xAPI enabled) 4.94 7. Estimated Cost (Low < 10K, Medium 10K-20K, High > 20K)" 4.92 8. Production of open source software (yes / no) 4.83 6/30/2014 LASI-Utrecht 2014 9
  • 10. Rated Evaluation Criteria (1: Not at all important-7 - Extremely important) 9. Sustainability (No chance of adoption, planned as a local service, planned as an UvA wide service) 4.83 10. Project risk (such as bottleneck with resources, project deadline (low, medium, high) 4.78 11. Dissemination of knowledge (whitepaper, scientific publication) 4.78 12. Reusability from external projects (none, some, totally) 4.69 13. Product Risk such as nonsense feedback being provided to students (low, medium, high) 4.61 14. Complexity (low, medium, high) 3.56 15. Novel research? (yes/no) 3.33 6/30/2014 LASI-Utrecht 2014 10
  • 11. UvAInform Process The process Focus Group: evaluation criteria Definition of initial pilots Discussion with an external expert (Erik Duval) Dilemma – how to agree on the separate pilots Dilemma – To LRS or not to LRS? 6/30/2014 LASI-Utrecht 2014 11
  • 13. Learning Record Store • Community sourced, secure, scalable repository/infrastructure • Store and retrieve statement data reliably and ensures a good scalable storage layer for various types of data and data streams • Scales above 100 billion records. • These data can be made available in a secure and consistent way for further analysis. • Upon this LRS infrastructure dashboards can be built or developed for the delivery of (analysed) data to students, educators, and researchers http://tincanapi.com 6/30/2014 LASI-Utrecht 2014 13
  • 14. UvAInform Process The process Focus Group: evaluation criteria Definition of initial pilots Discussion with an external expert (Erik Duval) Dilemma – how to agree on the separate pilots Dilemma – To LRS or not to LRS Definition of LRS as separate project All faculties present their research ideas (Pecha Kucha Format) Dilemma – How to agree on pilot requirements and associated budgets Result: 7 small pilots in 3 pilot clusters 6/30/2014 LASI-Utrecht 2014 14
  • 15. Cluster 1: Mirroring of traditional and non- traditional study performance to students • UvA Mirror (COACH) Visualize the position of individuals in the context of the group • Cluster Exam Feedback (qDNA) More fine grained mirroring of exam results to provide students and teachers with insight in the development on four competencies and knowledge goals 6/30/2014 LASI-Utrecht 2014 15
  • 16. Cluster 1: Goal Setting in Education • Building on functionality of schedules (www.roosters.uva.nl) to include goal setting and goal tracking. • Students will be instructed/taught to formulate goals in a concrete manner such that they are specific, measurable, attainable, relevant and time-bound (SMART). • Dashboard will facilitate individual students to choose from, and set their own goals (and deadlines) against specific course events deadlines based on mirrored data. • Dashboard shows them in a glance how they are scoring/succeeding in attaining the goals compared to their fellows 6/30/2014 LASI-Utrecht 2014 16
  • 17. Cluster 1 Side project: Labour market oriented learning trajectories Match learners’ pathways to those of alumni Help incoming students find a long term focus during their university education Mirror alumni data to current students based on desired/acquired job roles 6/30/2014 LASI-Utrecht 2014 17
  • 18. Cluster 2: Using specific student data to provide feedback for teachers • Assignments’ criteria validation Visualization of Turnitin scores for students and teachers over (partial) assignments or courses • Web lecture statistics Visualization of use of video’s/weblectures to find potential problems, difficult material to improve teaching, provide better remedial material 6/30/2014 LASI-Utrecht 2014 18
  • 19. Cluster 3: Using other people’s data to provide recommendation system to students • Reduce dropouts in Bachelor programs Get existing and new data in gathering as much demographic personal info on learning styles, motivation, personality to optimize prediction of study success Aim: First find predictors, then start developing interventions • Still under negotiation 6/30/2014 LASI-Utrecht 2014 19
  • 20. UvAInform Process The process Focus Group: evaluation criteria Definition of initial pilots Discussion with an external expert (Erik Duval) Dilemma – How to agree on the separate pilots Dilemma – To LRS or not to LRS Definition of LRS as separate project All faculties present their research ideas (Pecha Kucha Format) Dilemma – How to agree on pilot requirements and associated budgets Result: 7 small pilots in 3 pilot clusters Challenge: Clarifying requirements and exploiting synergies 6/30/2014 LASI-Utrecht 2014 20
  • 21. Organization and Project management 6/30/2014 LASI-Utrecht 2014 21
  • 23. Engaging critical stakeholders • Potential users • Students • Teachers • Legal/Ethical Committees • University Administration • Researchers • Communities outside of the organisation 6/30/2014 LASI-Utrecht 2014 23
  • 24. Data owners • Who is owning what data? • Highly political issue • Organizational resistance • Gatekeepers resistance • Complex infrastructure 6/30/2014 LASI-Utrecht 2014 24
  • 26. Positioning a learning analytics project • Centralized vs. decentralized • Research vs. practice • Imposed vs. desired • Technology vs. Pedagogy • Make or buy (resource oriented) • Generic infrastructure vs. pilot specific infrastructure in relation to UvA learning analytics lifecycle • Adoption of best practices or local identification thereof 6/30/2014 LASI-Utrecht 2014 26
  • 27. Lessons Learnt • Organizational awareness is growing • LA is on the agenda in many stakeholder groups • Data is everywhere, but accessibility is an issue • Need to document failures and agreements properly • Legal and ethical concerns are high on the agenda 6/30/2014 LASI-Utrecht 2014 27
  • 28. Towards a Center of Data Governance and Innovation • How do researchers/project members involved in data science projects ensure full compliance with the UvA’s ethical standards? • How do these people, whose expertise is seldom or never in the legal domain, ensure that their work does not infringe on the law? • Threat: bad publicity, legal costs due to litigation caused by uninformed data management, efficiency losses due to different parties having to reinvent the wheel (left hand does not know what the right hand is doing) 6/30/2014 LASI-Utrecht 2014 28
  • 29. Organizational Lessons • Responsibilities are not clear – not connected to the budget • Decisions are hard to make • Internal policies and processes should be clearer • Keep the university’s vision in focus 6/30/2014 LASI-Utrecht 2014 29