Bringing e-assessment to a higher maturity on a national scale


Published on

Presentatie gegeven op de Educause 2011,
door Alfred Hartoog en Nynke Bos

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • This ends our presentation,feel free to react, ask questions ....
  • Bringing e-assessment to a higher maturity on a national scale

    1. 1. Bringing e-assessment to a higher maturity on a national scale dr. A. Hartoog, SURF, Utrecht drs. N.R. Bos, Fontys, EindhovenEducause october 2011
    2. 2. Program introduction 1 problems, questions, all somehow 10 min related to e-assessment discussion 2 how is this situation in the US and/or 15 min Canada? elaboration 3 how are we tackling these issues in the 15 min Netherlands, on a national scale 4 questions/answers 10 min 1Educause october 2011
    3. 3. context The Netherlands: - surface area 40.000 km2 (US is 270 times bigger) - population density: 491 people/km2 / (US: 34 people/km2) - about 60 universities, - about 600.000 students - two handfuls of government funded and commercial R&D institutes 2Educause october 2011
    4. 4. Problems, questions…(1) - many students drop out in their first year of study - and if they don’t, their success rate is less than optimal - can we improve this situation via (digital) testing? 3Educause october 2011
    5. 5. Problems, questions…(2) - Assessment is a key proces in education - Testing students properly is tedious and time- consuming - Teachers in general are no trained specialists in the assessment proces, and are usually already overloaded - How to improve this situation? 4Educause october 2011
    6. 6. Problems, questions…(3) - Clinical reasoning is a key competence for doctors - proper assessment in their education is crucial - given the high number of students in medical schools, - how on earth do you assess this competency reliably 5Educause october 2011
    7. 7. Problems, questions…(4) - teaching statistics is a key component in many educations - statistics is often a stumbling block for students - statistics typically has slow and fast students - how do you make statistics education, training and assessment more effective and fun for the slow and fast alike 6Educause october 2011
    8. 8. Problems, questions…(5) - apprentices in nursing experience a gap between education and practise, leading to drop-outs - there are many educations in nursing, with the associated variety in end-terms - how to tackle this? 7Educause october 2011
    9. 9. Discussion- can e-assessment reduce teacher workload?- does information technology raise the quality of testing?- can e-assessment reduce students dropping out?- is it possible to test complex skills with closed questions?- how could e-assessment improve statistics education?- could e-assessment reduce the reality gap in nursing?- would a national infrastructure for e-assessment be useful?- how can you stimulate collaboration between educations?- how do you introduce e-assessment in law education? all the above on a national scale! 8 Educause october 2011
    10. 10. So, what we did? Context: Role of SURF in Higher Education in the Netherlands Program: Testing and test driven learning Impact: Things that work, things that don’t 9Educause october 2011
    11. 11. SURF: ICT innovation by and for higher education and research Research universities, universities of applied sciences, and research institutions work together in SURF in order to: • improve the quality of research and education by: • creating pioneering ICT innovations. For more than twenty years, collaboration within SURF has led to products and services that the participating institutions could not have achieved on their own. 10Educause october 2011
    12. 12. SURF is a partnershipbetween all Dutch institutions for higher education & research 42 Universities of applied sciences 14 National Library Universities Smaller Collaboration Netherlands Organisation for institutions ofhigher education = Scientific Research (NWO) Success factor! Royal Netherlands Commercial Academy of Arts R&D institutes and Sciences Dutch Organisation (KNAW) for Applied Scientific Research (TNO), Telematics Institute 11Educause october 2011
    13. 13. SURF as an innovatorCollaboration within SURF makes it possible to: • carry out a joint (public/private) survey of what higher education & research requires/will receive • cluster ICT requirements on the demand side (demand pull) • arrive at sustainable ICT services suitable for joint use • be innovative in contracting out services • encourage the use of innovative services (supply push).Educause october 2011 12
    14. 14. SURF subjects - advanced networks - security/identity management/federation - collaborative environments - software procurement/licence management - digital rights & open access - open educational resources - SURF academy - information architecture - national information brokerage 13Educause october 2011
    15. 15. The SURF methodImplementation SURF directs and coordinates the implementation of funded projects: • Projects are carried out close to the source of expertise: at institutions of higher education and in enterprise. • SURF sets up the programme, invites tenders, ensures independent quality assessment and monitoring, makes project results available.Educause october 2011 14
    16. 16. Program “Testing and test driven learning” Problems in higher education in the Netherlands: - high drop-out rate - high workload on teachers SURF launched a national program to tackle these problems with e-assessment 15Educause october 2011
    17. 17. Purpose of this program - improving transition from secondary higher education with diagnostic tests and revision material, to be jointly developed by institutions, - improving students’ progress by means of progress tests, - reducing the time needed to develop tests by collaboration, i.e. developing collections of digital tests, - reducing the time taken to mark tests by deploying computer-based testing, - providing expertise and professional development for instructors in the field of testing. 16Educause october 2011
    18. 18. programme compositionthe program focusses onthree main area’s: 1. enabling institutions to jointly develop computer-based tests and assessments 2. realising an expertise network, aimed at improving quality of testing 3. realising a national technical infrastructure to combine successful national initiatives for computer based testing 17 Educause october 2011
    19. 19. What kind of projects? bachelor nursing: - “body of knowledge” medicine: - interuniversity progress test medicine (2) - testing of clinical reasoning - testing with 3D images (research) math and statistics - foundation of national test database - research into effects of progresstesting - test-based learning (adaptive testing) (2) teacher education - research into testing abilities of teacher educators - research into methods for portfolio assessments -> and: new tender by 1 november 18Educause october 2011
    20. 20. Math & Statistics projects- Stumbling block- Faster and slower students- Based on earlier project “Mathematics Garden”-> computer adaptive monitoring- Students: can practice can adjust their learning process - Reduction in drop out- Teachers: can adjust their teaching process - Reduction in workloadUniversity of Amsterdam, University of Utrecht, Twente University
    21. 21. Medical projects- Low dropout rates students but high workload teachers (doctors)- Development of questions on a national scale- Sharing these in national databases- Different goals: - Measuring progress - Measuring Clinical Reasoning (complex skills) - Improvement gap practice/ theory- Involvement: all nursing schools, all medical schools
    22. 22. Research projects3D Testing (CT/MRI) The project aims at improving the psychometric quality and validity of assessment of 3D images. Also gain insight into the cognitive processes required in the interpretation of 3D imagesProgress testing and study success- First year economic students will get weekly progress tests- Will this increase [retention rates]?- Does this also effect workload for teachers, costs of assessment, quality of testing?
    23. 23. Network of expertise Fysically: - a Special Interest Group of inspired specialist - seminars, schools, master classes (SURF Academy) - small scale practical research projects Virtually: web portal that acts as a centre of expertise: - good practises - white papers - links to literature - ask and find specialists - blogs, discussions forum - agenda free accessible and maintained by the higher education community 22Educause october 2011
    24. 24. National infrastructure ambition: investigate, realize and exploit the advantages of a common infrastructure for computer based assessment on a national scale healthy in all respects, to stand on its own when the program finishes, and external funding is discontinued 23Educause october 2011
    25. 25. what’s to gain? (1) sharing systems: B user i/f A application user i/f s combi application C middleware user i/f s user i/f application middleware application ? s operating s operating systems & middleware middleware systems & networks operating operating networks systems & systems & networks networks hardware hardware hardware hardware 24Educause october 2011
    26. 26. what’s to gain? (2) sharing system management: B helpdesk A funct.mgnt combi helpdesk C helpdesk funct.mgnt appl.mgnt helpdesk ? funct.mgnt funct.mgnt appl.mgnt appl.mgnt technical appl.mgnt technical mgnt technical mgnt technical mgnt mgnt infra.mgnt infra.mgnt infra.mgnt infra.mgnt 25Educause october 2011
    27. 27. lessons learned up to now - subject domains in education (math, health, law, humaniora, etc) are very different - educational institutions are very different - the advantages of sharing an infrastructure are not always obvious - computer based testing is far from mature 26Educause october 2011
    28. 28. so, what do we do - confine within domains - investigate and explore along the functional lines - make advantages explicit - join already successful initiatives - take small steps, but think big - develop business models and this might actually work: - example 27Educause october 2011
    29. 29. collaboration in medical and nursing education BoKS Clinical Body of Knowledge Reasoning & Skills education in nursing and ? medicine ? Testing iPTM ? in 3D inter University Progress Test Medicine AdaPT 28Educause october 2011
    30. 30. Questions, answers shoot!Educause october 2011 29
    31. 31. Alfred Hartoog Nynke Bos 30Educause october 2011
    32. 32. Impact, how to measure - this four-year program is one year underway - high level goals: - reduce number of students that drop-out - reduce workload on teaching staff - raise quality level of e-assessment How do you measure this? basic problem: there are many factors (not related to the program) that may have a dominant influence on above subjects. 31Educause october 2011
    33. 33. evaluation and impact - decompose high-level program goals to manageable project proportions - find ways to measure these parameters, and assess them during the project - develop a model to arrange the findings in levels of evidence (based on the knowledge-pyramid from Kennisnet) - inspiration (this is great: it should work) - perception (this actually works for me) - existence (others confirm that it works) - evidence (scientific proof that it works) 32Educause october 2011
    34. 34. Evidence: measured benefits Perception: experienced benefits Existence: deployment of the project Inspiration: Idea 33Educause october 2011