Confidence, standards and technology  A view from Ofqual  Isabel Nisbet, Acting CEO 11 November 2009
Outline <ul><ul><li>About Ofqual – relevance to use of technology  </li></ul></ul><ul><ul><ul><ul><li>Standards </li></ul>...
Some definitions  <ul><li>“E-assessment” describes end-to-end electronic assessment processes where ICT is used for the pr...
Ofqual   <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009  </li></ul><ul><li>Independent of...
Ofqual   <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009  </li></ul><ul><li>Independent of...
The standards objective for qualifications and assessments  <ul><li>…To secure that regulated qualifications/assessments –...
Standards that we apply to all qualifications/assessments  <ul><li>Validity  </li></ul><ul><li>Reliability  </li></ul><ul>...
Standards that we apply to all qualifications/assessments  <ul><li>Validity   </li></ul><ul><li>Reliability  </li></ul><ul...
Validity  <ul><li>Needs to assess the knowledge and skills it’s supposed to  </li></ul><ul><li>Matches the normal mode of ...
Validity problems can arise when…. <ul><li>…  learning undertaken through technology is assessed in a different medium  </...
Use of the internet (Denmark)
<ul><li>&quot;As a nation [the UK] has been really good at embracing technology - we've been really at the forefront of do...
Standards that we apply to all qualifications/assessments  <ul><li>Validity  </li></ul><ul><li>Reliability  </li></ul><ul>...
Worries about cheating  <ul><li>“ Digital dishonesty and technology in exams and tests” (Professor Jean Underwood)  </li><...
Standards that we apply to all qualifications/assessments  <ul><li>Validity  </li></ul><ul><li>Reliability  </li></ul><ul>...
Manageability  <ul><li>Small cohort + frequent assessment sessions: EASIER </li></ul><ul><li>Large cohort + infrequent ses...
Ofqual   <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009  </li></ul><ul><li>Independent of...
Public confidence  <ul><li>“The public confidence objective is to promote public confidence in regulated qualifications an...
E-regulation – issues around confidence  <ul><li>Listening to learners </li></ul>
E-regulation – more issues around confidence  <ul><li>Public unfamiliarity </li></ul><ul><ul><ul><ul><li>Generational diff...
Ofqual   <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009  </li></ul><ul><li>Independent of...
Innovation  <ul><li>“So far as is relevant, in performing its functions Ofqual must have regard to –  </li></ul><ul><li>…....
Avoiding barriers – Open letter to awarding bodies, 13 October 2008  <ul><li>No requirement of a single national IT platfo...
Ofqual’s 4 projects  <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></u...
Ofqual’s 4 projects  <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></u...
e-assessment use in VRQs
Numbers of centres using e-assessment in a GCSE or A level
Three forecasts of e-assessment diffusion
Uptake of e-assessment in vocational qualifcations <ul><li>Use of e-assessment in vocational qualifications is increasing ...
Ofqual’s 4 projects  <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></u...
On-demand testing  <ul><li>Flavours of “on-demand” </li></ul><ul><ul><ul><ul><li>Unique to candidate – any time </li></ul>...
On-demand testing (ctd)  <ul><ul><ul><li>Ofqual is launching a project to develop and consult on a regulatory response tha...
Some challenges TO Ofqual  <ul><li>Are you doing enough regulation at the level of the whole system – is it fit for the mo...
Some challenges FROM Ofqual  <ul><li>Educationalists should challenge the E-industry  </li></ul><ul><ul><ul><ul><li>E-asse...
Conclusions  <ul><li>Ofqual will place E-assessment at the heart of our regulation – it is not a separate specialist area ...
Ofqual’s vision <ul><li>The independent regulator of qualifications and assessments that are valued and trusted by learner...
<ul><li>Thank you  </li></ul><ul><li>[email_address] </li></ul><ul><li>www.ofqual.gov.uk   </li></ul>
 
Upcoming SlideShare
Loading in...5
×

Westminster Forum. Confidence, standards and technology: A view from Ofqual

884

Published on

Isabel Nisbet's presentation at the Westminster Forum on 11 November 2009. The presentation focussed on confidence, standards and technology.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
884
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • We know that there is a lot of e-assessment use in some assessment that Ofqual does not regulate. For example, in the hazard perception test in the theory test for drivers and riders, or in the Qualified Teacher Status (QTS) skills tests run on behalf of the Training and Development Agency for Schools. But how much e-assessment is being used in qualifications accredited by Ofqual and other UK regulators? One way of addressing this question is by looking at data on qualifications that have been accredited via Web-Based Accreditation (WBA). WBA accreditation requires awarding bodies to describe the assessment methods for their qualifications, and one of the options in ‘assessment methods’ is e-assessment. This allows us to count the number of quals accredited via WBA that are using some e-assessment. Vocationally-Related Qualifications (VRQs) are an area in which there has been steady growth. Similarly, there a quite a lot of Other General Qualifications (things like graded music and language qualifications) that contain e-assessment. There are currently fewer general qualifications (GCSEs and A levels) that contain e-assessment (less than 50 in total). But it is likely that the number of these qualifications that contain e-assessment will grow.
  • On diffusion … How many centres entered at least one candidate for at least one GCSE or GCE that used some e-assessment in summer 2008? The answer is 421. This is a small proportion of all centres doing exams in England, Wales and Northern Ireland, but it does show that e-assessment is here now – it is not negligible or pie in the sky. 400 centres are dealing with this here and now.
  • Here are three forecasts for how quickly e-assessment might get taken up in GCSE &amp; GCE. They were done before we received the figures on use of e-assessment in the summer 2008 series and are based on three sources: (the top/fastest uptake) based on data provided by a colleague in the Netherlands An ‘average’ value – based on experience of how quickly many different innovations (from outside education) have been adopted in the United States. A slow uptake/’worst case scenario’ – based on a sceptical reading of the relatively light use of IT technologies in UK schools and the risk averse nature of our general qualifications business. As you can see, there are big differences between the forecasts. In the most ‘optimistic’ scenario, we are getting to over 90 per cent of centres using e-assessment for GCSE in two or three years. In the most pessimistic, we are still below 90 per cent in 15 years. Of course, forecasts of the future may not be right – and the more we have data on e-assessment uptake, the more we will be able to make accurate forecasts, but this is a task worth carrying out. The one area that we do not yet have data on is ‘infusion’. What the theory says is that infusion follows diffusion – often with a similar s-shaped curve. In terms of large cohort e-assessment one issue may be that infusion issues may inhibit diffusion – that is, centres may say ‘if we can’t test all the kids using e-assessment, then we won’t test any of them’. Reasons why they might not be able to test all kids in one sitting include (un)availability of machines, issues with network speed, space between workstations, etc. These are the kind of reasons that make ‘on-demand testing’ an important issue for us as regulator.
  • Westminster Forum. Confidence, standards and technology: A view from Ofqual

    1. 1. Confidence, standards and technology A view from Ofqual Isabel Nisbet, Acting CEO 11 November 2009
    2. 2. Outline <ul><ul><li>About Ofqual – relevance to use of technology </li></ul></ul><ul><ul><ul><ul><li>Standards </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Confidence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Innovation </li></ul></ul></ul></ul><ul><ul><li>The regulation of e-assessment </li></ul></ul><ul><ul><ul><ul><li>Ofqual’s 4 projects </li></ul></ul></ul></ul><ul><ul><ul><li>Looking forward </li></ul></ul></ul><ul><ul><ul><ul><li>Challenges to the regulator </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Challenges from the regulator </li></ul></ul></ul></ul>
    3. 3. Some definitions <ul><li>“E-assessment” describes end-to-end electronic assessment processes where ICT is used for the presentation of assessment activity, the recording of responses and marking [NB not just on-screen tests] </li></ul><ul><li>“E-marking” , or “on-screen marking” describes processes whereby student responses (which may be on paper) are marked on-screen </li></ul><ul><li>Both are different from “assessment of ICT skills” </li></ul>
    4. 4. Ofqual <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009 </li></ul><ul><li>Independent of </li></ul><ul><ul><ul><ul><li>Government </li></ul></ul></ul></ul><ul><ul><ul><ul><li>The QCDA </li></ul></ul></ul></ul><ul><ul><ul><li>Five statutory objectives: </li></ul></ul></ul><ul><ul><ul><ul><li>Standards in qualifications </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Standards in [National Curriculum] assessments </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Public confidence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Awareness </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Efficiency </li></ul></ul></ul></ul><ul><ul><ul><li>General duty regarding innovation </li></ul></ul></ul>
    5. 5. Ofqual <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009 </li></ul><ul><li>Independent of </li></ul><ul><ul><ul><ul><li>Government </li></ul></ul></ul></ul><ul><ul><ul><ul><li>The QCDA </li></ul></ul></ul></ul><ul><ul><ul><li>Five statutory objectives: </li></ul></ul></ul><ul><ul><ul><ul><li>Standards in qualifications </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Standards in [National Curriculum] assessments </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Public confidence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Awareness </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Efficiency </li></ul></ul></ul></ul><ul><ul><ul><li>General duty regarding innovation </li></ul></ul></ul>
    6. 6. The standards objective for qualifications and assessments <ul><li>…To secure that regulated qualifications/assessments – </li></ul><ul><li>(a) give a reliable indication of knowledge, skills and understanding </li></ul><ul><li>(b) indicate a consistent level of attainment (including over time) between comparable regulated qualifications/assessments </li></ul><ul><li>(Clause 127 (1) and (2)) </li></ul>
    7. 7. Standards that we apply to all qualifications/assessments <ul><li>Validity </li></ul><ul><li>Reliability </li></ul><ul><li>Security </li></ul><ul><li>Absence of bias </li></ul><ul><li>Accessibility </li></ul><ul><li>Manageability </li></ul>
    8. 8. Standards that we apply to all qualifications/assessments <ul><li>Validity </li></ul><ul><li>Reliability </li></ul><ul><li>Security </li></ul><ul><li>Absence of bias </li></ul><ul><li>Accessibility </li></ul><ul><li>Manageability </li></ul>
    9. 9. Validity <ul><li>Needs to assess the knowledge and skills it’s supposed to </li></ul><ul><li>Matches the normal mode of learning </li></ul><ul><li>…. Supports the best teaching and learning </li></ul><ul><li>………Good teaching and learning supports the needs of students in the real world (employment, FE/HE) </li></ul>
    10. 10. Validity problems can arise when…. <ul><li>… learning undertaken through technology is assessed in a different medium </li></ul><ul><li>eg writing on paper </li></ul><ul><li>special ICT technology just for the assessment </li></ul>
    11. 11. Use of the internet (Denmark)
    12. 12. <ul><li>&quot;As a nation [the UK] has been really good at embracing technology - we've been really at the forefront of doing this well in the classroom. </li></ul><ul><li>&quot;Then they go into the exam room and all that's taken away and they're given a fountain pen and a sheet of lines paper and a three hour time limit. It's time to get real, isn't it?&quot; </li></ul><ul><li>Stephen Heppell, Professor of New Media Environments, Bournemouth University </li></ul>
    13. 13. Standards that we apply to all qualifications/assessments <ul><li>Validity </li></ul><ul><li>Reliability </li></ul><ul><li>Security </li></ul><ul><li>Absence of bias </li></ul><ul><li>Accessibility </li></ul><ul><li>Manageability </li></ul>
    14. 14. Worries about cheating <ul><li>“ Digital dishonesty and technology in exams and tests” (Professor Jean Underwood) </li></ul><ul><li>Really about education and ethics </li></ul><ul><li>Guidance </li></ul><ul><li>Cheap safeguards available (eg plagiarism checks) </li></ul><ul><li>Controlled assessment rules </li></ul>
    15. 15. Standards that we apply to all qualifications/assessments <ul><li>Validity </li></ul><ul><li>Reliability </li></ul><ul><li>Security </li></ul><ul><li>Absence of bias </li></ul><ul><li>Accessibility </li></ul><ul><li>Manageability </li></ul>
    16. 16. Manageability <ul><li>Small cohort + frequent assessment sessions: EASIER </li></ul><ul><li>Large cohort + infrequent sessions: HARDER </li></ul><ul><li>Patrick Craven, Cambridge Assessment </li></ul><ul><li>More special/unusual – harder </li></ul>
    17. 17. Ofqual <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009 </li></ul><ul><li>Independent of </li></ul><ul><ul><ul><ul><li>Government </li></ul></ul></ul></ul><ul><ul><ul><ul><li>The QCDA </li></ul></ul></ul></ul><ul><ul><ul><li>Five statutory objectives: </li></ul></ul></ul><ul><ul><ul><li>- Standards in qualifications </li></ul></ul></ul><ul><ul><ul><ul><li>Standards in [National Curriculum] assessments </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Public confidence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Awareness </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Efficiency </li></ul></ul></ul></ul><ul><ul><ul><li>General duty regarding innovation </li></ul></ul></ul>
    18. 18. Public confidence <ul><li>“The public confidence objective is to promote public confidence in regulated qualifications and regulated assessment arrangements” </li></ul><ul><li>(Clause 127(4)) </li></ul>
    19. 19. E-regulation – issues around confidence <ul><li>Listening to learners </li></ul>
    20. 20. E-regulation – more issues around confidence <ul><li>Public unfamiliarity </li></ul><ul><ul><ul><ul><li>Generational differences </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Views about “seriousness”/permanence </li></ul></ul></ul></ul><ul><ul><ul><li>Concerns that the technology may encourage the “wrong” types of assessment </li></ul></ul></ul><ul><ul><ul><ul><li>Too much multiple choice </li></ul></ul></ul></ul><ul><ul><ul><li>Worries about marking </li></ul></ul></ul><ul><ul><ul><li>Worries about consistency over time between exams taken on paper and on screen </li></ul></ul></ul><ul><ul><ul><li>Worries about whether E-assessments will be respected </li></ul></ul></ul>
    21. 21. Ofqual <ul><li>Established by Apprenticeships, Skills, Children and Learning Bill 2009 </li></ul><ul><li>Independent of </li></ul><ul><ul><ul><ul><li>Government </li></ul></ul></ul></ul><ul><ul><ul><ul><li>The QCDA </li></ul></ul></ul></ul><ul><ul><ul><li>Five statutory objectives: </li></ul></ul></ul><ul><ul><ul><li>- Standards in qualifications </li></ul></ul></ul><ul><ul><ul><ul><li>Standards in [National Curriculum] assessments </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Public confidence </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Awareness </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Efficiency </li></ul></ul></ul></ul><ul><ul><ul><li>General duty regarding innovation </li></ul></ul></ul>
    22. 22. Innovation <ul><li>“So far as is relevant, in performing its functions Ofqual must have regard to – </li></ul><ul><li>….. </li></ul><ul><li>(g) The desirability of facilitating innovation in connection with the provision of regulated qualifications” </li></ul><ul><li>(Clause 128(2)) </li></ul>
    23. 23. Avoiding barriers – Open letter to awarding bodies, 13 October 2008 <ul><li>No requirement of a single national IT platform for delivering E-assessment </li></ul><ul><ul><ul><ul><li>But encouragement to share good practice </li></ul></ul></ul></ul><ul><li>Ofqual committed to embed regulatory e-strategy in our day-to-day practices </li></ul><ul><li>Commitment to consider where existing rules appear to prevent innovation </li></ul>
    24. 24. Ofqual’s 4 projects <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></ul><ul><li>Measuring/forecasting uptake of E-assessment </li></ul><ul><li>On-demand testing </li></ul><ul><li>E-futures (sharing good practice) </li></ul>
    25. 25. Ofqual’s 4 projects <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></ul><ul><li>Measuring/forecasting uptake of E-assessment </li></ul><ul><li>On-demand testing </li></ul><ul><li>E-futures (sharing good practice) </li></ul>
    26. 26. e-assessment use in VRQs
    27. 27. Numbers of centres using e-assessment in a GCSE or A level
    28. 28. Three forecasts of e-assessment diffusion
    29. 29. Uptake of e-assessment in vocational qualifcations <ul><li>Use of e-assessment in vocational qualifications is increasing more rapidly and includes the use of e-testing and e-portfolios </li></ul><ul><li>In the vocational world e-assessment is often driven by demands of the learner and the technology available to them </li></ul><ul><li>A range of everyday technology opens up opportunities for the increasing use of e-assessment in the vocational world </li></ul>
    30. 30. Ofqual’s 4 projects <ul><li>Consensus statement on comparability between E-assessment and pencil-and-paper tests </li></ul><ul><li>Measuring/forecasting uptake of E-assessment </li></ul><ul><li>On-demand testing </li></ul><ul><li>E-futures (sharing good practice) </li></ul>
    31. 31. On-demand testing <ul><li>Flavours of “on-demand” </li></ul><ul><ul><ul><ul><li>Unique to candidate – any time </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unique to session – many sessions </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unique to session – few sessions </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Re-usable – centre selected dates </li></ul></ul></ul></ul><ul><ul><ul><li>Many of the organisations we have consulted suggested on-demand testing is essential for e-assessment to move forward. </li></ul></ul></ul><ul><ul><ul><li>Ofqual commissioned AQA report on principles and practice of on-demand testing: </li></ul></ul></ul><ul><ul><ul><ul><li>Examination standards, accessibility, the burden of assessment, communication </li></ul></ul></ul></ul>
    32. 32. On-demand testing (ctd) <ul><ul><ul><li>Ofqual is launching a project to develop and consult on a regulatory response that facilitates on-demand testing </li></ul></ul></ul><ul><ul><ul><li>Difficult issues of standards, accessibility, comparability, manageability </li></ul></ul></ul><ul><ul><ul><li>BUT could be a key to pervasive use of E-assessment </li></ul></ul></ul>
    33. 33. Some challenges TO Ofqual <ul><li>Are you doing enough regulation at the level of the whole system – is it fit for the modern world? </li></ul><ul><li>Are you too worried about comparability and not enough about flexibility and validity for a changing curriculum? </li></ul><ul><li>Why are you accrediting pencil-and-paper GCSEs for use until 2013 (and beyond)? </li></ul><ul><li>Why is there so little E-assessment required in the Diplomas? </li></ul><ul><li>What are you doing to encourage MAINSTREAM 14-19 qualifications to use E-assessment? </li></ul><ul><li>Are you giving awarding bodies enough freedom to innovate? </li></ul>
    34. 34. Some challenges FROM Ofqual <ul><li>Educationalists should challenge the E-industry </li></ul><ul><ul><ul><ul><li>E-assessment need not be low-quality </li></ul></ul></ul></ul><ul><ul><ul><li>Listen to learners </li></ul></ul></ul><ul><ul><ul><li>Engage with the public </li></ul></ul></ul><ul><ul><ul><li>Celebrate examples of good practice </li></ul></ul></ul><ul><ul><ul><li>Awarding bodies – make your products manageable in large numbers by schools and colleges </li></ul></ul></ul><ul><ul><ul><li>Are your assessments supporting the best teaching and learning in schools, colleges, workplaces…. </li></ul></ul></ul><ul><ul><ul><li>Technology in assessment should be educationally led – are you giving a strong lead? </li></ul></ul></ul>
    35. 35. Conclusions <ul><li>Ofqual will place E-assessment at the heart of our regulation – it is not a separate specialist area </li></ul><ul><li>Linked to our statutory objectives </li></ul><ul><ul><ul><ul><li>Standards </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Confidence </li></ul></ul></ul></ul><ul><ul><ul><li>And our general duty to promote innovation </li></ul></ul></ul><ul><ul><ul><li>4 projects: </li></ul></ul></ul><ul><ul><ul><ul><li>Comparability between modes </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Measurement/forecasting </li></ul></ul></ul></ul><ul><ul><ul><ul><li>On-demand tests </li></ul></ul></ul></ul><ul><ul><ul><ul><li>E-futures </li></ul></ul></ul></ul>
    36. 36. Ofqual’s vision <ul><li>The independent regulator of qualifications and assessments that are valued and trusted by learners, users and the wider public. </li></ul>
    37. 37. <ul><li>Thank you </li></ul><ul><li>[email_address] </li></ul><ul><li>www.ofqual.gov.uk </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×