Regulating for quality in assessment  AAIA Conference 2010 Isabel Nisbet 15 September 2010
Summary <ul><ul><ul><li>Ofqual  </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><...
Overview <ul><li>Ofqual’s role in relation to qualifications and [National Curriculum] assessments: </li></ul><ul><ul><ul>...
Ofqual’s Reliability Programme <ul><ul><ul><li>“..As the regulator of qualifications in England I believe that it is essen...
A perfect assessment? <ul><li>&quot;We have really got to begin to engage with the public in a dialogue about the nature o...
Reliability  <ul><ul><ul><li>Reliability is an indicator of the quality of an assessment  </li></ul></ul></ul><ul><ul><ul>...
<ul><li>To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinat...
<ul><li>Strand 1: Generating evidence of reliability </li></ul><ul><li>Strand 2: Interpreting and communicating evidence o...
Work in progress <ul><li>AQA: The reliabilities of GCSE and GCE components </li></ul><ul><li>Cambridge Assessment: The rel...
Reliability – lessons  <ul><li>Concept of “error” misunderstood  </li></ul><ul><ul><li>- Variability doesn’t mean “mistake...
Standards – some provocative remarks  <ul><li>Tired old arguments about standards over time </li></ul><ul><li>Important ne...
Maintaining standards in practice – GCSEs and A levels  <ul><ul><ul><li>Ofqual sets conditions awarding organisations must...
Checking standards in all regulated qualifications  <ul><ul><ul><li>Monitoring the questions set and the marking of examin...
Innovation <ul><li>Ofqual aims to facilitate innovation – e-assessment? On-demand? </li></ul><ul><li>Need for valid assess...
Regulatory principles for e-assessment <ul><li>Validity and reliability  </li></ul><ul><li>Security </li></ul><ul><li>Data...
The extent of e-marking across Awarding Bodies: 2007  <ul><li>36% of GCSE and A level scripts electronically marked (wide ...
The big step  <ul><li>From E-marking to E-testing  </li></ul>
Over to you <ul><ul><ul><li>How we can help you </li></ul></ul></ul><ul><ul><ul><li>-  Providing essential information on ...
Summary <ul><ul><ul><li>Ofqual  </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><...
Thank you  <ul><li>www.ofqual.gov.uk </li></ul><ul><li>[email_address] </li></ul>
Upcoming SlideShare
Loading in …5
×

Regulating for quality in assessment

645
-1

Published on

Regulating for quality in assessment speech delivered by Isabel Nisbet at the AAIA (Association for Achievement and Improvement through Assessment) conference on 15 September 2010.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
645
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Regulating for quality in assessment

  1. 1. Regulating for quality in assessment AAIA Conference 2010 Isabel Nisbet 15 September 2010
  2. 2. Summary <ul><ul><ul><li>Ofqual </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><ul><ul><li>Standards </li></ul></ul></ul><ul><ul><ul><li>Our approach in action </li></ul></ul></ul><ul><ul><ul><ul><li>- GCSEs and A levels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Checking that standards are maintained </li></ul></ul></ul></ul><ul><ul><ul><li>Innovation and E-assessment and on-demand testing </li></ul></ul></ul>
  3. 3. Overview <ul><li>Ofqual’s role in relation to qualifications and [National Curriculum] assessments: </li></ul><ul><ul><ul><ul><li>encourage and support the development and implementation of [qualifications] [regulated assessments] that give a reliable indication of achievement , and that indicate a consistent level of attainment (including over time) between comparable [qualifications] [assessments] </li></ul></ul></ul></ul><ul><ul><ul><ul><li>promote public confidence in [regulated qualifications] [assessment arrangements] </li></ul></ul></ul></ul>
  4. 4. Ofqual’s Reliability Programme <ul><ul><ul><li>“..As the regulator of qualifications in England I believe that it is essential for us to understand better the reliability of assessments in our national systems… Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments in this country.” </li></ul></ul></ul><ul><ul><ul><li> Kathleen Tattersall, May 2008 </li></ul></ul></ul>
  5. 5. A perfect assessment? <ul><li>&quot;We have really got to begin to engage with the public in a dialogue about the nature of the examination system. </li></ul><ul><li>&quot;There's a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on. </li></ul><ul><li>&quot;Would a student have received the same result if she happened to have taken a different version of the examination, on a different day, with a different examiner marking her work?” </li></ul><ul><li> Kathleen Tattersall, 2008 </li></ul>
  6. 6. Reliability <ul><ul><ul><li>Reliability is an indicator of the quality of an assessment </li></ul></ul></ul><ul><ul><ul><li>Reliability work in England has generally been </li></ul></ul></ul><ul><ul><ul><ul><li>- Isolated </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Partial </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Under-theorised </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Under-reported </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Misunderstood </li></ul></ul></ul></ul><ul><ul><ul><li>Ofqual’s reliability programme will help to improve the situation </li></ul></ul></ul>
  7. 7. <ul><li>To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinations and qualifications </li></ul>Aims
  8. 8. <ul><li>Strand 1: Generating evidence of reliability </li></ul><ul><li>Strand 2: Interpreting and communicating evidence of reliability </li></ul><ul><li>Strand 3: Developing reliability policy </li></ul><ul><li>Strand 3a: Exploring public understanding of reliability </li></ul><ul><li>Strand 3b: Developing Ofqual policy on reliability </li></ul>Programme structure
  9. 9. Work in progress <ul><li>AQA: The reliabilities of GCSE and GCE components </li></ul><ul><li>Cambridge Assessment: The reliabilities of GCSE and GCE components and qualifications </li></ul><ul><li>City & Guilds: The reliabilities of vocational qualifications: </li></ul><ul><li>Assessment Europe: Quantifying and interpreting component reliability using G-theory analysis </li></ul><ul><li>Assessment Europe: Literature review of reliability studies of teacher assessments </li></ul><ul><li>TAG Report: Summary, further study and advice </li></ul><ul><li>Ofqual: Quantitative investigation of public perceptions using online questionnaire </li></ul>
  10. 10. Reliability – lessons <ul><li>Concept of “error” misunderstood </li></ul><ul><ul><li>- Variability doesn’t mean “mistakes” </li></ul></ul><ul><li>Public concerns but understanding about use of judgement </li></ul><ul><li>Issues around implications for progression and accountability of small differences in marks </li></ul><ul><li>Limits to reliability of other methods of assessment too </li></ul><ul><ul><li>- Pre-university selection tests </li></ul></ul><ul><ul><li>- Interviews/vivas </li></ul></ul>
  11. 11. Standards – some provocative remarks <ul><li>Tired old arguments about standards over time </li></ul><ul><li>Important new arguments about whether qualifications (academic, professional or vocational) are FIT FOR PURPOSE </li></ul><ul><li>Time to challenge: </li></ul><ul><ul><ul><ul><li>Unevidenced prejudices </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>against some qualifications </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>against some awarding organisations </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>Urban myths </li></ul></ul></ul></ul><ul><ul><ul><ul><li>National inferiority complexes </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unattainable ideals about reliability </li></ul></ul></ul></ul>
  12. 12. Maintaining standards in practice – GCSEs and A levels <ul><ul><ul><li>Ofqual sets conditions awarding organisations must meet to be recognised – governance; expertise; quality assurance </li></ul></ul></ul><ul><ul><ul><li>Ofqual recognises awarding organisations that show they can meet the conditions </li></ul></ul></ul><ul><ul><ul><li>Ofqual sets criteria for specific groups of qualifications (eg “GCSE history”) </li></ul></ul></ul><ul><ul><ul><li>Ofqual accredits qualifications that meet these criteria </li></ul></ul></ul><ul><ul><ul><li>Ofqual makes sure that different versions of the same qualification are comparable </li></ul></ul></ul><ul><ul><ul><li>Ofqual issues codes of practice - principles, processes and practices to be followed by organisations developing and delivering GCSEs and A levels </li></ul></ul></ul><ul><ul><ul><li>Ofqual makes sure that action is taken in real time to ensure that each set of awards is consistent and fair and that standards are maintained over time </li></ul></ul></ul>
  13. 13. Checking standards in all regulated qualifications <ul><ul><ul><li>Monitoring the questions set and the marking of examiners </li></ul></ul></ul><ul><ul><ul><li>Checking that awarding organisations carry out their roles correctly, through </li></ul></ul></ul><ul><ul><ul><li>- scrutiny programme </li></ul></ul></ul><ul><ul><ul><li>- code of practice monitoring programme </li></ul></ul></ul><ul><ul><ul><li>- audits of awarding organisations </li></ul></ul></ul><ul><ul><ul><li>Comparability studies (reviewing qualification standards over time, across organisations and between subjects) </li></ul></ul></ul><ul><ul><ul><li>Planned international comparison study </li></ul></ul></ul><ul><ul><ul><li>ALL qualifications, selected by risk </li></ul></ul></ul>
  14. 14. Innovation <ul><li>Ofqual aims to facilitate innovation – e-assessment? On-demand? </li></ul><ul><li>Need for valid assessments of 21 st Century learning </li></ul><ul><li>What might drive innovation? </li></ul><ul><li>- Efficiency </li></ul><ul><li>- Demand </li></ul><ul><li>- Quality </li></ul><ul><li>- Changes in teaching and learning </li></ul><ul><li>- Technological change </li></ul><ul><li>Ofqual has published principles to govern E-assessment to support innovation while protecting standards and integrity </li></ul>
  15. 15. Regulatory principles for e-assessment <ul><li>Validity and reliability </li></ul><ul><li>Security </li></ul><ul><li>Data integrity </li></ul><ul><li>Access to e-assessment </li></ul><ul><li>Business continuity and disaster recovery </li></ul><ul><li>Use of e-portfolios </li></ul>
  16. 16. The extent of e-marking across Awarding Bodies: 2007 <ul><li>36% of GCSE and A level scripts electronically marked (wide variation across awarding bodies) </li></ul><ul><li>Awarding Body trials found: </li></ul><ul><ul><li>- Reliable </li></ul></ul><ul><ul><li>- Timely </li></ul></ul><ul><ul><li>- Scanning and supporting technologies did not fail </li></ul></ul><ul><ul><li>- Accurate assembly of results and archiving </li></ul></ul><ul><ul><li>- (Senior) Examiner satisfaction </li></ul></ul><ul><ul><li>No extra admin burden on centres </li></ul></ul><ul><li>No link to concerns over quality of marking </li></ul>
  17. 17. The big step <ul><li>From E-marking to E-testing </li></ul>
  18. 18. Over to you <ul><ul><ul><li>How we can help you </li></ul></ul></ul><ul><ul><ul><li>- Providing essential information on assessments </li></ul></ul></ul><ul><ul><ul><li>- Providing information on what we require from those involved in delivering assessments </li></ul></ul></ul><ul><ul><ul><li>How you can help us </li></ul></ul></ul><ul><ul><ul><li>- Give us your view on what assessments should deliver </li></ul></ul></ul><ul><ul><ul><li>- Help us to make assessments as fair and accurate as possible </li></ul></ul></ul><ul><ul><ul><li>Contact us: www.ofqual.gov.uk </li></ul></ul></ul>
  19. 19. Summary <ul><ul><ul><li>Ofqual </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><ul><ul><li>Standards </li></ul></ul></ul><ul><ul><ul><li>Our approach in action </li></ul></ul></ul><ul><ul><ul><ul><li>- GCSEs and A levels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Checking that standards are maintained </li></ul></ul></ul></ul><ul><ul><ul><li>Innovation and E-assessment and on-demand testing </li></ul></ul></ul>
  20. 20. Thank you <ul><li>www.ofqual.gov.uk </li></ul><ul><li>[email_address] </li></ul>
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×