Your SlideShare is downloading. ×
0
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Regulating for quality in assessment
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Regulating for quality in assessment

597

Published on

Regulating for quality in assessment speech delivered by Isabel Nisbet at the AAIA (Association for Achievement and Improvement through Assessment) conference on 15 September 2010.

Regulating for quality in assessment speech delivered by Isabel Nisbet at the AAIA (Association for Achievement and Improvement through Assessment) conference on 15 September 2010.

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
597
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Regulating for quality in assessment AAIA Conference 2010 Isabel Nisbet 15 September 2010
  • 2. Summary <ul><ul><ul><li>Ofqual </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><ul><ul><li>Standards </li></ul></ul></ul><ul><ul><ul><li>Our approach in action </li></ul></ul></ul><ul><ul><ul><ul><li>- GCSEs and A levels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Checking that standards are maintained </li></ul></ul></ul></ul><ul><ul><ul><li>Innovation and E-assessment and on-demand testing </li></ul></ul></ul>
  • 3. Overview <ul><li>Ofqual’s role in relation to qualifications and [National Curriculum] assessments: </li></ul><ul><ul><ul><ul><li>encourage and support the development and implementation of [qualifications] [regulated assessments] that give a reliable indication of achievement , and that indicate a consistent level of attainment (including over time) between comparable [qualifications] [assessments] </li></ul></ul></ul></ul><ul><ul><ul><ul><li>promote public confidence in [regulated qualifications] [assessment arrangements] </li></ul></ul></ul></ul>
  • 4. Ofqual’s Reliability Programme <ul><ul><ul><li>“..As the regulator of qualifications in England I believe that it is essential for us to understand better the reliability of assessments in our national systems… Ofqual will undertake an in-depth programme of work – call it a health check – on the reliability of tests, examinations and teacher assessments in this country.” </li></ul></ul></ul><ul><ul><ul><li> Kathleen Tattersall, May 2008 </li></ul></ul></ul>
  • 5. A perfect assessment? <ul><li>&quot;We have really got to begin to engage with the public in a dialogue about the nature of the examination system. </li></ul><ul><li>&quot;There's a broad expectation that assessment should be absolutely perfect and accurate, that a mark of 50 is a mark of 50, regardless of who marks, the time at which it is marked and so on. </li></ul><ul><li>&quot;Would a student have received the same result if she happened to have taken a different version of the examination, on a different day, with a different examiner marking her work?” </li></ul><ul><li> Kathleen Tattersall, 2008 </li></ul>
  • 6. Reliability <ul><ul><ul><li>Reliability is an indicator of the quality of an assessment </li></ul></ul></ul><ul><ul><ul><li>Reliability work in England has generally been </li></ul></ul></ul><ul><ul><ul><ul><li>- Isolated </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Partial </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Under-theorised </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Under-reported </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Misunderstood </li></ul></ul></ul></ul><ul><ul><ul><li>Ofqual’s reliability programme will help to improve the situation </li></ul></ul></ul>
  • 7. <ul><li>To gather evidence for Ofqual to develop regulatory policy on reliability of results from national tests, examinations and qualifications </li></ul>Aims
  • 8. <ul><li>Strand 1: Generating evidence of reliability </li></ul><ul><li>Strand 2: Interpreting and communicating evidence of reliability </li></ul><ul><li>Strand 3: Developing reliability policy </li></ul><ul><li>Strand 3a: Exploring public understanding of reliability </li></ul><ul><li>Strand 3b: Developing Ofqual policy on reliability </li></ul>Programme structure
  • 9. Work in progress <ul><li>AQA: The reliabilities of GCSE and GCE components </li></ul><ul><li>Cambridge Assessment: The reliabilities of GCSE and GCE components and qualifications </li></ul><ul><li>City & Guilds: The reliabilities of vocational qualifications: </li></ul><ul><li>Assessment Europe: Quantifying and interpreting component reliability using G-theory analysis </li></ul><ul><li>Assessment Europe: Literature review of reliability studies of teacher assessments </li></ul><ul><li>TAG Report: Summary, further study and advice </li></ul><ul><li>Ofqual: Quantitative investigation of public perceptions using online questionnaire </li></ul>
  • 10. Reliability – lessons <ul><li>Concept of “error” misunderstood </li></ul><ul><ul><li>- Variability doesn’t mean “mistakes” </li></ul></ul><ul><li>Public concerns but understanding about use of judgement </li></ul><ul><li>Issues around implications for progression and accountability of small differences in marks </li></ul><ul><li>Limits to reliability of other methods of assessment too </li></ul><ul><ul><li>- Pre-university selection tests </li></ul></ul><ul><ul><li>- Interviews/vivas </li></ul></ul>
  • 11. Standards – some provocative remarks <ul><li>Tired old arguments about standards over time </li></ul><ul><li>Important new arguments about whether qualifications (academic, professional or vocational) are FIT FOR PURPOSE </li></ul><ul><li>Time to challenge: </li></ul><ul><ul><ul><ul><li>Unevidenced prejudices </li></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>against some qualifications </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><ul><li>against some awarding organisations </li></ul></ul></ul></ul></ul><ul><ul><ul><ul><li>Urban myths </li></ul></ul></ul></ul><ul><ul><ul><ul><li>National inferiority complexes </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Unattainable ideals about reliability </li></ul></ul></ul></ul>
  • 12. Maintaining standards in practice – GCSEs and A levels <ul><ul><ul><li>Ofqual sets conditions awarding organisations must meet to be recognised – governance; expertise; quality assurance </li></ul></ul></ul><ul><ul><ul><li>Ofqual recognises awarding organisations that show they can meet the conditions </li></ul></ul></ul><ul><ul><ul><li>Ofqual sets criteria for specific groups of qualifications (eg “GCSE history”) </li></ul></ul></ul><ul><ul><ul><li>Ofqual accredits qualifications that meet these criteria </li></ul></ul></ul><ul><ul><ul><li>Ofqual makes sure that different versions of the same qualification are comparable </li></ul></ul></ul><ul><ul><ul><li>Ofqual issues codes of practice - principles, processes and practices to be followed by organisations developing and delivering GCSEs and A levels </li></ul></ul></ul><ul><ul><ul><li>Ofqual makes sure that action is taken in real time to ensure that each set of awards is consistent and fair and that standards are maintained over time </li></ul></ul></ul>
  • 13. Checking standards in all regulated qualifications <ul><ul><ul><li>Monitoring the questions set and the marking of examiners </li></ul></ul></ul><ul><ul><ul><li>Checking that awarding organisations carry out their roles correctly, through </li></ul></ul></ul><ul><ul><ul><li>- scrutiny programme </li></ul></ul></ul><ul><ul><ul><li>- code of practice monitoring programme </li></ul></ul></ul><ul><ul><ul><li>- audits of awarding organisations </li></ul></ul></ul><ul><ul><ul><li>Comparability studies (reviewing qualification standards over time, across organisations and between subjects) </li></ul></ul></ul><ul><ul><ul><li>Planned international comparison study </li></ul></ul></ul><ul><ul><ul><li>ALL qualifications, selected by risk </li></ul></ul></ul>
  • 14. Innovation <ul><li>Ofqual aims to facilitate innovation – e-assessment? On-demand? </li></ul><ul><li>Need for valid assessments of 21 st Century learning </li></ul><ul><li>What might drive innovation? </li></ul><ul><li>- Efficiency </li></ul><ul><li>- Demand </li></ul><ul><li>- Quality </li></ul><ul><li>- Changes in teaching and learning </li></ul><ul><li>- Technological change </li></ul><ul><li>Ofqual has published principles to govern E-assessment to support innovation while protecting standards and integrity </li></ul>
  • 15. Regulatory principles for e-assessment <ul><li>Validity and reliability </li></ul><ul><li>Security </li></ul><ul><li>Data integrity </li></ul><ul><li>Access to e-assessment </li></ul><ul><li>Business continuity and disaster recovery </li></ul><ul><li>Use of e-portfolios </li></ul>
  • 16. The extent of e-marking across Awarding Bodies: 2007 <ul><li>36% of GCSE and A level scripts electronically marked (wide variation across awarding bodies) </li></ul><ul><li>Awarding Body trials found: </li></ul><ul><ul><li>- Reliable </li></ul></ul><ul><ul><li>- Timely </li></ul></ul><ul><ul><li>- Scanning and supporting technologies did not fail </li></ul></ul><ul><ul><li>- Accurate assembly of results and archiving </li></ul></ul><ul><ul><li>- (Senior) Examiner satisfaction </li></ul></ul><ul><ul><li>No extra admin burden on centres </li></ul></ul><ul><li>No link to concerns over quality of marking </li></ul>
  • 17. The big step <ul><li>From E-marking to E-testing </li></ul>
  • 18. Over to you <ul><ul><ul><li>How we can help you </li></ul></ul></ul><ul><ul><ul><li>- Providing essential information on assessments </li></ul></ul></ul><ul><ul><ul><li>- Providing information on what we require from those involved in delivering assessments </li></ul></ul></ul><ul><ul><ul><li>How you can help us </li></ul></ul></ul><ul><ul><ul><li>- Give us your view on what assessments should deliver </li></ul></ul></ul><ul><ul><ul><li>- Help us to make assessments as fair and accurate as possible </li></ul></ul></ul><ul><ul><ul><li>Contact us: www.ofqual.gov.uk </li></ul></ul></ul>
  • 19. Summary <ul><ul><ul><li>Ofqual </li></ul></ul></ul><ul><ul><ul><li>Responsibilities for assessments </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><ul><ul><li>Standards </li></ul></ul></ul><ul><ul><ul><li>Our approach in action </li></ul></ul></ul><ul><ul><ul><ul><li>- GCSEs and A levels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>- Checking that standards are maintained </li></ul></ul></ul></ul><ul><ul><ul><li>Innovation and E-assessment and on-demand testing </li></ul></ul></ul>
  • 20. Thank you <ul><li>www.ofqual.gov.uk </li></ul><ul><li>[email_address] </li></ul>

×