SQA Adapted Digital Question Papers - the story so far...


Published on

Published in: Education, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Main point here is that the schools that tried them for real in 2006 wanted more in 2007, and requests for 2008 are up again.
  • These have been very popular in schools: many pupils prefer them to scribes and readers; staff like them because you need far fewer rooms, staff and invigilators, SQA like them because pupils are more independent.
  • Reasons why pupils needed digital papers in 2007 – the most common reason is because of specific learning difficulties.
  • Comments from pupils who used the papers in exams in 2006
  • Comments from pupils in 2006
  • Staff evaluations in 2006 – staff felt pupils were more confident, independent, motivated and skilled with DPs than with other methods like readers/scribes (as we would expect given that the pupils chose to use the DPs in preference to other methods)
  • ADPs don’t appear to give unfair advantage, or disadvantage
  • PDF is pretty reliable. Still computers though, so need to plan, prepare, check, and have contingency arrangements
  • Fewer staff are required than when using readers/scribes
  • If we look at the different requests for ways to support pupils with reading difficulties across the 12 schools, though, we see that readers still dominate. A small number of exam entries were completed using text reader software (TRS) but there’s a long way to go before we can replace readers with ICT (if indeed, this is possible or in the best interests of the candidates).
  • Here’s a wee cost analysis of School H in 2007. 5 pupils sat SG English using DPs, and used TTS as well as typed their answers. This is an estimate of how much staff time would have been needed with readers/scribes. – 11.25 hours (and 11.25 hours worth of invigilator), and 15 rooms.
  • Here’s what was actually needed for the DPs – OK we could ague about the time – could be more, could be less. 5.25 for invigilator as well. 1 room at a time instead of 5
  • Cost comparison. School H buys in their reader/scribes so this is real money. They pay their reader/scribes £25 per half day – not per hour – so it depends on when the exams are during the day. Overall in 2008, School H had pupils using digital papers with text to speech for 57 separate papers instead of using reader/scribes. Cost saving depends on what staff cost. If it’s hourly, I’ve estimated the hourly rate at £20/hour to include employers costs. School H pay £200/day for supply cover if it was SfL staff reading/scribing who needed covered. May be more or less depending on whether the staff are teachers or support staff.
  • And nationally, it’s a bit shocking – is this really the best, most cost effective way to support pupils with reading and writing difficulties when there are clearly better, more independent and cheaper options?
  • SQA Adapted Digital Question Papers - the story so far...

    1. 1. SQA Adapted Digital Question Papers: The story so far…. Paul D. Nisbet CALL Scotland University of Edinburgh
    2. 2. Adapted Digital Papers 2006 (pilot) 2007 (pilot) 2008 2009 2010 Number of centres 8 12 46 73 Number of candidates 34 80 204 422 Number of requests 65 265 514 1,167 1,962 Number of candidates / centre 4.25 6.67 4.27 5.78 Number of requests / candidates 1.91 2.50 2.51 2.76
    3. 3. Requests for Adapted Digital Papers 2006-2010
    4. 4. 2009 requests by local authority Authority Requests Authority Requests Fife Council 149 The Moray Council 31 The City of Edinburgh 122 Highland Council 22 Falkirk 97 Shetland Islands Council 18 Dumfries and Galloway 89 North Ayrshire Council 16 Aberdeenshire 65 North Lanarkshire Council 15 Dundee City 62 Renfrewshire Council 13 South Ayrshire 60 East Dunbartonshire Council 7 Perth & Kinross 59 East Renfrewshire Council 4 Aberdeen City 52 Midlothian Council 4 West Lothian 45 Western Isles Council 2 City of Glasgow 42 East Ayrshire Council 1 Scottish Borders 41 South Lanarkshire Council 1 Argyll and Bute 38 Stirling 36 FE Colleges 21 Inverclyde 33 Independent Secondary Schools 22
    5. 5. 2009 requests by schools 2009 Centres Requests Local Authority Queen Anne High School 122 Fife The Royal High School 82 City of Edinburgh Bo'ness Academy 68 Stranraer Academy 62 Dumfries & Galloway Prestwick Academy 60 South Ayrshire Kinross High School 59 Perth & Kinross Mintlaw Academy 48 Aberdeenshire Ashcraig School 37 Glasgow City Campbeltown Grammar School 37 Argyll & Bute Peebles High School 35 Scottish Borders Aboyne Academy 34 Aberdeenshire St Columba's High School (Gourock) 33 Inverclyde St Paul's RC Academy 32 Dundee City Holy Rood High School 30 City of Edinburgh Buckie High School 29 Moray
    6. 6. 2009 requests by subject (subjects with requests > 10) Subject Number of requests Subject Number of requests English 284 French 39 Biology 98 Modern Studies 37 History 92 Physical Education 36 Computing Studies 91 Mathematics 29 Geography 76 Business Management 28 Chemistry 50 Drama 28 Craft & Design 50 Home Economics 22 Administration 48 Mathematics: Maths 1, 2 and 3 12 Physics 42 German 11
    7. 7. Reasons for requesting ADPs, 2009
    8. 8. Pupils’ comments 2006 & 2007 Prefer it to using a scribe. Easier to give answers. Easy to do. You can see both the questions and the text at the same time. Enjoyed using it! I would prefer to have split screens so that I could see questions and tables together instead of having to refer to another page.
    9. 9. Pupils’ comments 2006 & 2007 Papers are simpler to use. You do not have to fiddle about with different papers. By switching windows, you can see both the question and the text at the same time. Because it is much easier to use than a reader
    10. 10. Staff evaluations 2006 trials
    11. 11. Pupils’ results with digital papers <ul><li>“ Candidates’ results from digital papers are similar to their teachers’ estimates” </li></ul><ul><li>“ there appears to be little difference between [marks awarded for] entries using digital papers and the other entries sat by the same candidates” </li></ul><ul><li>Nisbet, P.D., Shearer, N., Balfour, F., Aitken, S., (2006) SQA Adapted Examination Papers in Digital Format: Feasibility Report 2005-2006 . www.adapteddigitalexams.org.uk </li></ul>
    12. 12. Reliability <ul><li>“ Digital exams were the only part of the 2007 diet which did not cause significant problems – they simply worked. Probably this was assisted by more stringent attention to detail, adequate risk assessment, etc.” </li></ul><ul><li>“ I personally would like to see digital exams as the default choice for pupils with physical disabilities, and paper, helpers, scribes etc as options which would need to be specially requested, because I believe they are empowering, less difficult to administer and cost-effective.” </li></ul>
    13. 13. Accommodation <ul><li>“ We had 2 rooms set up with double sized tables to allow a comfortable amount of room for laptop and papers. Each candidate was allocated a table which was the same for each exam. All tables were numbered and we made use of screen dividers which created a booth effect and cut down distractions and gave a certain amount of privacy in the small classroom. We had 9 candidates sitting the exam in digital format and 2 writing in the same room for the Standard Grade English exam. Candidates names and numbers were on the desks, and times.” </li></ul>
    14. 14. % of AA candidates using ADPs (12 schools 2006-08)
    15. 15. 2008 requests to support writing (46 centres using ADPs) “ We really appreciate this format of exam paper. The pupils are generally much happier to be independent rather than depend on scribes and readers. This year all our S3 and S4s sat the SG English examination at the same time. If we had had to provide readers and/or scribes for this we would not have been able to staff it.”
    16. 16. Support for writing 2007-08 (12 centres using ADPs)
    17. 17. Support for reading (data from 18 centres in 2008)
    18. 18. Text-reading software 2007-2008
    19. 19. Use of readers 2007-2008
    20. 20. Cost analysis: School ‘H’ English Standard Grade, 5 pupils using ADPs instead of reader/scribes Rooms Time Reader/scribe time Reading General 5 1 hour 5 hours Reading Credit 5 1 hour 5 hours Writing F/G/C 5 1.5 hours 7.5 hours Total 15 3.5 hours/ pupil 17.5 hours
    21. 21. School ‘H’ Standard Grade: digital papers for 5 pupils <ul><li>5 pupils, 5 computers, 1 room </li></ul><ul><ul><li>Preparation time (SfL): 1 hour? </li></ul></ul><ul><ul><li>Technician time: 2 hours? </li></ul></ul><ul><ul><li>Running time (SfL): 3.15 hours? </li></ul></ul><ul><ul><li>Total: 7.15 hours </li></ul></ul>
    22. 22. School ‘H’ readers/scribes vs. digital School H buys in all readers/scribes at £25 per half day. The school saved £1000 by using ADPs in 2008. * Cost calculated at £20/hour Readers / scribes Digital Papers Saving Staff hours 17.5 7.15 10 hours Staff cost @ £25/half day £250 £143 * £107 Staff cost @ £20/hour £350 £143 £207 Staff cost @ £200/day £1000 £143 £857 Rooms 5 1 4 rooms
    23. 23. Costs of readers/scribes <ul><li>16,554 reader requests; 14,197 scribe requests </li></ul><ul><li>~ 30,000 individual exams </li></ul><ul><li>~ 60,000 hours </li></ul><ul><li>Say average £20/hour for reader/scribe? </li></ul><ul><li>= £1,200,000 </li></ul><ul><li>Say £10/hour for invigilator? </li></ul><ul><li>= £600,000 </li></ul><ul><li>TOTAL = £1.8 m in Scotland per annum </li></ul>