Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Paul Nisbet, CALL Scotland
University of Edinburgh
Abi James, BDA
USING TEXT-TO-SPEECH IN EXAMS -
PITFALLS AND PRACTICAL S...
Reading difficulties in secondary
• “20% of 11 year olds have poor reading
comprehension” (data from SAT results)
• “by 14...
“While four-fifths of pupils at Key Stage 2
reached national expectations over the last
three years, one in five primary p...
SQA Assessment Arrangements
“Assessment arrangements allow candidates
who are disabled and/or who have been
identified as ...
Assessment Arrangements 2013
• Requests for Assessment Arrangements
made for
– Scotland: 17,263 candidates (11.3% of
candi...
Assessment Arrangements 2013
Scotland Rest of UK
Requests for Access
Arrangements
17,263
candidates
(11.3%)
145,430 candid...
Computer Reader (TTS) vs. Human
Reader
• TTS offers an independent means to decoding text.
• TTS provides greater consiste...
A 6th year pupil at Denny High in Falkirk was assessed using the Neale Analysis, reading with
and without text-to-speech. ...
2002 need for Digital Question Papers with
TTS identified
2003-04 research into specification;
development of papers
2004-...
Type your answers
Click to tick
Use on-screen drawing
tools
Read questions with
text-to-speech
SQA Digital Question Papers
Software and Apps
Windows
• Acrobat Reader (free)
• Text-to-speech:
– ClaroRead, Co:Writer, Ivona MiniReader ,
Penfriend, ...
Digital Paper Requests 2008-13
2008 2009 2010 2011 2012 2013
Number of requests 514 1,167 2,000 2,832 3,694 4,291
Number o...
Digital Question Papers 2008-2013
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
2008 2009 2010 2011 2012 2013
Number ...
Scottish assessment:
Costs of readers/scribes
• 19,058 reader requests; 14,905 scribe
requests
• ~ 36,000 individual exams...
1. Become confident with the technology and text-to-
speech tool they will be using in the exam.
- Use PDFs of past papers...
Resolving issues with use of digital
papers & text-to-speech
1. Voice quality
Problem Solution
“many of the students
commented that the
synthetic voice was of poor
quality and was dif...
2. Pronunciation
Problem Solution
“Text to speech doesn’t read every
word accurately, 1928 would be
read as one thousand n...
3. Unreadable image text
Problem Solution
Some text is an image and cannot be read by
the computer:
1. SQA desktop publish...
Impact on practice
Reliant on Readers?
Stuck with Scribes?
or
Independent with ICT?
Upcoming SlideShare
Loading in …5
×

Using text-to-speech in exams - practical solutions and pitfalls, UK perspective

1,272 views

Published on

Presented by Paul Nisbet and Abi James at the BDAN International Conference, March 2014.
Since 2012 the JCQ Access Arrangements have acknowledged that candidates using a Computer Reader or text-to-speech technology are reading independently making such provision available to candidates in exams that test reading skills for the first time. While use of digital exams with text-to-speech has been widely supported in Scotland through the work of SQA and CALL Scotland for a number of years, the rest of the UK has not had equivalent access. From 2013/14 exam boards in England, Wales and Northern Ireland propose to provide digital versions of exam papers to schools for text-to-speech users. This paper will draw on experiences in Scotland and the work of the BDA New Technology Committee to identify processes and best practices within schools for using these digital papers and to identify the best text-to-speech technology to maximise the benefits for students.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Using text-to-speech in exams - practical solutions and pitfalls, UK perspective

  1. 1. Paul Nisbet, CALL Scotland University of Edinburgh Abi James, BDA USING TEXT-TO-SPEECH IN EXAMS - PITFALLS AND PRACTICAL SOLUTIONS
  2. 2. Reading difficulties in secondary • “20% of 11 year olds have poor reading comprehension” (data from SAT results) • “by 14 years of age 33% of pupils have unsatisfactory reading comprehension” Chris Singleton, Special Children 182 April/May 2008
  3. 3. “While four-fifths of pupils at Key Stage 2 reached national expectations over the last three years, one in five primary pupils did not achieve the expected standard in English.” Moving English forward: action to raise standards in English, Ofsted 2012 Reading difficulties in secondary
  4. 4. SQA Assessment Arrangements “Assessment arrangements allow candidates who are disabled and/or who have been identified as having additional support needs appropriate arrangements to access the assessment without compromising its integrity.” Introduction to Assessment Arrangements for Schools and Colleges SQA January 2010, revised June 2013
  5. 5. Assessment Arrangements 2013 • Requests for Assessment Arrangements made for – Scotland: 17,263 candidates (11.3% of candidates) – Rest of UK: 145,430 candidates (6.9%) • Requests made for 61,680 entries (exams) (8.3% of all entries in Scotland)
  6. 6. Assessment Arrangements 2013 Scotland Rest of UK Requests for Access Arrangements 17,263 candidates (11.3%) 145,430 candidates (6.9% - many arrangements delegated to schools) Extra Time 76% 59% Reader 30% 27% Scribe 24% 15% Use of ICT including digital papers 19% n/a - use of word processing delegated to schools Computer readers 2% 0.4% Coloured paper (excludes coloured overlays) 3% 2%
  7. 7. Computer Reader (TTS) vs. Human Reader • TTS offers an independent means to decoding text. • TTS provides greater consistency. • Human readers can be more flexible (can read anything). • Some learners prefer to use TTS. • Studies have shown candidates are more likely to check questions with TTS than using a human reader, resulting in higher scores (Dolan et al, 2005) • Computer Readers are allowed in ALL exams, including all GCSE, A-level & Functional Skills exams including those testing reading skills “since it allows the candidate to independently meet the requirements of the reading standards”
  8. 8. A 6th year pupil at Denny High in Falkirk was assessed using the Neale Analysis, reading with and without text-to-speech. Reading herself, her comprehension age was 6 years 9 months. With text-to-speech, it was over 13 years.
  9. 9. 2002 need for Digital Question Papers with TTS identified 2003-04 research into specification; development of papers 2004-05 evaluation in 6 schools 2005-06 pilot trials #1 - 31 students used digital papers in 105 examinations 2006-07 pilot trials #2 - 80 candidates used 490 digital papers in 200 examinations 2014 Rest of UK – PDF papers will be provided by exam boards (not interactive); UKAAF guidance The Scottish Experience: Research, Development and Trial
  10. 10. Type your answers Click to tick Use on-screen drawing tools Read questions with text-to-speech SQA Digital Question Papers
  11. 11. Software and Apps Windows • Acrobat Reader (free) • Text-to-speech: – ClaroRead, Co:Writer, Ivona MiniReader , Penfriend, Read and Write Gold, Adobe Read Out Loud, etc iPad • ClaroPDF (£0.69 + £1.49 for extra voices) • PDF Expert (£6.99) • Adobe Reader (free) www.bdatech.org/what-technology/text-to- speech/exams/computer-readers
  12. 12. Digital Paper Requests 2008-13 2008 2009 2010 2011 2012 2013 Number of requests 514 1,167 2,000 2,832 3,694 4,291 Number of centres 46 73 101 149 173 188 Number of candidates 204 422 675 1,069 1,327 1,677 Mean number of requests per centre 11.17 15.99 19.80 19.01 21.35 22.82 Mean number of candidates per centre 4.43 5.78 6.68 7.17 7.67 8.92 Mean number of requests per candidate 2.52 2.77 2.96 2.65 2.78 2.56
  13. 13. Digital Question Papers 2008-2013 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 2008 2009 2010 2011 2012 2013 Number of requests Number of candidates
  14. 14. Scottish assessment: Costs of readers/scribes • 19,058 reader requests; 14,905 scribe requests • ~ 36,000 individual exams • ~ 54,000 hours • Say average £20/hour for reader/scribe? = £1,080,000 • Say £10/hour for invigilator? = £540,000 • TOTAL = £1.62min Scotland last year
  15. 15. 1. Become confident with the technology and text-to- speech tool they will be using in the exam. - Use PDFs of past papers to gain experience on how to navigate the papers - Use TTS in class 2. Centre applies for digital paper 1. In Scotland via the online AAR system. 2. In the rest of the UK through the Modified Paper Route 3. Digital exam paper arrives. 1. In Scotland on CD with papers 2. In the rest of the UK exam boards differ but most allow for download 1 hour before exam from secure website How do schools and candidates “use” a computer reader?
  16. 16. Resolving issues with use of digital papers & text-to-speech
  17. 17. 1. Voice quality Problem Solution “many of the students commented that the synthetic voice was of poor quality and was difficult to understand” Nisbet, P.D., Aitken, S., Shearer, N. (2004) Trial of External Papers in Accessible PDF for Candidates with Additional Support Needs. http://www.adapteddigitalexams.org.uk/ Downloads/Reports/ 1. Scottish ‘Heather’ voice licenced for use by Scottish Schools (2007). 2. Scottish ‘Stuart’ voice developed and licenced for Scottish schools (2011).
  18. 18. 2. Pronunciation Problem Solution “Text to speech doesn’t read every word accurately, 1928 would be read as one thousand nine hundred and twenty-eight. Words are mispronounced, in particular names and places, which can affect the smooth reading of longer passages. Words containing an apostrophe are not read correctly.” Nisbet, P.D. (2010) SQA Digital Papers 2010 Report. 1. 2011 and 2012 papers (154 papers; 2,044 pages; 235,205 words!) analysed for mispronunciations 2. 308 terms identified. 3. CereProc updated the voices to fix the pronunciation. 4. Or use a TTS tool with a pronunciation editor.
  19. 19. 3. Unreadable image text Problem Solution Some text is an image and cannot be read by the computer: 1. SQA desktop publishers now replace image text with text boxes with selectable, readable text. 2. By 2012, analysis showed that almost all image text elements were readable. 3. 2014 – UKAAF guidance on accessible PDF exam papers
  20. 20. Impact on practice
  21. 21. Reliant on Readers? Stuck with Scribes? or Independent with ICT?

×