Using Quality Management (QM) to Improve Language Testing


Published on

Dr. Nick Saville, language assessment specialist with University of Cambridge ESOL Examinations, presents at the 2011 Language Teaching Research Colloquium in Ann Arbor, MI.

Published in: Business, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Ann Arbor, LTRC June 2011 University of Cambridge Nick Saville and Michael Milanovic
  • Using Quality Management (QM) to Improve Language Testing

    1. 1. Using Quality Management (QM) to improve language testing Nick Saville and Michael Milanovic LTRC 2011
    2. 2. Outline – see handout <ul><li>Introduction - QMS </li></ul><ul><li>Historical perspectives and definition of QM </li></ul><ul><li>Key points in a QMS approach: </li></ul><ul><ul><li>People </li></ul></ul><ul><ul><li>Processes </li></ul></ul><ul><li>Quality Control (QC) and Quality Assurance (QA) in practice </li></ul><ul><ul><li>Illustrations of QC and QA in operational systems </li></ul></ul><ul><li>Conclusion - linking quality and validation </li></ul>
    3. 3. Quality management system <ul><li>“ Getting it right every time” </li></ul><ul><ul><li>Test developers need to adopt managerial practices which enable their organisation to implement error-free processes </li></ul></ul><ul><li>QMS provides a basis for this </li></ul><ul><ul><li>processes continually improved and standards raised </li></ul></ul><ul><ul><li>in keeping with validation as an ongoing activity </li></ul></ul><ul><li>Key point - take home message </li></ul><ul><li>convergence between the twin concepts of quality and validity </li></ul>
    4. 4. Definition of QMS <ul><li>Distinction is made between </li></ul><ul><li>quality management (QM) - overarching concept </li></ul><ul><li>quality control and quality assurance (QC and QA) - procedures for checking and assuring quality </li></ul><ul><li>QM concerned with the management of processes which lead to improvements being implemented </li></ul>
    5. 5. Definition of QMS – meeting needs <ul><li>In QM, meeting client needs is an important consideration </li></ul><ul><li>In language testing, clients are the test users: </li></ul><ul><ul><li>Primary users: test takers </li></ul></ul><ul><ul><li>Secondary users: sponsors of test takers or users of results to make decisions </li></ul></ul><ul><ul><ul><li>See the Code of Fair Testing Practices (1988) </li></ul></ul></ul><ul><li>Testing organisations need to guarantee &quot;fitness for purpose”: </li></ul><ul><li>to meet client needs for assessment in specific contexts of use, such as the workplace </li></ul>
    6. 6. Historical perspective - origins <ul><li>QM - origins in the manufacturing sector </li></ul><ul><li>Now applied across many types of organisation </li></ul><ul><li>Only recently within educational systems </li></ul><ul><ul><li>E.g. Wild and Ramaswamy (eds), 2008 </li></ul></ul><ul><ul><li> Improving Testing: Applying Process Tools and Techniques to Assure Quality </li></ul></ul>
    7. 7. Historical perspective - founders <ul><li>Founders - Taylor (1911), Shewhart (1931, 1939) </li></ul><ul><li>Shift from maintaining standards to improving standards </li></ul><ul><li>Continual improvement axiomatic within all QM systems </li></ul><ul><ul><li>processes defined in terms of inputs and outputs </li></ul></ul><ul><ul><li>links managed effectively between different parts of an organisation </li></ul></ul><ul><ul><li>reduce defective items before the final stage of a process </li></ul></ul><ul><ul><li>cut out high cost of post hoc inspections to check on quality </li></ul></ul>
    8. 8. Historical perspective - TQM <ul><li>W. Edwards Deming (1900-1993) </li></ul><ul><ul><li>leading figure in the movement </li></ul></ul><ul><ul><li>first coherent &quot;quality management systems“ (QMS) </li></ul></ul><ul><ul><ul><li>Total Quality Management - TQM </li></ul></ul></ul><ul><ul><ul><li>Deming, 1986: Out of the Crisis </li></ul></ul></ul>
    9. 9. Historical perspective - <ul><li>Quality standards </li></ul><ul><li>A mechanism of accountability </li></ul><ul><ul><li>dates back over 100 years </li></ul></ul><ul><ul><li>British Standards Institution (BSI), London, 1901 </li></ul></ul><ul><ul><li>original quality mark 1903 </li></ul></ul><ul><ul><ul><li>BSI “kite mark” </li></ul></ul></ul>
    10. 10. Historical perspective – ISO standards <ul><li>International Organization for Standardization (ISO) </li></ul><ul><ul><li>Quality Standards - ISO 9001: 2008 accreditation </li></ul></ul><ul><ul><li>Based on auditing </li></ul></ul><ul><ul><ul><ul><ul><li>Certifies that processes are being applied consistently and effectively to meet stated objectives of an organisation </li></ul></ul></ul></ul></ul>
    11. 11. Two key points in a QM approach <ul><li>People </li></ul><ul><li>the importance of leadership and staff development </li></ul><ul><ul><ul><li>In our field this includes specific training in assessment literacy </li></ul></ul></ul><ul><li>Processes </li></ul><ul><li>the importance of defining processes and managing them effectively </li></ul>
    12. 12. Key points - People <ul><li>A key asset in achieving error-free processes: </li></ul><ul><li>Managers ensure that staff are empowered to carry out their responsibilities </li></ul><ul><li>QC & QA procedures ensure all involved contribute to quality </li></ul><ul><li>Importance of: </li></ul><ul><ul><li>Expertise </li></ul></ul><ul><ul><li>Training </li></ul></ul><ul><ul><li>Ongoing professional development </li></ul></ul>
    13. 13. Key points - Processes <ul><li>Core processes - the assessment cycle </li></ul><ul><ul><li>Planning and design </li></ul></ul><ul><ul><li>Development - e.g. of systems for test assembly </li></ul></ul><ul><ul><li>Delivery - routine test assembly and administration </li></ul></ul><ul><ul><li>Processing – marking, grading, issue of results </li></ul></ul><ul><ul><li>Review and evaluation </li></ul></ul>
    14. 14. Key points - process interactions map <ul><ul><li>Figure 1 inputs and outputs </li></ul></ul>
    15. 15. Key points – Plan, Do, Check, Act <ul><li>Mistakes, inaccuracies, human error, malpractice, cheating, etc. – all threaten validity </li></ul><ul><li>Iterative processes provide feedback </li></ul><ul><ul><li>to evaluate effectiveness </li></ul></ul><ul><ul><li>to correct errors </li></ul></ul><ul><ul><li>to make improvements in subsequent iterations </li></ul></ul>
    16. 16. Key points - Figure 2 Aims, Inputs, Outputs
    17. 17. QC and QA in practice – check, act <ul><li>QC synonymous with checking </li></ul><ul><ul><li>systematic checks ensure that all aspects of a “work flow” meet the standards as defined and described in documentation for each stage of the overall process </li></ul></ul><ul><ul><li>checks are carried out by the people who have the responsibility for doing the tasks (not by a separate &quot;quality control department&quot;) </li></ul></ul><ul><li>QA carried out to monitor, evaluate and improve </li></ul><ul><ul><li>to ensure that all processes are maintained to the required standard </li></ul></ul><ul><ul><li>to monitor and adjust processes within the system </li></ul></ul><ul><ul><li>to evaluate whether core processes are working adequately </li></ul></ul>
    18. 18. QC in practice (1) <ul><li>Illustration 1 - Assembling test materials </li></ul><ul><li>production of test specifications at the development stage and instructions for item writers for assembling tests - essential prerequisite </li></ul><ul><li>checking against specifications - at various stages of the operational cycle before the assessment is finalised and ready to be administered </li></ul><ul><li>system to collect, store and process test materials </li></ul><ul><ul><li>particularly important if a large number of items and tasks are involved e.g. may require an item banking system </li></ul></ul>
    19. 19. QC in practice (1) <ul><li>Illustration 1 - Assembling test materials </li></ul><ul><li>A QC system for managing test materials might include: </li></ul><ul><ul><li>an ID and description of each task or set of materials </li></ul></ul><ul><ul><li>a checklist to track progress and to provide a complete record of the stages completed, changes made and information about decisions taken </li></ul></ul><ul><ul><li>test materials which are stored and are readily accessible with relevant documentation and management information accrued during the editing process </li></ul></ul><ul><ul><li>rules for confidentiality and security in handling materials </li></ul></ul>
    20. 20. QC in practice (2) <ul><li>Illustration 2 - Administering tests </li></ul><ul><li>in large-scale operations the administration of the assessment will be delegated to personnel at the testing venue </li></ul><ul><li>the test developer needs to be confident that assessments are administered in a standardised way </li></ul><ul><ul><li>Consistency is important </li></ul></ul><ul><ul><li>Uncontrolled variation undermines validity </li></ul></ul><ul><li>administrative procedures need to be clearly and comprehensively described and produced in a format which can be used under operational conditions: </li></ul>
    21. 21. QC in practice (2) <ul><li>QC should include: </li></ul><ul><li>physical setting </li></ul><ul><li>storage and handling of secure and confidential materials/information </li></ul><ul><li>type and number of personnel needed to oversee the administration </li></ul><ul><ul><li>to ensure safety and guarantee test security </li></ul></ul><ul><li>procedures for recruitment, training and management of invigilators /proctors and examiners </li></ul><ul><li>management of the interaction between test takers and administrators </li></ul><ul><ul><li>the checking of their identities and eligibility, seating arrangements, provision of accommodations to meet special needs or requirements </li></ul></ul><ul><li>management of the assessment procedures themselves </li></ul><ul><ul><li>providing instructions, handing out and collecting test papers and answer sheets, monitoring for malpractice (including all kinds of cheating), ensuring that timings are respected </li></ul></ul><ul><li>handling of unforeseen eventualities specific circumstances </li></ul><ul><ul><li>illness, fire alarms, power cuts, minor accidents, disruption from outside the venue </li></ul></ul>
    22. 22. <ul><li>Failure to maintain quality of any aspect of the system threatens validity </li></ul><ul><li>E.g. Undetected problems may lead to construct-irrelevant variance </li></ul><ul><li>This may affect all test takers, groups of test takers, or individuals and lead to unfair outcomes </li></ul><ul><li>see Kunnan on Fairness, (2000; 2004) </li></ul>Linking quality and validation
    23. 23. Linking quality and validation <ul><li>Validation is the process of accumulating evidence to support inferences made using test results </li></ul><ul><li>Synergy between test validation and QM concept of continual improvement </li></ul><ul><li>Accrual of evidence at all stages in the cycle </li></ul><ul><li>Figure 3 </li></ul>
    24. 24. Linking quality and validation Figure 3 Assessment cycle showing periodic review Follow the dotted red line _ _ _
    25. 25. Conclusion - putting principles into practice <ul><li>QMS can help </li></ul><ul><li>to bring about change and improvement in well-managed and principled ways </li></ul><ul><li>to connect good practice in language assessment with operational procedures which are both transparent and accountable </li></ul><ul><li>ISO 9001 in conjunction with an international Code of Practice provides the basis for error-free processes and achievement of auditable professional standards </li></ul>
    26. 26. References – on handout <ul><li>Council of Europe/ALTE (2011) Manual for language test development and examining. For use with the CEFR , Strasbourg: Language Policy Division, available online </li></ul><ul><li>Deming, W. E. (1986 ). Out of the Crisis. Cambridge : Cambridge University Press. </li></ul><ul><li>Downing, S.M. (2006).Twelve Steps for Effective Test Development. In Downing, S.M.; Haladyna, T.M. 2006. </li></ul><ul><li>Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment – an advanced resource book , Abingdon: Routledge. </li></ul><ul><li>Hatch, M. J. with Cunliffe, A. L. (2006). Organization Theory: modern, symbolic and post-modern perspectives. (2nd edition). Oxford: Oxford University Press. </li></ul><ul><li>Kane, M (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4th edition) . Washington, DC: American Council on Education/Praeger. </li></ul><ul><li>Kemp, S. (2006). Quality Management Demystified. New York, McGraw Hill. </li></ul><ul><li>Kuijper, H. (2003). QMS as a Continuous Process Of Self-Evaluation and Quality Improvement for Testing Bodies. </li></ul><ul><li>Kunnan, A. J. (2004). Test Fairness, in M. Milanovic and C. Weir (Eds). Studies in Language Testing 18: </li></ul><ul><li>Rose (2010) Research Notes, 39 (2-7). Downloadable from: </li></ul><ul><li>Saville, N. (2005).Setting and monitoring professional standards: A QMS approach. Research Notes 22, 2-5 . Cambridge: Cambridge ESOL. </li></ul><ul><li>Saville, N. (2010). Auditing the quality profile: From code of practice to standards. Research Notes, 39, 24-28. Cambridge: Cambridge ESOL. </li></ul><ul><li>Shewhart, W.A. (1931). Economic control of quality of manufactured product . New York: D. Van Nostrand Company. </li></ul><ul><li>Shewhart, W.A. (1939). Statistical method from the viewpoint of quality control. Washington , The Graduate School, the Department of Agriculture . </li></ul><ul><li>Taylor F.W. (1911). The Principles of Scientific Management. New York: Harper and Brothers Publishers,. </li></ul><ul><li>van Avermaet, P. (2003). QMS and The Setting of Minimum Standards: Issues of Contextualisation Variation between The Testing Bodies . Retrieved from </li></ul><ul><li>van Avermaet, P., Kuijper, H; and Saville, N. (2004 ). A Code of Practice and Quality Management System for International Language Examinations, Language Assessment Quarterly 1(2&3),137-150. </li></ul><ul><li>Wild, C.L. and Ramaswamy, R. (2008). Improving testing. Applying Process Tools and Techniques to Assure Quality . London: Routledge. </li></ul>
    27. 27. <ul><li>Thank You! </li></ul><ul><li>[email_address] </li></ul>