The document discusses the use of Quality Management Systems (QMS) to enhance language testing processes, emphasizing the importance of quality control (QC) and quality assurance (QA) in achieving error-free testing. It highlights the historical development of QMS frameworks and their application across various sectors, particularly in education. The key takeaway is the convergence of quality and validity in test design and implementation, ensuring that testing meets user needs and maintains high standards.

1. Using Quality Management (QM) to improve language testing Nick Saville and Michael Milanovic LTRC 2011

2. Outline – see handout Introduction - QMS Historical perspectives and definition of QM Key points in a QMS approach: People Processes Quality Control (QC) and Quality Assurance (QA) in practice Illustrations of QC and QA in operational systems Conclusion - linking quality and validation

3. Quality management system “ Getting it right every time” Test developers need to adopt managerial practices which enable their organisation to implement error-free processes QMS provides a basis for this processes continually improved and standards raised in keeping with validation as an ongoing activity Key point - take home message convergence between the twin concepts of quality and validity

4. Definition of QMS Distinction is made between quality management (QM) - overarching concept quality control and quality assurance (QC and QA) - procedures for checking and assuring quality QM concerned with the management of processes which lead to improvements being implemented

5. Definition of QMS – meeting needs In QM, meeting client needs is an important consideration In language testing, clients are the test users: Primary users: test takers Secondary users: sponsors of test takers or users of results to make decisions See the Code of Fair Testing Practices (1988) Testing organisations need to guarantee "fitness for purpose”: to meet client needs for assessment in specific contexts of use, such as the workplace

6. Historical perspective - origins QM - origins in the manufacturing sector Now applied across many types of organisation Only recently within educational systems E.g. Wild and Ramaswamy (eds), 2008 Improving Testing: Applying Process Tools and Techniques to Assure Quality

7. Historical perspective - founders Founders - Taylor (1911), Shewhart (1931, 1939) Shift from maintaining standards to improving standards Continual improvement axiomatic within all QM systems processes defined in terms of inputs and outputs links managed effectively between different parts of an organisation reduce defective items before the final stage of a process cut out high cost of post hoc inspections to check on quality

8. Historical perspective - TQM W. Edwards Deming (1900-1993) leading figure in the movement first coherent "quality management systems“ (QMS) Total Quality Management - TQM Deming, 1986: Out of the Crisis

9. Historical perspective - Quality standards A mechanism of accountability dates back over 100 years British Standards Institution (BSI), London, 1901 original quality mark 1903 BSI “kite mark”

10. Historical perspective – ISO standards International Organization for Standardization (ISO) Quality Standards - ISO 9001: 2008 accreditation Based on auditing Certifies that processes are being applied consistently and effectively to meet stated objectives of an organisation

11. Two key points in a QM approach People the importance of leadership and staff development In our field this includes specific training in assessment literacy Processes the importance of defining processes and managing them effectively

12. Key points - People A key asset in achieving error-free processes: Managers ensure that staff are empowered to carry out their responsibilities QC & QA procedures ensure all involved contribute to quality Importance of: Expertise Training Ongoing professional development

13. Key points - Processes Core processes - the assessment cycle Planning and design Development - e.g. of systems for test assembly Delivery - routine test assembly and administration Processing – marking, grading, issue of results Review and evaluation

14. Key points - process interactions map Figure 1 inputs and outputs

15. Key points – Plan, Do, Check, Act Mistakes, inaccuracies, human error, malpractice, cheating, etc. – all threaten validity Iterative processes provide feedback to evaluate effectiveness to correct errors to make improvements in subsequent iterations

16. Key points - Figure 2 Aims, Inputs, Outputs

17. QC and QA in practice – check, act QC synonymous with checking systematic checks ensure that all aspects of a “work flow” meet the standards as defined and described in documentation for each stage of the overall process checks are carried out by the people who have the responsibility for doing the tasks (not by a separate "quality control department") QA carried out to monitor, evaluate and improve to ensure that all processes are maintained to the required standard to monitor and adjust processes within the system to evaluate whether core processes are working adequately

18. QC in practice (1) Illustration 1 - Assembling test materials production of test specifications at the development stage and instructions for item writers for assembling tests - essential prerequisite checking against specifications - at various stages of the operational cycle before the assessment is finalised and ready to be administered system to collect, store and process test materials particularly important if a large number of items and tasks are involved e.g. may require an item banking system

19. QC in practice (1) Illustration 1 - Assembling test materials A QC system for managing test materials might include: an ID and description of each task or set of materials a checklist to track progress and to provide a complete record of the stages completed, changes made and information about decisions taken test materials which are stored and are readily accessible with relevant documentation and management information accrued during the editing process rules for confidentiality and security in handling materials

20. QC in practice (2) Illustration 2 - Administering tests in large-scale operations the administration of the assessment will be delegated to personnel at the testing venue the test developer needs to be confident that assessments are administered in a standardised way Consistency is important Uncontrolled variation undermines validity administrative procedures need to be clearly and comprehensively described and produced in a format which can be used under operational conditions:

21. QC in practice (2) QC should include: physical setting storage and handling of secure and confidential materials/information type and number of personnel needed to oversee the administration to ensure safety and guarantee test security procedures for recruitment, training and management of invigilators /proctors and examiners management of the interaction between test takers and administrators the checking of their identities and eligibility, seating arrangements, provision of accommodations to meet special needs or requirements management of the assessment procedures themselves providing instructions, handing out and collecting test papers and answer sheets, monitoring for malpractice (including all kinds of cheating), ensuring that timings are respected handling of unforeseen eventualities specific circumstances illness, fire alarms, power cuts, minor accidents, disruption from outside the venue

22. Failure to maintain quality of any aspect of the system threatens validity E.g. Undetected problems may lead to construct-irrelevant variance This may affect all test takers, groups of test takers, or individuals and lead to unfair outcomes see Kunnan on Fairness, (2000; 2004) Linking quality and validation

23. Linking quality and validation Validation is the process of accumulating evidence to support inferences made using test results Synergy between test validation and QM concept of continual improvement Accrual of evidence at all stages in the cycle Figure 3

24. Linking quality and validation Figure 3 Assessment cycle showing periodic review Follow the dotted red line _ _ _

25. Conclusion - putting principles into practice QMS can help to bring about change and improvement in well-managed and principled ways to connect good practice in language assessment with operational procedures which are both transparent and accountable ISO 9001 in conjunction with an international Code of Practice provides the basis for error-free processes and achievement of auditable professional standards

26. References – on handout Council of Europe/ALTE (2011) Manual for language test development and examining. For use with the CEFR , Strasbourg: Language Policy Division, available online www.coe.int/t/dg4/linguistic/ManualtLangageTest-Alte2011_EN.pdf Deming, W. E. (1986 ). Out of the Crisis. Cambridge : Cambridge University Press. Downing, S.M. (2006).Twelve Steps for Effective Test Development. In Downing, S.M.; Haladyna, T.M. 2006. Fulcher, G. and Davidson, F. (2007). Language Testing and Assessment – an advanced resource book , Abingdon: Routledge. Hatch, M. J. with Cunliffe, A. L. (2006). Organization Theory: modern, symbolic and post-modern perspectives. (2nd edition). Oxford: Oxford University Press. Kane, M (2006). Validation. In R. L. Brennan (Ed). Educational measurement (4th edition) . Washington, DC: American Council on Education/Praeger. Kemp, S. (2006). Quality Management Demystified. New York, McGraw Hill. Kuijper, H. (2003). QMS as a Continuous Process Of Self-Evaluation and Quality Improvement for Testing Bodies. www.alte.org/qa/index.php Kunnan, A. J. (2004). Test Fairness, in M. Milanovic and C. Weir (Eds). Studies in Language Testing 18: Rose (2010) Research Notes, 39 (2-7). Downloadable from: www.cambridgeesol.org/rs_notes/offprints Saville, N. (2005).Setting and monitoring professional standards: A QMS approach. Research Notes 22, 2-5 . Cambridge: Cambridge ESOL. Saville, N. (2010). Auditing the quality profile: From code of practice to standards. Research Notes, 39, 24-28. Cambridge: Cambridge ESOL. Shewhart, W.A. (1931). Economic control of quality of manufactured product . New York: D. Van Nostrand Company. Shewhart, W.A. (1939). Statistical method from the viewpoint of quality control. Washington , The Graduate School, the Department of Agriculture . Taylor F.W. (1911). The Principles of Scientific Management. New York: Harper and Brothers Publishers,. van Avermaet, P. (2003). QMS and The Setting of Minimum Standards: Issues of Contextualisation Variation between The Testing Bodies . Retrieved from www.alte.org van Avermaet, P., Kuijper, H; and Saville, N. (2004 ). A Code of Practice and Quality Management System for International Language Examinations, Language Assessment Quarterly 1(2&3),137-150. Wild, C.L. and Ramaswamy, R. (2008). Improving testing. Applying Process Tools and Techniques to Assure Quality . London: Routledge.

27. Thank You! [email_address]