Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Moving Towards Programmatic Assessment

3,542 views

Published on

Presented by Dame Lesley Southgate at the 2014 Annual Meeting

Published in: Education

Moving Towards Programmatic Assessment

  1. 1. Moving towards programmatic assessment challenges and opportunities Lesley Southgate St Georges Medical School University of London
  2. 2. Background • Over the past 30 years there has been significant change in approaches to assessment for the UK medical profession and wider. • From entry to medical school through postgraduate training and on to established practice, regular assessment has become a fact of life. 14/09/2014 lesley.southgate@dial.pipex.com
  3. 3. Background • The days are gone when progress to the next stage was determined largely by exit examinations comprising elements that were sometimes poorly constructed, unrelated to the taught curriculum and which could not be challenged. 14/09/2014 lesley.southgate@dial.pipex.com
  4. 4. Background • During the period of change the focus has principally been on improving the quality of methods of assessment and developing approaches to assessing performance in practice 14/09/2014 lesley.southgate@dial.pipex.com
  5. 5. Background • And more recently, it has become usual to combine various assessment instruments into a programme of assessment which may support a period of education or training over several years 14/09/2014 lesley.southgate@dial.pipex.com
  6. 6. Principles for good assessment design • Modern assessment programmes typically contain a range of assessment methods combined in a programme of assessment developed as part of a curriculum. They are selected in the light of the purpose and content of the assessment • Van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Edu 2005 Mar;39(3):309-17.. C.vanderVleuten@educ.unimaas.nl 14/09/2014 lesley.southgate@dial.pipex.com
  7. 7. Changes have come….. One of the cornerstones of the reform of assessment for UK undergraduate and postgraduate training, is the acceptance that a student/trainee will be assessed both by examinations, and in the workplace, within a programme of assessment methods which, taken together, ensures assessment of each domain of Good Medical Practice. (UK regulator guidance)
  8. 8. Changes have come This enables a rich picture of the performance of the student/trainee to be assembled and considered in the light of the type of decision that is to be made about career progression, learning needs, and professional development. • http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp 14/09/2014 lesley.southgate@dial.pipex.com
  9. 9. The purpose of assessing in the workplace In educational settings assessment for learning should take priority over assessment of learning. A programme of assessment should aim at building n:n relationships: each competency domain should be informed from various assessment sources and each assessment source should be used to inform about several competency domains.’ The GMC and other national bodies are currently grappling with this issue Programmatic assessment: From assessment of learning to assessment for learning. Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485
  10. 10. Introduction and context • From the beginning briefly – The UK scene from 1995……… • GMC Good Medical Practice effective from 1995 • Latest edition 2013 – The GMC performance procedures • A form of programmatic assessment introduced 1997 – PMETB 2005-2010 • Principles for assessment • The UK foundation programme
  11. 11. UK GMC standards: Good Medical Practice • Good clinical care • Maintaining good medical practice • Teaching and training, appraising and assessing • Relationships with patients • Working with colleagues • Probity • http://www.gmc-uk.org/guidance/good_medical_practice/duties_of_a_doctor.asp
  12. 12. (for example) Good medical practice • Good clinical care must include: – adequately assessing the patient's conditions, taking account of the history (including the symptoms, and psychological and social factors), the patient's views, and where necessary examining the patient – providing or arranging advice, investigations or treatment where necessary – referring a patient to another practitioner, when this is in the patient's best interests
  13. 13. UK GMC performance procedures Late 1997/2004 • Assessment of poorly performing doctors – At risk of losing licence to practice. • Questions we asked – What are the standards all practising physicians must reach whether they are in training or fully registered – What evidence must we collect to demonstrate that the standards have been reached – And where could we collect the evidence about performance
  14. 14. Which standards, what evidence, where……. • The conclusions we reached, and the design for the performance procedures, now well established in law, informed thinking in the establishment of the workplace assessments now in use in UK postgraduate training. Southgate, L., Cox, J., McAvoy, P., McCrorie, P.,et al. (2001) The General Medical Council’s Performance Procedures: peer review of performance in the workplace. Medical Education, vol.35 , Issue Supplement s1 Pages 1–78 When enough is enough: a conceptual basis for fair and defensible practice performance assessment. Schuwirth LW1, Southgate L, et al. Med Educ. 2002 Oct;36(10):925-30.
  15. 15. PMETB principles for assessment A working paper from the Postgraduate Medical Education and Training Board 14 September 2004 Lesley Southgate and Janet Grant In this paper, an assessment system refers to an integrated set of assessments which is in place for the entire postgraduate training programme and which supports the curriculum. It may comprise different methods, and be implemented either as national examinations, or as assessments in the workplace. The balance between these two approaches principally relates to the relationship between competence and performance. Competence (can do) is necessary but not sufficient for performance (does do), and as experience increases so performance based assessment in the workplace becomes more important http://evavalpa.org/modulos/modulo_04/principles__assessment.pdf
  16. 16. PMETB principles for assessment • Principle 1 The assessment system must be fit for a range of purposes • Principle 2 The content of the assessment will be based on curricula for postgraduate training which themselves are referenced to all of the areas of Good Medical Practice The blueprint from which assessments in the workplace or national examinations are drawn will be available to trainees and educators in addition to assessors/examiners • Principle 3 The methods used within the programme will be selected in the light of the purpose and content of that component of the assessment framework.
  17. 17. PMETB principles for assessment • Principle 4 The methods used to set standards for classification of trainee’s performance/competence must be transparent and in the public domain • Principle 5 Assessments must provide relevant feedback • Principle 6 Assessors/examiners will be recruited against criteria for performing the tasks they will undertake
  18. 18. PMETB principles for assessment • Principle 7 • There will be Lay input in the development of assessment • Lay opinion will be sought in relation to appropriate aspects of the development, implementation and use of assessments for classification of candidates. • Lay people may act as assessors/examiners for areas of competence they are capable of assessing. • Principle 8 • Documentation will be standardised and accessible nationally
  19. 19. PMETB principles for assessment • Principle 9 • There will be resources sufficient to support assessment • Resources will be made available for the proper training of assessors
  20. 20. Principles for good assessment design • Modern assessment programmes typically contain a range of assessment methods combined in a programme of assessment developed as part of a curriculum. They are selected in the light of the purpose and content of the assessment • Van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Edu 2005 Mar;39(3):309-17.. C.vanderVleuten@educ.unimaas.nl 14/09/2014 lesley.southgate@dial.pipex.com
  21. 21. And: the workplace • The workplace enables the trainee to encounter and resolve common and important clinical problems in real time. • The level of performance expected will depend on stage of training and feedback about progress • It gives an opportunity to observe a trainee demonstrate understanding of what it means to adopt Good Medical Practice as the basis for all aspects of professional life.
  22. 22. From the GMC…to summarise…… One of the cornerstones of the reform of assessment for UK postgraduate training, is the acceptance that a trainee will be assessed both by examinations, and in the workplace, within a programme of assessment methods which, taken together, ensures assessment of each domain of Good Medical Practice. This enables a rich picture of the performance of the trainee to be assembled and considered in the light of the type of decision that is to be made about career progression, learning needs, and professional development. • http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp
  23. 23. The purpose of assessing in the workplace In educational settings assessment for learning should take priority over assessment of learning. A programme of assessment should aim at building n:n relationships: each competency domain should be informed from various assessment sources and each assessment source should be used to inform about several competency domains.’ The GMC are currently grappling with this issue Programmatic assessment: From assessment of learning to assessment for learning. Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485
  24. 24. Early days UK FOUNDATION PROGRAMME
  25. 25. The UK Foundation programme from 2007 onwards • The Foundation Programme is a two-year generic training programme which forms the bridge between medical school and specialist/general practice training.
  26. 26. Early days: Assessment in the Programme • Purpose – Determine fitness to progress to next stage of training – Identify trainees in difficulty – Provide focused feedback consistent with CQI – Meet needs for accountability • Four Methods – mini-Clinical Evaluation Exercise (mCEX) – Directly Observed Procedures (DOP) – Case-Based Discussion (CbD) – Peer Assessment (mini- PAT) • Refined versions of traditional measures
  27. 27. • The toolkit for workplace assessment 9/14/2014 DOPS MiniCex CbD MSF Prof Dame Lesley Southgate lsouthga@sgul.ac.uk Purpose Observation Observe and assess the conduct of a practical procedure Observation Observe and assess a clinical encounter Conversation/discussion Discuss an outcome/ output from workplace activity using a record the trainees has made a contribution to Review by others/colleagues Professionalism Interpersonal skills/Team working Communication Takes place Process Reviewed and documented with feedback in the moment/ as it is happening Process Reviewed and documented with feedback in the moment/ as it is happening Outcome/output Discussing, explaining, justifying aspects of the report/record/result. Including aspects of professionalism Reflecting on comments of others within the framework of constructive feedback
  28. 28. Assessment: early days • Foundation Programme Year 1 – 3640 trainees had at least one of instruments completed – 2929 submitted at least one encounter for all four tools – There were 7 to 11 questions per instrument • All used a 6-point scale where 4 is “meets expectations”
  29. 29. CbD: Basic data
  30. 30. CbD:
  31. 31. CbD
  32. 32. Encounters: Complexity 70 60 50 40 30 20 10 0 mCEX CbD DOPs Low Average High
  33. 33. Early days: References • Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x. • Initial evaluation of the first year of the Foundation Assessment Programme. • Davies H1, Archer J, Southgate L, Norcini J. • Adv Health Sci Educ Theory Pract. 2008 May;13(2):181-92. Epub 2006 Oct 12. • mini-PAT (Peer Assessment Tool): a valid component of a national assessment programme in the UK? • Archer J1, Norcini J, Southgate L, Heard S, Davies H. • Med Educ. 2008 Oct;42(10):1014-20. doi: 10.1111/j.1365-2923.2008.03162.x. • Specialty-specific multi-source feedback: assuring validity, informing training. • Davies H1, Archer J, Bateman A, Dewar S, Crossley J, Grant J, Southgate L.
  34. 34. Early warnings 1 • The FAP was implemented in a very short time-frame in response to a central mandate and there was understandably significant concern about feasibility and the time it would require. Despite this, a mean of 16.6 case-focused assessments were submitted by each F1 trainee, although 40% of these were submitted in the last 6 weeks. It is likely that this reflects anxiety about achieving low scores early on in the year. Although the programme explicitly states that some scores of < 4 would be expected early in the year, this represents a major cultural shift in assessment. • Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x. • Initial evaluation of the first year of the Foundation Assessment Programme. • Davies H1, Archer J, Southgate L, Norcini J.
  35. 35. Early warnings 2 • It is also important that training is directed at all the health professionals involved in assessments and that it includes senior trainees and nurse specialists. In order to fully meet the PMETB principles, not only will assessors need to be trained, but there will need to be systematic processes in place to provide them with feedback on their performance
  36. 36. Feedback, judgement and training assessors • ‘What was striking during these discussions was the expectation among trainees that WPBA should be about helping them to become better doctors and their corresponding openness to feedback. However, expectations are dashed by a system that is seen to be open to bias and corruption, with assessors who are untrained and too busy, and which is thus failing to deliver high quality, honest feedback. In turn, the enormous potential benefit of helping trainee doctors learn from their performance is being lost’. • Abigail Sabey, Centre for Learning and Workforce Research University of the West of England, Bristol
  37. 37. The UK Foundation programme fast forward to 2013 • Foundation year 1 (F1) enables medical graduates to begin to take supervised responsibility for patient care and consolidate the skills that they have learned at medical school. Satisfactory completion of F1 allows the relevant university (or their designated representative in a postgraduate deanery or foundation school) to recommend to the GMC that the foundation doctor can be granted full registration. • http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/ curriculum2012
  38. 38. The UK Foundation programme 2013 • Foundation year 2 (F2) doctors remain under clinical supervision (as do all doctors in training) but take on increasing responsibility for patient care. In particular they begin to make management decisions as part of their progress towards independent practice. F2 doctors further develop their core generic skills and contribute more to the education and training of the wider healthcare workforce e.g. nurses, medical students and less experienced doctors. • http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/ curriculum2012
  39. 39. The UK Foundation programme 2013 • At the end of F2 they will have begun to demonstrate clinical effectiveness, leadership and the decision-making responsibilities that are essential for hospital and general practice specialty training. Satisfactory completion of F2 will lead to the award of a foundation achievement of competence document (FACD) which indicates that the foundation doctor is ready to enter a core, specialty or general practice training programme. • http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/ curriculum2012
  40. 40. And what happened to the workplace assessments? • SLEs (structured learning events) will continue to use the established set of tools of mini-clinical evaluation exercise (mini-CEX), acute care assessment tool (ACAT) and case-based discussion (CbD) and the forms will focus on constructive feedback and action plans. • Trainees may link SLEs and other evidence to curriculum competencies in order to demonstrate engagement with and exploration of the curriculum. The trainee has to make a judgement as to the evidence needed • Supervisors should sample the evidence linked to competencies in the ePortfolio. It is not necessary to examine all the competencies to determine a trainee’s engagement with the curriculum and to make a judgement on the trainee’s progress • http://www.jrcptb.org.uk/assessment/Documents/STAR%20report%20fina l%2029%20April%202014.pdf
  41. 41. A happy ending? 2014 • Supervised learning event Recommended minimum number* Direct observation of doctor/patient interaction: • Mini-CEX • DOPS • 3 or more per placement* • (minimum of nine observations; • at least six must be mini-CEX) • Case-based discussion (CBD) • 2 or more per placement* • Developing the clinical teacher • 1or more per year

×