Download to read offline
![Assessment of Competence -
Assessment of Competence -
Considerations
Considerations
• Use of both “formative” and
Use of both “formative” and
“summative” evaluation methods
“summative” evaluation methods
• Authentic evaluations
Authentic evaluations
• Real world tasks, meaningful activities
Real world tasks, meaningful activities
• [High] Fidelity: Degree to which
[High] Fidelity: Degree to which
the assessment represents
the assessment represents
actual performance
actual performance](https://image.slidesharecdn.com/toolkitgrus-250325152821-f1240757/85/Toolkit-on-competency-assessment-tools-and-issues-7-320.jpg)
Toolkit on competency assessment tools and issues
- 1. The Competency Assessment TheCompetency Assessment Toolkit for Professional Toolkit for Professional Psychology Psychology Catherine L. Grus, Ph.D., Catherine L. Grus, Ph.D., Associate Executive Director for Associate Executive Director for Professional Education and Training Professional Education and Training Education Directorate Education Directorate American Psychological Association American Psychological Association
- 2. THANK YOU THANK YOU Manythanks to Many thanks to • Nadya Fouad, University of Wisconsin Nadya Fouad, University of Wisconsin • Nadine Kaslow, Emory University Nadine Kaslow, Emory University • Benchmarks Workgroups Benchmarks Workgroups • Assessment “Toolkit” Workgroup Assessment “Toolkit” Workgroup
- 3. Overview Overview • Considerations inthe assessment of Considerations in the assessment of competence competence • Development of the Toolkit Development of the Toolkit • Putting the toolkit into practice Putting the toolkit into practice
- 4. Competence … Competence … •“ … “ … the habitual and judicious use of the habitual and judicious use of communication, knowledge, technical communication, knowledge, technical skills, clinical reasoning, emotions, skills, clinical reasoning, emotions, values, and reflection in daily practice values, and reflection in daily practice for the benefit of the individual and for the benefit of the individual and community being served.” community being served.” From Epstein and Hundert (2002) From Epstein and Hundert (2002)
- 5. Competence … Competence … •… … is developmental depending on stage is developmental depending on stage of professional functioning of professional functioning • … … context dependent context dependent • … … executed according to ethical executed according to ethical principles, guidelines, standards, and principles, guidelines, standards, and values of the profession and values of the profession and • … … requires public verification requires public verification
- 6. Competencies … Competencies … –Observable Observable – Measurable Measurable – Containable Containable – Practical Practical – Derived by experts Derived by experts – And flexible And flexible are elements of competence that are …. are elements of competence that are ….
- 7. Assessment of Competence- Assessment of Competence - Considerations Considerations • Use of both “formative” and Use of both “formative” and “summative” evaluation methods “summative” evaluation methods • Authentic evaluations Authentic evaluations • Real world tasks, meaningful activities Real world tasks, meaningful activities • [High] Fidelity: Degree to which [High] Fidelity: Degree to which the assessment represents the assessment represents actual performance actual performance
- 8. • Employ arange of perspectives (multi- Employ a range of perspectives (multi- informant) and methods informant) and methods • Challenge of evaluating attitudes and Challenge of evaluating attitudes and values values • Psychometric issues Psychometric issues • Feasibility/costs Feasibility/costs • Lifelong self-assessment Lifelong self-assessment
- 9. What methods arecommonly What methods are commonly used to assess students/ used to assess students/ trainees? trainees? • In class examinations and course In class examinations and course “grades” “grades” • Faculty/supervisor completes rating Faculty/supervisor completes rating scale scale • Letters of recommendation Letters of recommendation • EPPP, state jurisprudence exams EPPP, state jurisprudence exams
- 10. Developing Faculty/Supervisor Developing Faculty/Supervisor Competencies Competencies •Training in “difficult conversations” Training in “difficult conversations” – How to “critique constructively” How to “critique constructively” • Develop awareness of potential dual roles in Develop awareness of potential dual roles in evaluation and how that could influence evaluation and how that could influence results results – Benefits of having other faculty evaluate students’ Benefits of having other faculty evaluate students’ performance performance – Explicit obligations to make evaluations accurate Explicit obligations to make evaluations accurate • Grade inflation Grade inflation • Letters of recommendation to “market” vs. evaluate Letters of recommendation to “market” vs. evaluate • Knowledge of a range of assessment tools, Knowledge of a range of assessment tools, access to resources “toolkits” access to resources “toolkits”
- 11. Competency Assessment for CompetencyAssessment for “Toolkit” for Professional “Toolkit” for Professional Psychology Psychology – Charge from APA Board of Educational Affairs: Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional psychology Develop a “Toolkit” for professional psychology – Purpose: Promote broader implementation of Purpose: Promote broader implementation of competence assessment and provide competence assessment and provide information about application of assessment information about application of assessment methods to the assessment of competence methods to the assessment of competence – Coordinated with Benchmarks Work Group Coordinated with Benchmarks Work Group
- 12. Benchmarks Core Competencies BenchmarksCore Competencies Foundational Foundational Competencies: Competencies: • Professionalism Professionalism • Reflective practice Reflective practice • Scientific knowledge and Scientific knowledge and methods methods • Relationships Relationships • Individual and cultural Individual and cultural diversity diversity • Ethical and legal Ethical and legal standards and policy standards and policy • Interdisciplinary systems Interdisciplinary systems Functional Competencies: Functional Competencies: • Assessment Assessment • Intervention Intervention • Consultation Consultation • Research and Research and evaluation evaluation • Supervision Supervision • Teaching Teaching • Administration Administration • Advocacy Advocacy
- 13. How are theBenchmarks How are the Benchmarks Organized? Organized? • Core Foundational and Functional Core Foundational and Functional Competencies Competencies • Essential Component Essential Component: what are the critical : what are the critical elements of/what elements of/what knowledge/skills/attitudes that make up knowledge/skills/attitudes that make up this competency? this competency? • Behavioral Anchor Behavioral Anchor: what would it look like : what would it look like if you saw it (essential component)? if you saw it (essential component)?
- 14. Components of Toolkit Componentsof Toolkit • Background and Introduction Background and Introduction • Assessment Method Fact Sheets Assessment Method Fact Sheets • Grid of Assessment Methods and Grid of Assessment Methods and Competencies/Essential Components Competencies/Essential Components Best used for Best used for
- 15. Fact Sheet: PortfolioReview Description A portfolio is a collection of products, gathered by the person being assessed … Application Competency Applies to1 Predominant Use of2 Development al Level3 Professionalism E F,S 2,3,4 Reflective Practice E F,S 1,2,3,4 Implementation Deciding on form (e.g., web-based or hard copy) Psychometric Properties Reliability has not been well established due to the variable content included in a portfolio Strengths Provides a low cost assessment strategy Challenges Requires intense commitment of time and labor intensive for the assessors and the person being assessed Future directions Create and implement training modules for assessors and people being assessed in the portfolio assessment strategy
- 16. Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations Intervention Knowledge interventions 2 33 2 1 2 1 1 Intervention planning 2 1 2 1 2 2 3 1 2 2 1 Skills 2 2 1 3 2 1 2 1 2 1 1 2 Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1 Progress evaluation 3 1 2 2 2 3 2 2 1 2 2 Grid of Possible Assessment Methods Grid of Possible Assessment Methods (Toolkit) (Toolkit)
- 17. Assessment Methods inToolkit Assessment Methods in Toolkit • 360 evaluation 360 evaluation • Portfolio Portfolio • OSCE OSCE • Structured Written & Structured Written & Oral Exams Oral Exams • Case Presentation Case Presentation • Simulation/Role Play Simulation/Role Play • Competence Evaluation Competence Evaluation Rating Form Rating Form • Self-Assessment Self-Assessment • Ratings of live or Ratings of live or recorded performance recorded performance • Standardized Client Standardized Client Interview Interview • Client/Patient Client/Patient Process/Outcome Process/Outcome Measure Measure • Consumer Satisfaction Consumer Satisfaction Survey Survey • End of Rotation End of Rotation Performance Review Performance Review
- 18. Putting it allTogether … Putting it all Together … Benchmarks Toolkit Using the Toolkit and Using the Toolkit and Benchmarks to Guide the Benchmarks to Guide the Assessment of Student Learning Assessment of Student Learning Outcomes Outcomes
- 19. Identify expected student learning outcomesusing competency model (e.g., Benchmarks) Select assessment method(s) Expected level of competence attainment and training plan developed with trainee Formative assessment Student demonstrates problems with competencies acquisition Student demonstrates emerging competencies Training plan continues Remediation plan developed Summative evaluation to assess readiness for next level of training
- 20. Identify expected student learning outcomesusing competency model (e.g., Benchmarks)
- 21. Program Program Outcome Outcome: Studentswill demonstrate the : Students will demonstrate the following intervention (functional) competencies following intervention (functional) competencies prior to going on internship. prior to going on internship. 1. 1. Knowledge of scientific, theoretical, empirical Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including and contextual bases of intervention, including theory, research, and practice theory, research, and practice 2. 2. Formulates and conceptualizes cases and plan Formulates and conceptualizes cases and plan interventions utilizing at least one consistent interventions utilizing at least one consistent theoretical orientation theoretical orientation 3. 3. Clinical skills Clinical skills 4. 4. Implements evidence-based interventions that Implements evidence-based interventions that take into account empirical support, clinical take into account empirical support, clinical judgment, and client diversity (e.g., client judgment, and client diversity (e.g., client characteristics, values, and context) characteristics, values, and context) 5. 5. Evaluate treatment progress and modify Evaluate treatment progress and modify treatment planning as indicated, utilizing treatment planning as indicated, utilizing established outcome measures established outcome measures
- 22. Identify expected student learning outcomesusing competency model (e.g., Benchmarks) Select assessment method(s)
- 23. Annual review Case reviews Process/outcome data Rating Forms Consumer surveys OSCE Portfolio reviews Ratings Performance Record reviews Self-assessment Simulations/role plays Stand. patient interview Stand. oral exams Written exams 360 Evaluations Intervention Knowledge interventions 2 33 2 1 2 1 1 Intervention planning 2 1 2 1 2 2 3 1 2 2 1 Skills 2 2 1 3 2 1 2 1 2 1 1 2 Implementation 1 1 1 1 1 1 2 1 2 1 2 2 2 3 1 Progress evaluation 3 1 2 2 2 3 2 2 1 2 2 Grid of Possible Assessment Methods Grid of Possible Assessment Methods (Toolkit) (Toolkit)
- 24. Identify expected student learning outcomesusing competency model (e.g., Benchmarks) Select assessment method(s) Expected level of competence attainment and training plan developed with trainee
- 25. Developmental Level A. Knowledgeof Interventions Readiness for Practicum Readiness for Internship Essential Component: Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence- based practice and its role in scientific psychology Behavioral anchor: Articulates the relationship of EBP to the science of psychology Identifies basic strengths and weaknesses of intervention approaches for different problems and populations Essential Component: Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice Behavioral Anchor: Demonstrates knowledge of interventions and explanations for their use based on EBP Demonstrates the ability to select interventions for different problems and populations related to the practice setting Investigates existing literature related to problems and client issues Writes a statement of one’s own theoretical perspective regarding intervention strategies
- 26. Identify expected student learning outcomesusing competency model (e.g., Benchmarks) Select assessment method(s) Expected level of competence attainment and training plan developed with trainee Formative assessment Student demonstrates problems with competencies acquisition Student demonstrates emerging competencies Training plan continues Remediation plan developed
- 27. Remediation Plan Competenc y Domain/ Essential Component s Problem Behaviors Expectations forAcceptable Performance Trainee’s Responsibilitie s/ Actions Supervisors’/ Faculty Responsibilities / Actions Timeframe for Acceptable Performance Assessmen t Methods Dates of Evaluatio n Consequence s for Unsuccessful Remediation
- 28. Identify expected student learning outcomesusing competency model (e.g., Benchmarks) Select assessment method(s) Expected level of competence attainment and training plan developed with trainee Formative assessment Student demonstrates problems with competencies acquisition Student demonstrates emerging competencies Training plan continues Remediation plan developed Summative evaluation to assess readiness for next level of training
- 29. For more information: Formore information: http://www.apa.org/ed/graduate/competency.html http://www.apa.org/ed/graduate/competency.html Coming soon: Coming soon: Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Psychology. Assessment Toolkit for Professional Psychology. Training and Education in Professional Psychology. Training and Education in Professional Psychology.