A review of suggested measures to assess student competence de marinis

Do not copy without permission
A Review of Suggested Measures to
Assess Student Competence for
Professional Practice
2015 AOTA / OTCAS Education Summit
Denver, Colorado
October 18, 2015
Jessica DeMarinis
Ellen S. Cohn, ScD, OTR/L, FAOTA

Boston University Slideshow Title Goes Here
!  Review project background and objectives
!  Review process
!  Present 3 suggested measures to assess student
competence
!  Discuss next steps
2
Agenda

Agenda
!  Review process
competence
3

OT Education Research Agenda
1.  Theory building
2.  Pedagogy
3.  Instructional methods
4.  Learner characteristics and competencies
5.  Socialization to the profession
6.  Faculty development and resources
4
Project background and objectives
(AOTA, 2014)

OT Education Research Agenda
Goals for “Learner characteristics and competencies”
1.  Identify learner characteristics consistent with success across diverse
populations
2.  Identify educational program characteristics and practices that
effectively support diverse students’ success
3.  Identify supports for and barriers to collaborations among national data
sources
4.  Create performance-based tools for measurement of different levels of
competency
5.  Identify behavioral, social, and professional entry-level competencies
5
(AOTA, 2014)

Objectives of this project
6
•  Describe the evidence within OT on assessment of student
competencies for contemporary practice
•  Identify psychometrically sound measures used in other
healthcare disciplines, which may be transferable to OT
•  Develop a plan to use assessments at discrete points in program
curriculum to document:
•  Changes in student competence over the course of the program
•  Relationships between student competence and performance in the
practice setting

Agenda
!  Review process
competence
7

8
Approach and results
Define constructs
1
Therapist Factors
• Evidence-based practice skills and habits
• Critical thinking skills
•  Cognitive / Problem-solving skills
• Interpersonal / communication skills
•  Professionalism
• Motor skills, e.g. for MMT
• Subject matter knowledge
• Empathy
• Emotional intelligence
Client Factors
• Client personality and appearance
• Client abilities
• Client values / worldview / experiences
Environment / Context Factors
• Workplace: space, equipment, protocols
• Influence of managers and supervisors
• Power / political dynamics
• Social factors: SES, race, gender
• Intuition
• Level of experience
• Learner interest
• Opportunism: ability to make use of
resources
• Self-efficacy
• Common sense
• Therapist values / worldview / ethics
(Carrier et al., 2010; Thomas et al., 2011; Unsworth,
2008; Schell, Unsworth, & Schell, 2008)
Factors Influencing Clinical Competence

9
Define constructs
1
We prioritized 3 areas of student competence
that our curriculum seeks to develop:
Critical thinking skills
Skills and habits of evidence-based reasoning
Interpersonal communication skills
http://vectorpage.com/uploads/2014/05/Business-Icons-11.jpg
http://vectorpage.com/uploads/2015/02/Business-Icons-3.png
http://vectorpage.com/uploads/2014/12/Business-Different-Icons-2.png

10
Definition
Corresponding BU
Coursework
Appraisal and
communication
of evidence
“EBP requires that decisions about healthcare are based on
best available, current, valid, and relevant evidence.
These decisions should be made by those receiving care,
informed by the tacit and explicit knowledge of those
providing care within the context of available
resources” (Dawes et al., 2005, p. 4)
•  OT520: EBP I
•  OT620: EBP II
•  OT621: EBP III
Habits and
routines to
support
application in
practice
Implementation of EBP requires mastery of specific skills
appraising the research evidence and self-directed habits
to sustain routine use of current evidence in one’s
practice (Sackett et al., 1997)
•  OT502: Int. Seminar III
•  OT524: OT Process
•  OT564 & 565: Skills I, II
•  OT563: Context
•  OT530: GLE
•  OT556: Client Factors
Define constructs
1
Skills and habits of evidence-based reasoning
(Cohn, Coster, & Kramer, 2014)

11
Definition
Corresponding BU
Coursework
“Purposeful, self-regulatory judgment which
results in interpretation, analysis, evaluation,
and inference, as well as explanation of the
evidential, conceptual, methodological,
criteriological, or contextual considerations upon
which that judgment is based”
(Facione, 1990, p. 3)
Entire MSOT Curriculum
Define constructs
1

12
Definition
Corresponding BU
Coursework
Communication
skills
Skills in verbal and non-verbal communication,
therapeutic listening, engaging the client in a
collaborative goal-setting process, seeking and
responding to client feedback, providing clients with
structured direction and feedback, facilitating
information-sharing (Taylor, 2008)
•  OT520: EBP I
•  OT500: Int. Seminar I
•  OT501: Int. Seminar II
•  OT502: Int. Seminar III
•  OT530: GLE
Application to
client / simulated
client encounters
Therapeutic use of self is the planned use of
practitioner’s personality, insights, perceptions and
judgments as part of the therapeutic process
(Punwar & Peloquin, 2000)
•  OT513: Analysis & Adaptation
of Occupation
•  OT524: OT Process
•  OT526: Functional Movement
•  OT564 & 565: Skills I, II
•  OT530: GLE
Define constructs
1
Interpersonal communication skills

13
Search OT literature
2
Method Example Measures Limitations
Qualitative:
Clinical reasoning
process
•  Qualitative analysis of themes from
treatment sessions:
•  Oral / written reports
•  Video / audio analysis
(Unsworth, 2008)
• Validity depends on OT ability to
remember and write out own thoughts
• Unable to measure changes over time
• Non-objective performance measure
• Time-consuming to analyze
Quantitative:
Student self-
report
•  Self-Assessment of Clinical Reflection
and Reasoning (Royeen et al., 2000)
•  Simulated patient encounter, student self-
score Poor-Very Well (Herge et al., 2013)
• Non-objective performance measure
• Risk of respondent self-editing
Quantitative:
Professor /
Supervisor rater
•  Fieldwork Performance Evaluation Form
(AOTA, 2002)
•  Simulated patient encounter, prof.-
determined grading criteria, (Giles, 2014)
• Many of the quantitative measures in
OT literature lack evidence to support
psychometric properties
Takeaway: OT literature review suggests a paucity of evidence using
psychometrically sound measures to assess student preparedness for practice

14
Search healthcare professions literature
3
!  No limitations on specific profession
! Most results from Medicine and Nursing
! Other fields: pharmacy, psychology, PT
! Re-examined OT literature as well
!  Focused on studies that used
quantitative measures to assess our 3
priority areas of student competence:
! Skills and habits of evidence-based reasoning
! Critical thinking skills
! Interpersonal communication skills
!  Focused on measures that were most
relevant and transferable to OT
http://www.mc.vanderbilt.edu/root/vumc.php?site=capnah&doc=42879

Agenda
!  Review process
competence for professional practice
15

16
Construct Suggested Measure
Skills and habits of
evidence-based reasoning
Adapted Fresno Test
(McCluskey & Bishop, 2009)
Health Sciences Reasoning Test
(Insight Assessment, 2015)
Interpersonal
communication skills
Communication Assessment Tool
(Makoul, Krupat, & Chang, 2007)
Three suggested measures to assess
OT student competency for practice
Suggested measures

The Adapted Fresno Test (AFT)
17
Measures
! Competency in evidence-based practice knowledge and skills
! Adapted for use in OT (McCluskey & Lovarini, 2005; McCluskey & Bishop,
2009) from Ramos et al.’s (2003) original version for physicians
Admin
! 7-item, free-response questions based on brief clinical vignettes
! Grading rubric for point allocation (156 maximum points)
! Timing: 20 min to complete and 20 min to score
! 3 versions to facilitate testing at discrete time points
Study
Contexts
! Practicing OTs (Australia) pre/post a workplace-based EBP training
(McCluskey & Lovarini, 2005; McCluskey & Bishop, 2009)
! OT students (US) pre/post an EBP course and post-fieldwork (Crabtree et al.,
2012)
! School-based OTs pre/post a professional training on EBP (Cahill et al., 2015)
Advantages
! Psychometric properties established (Glegg & Holsti, 2010; McCluskey &
Bishop, 2009)
! Freely available (A. McCluskey, personal communication, Oct. 8, 2014)
Dis-
advantages
! Inter-rater reliability: time-consuming to establish consensus
! Vignettes, while adapted for use in OT, contain non-ideal content choices
Suggested measures

18
(McCluskey & Bishop, 2009)
The Adapted Fresno Test (AFT)
Suggested measures

Health Sciences Reasoning Test (HSRT)
19
Measures
! Critical thinking (CT) (total score) as well as five subscale scores for inference,
induction, deduction, analysis, and evaluation
! Based on constructs of California Critical Thinking Skills Test, a long-used
measure for CT; adapted for healthcare context (Insight Assessment, 2013)
Admin
! Health-based vignettes assess ability to think critically (but no previous domain-
specific knowledge is needed) (Huhn et al., 2011)
! 33-item multiple choice survey
! Online or paper-and-pencil formats available
Study
Contexts
! In a variety of health professions: used to evaluate change in students’ CT
performance (Huhn et al., 2011; Insight Assessment, 2015; Pitt et al., 2015)
! In unpublished OT doctoral dissertations: relationship to NBCOT exam score
(Inda, K., 2011); student change in CT after a course (Rodriguez, E.L., 2009)
Advantages
! Healthcare context connects to real-world practice (Huhn et al., 2011)
! Acceptable psychometric properties (Pitt et al., 2015)
Dis-
advantages
! Cannot receive item-specific data, only sub-scale and total scores
! Cost: $165 set-up fee + $10.77 per admin (Insight Assessment, personal
communication, Feb. 4, 2015)
Suggested measures

Communication Assessment Tool (CAT)
20
Measures
! Patient perceptions of physician performance on interpersonal and
communication skills
Admin
!  15 items, uses a 5-point scale (1 = poor - 5 = excellent)
!  Fourth-grade reading level
!  Timing: 1-2 minutes to complete
!  Online and telephone options studied (but written seems feasible)
Study
Contexts
! Developed with physicians-in-training and physicians-in-practice
across specialties (US) (Makoul et al., 2007)
! Learning outcomes: over time for medical residents (Stausmire et al.,
2015); in a nursing education program (Yoo et al., 2010)
Advantages
! Client-centered perspective; quick and simple format
! May work nicely with a simulated patient encounter teaching method
! Acceptable psychometric properties (Makoul et al., 2007)
Dis-
advantages
! Not designed for or studied in OT (but items seem transferable)
! Only measures 1 facet of communication skills: the client's
perception of the clinician-client encounter
Suggested measures

Communication Assessment Tool (CAT)
21
(Makoul et al., 2007)
Suggested measures

Boston University: Our next steps to
assess student progress
22
Fall ‘15 Spring ‘16 Fall ‘16 Spring ‘17
Beginning with MSOT class entering Fall ’15:
• AFT Version 1 and HSRT
during first week of classes
(before OT520 EBP I begins)
• CAT for OT526 Functional
Movement simulated client
practical exam
• AFT Version 2 mid-way
through program (after
OT520 & 521, EBP I & II)
• TBD: Documentation
assessment before
OT524: OT Process
begins
• CAT for OT565 Skills II
practical exam
• AFT Version 3 and HSRT at
conclusion of the didactic
program (after OT621 EBP III
and OT530 Group Leadership
Experience; before LIIFW)
Process continues with first entry-level OTD class entering Fall ’16
Example curriculum evaluation plan

References (p. 1 of 5)
American Occupational Therapy Association. (2014). Occupational therapy education research agenda. American Journal of
Occupational Therapy, 68(Supplement_2), S83.
American Occupational Therapy Association. (2002). Fieldwork Performance Evaluation for the Occupational Therapy Student.
Bethesda, MD: AOTA Press.
Cahill, S. M., Egan, B. E., Wallingford, M., Huber-Lee, C., & Dess-McGuire, M. (2015). Results of a school-based evidence-based
practice initiative. American Journal of Occupational Therapy, 69(2), 6902220010p1–6902220010p5.
Carrier, A., Levasseur, M., Bédard, D., & Desrosiers, J. (2010). Community occupational therapists’ clinical reasoning: Identifying tacit
knowledge. Australian Occupational Therapy Journal, 57(6), 356–365. http://doi.org/10.1111/j.1440-1630.2010.00875.x
Crabtree, J. L., Justiss, M., & Swinehart, S. (2012). Occupational therapy master-level students’ evidence-based practice knowledge
and skills before and after fieldwork. Occupational Therapy in Health Care, 26(2-3), 138–149. doi:
10.3109/07380577.2012.694584
Cohn, E.S., Coster, W.J., & Kramer, J.M. (2014). Conference proceedings – Facilitated learning model to teach habits of evidence-
based reasoning across an integrated master of science in occupational therapy curriculum. American Journal of Occupational
Therapy, 68, S73-S82. http://dx.doi.org/10.5014/ajot.2014.685S05
23
References

24
Dawes, M., Summerskill, W., Glasziou, P., Cartabellotta, A., Martin, J., Hopayian, K., … Osborne, J. (2005). Sicily statement on
evidence-based practice. BMC Medical Education, 5(1), 1. http://doi.org/10.1186/1472-6920-5-1
Facione, P. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction.
Research Findings and Recommendations. American Phiolosphical Association, Newark, DE.
Giles, A. K., Carson, N. E., Breland, H. L., Coker-Bolt, P., & Bowman, P. J. (2014). Use of simulated patients and reflective video
analysis to assess occupational therapy students’ preparedness for fieldwork. American Journal of Occupational Therapy,
68(Supplement_2), S57. doi:10.5014/ajot.2014.685S03
Glegg, S. M. N., & Holsti, L. (2010). Measures of knowledge and skills for evidence-based practice: A systematic review. Canadian
Journal of Occupational Therapy, 77, 219-232. doi: 10.2182/cjot.2010.77.4.4
Herge, E. A., Lorch, A., DeAngelis, T., Vause-Earland, T., Mollo, K., & Zapletal, A. (2013). The standardized patient encounter: A
dynamic educational approach to enhance students’ clinical healthcare skills. Journal of Allied Health, 42(4), 229–35.
Huhn, K., Black, L., Jensen, G. M., & Deutsch, J. E. (2011). Construct Validity of the Health Science Reasoning Test. Journal of
Allied Health, 40(4), 181–186.
References

25
Inda KC. (2007). Relationship between clinical reasoning skills and certification exam performance in occupational therapy
candidates (Unpublished doctoral dissertation). Nova Southeastern University, Fort Lauderdale, FL.
Insight Assessment. (2013). Health Sciences Reasoning Test. Millbrae, CA.
Makoul, G., Krupat, E., & Chang, C.-H. (2007). Measuring patient views of physician communication skills: Development and
testing of the Communication Assessment Tool. Patient Education and Counseling, 67(3), 333–342.
McCluskey, A., & Lovarini, M. (2005). Providing education on evidence-based practice improved knowledge but did not change
behaviour: a before and after study. BMC Medical Education, 5(1), 40. doi:10.1186/1472-6920-5-40
McCluskey, A., & Bishop, B. (2009). The Adapted Fresno Test of competence in evidence-based practice. Journal of Continuing
Education in the Health Professions, 29(2), 119–126.
Pitt, V., Powis, D., Levett-Jones, T., & Hunter, S. (2015). The influence of critical thinking skills on performance and progression in
a pre-registration nursing program. Nurse Education Today, 35(1), 125–131. http://doi.org/10.1016/j.nedt.2014.08.006
References

26
References
Punwar, A.J. & Peloquin, S. (2000). Occupational therapy: Principles and practice (3rd ed.). Philadelphia, PA: Lippincott Williams &
Wilkins.
Rodriguez, E. (2009). The use of blended learning to facilitate critical thinking in entry level occupational therapy students (Unpublished
doctoral dissertation). Capella University.
Royeen, C. B., Mu, K., Barrett, K., & Luebben, A. J. (2000). Pilot investigation: Evaluation of clinical reflection and reasoning before and
after workshop intervention. In P. Crist (Ed.), Innovations in occupational therapy education (pp. 107-114). Bethesda, MD:
American Occupational Therapy Association.
Sackett, D. L., Rosenberg, W., Richardson, W. S., & Haynes, R. B. (1997). Evidence-based medicine: How to practice and teach EBM.
Edinburgh, Scotland: Churchill Livingstone.
Schell, B. A. B., Unsworth, C. A., & Schell, J. W. (2008). Theory and practice: new directions for research in professional reasoning. In
B. A. B. Schell & J. W. Schell (Eds.), Clinical and professional reasoning in occupational therapy (pp. 401–431).

27
References
Stausmire, J. M., Cashen, C. P., Myerholtz, L., & Buderer, N. (2015). Measuring general surgery residents’ communication skills
from the patient’s perspective using the Communication Assessment Tool (CAT). Journal of Surgical Education, 72(1),
108–116. http://doi.org/10.1016/j.jsurg.2014.06.021
Taylor, R. R. (2008). The intentional relationship: Occupational therapy and use of self. Philadelphia, PA: F.A. Davis Company
Thomas, A., Saroyan, A., & Dauphinee, W. D. (2011). Evidence-based practice: A review of theoretical assumptions and
effectiveness of teaching and assessment interventions in health professions. Advances in Health Sciences Education,
16(2), 253–276. doi:10.1007/s10459-010-9251-6
Unsworth, C. A. (2008). Review of methodologies for researching clinical reasoning. In B. A. B. Schell & J. W. Schell (Eds.),
Clinical and professional reasoning in occupational therapy (pp. 371–400). Baltimore, MD: Lippincott Williams & Wilkins.
Yoo, M. S., Yoo, I. Y., & Lee, H. (2010). Nursing Students’ Self-Evaluation Using a Video Recording of Foley Catheterization:
Effects on Students’ Competence, Communication Skills, and Learning Motivation. Journal of Nursing Education, 49(7),
402–405.

A review of suggested measures to assess student competence de marinis

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to A review of suggested measures to assess student competence de marinis

Similar to A review of suggested measures to assess student competence de marinis (20)

More from The American Occupational Therapy Association

More from The American Occupational Therapy Association (20)

Recently uploaded

Recently uploaded (20)

A review of suggested measures to assess student competence de marinis