Presented by Karen Bobak, DC, EdD, Dean of Chiropractic, and Wendy Maneri, MS, DC, Associate Dean of Chiropractic Clinical Education and Health Centers, of New York Chiropractic College, discussed ways to effectively assessing student competency in a clinical setting is an essential element in professional healthcare education. Effectively assessing student competency in a clinical setting is an essential element in professional healthcare education. Moreover, the use of assessment data to improve student learning is essential in order to meet program goals, professional standards, and provide quality patient care. Examples of strategies used to develop and implement a process of assessment / analysis / communication and change will be shared. Participants will be encouraged to develop a process framework while considering the challenges and opportunities that exist within their programs.

1. CLOSING THE LOOP
ON COMPETENCY
BASED
ASSESSMENTS
Subtitle
Wendy L. Maneri, MS, DC
Associate Dean of Chiropractic Clinical Education
New York Chiropractic College
Karen A. Bobak, DC, EdD
Dean of Chiropractic
New York Chiropractic College

2. New York Chiropractic College
• 286 acre site located in upstate New York
• Multiple Degree programs to include Doctor
of Chiropractic, Masters degree in
Acupuncture and Oriental Medicine, Masters
Degree in Nutrition, Master’s Degree in
Clinical Anatomy, Master’s Degree in
Anatomy and Physiology Instruction, Master’s
Degree in Diagnostic Imaging
New York Chiropractic College is committed to academic excellence, quality
patient care, and professional leadership
Bobak/Maneri 2017 2

3. Multiple Locations with Faculty and Students Across the Country
Bobak/Maneri 2017 3

4. Where are You on Your Assessment Journey?
 How do you know that what you promise a student on the front-end is actually
achieved on the back-end??
Bobak/Maneri 2017 4

5. The Assessment Journey
Bobak/Maneri 2017
5
James Madison University, 2017

6. The Challenge of Assessing Competence
“Professional competence is more than factual knowledge
and the ability to solve problems with clear-cut solutions: it is
defined by the ability to manage ambiguous problems,
tolerate uncertainty, and make decisions with limited
information.”
Epstein, R. M., Hundert, E. M., Defining and Assessing Professional Competence, Journal of the American Medical Association, 2002, Vol. 287, No. 2, pp. 226-234
Bobak/Maneri 2017
6

7. How Does NYCC Construct Evidence of Student Competency
 Curriculum mapping
 Varying levels of Bloom’s taxonomy:
 Low in early trimesters – not all competencies
can be evaluated
 Highest levels in clinical phase
 Accreditation Meta-competency outcomes measured
across the curriculum
 Linking meta-competency outcomes to course
content
 Clinic courses: clinical activities are linked to meta-
competency outcomes
7Bobak/Maneri 2017

8. Bobak/Maneri 2017 8

9. Assessments Tell Us as Much About the Program as They Do About the Students
Student Performance Data
Excellent Assessments Excellent Curriculum Excellent Clinicians
Bobak/Maneri 2017 9

10. Program Improvement Comes From the Application of Knowledge
• Know why your program needs to improve and where
• Develop an effective change that can result in improvement
• Test the change
• Have a feedback mechanism to tell if improvement is happening
The Improvement Guide, 2009
When a program has a feedback mechanism, it is free to take
risks with assessment innovation
Bobak/Maneri 2017 10

11. Developing an Assessment and Academic Communication Plan
You must have a plan and a process to keep the program focused and aligned.
In an educational setting, each stakeholder group has unique perspectives on
academic success, however, the Program Goals must be maintained as the
central focus and the common vision of program excellence.
NYCC followed several specific steps to build a process designed to align
assessments, demonstrate student competency and close an internal feedback
loop in order to facilitate program improvement.
Bobak/Maneri 2017
11

12. Program Goals
1. To embody the principles of academic excellence and patient-centered care through the analysis of evidence, utilization
of best practices and incorporation of patient values in clinical assessment, diagnosis, chiropractic technique, case
management, and documentation.
2. To exhibit and value integrity, compassion and ethical principles in alignment with chiropractic professional standards
and consistent with the role of a healthcare provider while accepting responsibility for personal actions.
3. To demonstrate effective leadership skills.
4. To contribute positively to the chiropractic profession and broader community by utilizing knowledge and skills to
assess critical issues, adapt to change, and communicate effectively with diverse populations.
Where We Started
1. Establish or review the program goals for student learning
2. Identify external standards that must be met (Accreditation, professional standards, etc.)
3. Identify what methods are currently used to assess student learning & competency
4. Determine if assessments are linked to program goals and standards
12

13. Program Objective
Meta-competency
Outcome Alignment
Capstone
4
(10th tri)
Milestone
3
(7th tri)
Milestone
2
(2nd year)
Milestone
1
(1st year)
1b. Critically evaluate
clinical data
MC1
MC1.2
MC1.3
MC2.2
Able to efficiently gather
essential and accurate
patient information in
complex presentations and
reach precise working
diagnosis. Demonstrates a
command of all physical,
neurological, and orthopedic
examination procedures and
accurately selects the
examination appropriate to a
patient presentation.
Correctly interprets and
integrates examination
findings on a consistent
level. Accurately selects the
appropriate lab, diagnostic
imaging, and other pertinent
diagnostic studies as
appropriate. Demonstrates
the ability to determine when
medical referral is
appropriate.
Able to filter and prioritize
patient information while
synthesizing specific
diagnostic considerations and
developing differential
diagnoses. Performs physical,
neurological and orthopedic
examinations accurately.
Accurately recognizes and
interprets examination
findings on a regular basis.
Able to link signs and symptoms
of patient presentation through
analytic reasoning. Filters,
prioritizes and synthesizes
broad diagnostic categories.
Performs basic physical
examination maneuvers
correctly but does not regularly
recognize or interpret abnormal
findings.
Begins the process of gathering
relevant patient information.
Demonstrates basic ability to
filter, prioritize and connect
segments of clinical information.
DC Program Rubric
Bobak/Maneri 2017
13

14. Council on Chiropractic Education 7 Meta-competencies
META-COMPETENCY 1 - ASSESSMENT & DIAGNOSIS
META-COMPETENCY 2 - MANAGEMENT PLAN
META-COMPETENCY 3 - HEALTH PROMITON AND DISEASE PREVENTION
META-COMPETENCY 4 - COMMUNICATION AND RECORDKEEPING
META-COMPETENCY 5 - PROFESSIONAL ETHICS AND JURISPRUDENCE
META-COMPETENCY 6 - INFORMATIONA ND TECHNOLOGY LITERACY
META-COMPETENCY 7 - INTELLECTUAL AND PROFESSIONAL DEVELOPMENT
Bobak/Maneri 2017 14

15. • Primarily rubrics for clinical patient interactions
• Occur at different times for each student
• OSCE type exams
• Professionalism/Incident reporting
• Quizzes
• Self reflection
Our Rubrics and Assessments
Bobak/Maneri 2017
15

16. Our Next Steps
5. Determine if our assessments provided the necessary
evidence of competency.
6. Identify how to collect and format student performance
data.
7. Determine how often data should be reviewed to verify
that standards and goals are being met.
Bobak/Maneri 2017
16

17. • A complete and thorough evaluation was performed.
• We were able to identify and track the performance of
aspects of clinical service through the use of tagging within
ExamSoft.
• Performance criteria and outcome measures were also
tagged and correlated to Meta-competency goals.
To see where we stand, a study was performed
Bobak/Maneri 2017 17

18. Analysis was thorough and
linked to key outcomes.
Bobak/Maneri 2017 18

19. Analysis was thorough and
linked to key outcomes.
Bobak/Maneri 2017 19

20. New Patient CRW 2 1 0 Met.
Outcom
e
1. Did the student
determine all of
the correct
differential
diagnoses and
provide rationale
why each were
eliminated?
Yes all major
differential
dx. were
considered
with
rationale.
Missing some
differentials or
wrong
rationale.
No, the
student
was
missing
major
differential
s or
rationale.
MC1.1
MC1.2
2. Did the student
correctly identify
complicating
factors or
comorbidities and
(if applicable) need
for follow up?
Yes the
student was
able to
correctly
identify
complicating
factors and
follow up as
indicated with
patient care
handouts.
Complicating
factors not
identified
correctly or
missing
appropriate
follow up or
handouts.
No, missed
major
complicatin
g factors or
need for
follow up.
MC1.2
MC3.1
What is/are the area/areas of complaint? (Indicate with an “X” )
☐Head ☐Cervical ☐Thoracic/Rib ☐Lumbar
☐Pelvic/Sacroiliac ☐Upper Extremity ☐Lower Extremity ☐General
NMSK
☐Clearance ☐Wellness ☐Nutrition
Does the patient present with a similar complaint as the previous visit(s) or is it a
different complaint?
☐Similar Complaint ☐Different Complaint
See the student version of the SOAP note in the EHR for the complete history and
examination procedures selected for this patient.
Is the diagnosis the same?
☐Yes Enter Diagnosis Click here to enter text.
☐No List new diagnosis(es) below:
Documentation of a list of Differential Dx. including any organic, visceral or systemic
issues that were considered and why they were eliminated.
List in order from most likely to least likely. (This should not be the ICD 10 code)
1. Click here to enter text.
1. Click here to enter text.
1. Click here to enter text.
1. Click here to enter text.
Performance of the
Established Patient Visit
Bobak/Maneri 2017 20

21. Longitudinal Analysis
StudentID MC1.1 Documentation of a List of Differential diagnosis(es) % PointsMC1.2 Determination of Physical Findings % PointsMC1.3 Generation of a problem list % Points
# Assessments 3 2 3
# Items 4 3 5
Group Average 90.37% 87.54% 89.88%
4760 100% 100% 100%
6567 87.20% 83.45% 79.83%
6581 87.07% 100% 89.92%
6765 87.20% 66.72% 79.62%
6832 100% 100% 100%
6880 100% 100% 79.83%
6942 74.27% 83.45% 100%
6986 87.20% 66.72% 89.81%
Summary score of all assessments
Set at 70% for each MC for an individual student
Bobak/Maneri 2017
21

22. Bobak/Maneri 2017
22

23. The Process Continues
8. Determine who will have input into data analysis
9. Determine how and how often data will be shared with different
stakeholder groups
10. Document any changes made based on the analysis of student
performance data
11. Verify if changes made have resulted in improvement
Bobak/Maneri 2017 23

24. We were able to identify student issues by trimester using data from
each criteria:
-Appropriate levels of clinical skills
-We were also able to identify challenges with faculty grading
-We were able to develop a process to remediate students in need of support
-We were able to show that every student was able to meet every meta-competency
outcome
What we learned…
Bobak/Maneri 2017
24

25. What we learned…
Bobak/Maneri 2017
25
• A data review cycle should engage multiple perspectives and reinforce the importance of
cumulative assessment of student learning in alignment with program goals
• Studying the longitudinal assessment results makes gaps in the curriculum more
transparent and the identification more precise
• Cyclic data analysis provides the program the opportunity to monitor changes made and
determine if changes have resulted in improvements
We were able to identify program issues by engaging multiple stakeholders
in data review

26. Action Steps-Closing the Loop
• Students
– Development of a specific remediation process
– Evidence of meeting criteria
• Faculty
– Integral in the student remediation process
– Discussion in clinic meetings
– Academic Chart review
• Program
– Engage all aspects of the curriculum in regular data review
– Improve communication in support of program goals
– Improved ability to analyze program gaps
Cyclic process allows for modification and fluidity
Bobak/Maneri 2017
26

27. Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation
NBCE Licensure Exams, CCE Meta-competencies, NYCC Signature Competencies
Bobak/Maneri 2017
27

28. Living in a Constant State of Assessment
• Collecting data
• Managing data
• Analyzing data
• Communicating findings
• Making changes
• Collecting new data…..
The cycle must be continuous and transparent to support program success
Bobak/Maneri 2017 28

29. Karen Bobak, DC, EdD
kbobak@nycc.edu
Wendy Maneri, DC, MS
wmaneri@nycc.edu
Questions?
Thank You! New York Chiropractic College
2360 State Route 89
Seneca Falls, N.Y. 13148
Bobak/Maneri 2017 29