SlideShare a Scribd company logo
Dr. Ara Tekian
University of Illinois
Plenary II: The Process of
Blueprinting National Examinations
The Process of Blueprinting
National Examinations
Ara Tekian, PhD, MHPE
University of Illinois at Chicago
tekian@uic.edu
TOUCHSTONE Institute – Annual Symposium
January 22, 2018, Toronto
Outline
•Process of developing blueprints
•Experience of the National Board of the Medical
Examiners (NBME) in Blueprinting
• Two committees of content experts
• Five step process
•American Board of Internal Medicine (ABIM)
• Blueprint for MOC exam
• Sample content distribution
• Detailed blueprint with levels of importance
The Blueprint
•A document that describes the CONTENT to be
included in an examination.
• Indicates the proportion of questions
• General and specific content areas
•It serves as a guide for the design of the
examination and for developing a pool of potential
test questions or items.
Components of a Faculty Development Program
in Assessment
(Chapter 16)
1. Basic principles
2. Methods and their alignment with
competencies
3. Blueprinting and test construction
4. Assessor training
5. Scoring and standard setting
System History Explanation Examination Procedures
Cardiovascular Chest pain Discharge drugs Cardiac Blood pressure
management
Respiratory Hemoptysis Respiratory Peak flow
Gastro-intestinal Abdominal pain Gastroscopy Abdominal Rectal examination
Reproductive Amenorrhea Abnormal smear Cervical smears
Nervous Headache Eyes Ophthalmology
Musculo-skeletal Backache Hip
Generic Pre-op
assessment
Consent for post-
mortem
IV cannulation
Example of an Objective Structured
Clinical Examination (OSCE) Blueprint
A Sample Blueprint
for National Examination
Experience of NBME in
Blueprinting*
Performing a job or task analysis is expensive and time
consuming
Therefore -
• If content experts can identify the tasks involved in
performing a job
• They should be able to identify what knowledge one must
have in order to perform that job.
• They should be able to identify the knowledge base that
needed to be tested in order to show competency in
performing the job.
*NBME Blueprinting, 2016
Acceptance of the rationale
•Asking content experts:
• What should be on the test?
• How much of it?
•Immediate appeal for the following reasons:
• addressed the blueprint question
• concerned only with concepts that could be measured by
MCQs
• relatively simple, practical, and cost-effective
Before beginning to develop a
content outline
•Establish the primary purpose of the examination
and the inferences to be made about examinees
from their scores
•Ensure that the content scope and breadth
represented in the blueprint support appropriate
score interpretations
• Common inference:
Examinees who pass a certification examination should have
demonstrated adequate content mastery to practice in a
specialty or subspecialty area.
Main questions in developing
a blueprint
•What areas of knowledge should be included in the
test?
•How much of each area should be included?
Note: While a practice analysis study can tell us what
the target examinees do on the job, what actions
they perform, and what decisions they must make, it
does not directly address the question of what should
be on the test.
Recruitment of two committees of
content experts
•The Advisory Committee will develop the
prototype, evaluate comments, and finalize the
blueprint.
• academic and industry leaders in the subject area
• approximately 7 members
•Subject matter experts will review the prototype
draft and suggest specific content areas to add,
remove, or expand.
• provide overall comments on the breadth, depth, and
balance of the content.
• 30 to 50 experts
Expertise of the two committees
•Advisory Committee
•Subject matter experts
•Experienced practitioners who are familiar with the
knowledge, skills, and abilities needed by the target
users (examinees / practitioners / educators /
influential individuals in the field).
5 Step Process
•1. Information gathering
•2. Initial meeting and draft development
•3. Confirmation and review
•4. Data gathering and compilation
•5. Finalizing the blueprint
5 Steps Process
1. Information gathering
•Compile background materials:
• Teaching curricula
• Job analyses
• Relevant articles
• Training manuals
• Milestones and required competencies
• Blueprints for similar existing exams
• Any professional standards
Note: this should be a collaborative process between the
client and the “national board.”
2. Initial meeting and draft
development
•The advisory committee convenes at a one-day
meeting to go over the background materials and
produce a draft of the content areas and relative
weights that will make up the test blueprint
(content specifications).
•The activities of the meeting are:
• Presentation/orientation by the psychometrician,
explaining the tasks of the day.
• Review and discussion of the background materials by the
advisory committee.
2. Initial meeting and draft
development (cont.)
•The activities of the meeting are (cont.):
• Discussion and definition of the major areas of content to
be included in the examination. (Test Development staff ensure
the resulting draft blueprint meets their required guidelines.)
• Definition of specific sub-areas within each major area on
which questions should be written.
• Assignment of weights to each major and sub-area in
terms of percentage and number of items to be included
in the test from that area (more important areas will have
higher percentages of items).
• Review and finalization of the draft of the blueprint.
3. Confirmation and review
•Send the blueprint draft assembled by the advisory
committee to a group of subject matter experts for
suggestions, comments, critiques, alternatives, and
revisions of points and weights.
4. Data gathering and compilation
•Consolidate the results of the survey into a single
document and send to the advisory committee for
review.
5. Finalizing the blueprint
•The advisory committee discusses the survey
results and finalizes the blueprint (via a series of
conference calls or other means.)
•This blueprint will then be used to analyze any
existing item pools and to establish areas where
items need to be developed, as well as guide the
assembly of the examination.
Practice analyses
(Two parts)
•An initial data generation phase with a focused
group of experts (e.g., blueprint development
process utilizing an advisory committee)
•A confirmation phase to validate the information by
a larger group of stakeholders.
Note: This validation phase typically incorporates a
survey sent out to identified individuals.
Necessity of the confirmation
phase
•A small group of experts may be biased
•The initial data generation phase is based on human
judgment rather than objective reality, and is
therefore not immune to the influence of raters
who may be subject to numerous sources of
inaccuracy
Note: Most practice analyses include this subsequent
questionnaire step to have a larger group of outside experts
review the materials created by the initial advisory committee
during the data generation phase.
Survey
• Members of the expert survey group are selected
by the advisory committee as a broader coalition of
stakeholders with the necessary expertise to
provide input into the blueprint.
•The expert survey group also serves as a vehicle to
create buy-in from the larger professional
community.
•The survey itself provides the proposed blueprint
along with instructions to the group to provide
input. It should be stressed that the blueprint needs
to reflect the content intended to be tested.
American Board of Internal Medicine (ABIM)*
Maintenance of Certification (MOC) Exam Blueprint
• In 2015, ABIM invited certified general internists to provide
ratings of the relative frequency and importance of blueprint
topics in practice.
• A sample of over 300 physicians, similar to the total invited
population of internists in age, time spent in direct patient
care, and practice setting, provided the blueprint topic
ratings.
• ABIM considered the average respondent ratings of topic
frequency and importance in each of the content categories.
• A second source of information was the relative frequency of
patient conditions in the content categories, as seen by
certified Internists and documented by national health care
data.
*ABIM: MOC Exam Blueprint, 2015
Purpose of the
Internal Medicine MOC exam
•Evaluate whether a certified internist has
maintained competence and currency in the
knowledge and judgment required for practice.
•The exam emphasizes diagnosis and management
of prevalent conditions, particularly in areas where
practice has changed in recent years.
Exam format
•The exam is composed of 240 single-best-answer
MCQs, of which 40 are new questions that do not
count in the examinee’s score.
•All questions describe clinical scenarios and ask
about the work done by physicians in the course of
practice:
• Diagnosis
• Testing
• Treatment/ Care Decisions
• Risk Assessment/ Prognosis/ Epidemiology
• Pathophysiology/ Basic Science
How the blueprint ratings are used
to assemble the MOC exam
• Blueprint reviewers provided ratings of relative
frequency in practice for each of the detailed content
topics in the blueprint and provided ratings of the
relative importance of the topics for each of the tasks
described.
• In rating importance, reviewers are asked to consider
factors such as the following:
• High risk of a significant adverse outcome
• Cost of care and stewardship of resources
• Common errors in diagnosis or management
• Effect on population health
• Effect on quality of life
Frequency and Importance
• Rated on a three-point scale corresponding to:
•low, medium, or high
• parameters for selecting MOC exam questions:
• At least 75% of exam questions will address high-importance
content (indicated in green)
• No more than 25% of exam questions will address medium-
importance content (indicated in yellow)
• No exam questions will address low-importance content
(indicated in red)
• Independent of the importance and task ratings, no
more than 18% of exam questions will address low-
frequency content (indicated by “LF” following the topic
description).
In Conclusion
• Blueprinting of a national examination is extremely
important
• It might involve up to 5 steps and 2 committees
(Advisory and Subject matter experts)
• Practice analyses includes two parts: an initial data
generation phase, and a confirmation phase (to validate
the information (by survey)
• For the initial data generation phase, patient issues and
problems could be utilized from national databases
(e.g., Medicare, ambulatory care database) or any
billing database
• Ratings for relative frequency and relative importance
are essential considerations
Bibliography
• Connell, A. F., & Nord, W. R. (1996). The bloodless coup: The infiltration of organization
science by uncertainty and values. Journal of Applied Behavioral Science, 32, 407-427.
• Goldstein, I. L., Zedeck, S. & Schneider, B. (1993). An exploration of the job analysis-content
validity process. In: N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations.
San Francisco, CA: Jossey-Bass.
• Knapp, J. E., & Knapp, L. G. (1995). Practice analysis: building the foundation for validity. In:
J. C. Impara, (Ed.), Buros-Nebraska series on measurement and testing. Lincoln, NE: Buros
Institute of Mental Measurements, 93-116.
• Morgeson, F. P, & Campion, M. A. (2004). A framework of potential sources of inaccuracy in
job analysis. In: M. Wilson, R. Harvey, G. Alliger, &W. Bennett (Eds.), The handbook of work
analysis: The methods, systems, applications, and science of work measurement in
organizations. New York: Psychology Press/Taylor and Francis Group.
• Newman, L. S., Slaughter, R. C., & Taranath, S. N. (1999, April). The selection and use of
rating scales in task surveys: A review of current job analysis practice. Paper presented at
the meeting of the National Council on Measurement in Education, Montreal, Canada.
• Raymond, M. R. (2002). A practical guide to practice analysis for credentialing
examinations. Educational Measurement: Issues and Practice, 21, 25-37.
2018 Perspectives Symposium
January 22, 2018
#TSINPerspectives2018

More Related Content

What's hot

03 salusky ctsi kl2 workshop july 2012
03 salusky ctsi kl2 workshop july 201203 salusky ctsi kl2 workshop july 2012
03 salusky ctsi kl2 workshop july 2012
UCLA CTSI
 
The Evaluation Checklist
The Evaluation ChecklistThe Evaluation Checklist
The Evaluation Checklist
wmartz
 
Delphi in community assessment na
Delphi in community assessment naDelphi in community assessment na
Delphi in community assessment na
Hibsah Ridwan
 
Monitoring Plan Template
Monitoring Plan TemplateMonitoring Plan Template
Monitoring Plan Template
www.datatrak.com
 
Meta Evaluation
Meta EvaluationMeta Evaluation
Meta Evaluation
Mohsen Sharifirad
 
Meta Evaluation and Evaluation Dissemination(Manual 3)
Meta Evaluation and Evaluation Dissemination(Manual 3)Meta Evaluation and Evaluation Dissemination(Manual 3)
Meta Evaluation and Evaluation Dissemination(Manual 3)
Dawit Wolde
 
Validation and moderation_-_guide_for_developing_assessment_tools
Validation and moderation_-_guide_for_developing_assessment_toolsValidation and moderation_-_guide_for_developing_assessment_tools
Validation and moderation_-_guide_for_developing_assessment_tools
Kaztiania Andig
 
Clinical auditing in pharmacology
Clinical auditing  in pharmacologyClinical auditing  in pharmacology
Clinical auditing in pharmacology
Dr. Rupendra Bharti
 
Enhancing quality of research 4 development: Initial ideas for design and imp...
Enhancing quality of research 4 development: Initial ideas for design and imp...Enhancing quality of research 4 development: Initial ideas for design and imp...
Enhancing quality of research 4 development: Initial ideas for design and imp...
ILRI
 
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
CADTH Symposium
 
Development of Cochrane Response part 1
Development of Cochrane Response part 1Development of Cochrane Response part 1
Development of Cochrane Response part 1
Juliane Ried
 
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledkyBR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
MEYS, MŠMT in Czech
 
Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
Expertise, Consumer-Oriented, and Program-Oriented Evaluation ApproachesExpertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
dctrcurry
 
Stakeholder Involvement in KCE Processes
Stakeholder Involvement in KCE ProcessesStakeholder Involvement in KCE Processes
Stakeholder Involvement in KCE Processes
Patrice Chalon
 
Navigating the K Award Process CTSI K Award Workshop
Navigating the K Award Process CTSI K Award WorkshopNavigating the K Award Process CTSI K Award Workshop
Navigating the K Award Process CTSI K Award Workshop
UCLA CTSI
 
Management oriented evaluation approaches
Management oriented evaluation approachesManagement oriented evaluation approaches
Management oriented evaluation approaches
Jessica Bernardino
 
Validation and moderation workshop Session 1
Validation and moderation workshop Session 1Validation and moderation workshop Session 1
Validation and moderation workshop Session 1
SarinaRussoInstitute
 
NAAF Patient-Reported Outcomes Consortium
NAAF Patient-Reported Outcomes ConsortiumNAAF Patient-Reported Outcomes Consortium
NAAF Patient-Reported Outcomes Consortium
National Alopecia Areata Foundation
 
NR 505 Invent Yourself/newtonhelp.com
NR 505 Invent Yourself/newtonhelp.comNR 505 Invent Yourself/newtonhelp.com
NR 505 Invent Yourself/newtonhelp.com
lechenau109
 
Assessment Validation
Assessment ValidationAssessment Validation
Assessment Validation
alicanc
 

What's hot (20)

03 salusky ctsi kl2 workshop july 2012
03 salusky ctsi kl2 workshop july 201203 salusky ctsi kl2 workshop july 2012
03 salusky ctsi kl2 workshop july 2012
 
The Evaluation Checklist
The Evaluation ChecklistThe Evaluation Checklist
The Evaluation Checklist
 
Delphi in community assessment na
Delphi in community assessment naDelphi in community assessment na
Delphi in community assessment na
 
Monitoring Plan Template
Monitoring Plan TemplateMonitoring Plan Template
Monitoring Plan Template
 
Meta Evaluation
Meta EvaluationMeta Evaluation
Meta Evaluation
 
Meta Evaluation and Evaluation Dissemination(Manual 3)
Meta Evaluation and Evaluation Dissemination(Manual 3)Meta Evaluation and Evaluation Dissemination(Manual 3)
Meta Evaluation and Evaluation Dissemination(Manual 3)
 
Validation and moderation_-_guide_for_developing_assessment_tools
Validation and moderation_-_guide_for_developing_assessment_toolsValidation and moderation_-_guide_for_developing_assessment_tools
Validation and moderation_-_guide_for_developing_assessment_tools
 
Clinical auditing in pharmacology
Clinical auditing  in pharmacologyClinical auditing  in pharmacology
Clinical auditing in pharmacology
 
Enhancing quality of research 4 development: Initial ideas for design and imp...
Enhancing quality of research 4 development: Initial ideas for design and imp...Enhancing quality of research 4 development: Initial ideas for design and imp...
Enhancing quality of research 4 development: Initial ideas for design and imp...
 
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
Cadth 2015 c1 chelak cadth tr_ process_cadth symposium_2015_finalapril13
 
Development of Cochrane Response part 1
Development of Cochrane Response part 1Development of Cochrane Response part 1
Development of Cochrane Response part 1
 
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledkyBR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
BR 10 / Malé pilotní hodnocení: zpětná vazba a výsledky
 
Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
Expertise, Consumer-Oriented, and Program-Oriented Evaluation ApproachesExpertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
Expertise, Consumer-Oriented, and Program-Oriented Evaluation Approaches
 
Stakeholder Involvement in KCE Processes
Stakeholder Involvement in KCE ProcessesStakeholder Involvement in KCE Processes
Stakeholder Involvement in KCE Processes
 
Navigating the K Award Process CTSI K Award Workshop
Navigating the K Award Process CTSI K Award WorkshopNavigating the K Award Process CTSI K Award Workshop
Navigating the K Award Process CTSI K Award Workshop
 
Management oriented evaluation approaches
Management oriented evaluation approachesManagement oriented evaluation approaches
Management oriented evaluation approaches
 
Validation and moderation workshop Session 1
Validation and moderation workshop Session 1Validation and moderation workshop Session 1
Validation and moderation workshop Session 1
 
NAAF Patient-Reported Outcomes Consortium
NAAF Patient-Reported Outcomes ConsortiumNAAF Patient-Reported Outcomes Consortium
NAAF Patient-Reported Outcomes Consortium
 
NR 505 Invent Yourself/newtonhelp.com
NR 505 Invent Yourself/newtonhelp.comNR 505 Invent Yourself/newtonhelp.com
NR 505 Invent Yourself/newtonhelp.com
 
Assessment Validation
Assessment ValidationAssessment Validation
Assessment Validation
 

Similar to Perspectives 2018: Ara Tekian

Flexible Evaluation
Flexible EvaluationFlexible Evaluation
Flexible Evaluation
pasicUganda
 
Moderation workshop dietetics april 2017b
Moderation workshop dietetics april 2017bModeration workshop dietetics april 2017b
Moderation workshop dietetics april 2017b
Rita Ndagire Kizito
 
Evaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation MethodsEvaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation Methods
Debbie_at_IDS
 
Clinical protocol
Clinical protocolClinical protocol
Clinical protocol
Sreenivasa Reddy Thalla
 
Test construction
Test constructionTest construction
Test construction
munsif123
 
Project Formulation and Management - Project Scope Management
Project Formulation and Management - Project Scope ManagementProject Formulation and Management - Project Scope Management
Project Formulation and Management - Project Scope Management
Hrishikesh Satpute
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...
MedCouncilCan
 
Qi toolkit oct 2020
Qi toolkit oct 2020 Qi toolkit oct 2020
Qi toolkit oct 2020
JosephCope3
 
Feedback on the draft summary report
Feedback on the draft summary reportFeedback on the draft summary report
Feedback on the draft summary report
MEYS, MŠMT in Czech
 
Systematic Literature Review
Systematic Literature ReviewSystematic Literature Review
Systematic Literature Review
Tiago Silva da Silva
 
Quality Standards ERS 2017
Quality Standards ERS 2017Quality Standards ERS 2017
Quality Standards ERS 2017
Kathryn Brown
 
Philippines Field Management Training Program curriculum outline
Philippines Field Management Training Program curriculum outlinePhilippines Field Management Training Program curriculum outline
Philippines Field Management Training Program curriculum outline
SMDPalums
 
PDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation PlanPDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation Plan
kpravera
 
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
eaquals
 
Btec centre guide_to_internal_verification
Btec centre guide_to_internal_verificationBtec centre guide_to_internal_verification
Btec centre guide_to_internal_verification
Ibrahim Khleifat
 
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J SheldonMc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
Jeffrey Andrew Sheldon, M.A., Ed.M.
 
TIARA Module 2 Anne Sales Theory & Approaches 06192019
TIARA Module 2 Anne Sales Theory & Approaches 06192019TIARA Module 2 Anne Sales Theory & Approaches 06192019
TIARA Module 2 Anne Sales Theory & Approaches 06192019
Stacy Farr, PhD, MPH
 
Bioscience Laboratory Workforce Skills - part II
Bioscience Laboratory Workforce Skills - part IIBioscience Laboratory Workforce Skills - part II
Bioscience Laboratory Workforce Skills - part II
bio-link
 
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
shayamiticharles
 
Project auditing
Project auditingProject auditing
Project auditing
Libra chudry
 

Similar to Perspectives 2018: Ara Tekian (20)

Flexible Evaluation
Flexible EvaluationFlexible Evaluation
Flexible Evaluation
 
Moderation workshop dietetics april 2017b
Moderation workshop dietetics april 2017bModeration workshop dietetics april 2017b
Moderation workshop dietetics april 2017b
 
Evaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation MethodsEvaluability Assessments and Choice of Evaluation Methods
Evaluability Assessments and Choice of Evaluation Methods
 
Clinical protocol
Clinical protocolClinical protocol
Clinical protocol
 
Test construction
Test constructionTest construction
Test construction
 
Project Formulation and Management - Project Scope Management
Project Formulation and Management - Project Scope ManagementProject Formulation and Management - Project Scope Management
Project Formulation and Management - Project Scope Management
 
Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...Peeking behind the test: insights and innovations from the Medical Council of...
Peeking behind the test: insights and innovations from the Medical Council of...
 
Qi toolkit oct 2020
Qi toolkit oct 2020 Qi toolkit oct 2020
Qi toolkit oct 2020
 
Feedback on the draft summary report
Feedback on the draft summary reportFeedback on the draft summary report
Feedback on the draft summary report
 
Systematic Literature Review
Systematic Literature ReviewSystematic Literature Review
Systematic Literature Review
 
Quality Standards ERS 2017
Quality Standards ERS 2017Quality Standards ERS 2017
Quality Standards ERS 2017
 
Philippines Field Management Training Program curriculum outline
Philippines Field Management Training Program curriculum outlinePhilippines Field Management Training Program curriculum outline
Philippines Field Management Training Program curriculum outline
 
PDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation PlanPDE Week 5: Developing an Evaluation Plan
PDE Week 5: Developing an Evaluation Plan
 
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
Ludka Kotarska Mid-term self-assessment revisited - making the most of the Ea...
 
Btec centre guide_to_internal_verification
Btec centre guide_to_internal_verificationBtec centre guide_to_internal_verification
Btec centre guide_to_internal_verification
 
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J SheldonMc Cord Hosptial Final Products Report August 2005 J Sheldon
Mc Cord Hosptial Final Products Report August 2005 J Sheldon
 
TIARA Module 2 Anne Sales Theory & Approaches 06192019
TIARA Module 2 Anne Sales Theory & Approaches 06192019TIARA Module 2 Anne Sales Theory & Approaches 06192019
TIARA Module 2 Anne Sales Theory & Approaches 06192019
 
Bioscience Laboratory Workforce Skills - part II
Bioscience Laboratory Workforce Skills - part IIBioscience Laboratory Workforce Skills - part II
Bioscience Laboratory Workforce Skills - part II
 
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
2014_10_17_HowtoWriteanEvaluationPlanSlides_ORE.ppt
 
Project auditing
Project auditingProject auditing
Project auditing
 

More from Touchstone Institute

Symposium 2020: Sandra Monteiro Implications of Standard Setting
Symposium 2020: Sandra Monteiro Implications of Standard Setting Symposium 2020: Sandra Monteiro Implications of Standard Setting
Symposium 2020: Sandra Monteiro Implications of Standard Setting
Touchstone Institute
 
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
Touchstone Institute
 
Symposium 2020: Mark Reckase - Translation of Policy to Cut Score
Symposium 2020: Mark Reckase - Translation of Policy to Cut ScoreSymposium 2020: Mark Reckase - Translation of Policy to Cut Score
Symposium 2020: Mark Reckase - Translation of Policy to Cut Score
Touchstone Institute
 
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
Touchstone Institute
 
Perspectives 2019: Taivi Lobu
Perspectives 2019: Taivi LobuPerspectives 2019: Taivi Lobu
Perspectives 2019: Taivi Lobu
Touchstone Institute
 
Perspectives 2019: Leela Persaud
Perspectives 2019: Leela PersaudPerspectives 2019: Leela Persaud
Perspectives 2019: Leela Persaud
Touchstone Institute
 
Perspectives 2019: Doris Grinspun
Perspectives 2019: Doris GrinspunPerspectives 2019: Doris Grinspun
Perspectives 2019: Doris Grinspun
Touchstone Institute
 
Perspectives 2019: Sylvia Langlois
Perspectives 2019: Sylvia LangloisPerspectives 2019: Sylvia Langlois
Perspectives 2019: Sylvia Langlois
Touchstone Institute
 
Perspectives 2019: Meredith Vanstone
Perspectives 2019: Meredith VanstonePerspectives 2019: Meredith Vanstone
Perspectives 2019: Meredith Vanstone
Touchstone Institute
 
Perspectives 2019: Erika Abner
Perspectives 2019: Erika AbnerPerspectives 2019: Erika Abner
Perspectives 2019: Erika Abner
Touchstone Institute
 
Perspectives 2019: Adrian Frisina
Perspectives 2019: Adrian FrisinaPerspectives 2019: Adrian Frisina
Perspectives 2019: Adrian Frisina
Touchstone Institute
 
Perspectives 2018: Debra Sibbald & Tammie McParland
Perspectives 2018: Debra Sibbald & Tammie McParlandPerspectives 2018: Debra Sibbald & Tammie McParland
Perspectives 2018: Debra Sibbald & Tammie McParland
Touchstone Institute
 
Perspectives 2018: Karen Kirkham
Perspectives 2018: Karen KirkhamPerspectives 2018: Karen Kirkham
Perspectives 2018: Karen Kirkham
Touchstone Institute
 
Perspectives 2018: Marie-Claude Léveillé
Perspectives 2018: Marie-Claude LéveilléPerspectives 2018: Marie-Claude Léveillé
Perspectives 2018: Marie-Claude Léveillé
Touchstone Institute
 
Perspectives 2018: Brian Hodges
Perspectives 2018: Brian HodgesPerspectives 2018: Brian Hodges
Perspectives 2018: Brian Hodges
Touchstone Institute
 
Perspectives 2017: Jenna Bright
Perspectives 2017: Jenna BrightPerspectives 2017: Jenna Bright
Perspectives 2017: Jenna Bright
Touchstone Institute
 
Perspectives 2017: Sue VanDeVelde-Coke
Perspectives 2017: Sue VanDeVelde-CokePerspectives 2017: Sue VanDeVelde-Coke
Perspectives 2017: Sue VanDeVelde-Coke
Touchstone Institute
 
Perspectives 2017: Shafi Bhuiyan
Perspectives 2017: Shafi BhuiyanPerspectives 2017: Shafi Bhuiyan
Perspectives 2017: Shafi Bhuiyan
Touchstone Institute
 
Perspectives 2017: Inge Schabort
Perspectives 2017: Inge SchabortPerspectives 2017: Inge Schabort
Perspectives 2017: Inge Schabort
Touchstone Institute
 
Perspectives 2017: Ivy Lynn Bourgeault
Perspectives 2017: Ivy Lynn BourgeaultPerspectives 2017: Ivy Lynn Bourgeault
Perspectives 2017: Ivy Lynn Bourgeault
Touchstone Institute
 

More from Touchstone Institute (20)

Symposium 2020: Sandra Monteiro Implications of Standard Setting
Symposium 2020: Sandra Monteiro Implications of Standard Setting Symposium 2020: Sandra Monteiro Implications of Standard Setting
Symposium 2020: Sandra Monteiro Implications of Standard Setting
 
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
Perspectives 2020: Richard Tannenbaum - Interaction of Policy and Methodology...
 
Symposium 2020: Mark Reckase - Translation of Policy to Cut Score
Symposium 2020: Mark Reckase - Translation of Policy to Cut ScoreSymposium 2020: Mark Reckase - Translation of Policy to Cut Score
Symposium 2020: Mark Reckase - Translation of Policy to Cut Score
 
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
Perspectives 2020: Jason Frank- Setting Standards that Reflect the Profession...
 
Perspectives 2019: Taivi Lobu
Perspectives 2019: Taivi LobuPerspectives 2019: Taivi Lobu
Perspectives 2019: Taivi Lobu
 
Perspectives 2019: Leela Persaud
Perspectives 2019: Leela PersaudPerspectives 2019: Leela Persaud
Perspectives 2019: Leela Persaud
 
Perspectives 2019: Doris Grinspun
Perspectives 2019: Doris GrinspunPerspectives 2019: Doris Grinspun
Perspectives 2019: Doris Grinspun
 
Perspectives 2019: Sylvia Langlois
Perspectives 2019: Sylvia LangloisPerspectives 2019: Sylvia Langlois
Perspectives 2019: Sylvia Langlois
 
Perspectives 2019: Meredith Vanstone
Perspectives 2019: Meredith VanstonePerspectives 2019: Meredith Vanstone
Perspectives 2019: Meredith Vanstone
 
Perspectives 2019: Erika Abner
Perspectives 2019: Erika AbnerPerspectives 2019: Erika Abner
Perspectives 2019: Erika Abner
 
Perspectives 2019: Adrian Frisina
Perspectives 2019: Adrian FrisinaPerspectives 2019: Adrian Frisina
Perspectives 2019: Adrian Frisina
 
Perspectives 2018: Debra Sibbald & Tammie McParland
Perspectives 2018: Debra Sibbald & Tammie McParlandPerspectives 2018: Debra Sibbald & Tammie McParland
Perspectives 2018: Debra Sibbald & Tammie McParland
 
Perspectives 2018: Karen Kirkham
Perspectives 2018: Karen KirkhamPerspectives 2018: Karen Kirkham
Perspectives 2018: Karen Kirkham
 
Perspectives 2018: Marie-Claude Léveillé
Perspectives 2018: Marie-Claude LéveilléPerspectives 2018: Marie-Claude Léveillé
Perspectives 2018: Marie-Claude Léveillé
 
Perspectives 2018: Brian Hodges
Perspectives 2018: Brian HodgesPerspectives 2018: Brian Hodges
Perspectives 2018: Brian Hodges
 
Perspectives 2017: Jenna Bright
Perspectives 2017: Jenna BrightPerspectives 2017: Jenna Bright
Perspectives 2017: Jenna Bright
 
Perspectives 2017: Sue VanDeVelde-Coke
Perspectives 2017: Sue VanDeVelde-CokePerspectives 2017: Sue VanDeVelde-Coke
Perspectives 2017: Sue VanDeVelde-Coke
 
Perspectives 2017: Shafi Bhuiyan
Perspectives 2017: Shafi BhuiyanPerspectives 2017: Shafi Bhuiyan
Perspectives 2017: Shafi Bhuiyan
 
Perspectives 2017: Inge Schabort
Perspectives 2017: Inge SchabortPerspectives 2017: Inge Schabort
Perspectives 2017: Inge Schabort
 
Perspectives 2017: Ivy Lynn Bourgeault
Perspectives 2017: Ivy Lynn BourgeaultPerspectives 2017: Ivy Lynn Bourgeault
Perspectives 2017: Ivy Lynn Bourgeault
 

Recently uploaded

NARCOTICS- POLICY AND PROCEDURES FOR ITS USE
NARCOTICS- POLICY AND PROCEDURES FOR ITS USENARCOTICS- POLICY AND PROCEDURES FOR ITS USE
NARCOTICS- POLICY AND PROCEDURES FOR ITS USE
Dr. Ahana Haroon
 
Nano-gold for Cancer Therapy chemistry investigatory project
Nano-gold for Cancer Therapy chemistry investigatory projectNano-gold for Cancer Therapy chemistry investigatory project
Nano-gold for Cancer Therapy chemistry investigatory project
SIVAVINAYAKPK
 
Recent advances on Cervical cancer .pptx
Recent advances on Cervical cancer .pptxRecent advances on Cervical cancer .pptx
Recent advances on Cervical cancer .pptx
DrGirishJHoogar
 
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotesPromoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
PsychoTech Services
 
Breast cancer: Post menopausal endocrine therapy
Breast cancer: Post menopausal endocrine therapyBreast cancer: Post menopausal endocrine therapy
Breast cancer: Post menopausal endocrine therapy
Dr. Sumit KUMAR
 
Medical Quiz ( Online Quiz for API Meet 2024 ).pdf
Medical Quiz ( Online Quiz for API Meet 2024 ).pdfMedical Quiz ( Online Quiz for API Meet 2024 ).pdf
Medical Quiz ( Online Quiz for API Meet 2024 ).pdf
Jim Jacob Roy
 
The Nervous and Chemical Regulation of Respiration
The Nervous and Chemical Regulation of RespirationThe Nervous and Chemical Regulation of Respiration
The Nervous and Chemical Regulation of Respiration
MedicoseAcademics
 
Case presentation On Urinary tract infection
Case presentation On Urinary tract infectionCase presentation On Urinary tract infection
Case presentation On Urinary tract infection
KarthyAks1
 
Pharmacology of 5-hydroxytryptamine and Antagonist
Pharmacology of 5-hydroxytryptamine and AntagonistPharmacology of 5-hydroxytryptamine and Antagonist
Pharmacology of 5-hydroxytryptamine and Antagonist
Dr. Nikhilkumar Sakle
 
Travel Clinic Cardiff: Health Advice for International Travelers
Travel Clinic Cardiff: Health Advice for International TravelersTravel Clinic Cardiff: Health Advice for International Travelers
Travel Clinic Cardiff: Health Advice for International Travelers
NX Healthcare
 
Tele Optometry (kunj'sppt) / Basics of tele optometry.
Tele Optometry (kunj'sppt) / Basics of tele optometry.Tele Optometry (kunj'sppt) / Basics of tele optometry.
Tele Optometry (kunj'sppt) / Basics of tele optometry.
Kunj Vihari
 
What are the different types of Dental implants.
What are the different types of Dental implants.What are the different types of Dental implants.
What are the different types of Dental implants.
Gokuldas Hospital
 
Know the difference between Endodontics and Orthodontics.
Know the difference between Endodontics and Orthodontics.Know the difference between Endodontics and Orthodontics.
Know the difference between Endodontics and Orthodontics.
Gokuldas Hospital
 
DECLARATION OF HELSINKI - History and principles
DECLARATION OF HELSINKI - History and principlesDECLARATION OF HELSINKI - History and principles
DECLARATION OF HELSINKI - History and principles
anaghabharat01
 
What is Obesity? How to overcome Obesity?
What is Obesity? How to overcome Obesity?What is Obesity? How to overcome Obesity?
What is Obesity? How to overcome Obesity?
Healthmedsrx.com
 
Pollen and Fungal allergy: aeroallergy.pdf
Pollen and Fungal allergy: aeroallergy.pdfPollen and Fungal allergy: aeroallergy.pdf
Pollen and Fungal allergy: aeroallergy.pdf
Chulalongkorn Allergy and Clinical Immunology Research Group
 
How to Control Your Asthma Tips by gokuldas hospital.
How to Control Your Asthma Tips by gokuldas hospital.How to Control Your Asthma Tips by gokuldas hospital.
How to Control Your Asthma Tips by gokuldas hospital.
Gokuldas Hospital
 
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
shruti jagirdar
 
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
Université de Montréal
 
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
AyushGadhvi1
 

Recently uploaded (20)

NARCOTICS- POLICY AND PROCEDURES FOR ITS USE
NARCOTICS- POLICY AND PROCEDURES FOR ITS USENARCOTICS- POLICY AND PROCEDURES FOR ITS USE
NARCOTICS- POLICY AND PROCEDURES FOR ITS USE
 
Nano-gold for Cancer Therapy chemistry investigatory project
Nano-gold for Cancer Therapy chemistry investigatory projectNano-gold for Cancer Therapy chemistry investigatory project
Nano-gold for Cancer Therapy chemistry investigatory project
 
Recent advances on Cervical cancer .pptx
Recent advances on Cervical cancer .pptxRecent advances on Cervical cancer .pptx
Recent advances on Cervical cancer .pptx
 
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotesPromoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
Promoting Wellbeing - Applied Social Psychology - Psychology SuperNotes
 
Breast cancer: Post menopausal endocrine therapy
Breast cancer: Post menopausal endocrine therapyBreast cancer: Post menopausal endocrine therapy
Breast cancer: Post menopausal endocrine therapy
 
Medical Quiz ( Online Quiz for API Meet 2024 ).pdf
Medical Quiz ( Online Quiz for API Meet 2024 ).pdfMedical Quiz ( Online Quiz for API Meet 2024 ).pdf
Medical Quiz ( Online Quiz for API Meet 2024 ).pdf
 
The Nervous and Chemical Regulation of Respiration
The Nervous and Chemical Regulation of RespirationThe Nervous and Chemical Regulation of Respiration
The Nervous and Chemical Regulation of Respiration
 
Case presentation On Urinary tract infection
Case presentation On Urinary tract infectionCase presentation On Urinary tract infection
Case presentation On Urinary tract infection
 
Pharmacology of 5-hydroxytryptamine and Antagonist
Pharmacology of 5-hydroxytryptamine and AntagonistPharmacology of 5-hydroxytryptamine and Antagonist
Pharmacology of 5-hydroxytryptamine and Antagonist
 
Travel Clinic Cardiff: Health Advice for International Travelers
Travel Clinic Cardiff: Health Advice for International TravelersTravel Clinic Cardiff: Health Advice for International Travelers
Travel Clinic Cardiff: Health Advice for International Travelers
 
Tele Optometry (kunj'sppt) / Basics of tele optometry.
Tele Optometry (kunj'sppt) / Basics of tele optometry.Tele Optometry (kunj'sppt) / Basics of tele optometry.
Tele Optometry (kunj'sppt) / Basics of tele optometry.
 
What are the different types of Dental implants.
What are the different types of Dental implants.What are the different types of Dental implants.
What are the different types of Dental implants.
 
Know the difference between Endodontics and Orthodontics.
Know the difference between Endodontics and Orthodontics.Know the difference between Endodontics and Orthodontics.
Know the difference between Endodontics and Orthodontics.
 
DECLARATION OF HELSINKI - History and principles
DECLARATION OF HELSINKI - History and principlesDECLARATION OF HELSINKI - History and principles
DECLARATION OF HELSINKI - History and principles
 
What is Obesity? How to overcome Obesity?
What is Obesity? How to overcome Obesity?What is Obesity? How to overcome Obesity?
What is Obesity? How to overcome Obesity?
 
Pollen and Fungal allergy: aeroallergy.pdf
Pollen and Fungal allergy: aeroallergy.pdfPollen and Fungal allergy: aeroallergy.pdf
Pollen and Fungal allergy: aeroallergy.pdf
 
How to Control Your Asthma Tips by gokuldas hospital.
How to Control Your Asthma Tips by gokuldas hospital.How to Control Your Asthma Tips by gokuldas hospital.
How to Control Your Asthma Tips by gokuldas hospital.
 
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
STUDIES IN SUPPORT OF SPECIAL POPULATIONS: GERIATRICS E7
 
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
“Psychiatry and the Humanities”: An Innovative Course at the University of Mo...
 
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
Lecture 6 -- Memory 2015.pptlearning occurs when a stimulus (unconditioned st...
 

Perspectives 2018: Ara Tekian

  • 1. Dr. Ara Tekian University of Illinois Plenary II: The Process of Blueprinting National Examinations
  • 2. The Process of Blueprinting National Examinations Ara Tekian, PhD, MHPE University of Illinois at Chicago tekian@uic.edu TOUCHSTONE Institute – Annual Symposium January 22, 2018, Toronto
  • 3. Outline •Process of developing blueprints •Experience of the National Board of the Medical Examiners (NBME) in Blueprinting • Two committees of content experts • Five step process •American Board of Internal Medicine (ABIM) • Blueprint for MOC exam • Sample content distribution • Detailed blueprint with levels of importance
  • 4. The Blueprint •A document that describes the CONTENT to be included in an examination. • Indicates the proportion of questions • General and specific content areas •It serves as a guide for the design of the examination and for developing a pool of potential test questions or items.
  • 5.
  • 6. Components of a Faculty Development Program in Assessment (Chapter 16) 1. Basic principles 2. Methods and their alignment with competencies 3. Blueprinting and test construction 4. Assessor training 5. Scoring and standard setting
  • 7. System History Explanation Examination Procedures Cardiovascular Chest pain Discharge drugs Cardiac Blood pressure management Respiratory Hemoptysis Respiratory Peak flow Gastro-intestinal Abdominal pain Gastroscopy Abdominal Rectal examination Reproductive Amenorrhea Abnormal smear Cervical smears Nervous Headache Eyes Ophthalmology Musculo-skeletal Backache Hip Generic Pre-op assessment Consent for post- mortem IV cannulation Example of an Objective Structured Clinical Examination (OSCE) Blueprint
  • 8. A Sample Blueprint for National Examination
  • 9. Experience of NBME in Blueprinting* Performing a job or task analysis is expensive and time consuming Therefore - • If content experts can identify the tasks involved in performing a job • They should be able to identify what knowledge one must have in order to perform that job. • They should be able to identify the knowledge base that needed to be tested in order to show competency in performing the job. *NBME Blueprinting, 2016
  • 10. Acceptance of the rationale •Asking content experts: • What should be on the test? • How much of it? •Immediate appeal for the following reasons: • addressed the blueprint question • concerned only with concepts that could be measured by MCQs • relatively simple, practical, and cost-effective
  • 11. Before beginning to develop a content outline •Establish the primary purpose of the examination and the inferences to be made about examinees from their scores •Ensure that the content scope and breadth represented in the blueprint support appropriate score interpretations • Common inference: Examinees who pass a certification examination should have demonstrated adequate content mastery to practice in a specialty or subspecialty area.
  • 12. Main questions in developing a blueprint •What areas of knowledge should be included in the test? •How much of each area should be included? Note: While a practice analysis study can tell us what the target examinees do on the job, what actions they perform, and what decisions they must make, it does not directly address the question of what should be on the test.
  • 13. Recruitment of two committees of content experts •The Advisory Committee will develop the prototype, evaluate comments, and finalize the blueprint. • academic and industry leaders in the subject area • approximately 7 members •Subject matter experts will review the prototype draft and suggest specific content areas to add, remove, or expand. • provide overall comments on the breadth, depth, and balance of the content. • 30 to 50 experts
  • 14. Expertise of the two committees •Advisory Committee •Subject matter experts •Experienced practitioners who are familiar with the knowledge, skills, and abilities needed by the target users (examinees / practitioners / educators / influential individuals in the field).
  • 15. 5 Step Process •1. Information gathering •2. Initial meeting and draft development •3. Confirmation and review •4. Data gathering and compilation •5. Finalizing the blueprint
  • 16. 5 Steps Process 1. Information gathering •Compile background materials: • Teaching curricula • Job analyses • Relevant articles • Training manuals • Milestones and required competencies • Blueprints for similar existing exams • Any professional standards Note: this should be a collaborative process between the client and the “national board.”
  • 17. 2. Initial meeting and draft development •The advisory committee convenes at a one-day meeting to go over the background materials and produce a draft of the content areas and relative weights that will make up the test blueprint (content specifications). •The activities of the meeting are: • Presentation/orientation by the psychometrician, explaining the tasks of the day. • Review and discussion of the background materials by the advisory committee.
  • 18. 2. Initial meeting and draft development (cont.) •The activities of the meeting are (cont.): • Discussion and definition of the major areas of content to be included in the examination. (Test Development staff ensure the resulting draft blueprint meets their required guidelines.) • Definition of specific sub-areas within each major area on which questions should be written. • Assignment of weights to each major and sub-area in terms of percentage and number of items to be included in the test from that area (more important areas will have higher percentages of items). • Review and finalization of the draft of the blueprint.
  • 19. 3. Confirmation and review •Send the blueprint draft assembled by the advisory committee to a group of subject matter experts for suggestions, comments, critiques, alternatives, and revisions of points and weights.
  • 20. 4. Data gathering and compilation •Consolidate the results of the survey into a single document and send to the advisory committee for review.
  • 21. 5. Finalizing the blueprint •The advisory committee discusses the survey results and finalizes the blueprint (via a series of conference calls or other means.) •This blueprint will then be used to analyze any existing item pools and to establish areas where items need to be developed, as well as guide the assembly of the examination.
  • 22. Practice analyses (Two parts) •An initial data generation phase with a focused group of experts (e.g., blueprint development process utilizing an advisory committee) •A confirmation phase to validate the information by a larger group of stakeholders. Note: This validation phase typically incorporates a survey sent out to identified individuals.
  • 23. Necessity of the confirmation phase •A small group of experts may be biased •The initial data generation phase is based on human judgment rather than objective reality, and is therefore not immune to the influence of raters who may be subject to numerous sources of inaccuracy Note: Most practice analyses include this subsequent questionnaire step to have a larger group of outside experts review the materials created by the initial advisory committee during the data generation phase.
  • 24. Survey • Members of the expert survey group are selected by the advisory committee as a broader coalition of stakeholders with the necessary expertise to provide input into the blueprint. •The expert survey group also serves as a vehicle to create buy-in from the larger professional community. •The survey itself provides the proposed blueprint along with instructions to the group to provide input. It should be stressed that the blueprint needs to reflect the content intended to be tested.
  • 25. American Board of Internal Medicine (ABIM)* Maintenance of Certification (MOC) Exam Blueprint • In 2015, ABIM invited certified general internists to provide ratings of the relative frequency and importance of blueprint topics in practice. • A sample of over 300 physicians, similar to the total invited population of internists in age, time spent in direct patient care, and practice setting, provided the blueprint topic ratings. • ABIM considered the average respondent ratings of topic frequency and importance in each of the content categories. • A second source of information was the relative frequency of patient conditions in the content categories, as seen by certified Internists and documented by national health care data. *ABIM: MOC Exam Blueprint, 2015
  • 26. Purpose of the Internal Medicine MOC exam •Evaluate whether a certified internist has maintained competence and currency in the knowledge and judgment required for practice. •The exam emphasizes diagnosis and management of prevalent conditions, particularly in areas where practice has changed in recent years.
  • 27. Exam format •The exam is composed of 240 single-best-answer MCQs, of which 40 are new questions that do not count in the examinee’s score. •All questions describe clinical scenarios and ask about the work done by physicians in the course of practice: • Diagnosis • Testing • Treatment/ Care Decisions • Risk Assessment/ Prognosis/ Epidemiology • Pathophysiology/ Basic Science
  • 28.
  • 29. How the blueprint ratings are used to assemble the MOC exam • Blueprint reviewers provided ratings of relative frequency in practice for each of the detailed content topics in the blueprint and provided ratings of the relative importance of the topics for each of the tasks described. • In rating importance, reviewers are asked to consider factors such as the following: • High risk of a significant adverse outcome • Cost of care and stewardship of resources • Common errors in diagnosis or management • Effect on population health • Effect on quality of life
  • 30. Frequency and Importance • Rated on a three-point scale corresponding to: •low, medium, or high • parameters for selecting MOC exam questions: • At least 75% of exam questions will address high-importance content (indicated in green) • No more than 25% of exam questions will address medium- importance content (indicated in yellow) • No exam questions will address low-importance content (indicated in red) • Independent of the importance and task ratings, no more than 18% of exam questions will address low- frequency content (indicated by “LF” following the topic description).
  • 31.
  • 32.
  • 33. In Conclusion • Blueprinting of a national examination is extremely important • It might involve up to 5 steps and 2 committees (Advisory and Subject matter experts) • Practice analyses includes two parts: an initial data generation phase, and a confirmation phase (to validate the information (by survey) • For the initial data generation phase, patient issues and problems could be utilized from national databases (e.g., Medicare, ambulatory care database) or any billing database • Ratings for relative frequency and relative importance are essential considerations
  • 34. Bibliography • Connell, A. F., & Nord, W. R. (1996). The bloodless coup: The infiltration of organization science by uncertainty and values. Journal of Applied Behavioral Science, 32, 407-427. • Goldstein, I. L., Zedeck, S. & Schneider, B. (1993). An exploration of the job analysis-content validity process. In: N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations. San Francisco, CA: Jossey-Bass. • Knapp, J. E., & Knapp, L. G. (1995). Practice analysis: building the foundation for validity. In: J. C. Impara, (Ed.), Buros-Nebraska series on measurement and testing. Lincoln, NE: Buros Institute of Mental Measurements, 93-116. • Morgeson, F. P, & Campion, M. A. (2004). A framework of potential sources of inaccuracy in job analysis. In: M. Wilson, R. Harvey, G. Alliger, &W. Bennett (Eds.), The handbook of work analysis: The methods, systems, applications, and science of work measurement in organizations. New York: Psychology Press/Taylor and Francis Group. • Newman, L. S., Slaughter, R. C., & Taranath, S. N. (1999, April). The selection and use of rating scales in task surveys: A review of current job analysis practice. Paper presented at the meeting of the National Council on Measurement in Education, Montreal, Canada. • Raymond, M. R. (2002). A practical guide to practice analysis for credentialing examinations. Educational Measurement: Issues and Practice, 21, 25-37.
  • 35. 2018 Perspectives Symposium January 22, 2018 #TSINPerspectives2018