This document discusses how to create a valid certification examination through a process informed by psychometrics. It involves performing a job task analysis to determine the skills and knowledge to assess. Requirements are developed and published, then subject matter experts write exam items and a beta period tests them. An Angoff session sets the passing score. Ongoing maintenance through analysis ensures continued validity. Interested parties can get involved by assisting as subject matter experts or helping promote awareness of the certification.

1. BSD Certification:
How to Create a
Psychometrically Valid
Certification Examination
Dru Lavigne
Chair, BSD Certification Group Inc.
EuroBSDCon, 2011

2. Outline
What is psychometrics? (and why should I care?)
The Job Task Analysis (JTA)
Certification Requirements
Item Writing
Beta Period
Angoff Session
Exam Maintenance
Becoming Involved

3. Psychometrics
Science of assessment
Determines the skills and knowledge to be
assessed
Assesses understanding rather than
memorization
Increases the value of the assessment

4. Psychometrics
Value to:
Certificant: since the exam matches the
requirements, even if additional study aides are
not available, examinees can gauge their exam
preparedness by comparing their skillset to the
listed objectives
Employer: has a yardstick that validates the
minimum skillset of certified employees

5. Psychometrics
Value to:
Certifying Body: is in a position to successfully
defend the reliability of the assessment should an
examinee dispute the results of their examination
Community: the certificate is worth more than the
paper it is written on and certificants are not able
to pass the exam by memorizing the content
available from "exam cram" websites

6. Psychometrics
Signs of pyschometric validity:
Published exam requirements
Exam questions match the exam requirements
Assessment is repeatable and defendable
Exam does not try to “trick” the examinee or
introduce bias
Exam does not contain inaccuracies

7. JTA
First step in creating a new certification
Determines which tasks will be assessed
Goal is to define the content areas (domains) and
the importance and frequency of each task
Survey sample should be large and should
include working members of the target audience
and their employers

8. JTA
Results are analyzed and used to create the
Certification Requirements
BSDA: http://www.bsdcertification.org/
downloads/sr1_links.pdf
BSDP: http://www.bsdcertification.org/downloads/
delivery_survey.pdf

9. Certification Requirements
Also known as the exam blueprint
Contains the following information:
1. A clear definition of the intended audience
2. The knowledge domains and their percentages
(Table of Specifications)
3. Detailed exam objectives that clearly identify
the tasks to be assessed

10. Certification Requirements
4. The depth of knowledge to be assessed
according to Bloom's Taxonomy
(http://en.wikipedia.org/wiki/Blooms_taxonomy)
This document should be published and freely
available to examinees, employers, and the
creators of study aides
BSDA: http://www.bsdcertification.org/downloads/
pr_20051005_certreq_bsda_en_en.pdf
BSDP: http://www.bsdcertification.org/downloads/
certreq_bsdp_en.pdf

11. Item Writing
SMEs (Subject Matter Experts) use the blueprint
to create exam questions (items)
Item writing is an art and a skill
Items must be written to the skill level of intended
audience
Items must be clearly written and understandable,
even to non-native English speakers
An item can only assess one exam objective

12. Item Writing
Items must be consistent in punctuation, style,
and formatting to avoid distractions
Items can't contain cut/pastes from instructional
materials
Items shouldn't contain double negatives
Correct answers should not be in same position or
follow a pattern

13. Item Writing
Multiple choice item contains:
Stem: the question being asked
Key: the correct answer(s); appears as an option
Distractors: 3-4 incorrect answers which appear
as options; must be plausible and not contain
contextual cues

14. Item Writing
First draft of an item rarely looks like the final
version of question that appears on exam
Items undergo much scrutiny, rewriting, and
review by both the SMEs and the psychometrician
An exam form can not contain enemies (a
question that helps to answer another question)

15. Beta Period
Once the items have finished their review, the
psychometrician creates one or more exam forms
Goal of the beta period is to generate statistical
information on how examinees respond to the
exam forms and to identify anomalies
You can't pass or fail a beta as the passing (cut)
score can't be determined until the stastics of the
beta are analyzed, required questions are
rewritten or removed from the form, and the
Angoff session has occurred

16. Angoff Session
Goal of theAngoff session is to set the cut score
(passing score) by determining the number of
items a minimally competent examinee would
answer correctly
Each SME individually rates every item
Ratings are compared and any discrepancies are
discussed until concensus is achieved
Psychometrician uses final ratings to determine
exam's passing score; exam can now be
published

17. Exam Maintenance
Once exam is published, psychometrics does not
end
Statistics continue to be gathered for analysis on
how examinees respond to each exam item
Psychometrician will alert exam body of any
anomalies that may indicate cheating is occurring
Regular review period is scheduled (by time or
number of completed exams)

18. Exam Maintenance
For each established review period, the
psychometrician will identify which items should
be reviewed by the SMEs
Some items may be rewritten, removed, or
replaced, and pilot items may be introduced
New version of form may be published (usually
not advertised)
Period must be set for when to redo JTA and start
process all over again

19. Becoming Involved
Members of intended audience can act as SMEs
and participate in item writing, item review, and
Angoff sessions
Employers who include BSD certification in job
postings help to increase awareness of the
certification program while assisting the
organization in finding suitable candidates
Anyone who is interested in promoting BSD can
assist the BSDCG in spreading the word about
BSD certification

20. Additional Resources
What it Takes: 3 part series in the June to August
Issues of BSD Magazine (http://bsdmag.org)
A Case Study in Open Source Certification
http://blip.tv/bsdtv
The Importance of Psychometrics
http://www.bsdcertification.org/downloads/
psychometric.pdf

21. Contact & Questions
Contact:
dru@freebsd.org
URL to slides online:
http://www.slideshare.net/dlavigne/
eurobsdcon-2011
URL to formal paper:
http://www.slideshare.net/dlavigne/
lavigne-eurobsdcon11