BSD Certification:          How to Create a      Psychometrically Valid     Certification ExaminationDru LavigneChair, BSD...
OutlineWhat is psychometrics? (and why should I care?)The Job Task Analysis (JTA)Certification RequirementsItem WritingBet...
PsychometricsScience of assessmentDetermines the skills and knowledge to beassessedAssesses understanding rather thanmemor...
PsychometricsValue to:Certificant: since the exam matches therequirements, even if additional study aides arenot available...
PsychometricsValue to:Certifying Body: is in a position to successfullydefend the reliability of the assessment should ane...
PsychometricsSigns of pyschometric validity:Published exam requirementsExam questions match the exam requirementsAssessmen...
JTAFirst step in creating a new certificationDetermines which tasks will be assessedGoal is to define the content areas (d...
JTAResults are analyzed and used to create theCertification RequirementsBSDA: http://www.bsdcertification.org/downloads/sr...
Certification RequirementsAlso known as the exam blueprintContains the following information:1. A clear definition of the ...
Certification Requirements4. The depth of knowledge to be assessedaccording to Blooms Taxonomy(http://en.wikipedia.org/wik...
Item WritingSMEs (Subject Matter Experts) use the blueprintto create exam questions (items)Item writing is an art and a sk...
Item WritingItems must be consistent in punctuation, style,and formatting to avoid distractionsItems cant contain cut/past...
Item WritingMultiple choice item contains:Stem: the question being askedKey: the correct answer(s); appears as an optionDi...
Item WritingFirst draft of an item rarely looks like the finalversion of question that appears on examItems undergo much s...
Beta PeriodOnce the items have finished their review, thepsychometrician creates one or more exam formsGoal of the beta pe...
Angoff SessionGoal of theAngoff session is to set the cut score(passing score) by determining the number ofitems a minimal...
Exam MaintenanceOnce exam is published, psychometrics does notendStatistics continue to be gathered for analysis onhow exa...
Exam MaintenanceFor each established review period, thepsychometrician will identify which items shouldbe reviewed by the ...
Becoming InvolvedMembers of intended audience can act as SMEsand participate in item writing, item review, andAngoff sessi...
Additional ResourcesWhat it Takes: 3 part series in the June to AugustIssues of BSD Magazine (http://bsdmag.org)A Case Stu...
Contact & Questions                   Contact:                dru@freebsd.orgURL to slides online:http://www.slideshare.ne...
Upcoming SlideShare
Loading in...5
×

Eurobsdcon 2011

627

Published on

Slides for EuroBSDCon.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
627
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Eurobsdcon 2011

  1. 1. BSD Certification: How to Create a Psychometrically Valid Certification ExaminationDru LavigneChair, BSD Certification Group Inc.EuroBSDCon, 2011
  2. 2. OutlineWhat is psychometrics? (and why should I care?)The Job Task Analysis (JTA)Certification RequirementsItem WritingBeta PeriodAngoff SessionExam MaintenanceBecoming Involved
  3. 3. PsychometricsScience of assessmentDetermines the skills and knowledge to beassessedAssesses understanding rather thanmemorizationIncreases the value of the assessment
  4. 4. PsychometricsValue to:Certificant: since the exam matches therequirements, even if additional study aides arenot available, examinees can gauge their exampreparedness by comparing their skillset to thelisted objectivesEmployer: has a yardstick that validates theminimum skillset of certified employees
  5. 5. PsychometricsValue to:Certifying Body: is in a position to successfullydefend the reliability of the assessment should anexaminee dispute the results of their examinationCommunity: the certificate is worth more than thepaper it is written on and certificants are not ableto pass the exam by memorizing the contentavailable from "exam cram" websites
  6. 6. PsychometricsSigns of pyschometric validity:Published exam requirementsExam questions match the exam requirementsAssessment is repeatable and defendableExam does not try to “trick” the examinee orintroduce biasExam does not contain inaccuracies
  7. 7. JTAFirst step in creating a new certificationDetermines which tasks will be assessedGoal is to define the content areas (domains) andthe importance and frequency of each taskSurvey sample should be large and shouldinclude working members of the target audienceand their employers
  8. 8. JTAResults are analyzed and used to create theCertification RequirementsBSDA: http://www.bsdcertification.org/downloads/sr1_links.pdfBSDP: http://www.bsdcertification.org/downloads/delivery_survey.pdf
  9. 9. Certification RequirementsAlso known as the exam blueprintContains the following information:1. A clear definition of the intended audience2. The knowledge domains and their percentages(Table of Specifications)3. Detailed exam objectives that clearly identifythe tasks to be assessed
  10. 10. Certification Requirements4. The depth of knowledge to be assessedaccording to Blooms Taxonomy(http://en.wikipedia.org/wiki/Blooms_taxonomy)This document should be published and freelyavailable to examinees, employers, and thecreators of study aidesBSDA: http://www.bsdcertification.org/downloads/pr_20051005_certreq_bsda_en_en.pdfBSDP: http://www.bsdcertification.org/downloads/certreq_bsdp_en.pdf
  11. 11. Item WritingSMEs (Subject Matter Experts) use the blueprintto create exam questions (items)Item writing is an art and a skillItems must be written to the skill level of intendedaudienceItems must be clearly written and understandable,even to non-native English speakersAn item can only assess one exam objective
  12. 12. Item WritingItems must be consistent in punctuation, style,and formatting to avoid distractionsItems cant contain cut/pastes from instructionalmaterialsItems shouldnt contain double negativesCorrect answers should not be in same position orfollow a pattern
  13. 13. Item WritingMultiple choice item contains:Stem: the question being askedKey: the correct answer(s); appears as an optionDistractors: 3-4 incorrect answers which appearas options; must be plausible and not containcontextual cues
  14. 14. Item WritingFirst draft of an item rarely looks like the finalversion of question that appears on examItems undergo much scrutiny, rewriting, andreview by both the SMEs and the psychometricianAn exam form can not contain enemies (aquestion that helps to answer another question)
  15. 15. Beta PeriodOnce the items have finished their review, thepsychometrician creates one or more exam formsGoal of the beta period is to generate statisticalinformation on how examinees respond to theexam forms and to identify anomaliesYou cant pass or fail a beta as the passing (cut)score cant be determined until the stastics of thebeta are analyzed, required questions arerewritten or removed from the form, and theAngoff session has occurred
  16. 16. Angoff SessionGoal of theAngoff session is to set the cut score(passing score) by determining the number ofitems a minimally competent examinee wouldanswer correctlyEach SME individually rates every itemRatings are compared and any discrepancies arediscussed until concensus is achievedPsychometrician uses final ratings to determineexams passing score; exam can now bepublished
  17. 17. Exam MaintenanceOnce exam is published, psychometrics does notendStatistics continue to be gathered for analysis onhow examinees respond to each exam itemPsychometrician will alert exam body of anyanomalies that may indicate cheating is occurringRegular review period is scheduled (by time ornumber of completed exams)
  18. 18. Exam MaintenanceFor each established review period, thepsychometrician will identify which items shouldbe reviewed by the SMEsSome items may be rewritten, removed, orreplaced, and pilot items may be introducedNew version of form may be published (usuallynot advertised)Period must be set for when to redo JTA and startprocess all over again
  19. 19. Becoming InvolvedMembers of intended audience can act as SMEsand participate in item writing, item review, andAngoff sessionsEmployers who include BSD certification in jobpostings help to increase awareness of thecertification program while assisting theorganization in finding suitable candidatesAnyone who is interested in promoting BSD canassist the BSDCG in spreading the word aboutBSD certification
  20. 20. Additional ResourcesWhat it Takes: 3 part series in the June to AugustIssues of BSD Magazine (http://bsdmag.org)A Case Study in Open Source Certificationhttp://blip.tv/bsdtvThe Importance of Psychometricshttp://www.bsdcertification.org/downloads/psychometric.pdf
  21. 21. Contact & Questions Contact: dru@freebsd.orgURL to slides online:http://www.slideshare.net/dlavigne/eurobsdcon-2011URL to formal paper:http://www.slideshare.net/dlavigne/lavigne-eurobsdcon11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×