Eurobsdcon 2011


Published on

Slides for EuroBSDCon.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Eurobsdcon 2011

  1. 1. BSD Certification: How to Create a Psychometrically Valid Certification ExaminationDru LavigneChair, BSD Certification Group Inc.EuroBSDCon, 2011
  2. 2. OutlineWhat is psychometrics? (and why should I care?)The Job Task Analysis (JTA)Certification RequirementsItem WritingBeta PeriodAngoff SessionExam MaintenanceBecoming Involved
  3. 3. PsychometricsScience of assessmentDetermines the skills and knowledge to beassessedAssesses understanding rather thanmemorizationIncreases the value of the assessment
  4. 4. PsychometricsValue to:Certificant: since the exam matches therequirements, even if additional study aides arenot available, examinees can gauge their exampreparedness by comparing their skillset to thelisted objectivesEmployer: has a yardstick that validates theminimum skillset of certified employees
  5. 5. PsychometricsValue to:Certifying Body: is in a position to successfullydefend the reliability of the assessment should anexaminee dispute the results of their examinationCommunity: the certificate is worth more than thepaper it is written on and certificants are not ableto pass the exam by memorizing the contentavailable from "exam cram" websites
  6. 6. PsychometricsSigns of pyschometric validity:Published exam requirementsExam questions match the exam requirementsAssessment is repeatable and defendableExam does not try to “trick” the examinee orintroduce biasExam does not contain inaccuracies
  7. 7. JTAFirst step in creating a new certificationDetermines which tasks will be assessedGoal is to define the content areas (domains) andthe importance and frequency of each taskSurvey sample should be large and shouldinclude working members of the target audienceand their employers
  8. 8. JTAResults are analyzed and used to create theCertification RequirementsBSDA:
  9. 9. Certification RequirementsAlso known as the exam blueprintContains the following information:1. A clear definition of the intended audience2. The knowledge domains and their percentages(Table of Specifications)3. Detailed exam objectives that clearly identifythe tasks to be assessed
  10. 10. Certification Requirements4. The depth of knowledge to be assessedaccording to Blooms Taxonomy( document should be published and freelyavailable to examinees, employers, and thecreators of study aidesBSDA:
  11. 11. Item WritingSMEs (Subject Matter Experts) use the blueprintto create exam questions (items)Item writing is an art and a skillItems must be written to the skill level of intendedaudienceItems must be clearly written and understandable,even to non-native English speakersAn item can only assess one exam objective
  12. 12. Item WritingItems must be consistent in punctuation, style,and formatting to avoid distractionsItems cant contain cut/pastes from instructionalmaterialsItems shouldnt contain double negativesCorrect answers should not be in same position orfollow a pattern
  13. 13. Item WritingMultiple choice item contains:Stem: the question being askedKey: the correct answer(s); appears as an optionDistractors: 3-4 incorrect answers which appearas options; must be plausible and not containcontextual cues
  14. 14. Item WritingFirst draft of an item rarely looks like the finalversion of question that appears on examItems undergo much scrutiny, rewriting, andreview by both the SMEs and the psychometricianAn exam form can not contain enemies (aquestion that helps to answer another question)
  15. 15. Beta PeriodOnce the items have finished their review, thepsychometrician creates one or more exam formsGoal of the beta period is to generate statisticalinformation on how examinees respond to theexam forms and to identify anomaliesYou cant pass or fail a beta as the passing (cut)score cant be determined until the stastics of thebeta are analyzed, required questions arerewritten or removed from the form, and theAngoff session has occurred
  16. 16. Angoff SessionGoal of theAngoff session is to set the cut score(passing score) by determining the number ofitems a minimally competent examinee wouldanswer correctlyEach SME individually rates every itemRatings are compared and any discrepancies arediscussed until concensus is achievedPsychometrician uses final ratings to determineexams passing score; exam can now bepublished
  17. 17. Exam MaintenanceOnce exam is published, psychometrics does notendStatistics continue to be gathered for analysis onhow examinees respond to each exam itemPsychometrician will alert exam body of anyanomalies that may indicate cheating is occurringRegular review period is scheduled (by time ornumber of completed exams)
  18. 18. Exam MaintenanceFor each established review period, thepsychometrician will identify which items shouldbe reviewed by the SMEsSome items may be rewritten, removed, orreplaced, and pilot items may be introducedNew version of form may be published (usuallynot advertised)Period must be set for when to redo JTA and startprocess all over again
  19. 19. Becoming InvolvedMembers of intended audience can act as SMEsand participate in item writing, item review, andAngoff sessionsEmployers who include BSD certification in jobpostings help to increase awareness of thecertification program while assisting theorganization in finding suitable candidatesAnyone who is interested in promoting BSD canassist the BSDCG in spreading the word aboutBSD certification
  20. 20. Additional ResourcesWhat it Takes: 3 part series in the June to AugustIssues of BSD Magazine ( Case Study in Open Source Certification Importance of Psychometrics
  21. 21. Contact & Questions Contact: dru@freebsd.orgURL to slides online: to formal paper: