Eurobsdcon 2011
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Eurobsdcon 2011

on

  • 807 views

Slides for EuroBSDCon.

Slides for EuroBSDCon.

Statistics

Views

Total Views
807
Views on SlideShare
807
Embed Views
0

Actions

Likes
0
Downloads
9
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Eurobsdcon 2011 Presentation Transcript

  • 1. BSD Certification: How to Create a Psychometrically Valid Certification ExaminationDru LavigneChair, BSD Certification Group Inc.EuroBSDCon, 2011
  • 2. OutlineWhat is psychometrics? (and why should I care?)The Job Task Analysis (JTA)Certification RequirementsItem WritingBeta PeriodAngoff SessionExam MaintenanceBecoming Involved
  • 3. PsychometricsScience of assessmentDetermines the skills and knowledge to beassessedAssesses understanding rather thanmemorizationIncreases the value of the assessment
  • 4. PsychometricsValue to:Certificant: since the exam matches therequirements, even if additional study aides arenot available, examinees can gauge their exampreparedness by comparing their skillset to thelisted objectivesEmployer: has a yardstick that validates theminimum skillset of certified employees
  • 5. PsychometricsValue to:Certifying Body: is in a position to successfullydefend the reliability of the assessment should anexaminee dispute the results of their examinationCommunity: the certificate is worth more than thepaper it is written on and certificants are not ableto pass the exam by memorizing the contentavailable from "exam cram" websites
  • 6. PsychometricsSigns of pyschometric validity:Published exam requirementsExam questions match the exam requirementsAssessment is repeatable and defendableExam does not try to “trick” the examinee orintroduce biasExam does not contain inaccuracies
  • 7. JTAFirst step in creating a new certificationDetermines which tasks will be assessedGoal is to define the content areas (domains) andthe importance and frequency of each taskSurvey sample should be large and shouldinclude working members of the target audienceand their employers
  • 8. JTAResults are analyzed and used to create theCertification RequirementsBSDA: http://www.bsdcertification.org/downloads/sr1_links.pdfBSDP: http://www.bsdcertification.org/downloads/delivery_survey.pdf
  • 9. Certification RequirementsAlso known as the exam blueprintContains the following information:1. A clear definition of the intended audience2. The knowledge domains and their percentages(Table of Specifications)3. Detailed exam objectives that clearly identifythe tasks to be assessed
  • 10. Certification Requirements4. The depth of knowledge to be assessedaccording to Blooms Taxonomy(http://en.wikipedia.org/wiki/Blooms_taxonomy)This document should be published and freelyavailable to examinees, employers, and thecreators of study aidesBSDA: http://www.bsdcertification.org/downloads/pr_20051005_certreq_bsda_en_en.pdfBSDP: http://www.bsdcertification.org/downloads/certreq_bsdp_en.pdf
  • 11. Item WritingSMEs (Subject Matter Experts) use the blueprintto create exam questions (items)Item writing is an art and a skillItems must be written to the skill level of intendedaudienceItems must be clearly written and understandable,even to non-native English speakersAn item can only assess one exam objective
  • 12. Item WritingItems must be consistent in punctuation, style,and formatting to avoid distractionsItems cant contain cut/pastes from instructionalmaterialsItems shouldnt contain double negativesCorrect answers should not be in same position orfollow a pattern
  • 13. Item WritingMultiple choice item contains:Stem: the question being askedKey: the correct answer(s); appears as an optionDistractors: 3-4 incorrect answers which appearas options; must be plausible and not containcontextual cues
  • 14. Item WritingFirst draft of an item rarely looks like the finalversion of question that appears on examItems undergo much scrutiny, rewriting, andreview by both the SMEs and the psychometricianAn exam form can not contain enemies (aquestion that helps to answer another question)
  • 15. Beta PeriodOnce the items have finished their review, thepsychometrician creates one or more exam formsGoal of the beta period is to generate statisticalinformation on how examinees respond to theexam forms and to identify anomaliesYou cant pass or fail a beta as the passing (cut)score cant be determined until the stastics of thebeta are analyzed, required questions arerewritten or removed from the form, and theAngoff session has occurred
  • 16. Angoff SessionGoal of theAngoff session is to set the cut score(passing score) by determining the number ofitems a minimally competent examinee wouldanswer correctlyEach SME individually rates every itemRatings are compared and any discrepancies arediscussed until concensus is achievedPsychometrician uses final ratings to determineexams passing score; exam can now bepublished
  • 17. Exam MaintenanceOnce exam is published, psychometrics does notendStatistics continue to be gathered for analysis onhow examinees respond to each exam itemPsychometrician will alert exam body of anyanomalies that may indicate cheating is occurringRegular review period is scheduled (by time ornumber of completed exams)
  • 18. Exam MaintenanceFor each established review period, thepsychometrician will identify which items shouldbe reviewed by the SMEsSome items may be rewritten, removed, orreplaced, and pilot items may be introducedNew version of form may be published (usuallynot advertised)Period must be set for when to redo JTA and startprocess all over again
  • 19. Becoming InvolvedMembers of intended audience can act as SMEsand participate in item writing, item review, andAngoff sessionsEmployers who include BSD certification in jobpostings help to increase awareness of thecertification program while assisting theorganization in finding suitable candidatesAnyone who is interested in promoting BSD canassist the BSDCG in spreading the word aboutBSD certification
  • 20. Additional ResourcesWhat it Takes: 3 part series in the June to AugustIssues of BSD Magazine (http://bsdmag.org)A Case Study in Open Source Certificationhttp://blip.tv/bsdtvThe Importance of Psychometricshttp://www.bsdcertification.org/downloads/psychometric.pdf
  • 21. Contact & Questions Contact: dru@freebsd.orgURL to slides online:http://www.slideshare.net/dlavigne/eurobsdcon-2011URL to formal paper:http://www.slideshare.net/dlavigne/lavigne-eurobsdcon11