Openhelp11

Starting an Open Source Certification ProgramDru LavigneChair, BSD Certification Group Inc.Open Help 2011

This presentation will discuss:Brief introduction (why certification?)Questions to answer before startingDetails (10 step program)Summary (is it worth the effort?)

IntroductionWhat does this have to do withdocumentation?Doc team members often have the knowledgeand writing skills required by a cert programsSMEs (subject matter experts) and authors oftraining materials

IntroductionWhy consider a certification program?Increased project exposure and enterpriseadoption of that open source softwareIncreased business credibility; certification is arequirement in some environmentsPossible revenue stream

IntroductionIs it a good fit for our project?How mature is the project?Is there a corporate sponsor or foundation?What is the projects “product”?Are the “skills” for that product assessable and ofinterest to employers?Is there a strong documentation focus?

Before StartingDetermine the organizational structure:For profit? non-profit? incorporated?Part of the project or separate organization?Paid, volunteer, mixture of both?Create exam only or also provide trainingmaterials?

Before StartingDetermine the exam structure:Who is the target audience?Which skills are to be assessed?Are you aiming for psychometric validity?accreditation?How many exams must be passed to be certified?What value do certificants receive?

Before StartingPsychometrics:The science of assessmentNot cheap: $25k-100k+ to get started, at least $5k/yr to maintain each examProvides process and expertise--useful when youdont know what youre doingWorth every penny

Before StartingAccreditation:Not cheap: expect $25k to get started, $2-5k/yr tomaintainTime consuming, tedious, boring work (i.e. youwill document all processes to ISO standards)Exam program must already be psychometricallyvalid and recertification process is mandatoryhttps://www.ansica.org/wwwversion2/outside/PERgeneral.asp?menuID=2

Before StartingExam delivery:Paper or computer based?Multiple choice, virtual lab?Proctored environment?Who provides scoring services?Issues: cost, proprietary solutions

Step 1JTA:Job Task Analysis: a survey to determine theimportance and frequency of tasks--results areused to create exam objectivesRequires knowledge of which tasks are possiblefor that productSurvey should be completed by employers,employees, and hiring staff; aim for a large andbalanced sampling

Step 2Certification Requirements (Blueprint):Typically preceded by a report on the JTA results;results are analyzed to determine what belongson the examCertification Requirements document contains:description of the exam audience, study domainsand their %, and detailed exam objectives. Eachexam objective includes its importance andfrequency rating, blooms taxonomy classification,and a clear explanation of what is required of theexam candidate to successfully pass thatobjective

Step 3Question Writing:SMEs (subject matter experts) write examquestions to match exam blueprintAim for a large pool of questions, preferablyseveral per objectiveThere are only so many ways you can ask aquestionOriginal draft of question typically bears littleresemblance to the final form of the question

Step 3The Art of Question Writing:Items must be written to the skill level of theaudience defined in the CertificationRequirementsItems must be clearly written and understandable,even to non-native English speakersEach item assesses only one objectiveItems should be consistent in formatting style,punctuation, and grammar

Step 3The Art of Question Writing:Avoid items that require the examinee to selectmultiple answers or that include an answer of "allof the above" or "none of the above"If instructional materials exist, do not cut/pasteinformation to create itemsDo not place enemies on the same exam form;enemies are items that contain information thatcan be used to answer another question on theexam

Step 3The Art of Question Writing:Distractors should be of similar length andplausibility as the correct answer; someone whodoes not understand the objective should not beable to answer correctly from contextual cuesAvoid stems containing negatives or doublenegativesHave a standard for determining the order of theoptions and ensure that the correct answer is notalways in the same position

Step 4Review Period:Once a pool of questions is created, it undergoesscrutiny to make sure questions meet the criteriaidentified in Step 3SMEs can help review technical accuracy of examquestionsReview should include non-SMEs to look forcontextual cues, formatting/grammatical issues

Step 5Beta Period:Once the question review is complete,psychometrician will determine which questionsgo on which form: determination must matchexam blueprint and multiple forms must provideequal assessmentsBeta period is used to identify anomalies in howcandidates respond to questionsBeta participants should closely match the examsaudience and beta should be available incountries where the exam will be offered

Step 5Beta Period:Beta participants can not "pass" or "fail" the betaexam as the passing score can not be determinedbefore the Angoff sessionAny questions identified by the beta as beingproblematic (e.g. nearly everyone answerscorrectly or incorrectly, English speakers answercorrectly but non-native speakers do not) need tobe reviewed to determine if belong on exam orneed to be rewritten

Step 6Angoff session:Used to finalize questions and set the cut(passing) score by determining the number ofitems a minimally competent examinee wouldanswer correctly; anyone who correctly answersmore items passes the exam as they will havedemonstrated that they are more thanminimally competentSMEs and psychometrician should attend

Step 6Angoff session:Each SME reviews every exam item and rates iton a scale of 1-10 on the likelihood that aminimally competent person would answer thatquestion correctlyOnce all the items are individually rated, theratings are compared, any discrepancies arediscussed, and a concensus is reached on therating for every itemThe psychometrician uses the list of ratings todetermine the passing score for the exam

Step 7Launch Exam:Prior to launch, certification organization needs topublish their policies: cheating, recertification,handling disputes, etc. as well as finalize anycontracts regarding delivery and scoringNeed to write marketing materials, press releases,update website information, etc.If project will be providing training materials, arethey created in parallel with the exam creationprocess or serially after the exam becomesavailable?

Step 8Ongoing Metrics:Psychometrician collects ongoing metrics on howevery exam candidate responds to each examquestionPsychometrician will advise of any anomalies thatmay indicate cheating is occuring at a testingcenter or with a particular proctorCertification body determines frequency of reviewprocess (annually or after so many exams)

Step 9Periodic Review:Psychometrician will identify which items shouldbe reviewed by the SMEsThese will include items that aredisproportionately answered correctly orincorrectly and any items that are statisticallystrange (e.g. items which minimally competentexaminees tend to answer correctly while verycompetent examinees tend to answer incorrectly)

Step 9Periodic Review:SMEs may rewrite these items or replace themwith different itemsPsychometrician may decide to introduce "pilot"items which dont affect the examinees score andtherefore dont require a beta or angoff periodOnce the review is complete, the certifyingorganization will publish the new version of theexam (typically not announced)

Step 10Rinse and Repeat:In other words, an exam may be published but itis never “done”Organization needs to decide how often the JTAneeds to be repeated and the CertificationRequirements document is updated to reflectchanges in the productBrave organizations may even decide to undergothe whole process again with another certificationprogram

SummaryCreation and maintenance of certification examsis a lot of work and can cost a lot of moneyKnow your goals beforehandAim to have a few individuals who are committedto the long haulExpect the first exam to take at least twice as longto launch as you think it shouldDont reinvent the wheel--make relationships withother certification organizations

Questions & DiscussionSlides:http://www.slideshare.net/dlavigne/openhelp11BSD Certification website:http://www.bsdcertification.orgContact:dru@freebsd.org

Openhelp11

by Dru Lavigne on May 30, 2011

Accessibility

Upload Details

Usage Rights

1 Embed 8

Statistics

Openhelp11 Presentation Transcript