11Evaluating HRDEvaluating HRDProgramsProgramsProfessor Jayashree SadriProfessor Jayashree Sadriand Dr. Sorab Sadriand Dr. Sorab Sadri
22EffectivenessEffectiveness The degree to which a training (orThe degree to which a training (orother HRD program) achieves itsother HRD program) achieves itsintended purposeintended purpose Measures are relative to someMeasures are relative to somestarting pointstarting point Measures how well the desiredMeasures how well the desiredgoal is achievedgoal is achieved
44HRD EvaluationHRD EvaluationTextbook definition:Textbook definition:““The systematic collection ofThe systematic collection ofdescriptive and judgmentaldescriptive and judgmentalinformation necessary to makeinformation necessary to makeeffective training decisions relatedeffective training decisions relatedto the selection, adoption, value,to the selection, adoption, value,and modification of variousand modification of variousinstructional activities.”instructional activities.”
55In Other Words…In Other Words…Are we training:Are we training: the right peoplethe right people the right “stuff”the right “stuff” the right waythe right way with the right materialswith the right materials at the right time?at the right time?
66Evaluation NeedsEvaluation Needs Descriptive and judgmentalDescriptive and judgmentalinformation neededinformation needed– Objective and subjective dataObjective and subjective data Information gathered according toInformation gathered according toa plan and in a desired formata plan and in a desired format Gathered to provide decisionGathered to provide decisionmaking informationmaking information
77Purposes of EvaluationPurposes of Evaluation Determine whether the program isDetermine whether the program ismeeting the intended objectivesmeeting the intended objectives Identify strengths and weaknessesIdentify strengths and weaknesses Determine cost-benefit ratioDetermine cost-benefit ratio Identify who benefited most or leastIdentify who benefited most or least Determine future participantsDetermine future participants Provide information for improvingProvide information for improvingHRD programsHRD programs
88Purposes of EvaluationPurposes of Evaluation –– 22 Reinforce major points to be madeReinforce major points to be made Gather marketing informationGather marketing information Determine if training program isDetermine if training program isappropriateappropriate Establish management databaseEstablish management database
99Evaluation Bottom LineEvaluation Bottom Line Is HRD a revenue contributor or aIs HRD a revenue contributor or arevenue user?revenue user? Is HRD credible to line and upper-Is HRD credible to line and upper-level managers?level managers? Are benefits of HRD readily evidentAre benefits of HRD readily evidentto all?to all?
1010How Often are HRD EvaluationsHow Often are HRD EvaluationsConducted?Conducted? Not often enough!!!Not often enough!!! Frequently, only end-of-courseFrequently, only end-of-courseparticipant reactions are collectedparticipant reactions are collected Transfer to the workplace isTransfer to the workplace isevaluated less frequentlyevaluated less frequently
1111Why HRD Evaluations are RareWhy HRD Evaluations are Rare Reluctance to having HRD programsReluctance to having HRD programsevaluatedevaluated Evaluation needs expertise andEvaluation needs expertise andresourcesresources Factors other than HRD causeFactors other than HRD causeperformance improvementsperformance improvements –– e.g.,e.g.,– EconomyEconomy– EquipmentEquipment– Policies, etc.Policies, etc.
1212Need for HRD EvaluationNeed for HRD Evaluation Shows the value of HRDShows the value of HRD Provides metrics for HRD efficiencyProvides metrics for HRD efficiency Demonstrates value-addedDemonstrates value-addedapproach for HRDapproach for HRD Demonstrates accountability forDemonstrates accountability forHRD activitiesHRD activities Everyone else has it… why notEveryone else has it… why notHRD?HRD?
1313Make or Buy EvaluationMake or Buy Evaluation ““I bought it, therefore it is good.”I bought it, therefore it is good.” ““Since it’s good, I don’t need to post-Since it’s good, I don’t need to post-test.”test.” Who says it’s:Who says it’s:– Appropriate?Appropriate?– Effective?Effective?– Timely?Timely?– Transferable to the workplace?Transferable to the workplace?
1414Evolution of Evaluation EffortsEvolution of Evaluation Efforts1.1. AnecdotalAnecdotal approachapproach –– talk to othertalk to otherusersusers2.2. Try before buyTry before buy –– borrow and useborrow and usesamplessamples3.3. AnalyticalAnalytical approachapproach –– matchmatchresearch data to training needsresearch data to training needs4.4. HolisticHolistic approachapproach –– look at overalllook at overallHRD process, as well as individualHRD process, as well as individualtrainingtraining
1515Models and Frameworks ofModels and Frameworks ofEvaluationEvaluation Table 7-1 lists six frameworks forTable 7-1 lists six frameworks forevaluationevaluation The most popular is that of D.The most popular is that of D.Kirkpatrick:Kirkpatrick:– ReactionReaction– LearningLearning– Job BehaviorJob Behavior– ResultsResults
1616Kirkpatrick’s Four LevelsKirkpatrick’s Four Levels ReactionReaction– Focus on trainee’s reactionsFocus on trainee’s reactions LearningLearning– Did they learn what they were supposed to?Did they learn what they were supposed to? Job BehaviorJob Behavior– Was it used on job?Was it used on job? ResultsResults– Did it improve the organization’s effectiveness?Did it improve the organization’s effectiveness?
1717Issues Concerning Kirkpatrick’sIssues Concerning Kirkpatrick’sFrameworkFramework Most organizations don’t evaluateMost organizations don’t evaluateat all four levelsat all four levels Focuses only on post-trainingFocuses only on post-training Doesn’t treat inter-stageDoesn’t treat inter-stageimprovementsimprovements WHAT ARE YOUR THOUGHTS?WHAT ARE YOUR THOUGHTS?
1818Other Frameworks/ModelsOther Frameworks/Models CIPP: Context, Input, Process, ProductCIPP: Context, Input, Process, Product(Galvin, 1983)(Galvin, 1983) Brinkerhoff (1987):Brinkerhoff (1987):– Goal settingGoal setting– Program designProgram design– Program implementationProgram implementation– Immediate outcomesImmediate outcomes– Usage outcomesUsage outcomes– Impacts and worthImpacts and worth
1919Other Frameworks/Models – 2Other Frameworks/Models – 2 Kraiger, Ford, & Salas (1993):Kraiger, Ford, & Salas (1993):– Cognitive outcomesCognitive outcomes– Skill-based outcomesSkill-based outcomes– Affective outcomesAffective outcomes Holton (1996): Five Categories:Holton (1996): Five Categories:– Secondary InfluencesSecondary Influences– Motivation ElementsMotivation Elements– Environmental ElementsEnvironmental Elements– OutcomesOutcomes– Ability/Enabling ElementsAbility/Enabling Elements
2020Other Frameworks/Models – 3Other Frameworks/Models – 3 Phillips (1996):Phillips (1996):– Reaction and Planned ActionReaction and Planned Action– LearningLearning– Applied Learning on the JobApplied Learning on the Job– Business ResultsBusiness Results– ROIROI
2121A Suggested Framework – 1A Suggested Framework – 1 ReactionReaction– Did trainees like the training?Did trainees like the training?– Did the training seem useful?Did the training seem useful? LearningLearning– How much did they learn?How much did they learn? BehaviorBehavior– What behavior change occurred?What behavior change occurred?
2222Suggested Framework – 2Suggested Framework – 2 ResultsResults– What were the tangible outcomes?What were the tangible outcomes?– What was the return on investmentWhat was the return on investment(ROI)?(ROI)?– What was the contribution to theWhat was the contribution to theorganization?organization?
2323Data Collection for HRDData Collection for HRDEvaluationEvaluationPossible methods:Possible methods: InterviewsInterviews QuestionnairesQuestionnaires Direct observationDirect observation Written testsWritten tests Simulation/Performance testsSimulation/Performance tests Archival performance informationArchival performance information
2424InterviewsInterviewsAdvantagesAdvantages:: FlexibleFlexible Opportunity forOpportunity forclarificationclarification Depth possibleDepth possible Personal contactPersonal contactLimitations:Limitations: High reactiveHigh reactiveeffectseffects High costHigh cost Face-to-face threatFace-to-face threatpotentialpotential Labor intensiveLabor intensive Trained observersTrained observersneededneeded
2626Direct ObservationDirect ObservationAdvantagesAdvantages:: NonthreateningNonthreatening Excellent way toExcellent way tomeasure behaviormeasure behaviorchangechangeLimitationsLimitations:: Possibly disruptivePossibly disruptive Reactive effectsReactive effectsare possibleare possible May be unreliableMay be unreliable Need trainedNeed trainedobserversobservers
2727Written TestsWritten TestsAdvantagesAdvantages:: Low purchase costLow purchase cost Readily scoredReadily scored Quickly processedQuickly processed Easily administeredEasily administered Wide samplingWide samplingpossiblepossibleLimitationsLimitations:: May be threateningMay be threatening Possibly no relationPossibly no relationto job performanceto job performance Measures onlyMeasures onlycognitive learningcognitive learning Relies on normsRelies on norms Concern for racial/Concern for racial/ethnic biasethnic bias
2828Simulation/Performance TestsSimulation/Performance TestsAdvantagesAdvantages:: ReliableReliable ObjectiveObjective Close relation toClose relation tojob performancejob performance Includes cognitive,Includes cognitive,psychomotor andpsychomotor andaffective domainsaffective domainsLimitationsLimitations:: Time consumingTime consuming Simulations oftenSimulations oftendifficult to createdifficult to create High costs toHigh costs todevelopment anddevelopment anduseuse
2929Archival Performance DataArchival Performance DataAdvantagesAdvantages:: ReliableReliable ObjectiveObjective Job-basedJob-based Easy to reviewEasy to review Minimal reactiveMinimal reactiveeffectseffectsLimitationsLimitations:: Criteria forCriteria forkeeping/ discardingkeeping/ discardingrecordsrecords Information systemInformation systemdiscrepanciesdiscrepancies IndirectIndirect Not always usableNot always usable Records preparedRecords preparedfor other purposesfor other purposes
3030Choosing Data CollectionChoosing Data CollectionMethodsMethods ReliabilityReliability– Consistency of results, and freedom fromConsistency of results, and freedom fromcollection method bias and errorcollection method bias and error ValidityValidity– Does the device measure what we want toDoes the device measure what we want tomeasure?measure? PracticalityPracticality– Does it make sense in terms of theDoes it make sense in terms of theresources used to get the data?resources used to get the data?
3131Type of Data Used/NeededType of Data Used/Needed Individual performanceIndividual performance Systemwide performanceSystemwide performance EconomicEconomic
3434Economic DataEconomic Data ProfitsProfits Product liability claimsProduct liability claims Avoidance of penaltiesAvoidance of penalties Market shareMarket share Competitive positionCompetitive position Return on investment (ROI)Return on investment (ROI) Financial utility calculationsFinancial utility calculations
3535Use of Self-Report DataUse of Self-Report Data Most common methodMost common method Pre-training and post-training dataPre-training and post-training data Problems:Problems:– Mono-method biasMono-method bias Desire to be consistent between testsDesire to be consistent between tests– Socially desirable responsesSocially desirable responses– Response Shift Bias:Response Shift Bias: Trainees adjust expectations to trainingTrainees adjust expectations to training
3636Research DesignResearch DesignSpecifies in advance:Specifies in advance: the expected results of the studythe expected results of the study the methods of data collection to bethe methods of data collection to beusedused how the data will be analyzedhow the data will be analyzed
3737Research Design IssuesResearch Design Issues Pretest and PosttestPretest and Posttest– Shows trainee what training hasShows trainee what training hasaccomplishedaccomplished– Helps eliminate pretest knowledge biasHelps eliminate pretest knowledge bias Control GroupControl Group– Compares performance of group withCompares performance of group withtraining against the performance of atraining against the performance of asimilar group without trainingsimilar group without training
3838Recommended ResearchRecommended ResearchDesignDesign Pretest and posttest with controlPretest and posttest with controlgroupgroup Whenever possible:Whenever possible:– Randomly assign individuals to the testRandomly assign individuals to the testgroup and the control group to minimizegroup and the control group to minimizebiasbias– Use “time-series” approach to dataUse “time-series” approach to datacollection to verify performancecollection to verify performanceimprovement is due to trainingimprovement is due to training
3939Ethical Issues ConcerningEthical Issues ConcerningEvaluation ResearchEvaluation Research ConfidentialityConfidentiality Informed consentInformed consent Withholding training from controlWithholding training from controlgroupsgroups Use of deceptionUse of deception Pressure to produce positive resultsPressure to produce positive results
4040Assessing the Impact of HRDAssessing the Impact of HRD Money is the language of business.Money is the language of business. You MUST talk dollars, not HRDYou MUST talk dollars, not HRDjargon.jargon. No one (except maybe you) caresNo one (except maybe you) caresabout “the effectiveness of trainingabout “the effectiveness of traininginterventions as measured by andinterventions as measured by andanalysis of formal pretest, posttestanalysis of formal pretest, posttestcontrol group data.”control group data.”
4141HRD Program AssessmentHRD Program Assessment HRD programs and training areHRD programs and training areinvestmentsinvestments Line managers often see HR and HRD asLine managers often see HR and HRD ascostscosts –– i.e.,i.e., revenue users, not revenuerevenue users, not revenueproducersproducers You must prove your worth to theYou must prove your worth to theorganization –organization –– Or you’ll have to find anotherOr you’ll have to find anotherorganization…organization…
4242Two Basic Methods forTwo Basic Methods forAssessing Financial ImpactAssessing Financial Impact Evaluation of training costsEvaluation of training costs Utility analysisUtility analysis
4343Evaluation of Training CostsEvaluation of Training Costs Cost-benefit analysisCost-benefit analysis– Compares cost of training to benefitsCompares cost of training to benefitsgained such as attitudes, reduction ingained such as attitudes, reduction inaccidents, reduction in employee sick-accidents, reduction in employee sick-days, etc.days, etc. Cost-effectiveness analysisCost-effectiveness analysis– Focuses on increases in quality,Focuses on increases in quality,reduction in scrap/rework, productivity,reduction in scrap/rework, productivity,etc.etc.
4444Return on InvestmentReturn on Investment Return on investment =Return on investment =Results/CostsResults/Costs
4545Calculating Training Return OnCalculating Training Return OnInvestmentInvestment Results Results Operational How Before After Differences ExpressedResults Area Measured Training Training (+ or –) in $Quality of panels % rejected 2% rejected 1.5% rejected .5% $720 per day 1,440 panels 1,080 panels 360 panels $172,800 per day per day per yearHousekeeping Visual 10 defects 2 defects 8 defects Not measur- inspection (average) (average) able in $ using 20-item checklist Preventable Number of 24 per year 16 per year 8 per year accidents accidents Direct cost $144,000 $96,000 per $48,000 $48,000 per of each per year year year accident ReturnInvestment Total savings: $220,800.00ROI = = SOURCE: From D. G. Robinson & J. Robinson (1989). Training for impact. Training and Development Journal, 43(8), 41. Printed by permission.Operational ResultsTraining Costs= $220,800$32,564= 6.8
4646Types of Training CostsTypes of Training Costs Direct costsDirect costs Indirect costsIndirect costs Development costsDevelopment costs Overhead costsOverhead costs Compensation for participantsCompensation for participants
4747Direct CostsDirect Costs InstructorInstructor– Base payBase pay– Fringe benefitsFringe benefits– Travel and per diemTravel and per diem MaterialsMaterials Classroom and audiovisualClassroom and audiovisualequipmentequipment TravelTravel Food and refreshmentsFood and refreshments
4848Indirect CostsIndirect Costs Training managementTraining management Clerical/AdministrativeClerical/Administrative Postal/shipping, telephone,Postal/shipping, telephone,computers, etc.computers, etc. Pre- and post-learning materialsPre- and post-learning materials Other overhead costsOther overhead costs
4949Development CostsDevelopment Costs Fee to purchase programFee to purchase program Costs to tailor program toCosts to tailor program toorganizationorganization Instructor training costsInstructor training costs
5050Overhead CostsOverhead Costs General organization supportGeneral organization support Top management participationTop management participation Utilities, facilitiesUtilities, facilities General and administrative costs,General and administrative costs,such as HRMsuch as HRM
5151Compensation for ParticipantsCompensation for Participants Participants’ salary and benefits forParticipants’ salary and benefits fortime away from jobtime away from job Travel, lodging, and per-diem costsTravel, lodging, and per-diem costs
5252Measuring BenefitsMeasuring Benefits– Change in quality per unit measured inChange in quality per unit measured indollarsdollars– Reduction in scrap/rework measured inReduction in scrap/rework measured indollar cost of labor and materialsdollar cost of labor and materials– Reduction in preventable accidentsReduction in preventable accidentsmeasured in dollarsmeasured in dollars– ROI = Benefits/Training costsROI = Benefits/Training costs
5353Utility AnalysisUtility Analysis Uses a statistical approach toUses a statistical approach tosupport claims of trainingsupport claims of trainingeffectiveness:effectiveness:– N = Number of traineesN = Number of trainees– T = Length of time benefits are expected to lastT = Length of time benefits are expected to last– ddtt = True performance difference resulting from= True performance difference resulting fromtrainingtraining– SDSDyy = Dollar value of untrained job performance (in= Dollar value of untrained job performance (instandard deviation units)standard deviation units)– C = Cost of trainingC = Cost of training ∆∆U = (N)(T)(dU = (N)(T)(dtt)(Sd)(Sdyy) – C) – C
5454Critical Information for UtilityCritical Information for UtilityAnalysisAnalysis ddtt = difference in units between= difference in units betweentrained/untrained, divided bytrained/untrained, divided bystandard deviation in unitsstandard deviation in unitsproduced by trainedproduced by trained SDSDyy = standard deviation in= standard deviation indollars, or overall productivity ofdollars, or overall productivity oforganizationorganization
5555Ways to Improve HRDWays to Improve HRDAssessmentAssessment Walk the walk, talk the talk: MONEYWalk the walk, talk the talk: MONEY Involve HRD in strategic planningInvolve HRD in strategic planning Involve management in HRD planning andInvolve management in HRD planning andestimation effortsestimation efforts– Gain mutual ownershipGain mutual ownership Use credible and conservative estimatesUse credible and conservative estimates Share credit for successes and blame forShare credit for successes and blame forfailuresfailures
5656HRD Evaluation StepsHRD Evaluation Steps1.1. Analyze needs.Analyze needs.2.2. Determine explicit evaluation strategy.Determine explicit evaluation strategy.3.3. Insist on specific and measurableInsist on specific and measurabletraining objectives.training objectives.4.4. Obtain participant reactions.Obtain participant reactions.5.5. Develop criterion measures/instrumentsDevelop criterion measures/instrumentsto measure results.to measure results.6.6. Plan and execute evaluation strategy.Plan and execute evaluation strategy.
5757SummarySummary Training results must be measuredTraining results must be measuredagainst costsagainst costs Training must contribute to theTraining must contribute to the“bottom line”“bottom line” HRD must justify itself repeatedlyHRD must justify itself repeatedlyas a revenue enhanceras a revenue enhancer