Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Current Trends in Rater Training: A
Survey of Rater Training Programs in
U. S. Organizations
C. Allen Gorman
East Tennesse...
Introduction
• The accuracy of performance ratings is
important to the success of a performance
management system (Werner ...
Purpose
• No published research on the prevalence of
rater training programs in organizations
• Purpose is to fill the voi...
Rater Training
• In general, rater training is effective for improving the
quality of performance ratings (Smith, 1986; Sp...
Approaches to Rater Training
• From Woehr & Huffcutt (1994)
– Rater Error Training (RET)
– Performance Dimension Training ...
Rater Error Training
• Developed as a way to combat the
prevalence of psychometric errors in
performance appraisal ratings...
But….
• RET inadvertently lowers levels of rating
accuracy (Bernardin & Pence, 1980; Borman, 1979;
Landy & Farr, 1980)
• S...
Performance Dimension Training
• Criticisms of RET shifted focus of rater training
literature toward rating accuracy (Athe...
Frame-of-Reference Training
• Proposed by Bernardin & Buckley (1981) in
response to the disappointing results of RET
• Ess...
FORT
• Has emerged as the most popular approach
for improving rating accuracy (Roch et al., 2011)
• Meta-analytic effect s...
Criticisms of FORT
• Does not instruct raters on how to process behavior
information with goal of remembering the behavior...
Behavioral Observation Training
• Emphasizes the accuracy of behavioral
observations
• Important when considering that rat...
BOT
• Reduces rating errors (Bernardin & Walter, 1977;
Latham, Wexley, & Pursell, 1975)
• Leads to increased observational...
Combinations of Rater Training
Approaches
• RET + FORT = no significant increase in
rating accuracy (McIntyre et al., 1984...
Summary of Rater Training Research
• FORT has become the “go to” training
– Although may have limited generalizability
• R...
However….
• Lack of widespread adoption of rater training
programs in applied settings (Bernardin, Buckley, Tyler, &
Wiese...
The Present Study
• No scholarly evidence of the prevalence and
types of rater training programs in
organizations today
• ...
Method
• Procedure
– Survey part of a larger data collection effort on current
performance management practices (Gorman, R...
Measures
• Rater Training
– 8 items (e.g., Does your company train managers how to conduct
performance appraisals?)
• Perf...
Results
Do Organizations Conduct Rater Training?
Response Train Managers Train Non-Managers Refresher/Recalibration
Traini...
Results
Frequency of Rater Training Approaches
Rater Training Approach Frequency Percent
No training 24 23.76%
Rater error...
Results
Who Conducts Rater Training?
Training Conducted by Frequency Percent
External consultant 2 2.63%
Internal consulta...
Results
Frequency of Rater Training
Frequency of Rater
Training
Frequency Percent
Less than one time a year 6 8.00%
One ti...
Exploratory Analyses
• Control variable:
– Company size
• Performance appraisal systems that utilize
managerial rater trai...
Exploratory Results
• No significant difference in perceived fairness
of performance appraisal system
• Organizations that...
Rater Training Focus and Perceived
Performance Appraisal Effectiveness
Rater Training Focus and Company
Revenue
Discussion
• Rater training is alive and well
– 76% of organizations surveyed utilize
managerial rater training
– 31% trai...
Discussion
• Encouraging results
– In contrast to the presumed scientist-
practitioner gap in performance appraisal (Banks...
Areas for Improvement
• Only 22 of the 77 organizations that offer
rater training have evaluated the training
• Majority o...
Limitations
• Single source data
• Links between rater training and firm
performance are not causal
• Training programs in...
Thank You!
• If you would like a copy of the chapter,
please e-mail me
– gormanc@etsu.edu
Upcoming SlideShare
Loading in …5
×

Survey of Rater Training Programs

2,926 views

Published on

  • My brother found Custom Writing Service ⇒ www.HelpWriting.net ⇐ and ordered a couple of works. Their customer service is outstanding, never left a query unanswered.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • You can hardly find a student who enjoys writing a college papers. Among all the other tasks they get assigned in college, writing essays is one of the most difficult assignments. Fortunately for students, there are many offers nowadays which help to make this process easier. The best service which can help you is ⇒ www.WritePaper.info ⇐
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Easy and hassle free way to make money online! I have just registered with this site and straight away I was making money! It doesn't get any better than this. Thank you for taking out all the hassle and making money answering surveys as easy as possible even for non-techie guys like me! ●●● https://bit.ly/2Ruzr8s
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Survey of Rater Training Programs

  1. 1. Current Trends in Rater Training: A Survey of Rater Training Programs in U. S. Organizations C. Allen Gorman East Tennessee State University Joshua L. Ray Tusculum College John P. Meriac University of Missouri-St. Louis Thomas W. Roddy East Tennessee State University
  2. 2. Introduction • The accuracy of performance ratings is important to the success of a performance management system (Werner & Bolino, 1997) • Two general strategies for improving rating accuracy (Woehr & Huffcutt, 1994) – Rating scale development – Rater training • Rater training has become the most widely accepted strategy (Roch, Woehr, Mishra, & Kieszczynska, 2011)
  3. 3. Purpose • No published research on the prevalence of rater training programs in organizations • Purpose is to fill the void by conducting a survey of U.S. organizations to determine – Do organizations utilize rater training programs? – If so, what types of training programs?
  4. 4. Rater Training • In general, rater training is effective for improving the quality of performance ratings (Smith, 1986; Spool, 1978) • Two major benefits of rater training (McIntyre, Smith, & Hassett, 1984) – Enhance raters’ knowledge and skills for carrying out evaluations – Motivate raters to use the knowledge and skills learned in the training program • Two meta-analyses have empirically demonstrated the overall effectiveness of rater training programs for improving rating accuracy (Roch, et al., 2011; Woehr & Huffcutt, 1994)
  5. 5. Approaches to Rater Training • From Woehr & Huffcutt (1994) – Rater Error Training (RET) – Performance Dimension Training (PDT) – Frame-of-Reference Training (FORT) – Behavioral Observation Training (BOT)
  6. 6. Rater Error Training • Developed as a way to combat the prevalence of psychometric errors in performance appraisal ratings (Borman, 2001) • Generally focuses on recognizing and avoiding halo, leniency, and central tendency errors (Woehr & Huffcutt, 1994) • RET reduces halo and leniency errors (Smith, 1986)
  7. 7. But…. • RET inadvertently lowers levels of rating accuracy (Bernardin & Pence, 1980; Borman, 1979; Landy & Farr, 1980) • Smith (1986) argued that RET actually produces a meaningless redistribution of ratings • Rater errors may not be errors, but could actually reflect true score variance (Arvey & Murphy, 1998; Hedge & Kavanagh, 1988)
  8. 8. Performance Dimension Training • Criticisms of RET shifted focus of rater training literature toward rating accuracy (Athey & McIntyre, 1987) • PDT emphasizes the cognitive processing of raters as the key to the success of rater training • Typically involves having raters review the rating scale or participating in the development of the scale (Woehr & Huffcutt, 1994) • Generally effective for improving rating accuracy (Woehr & Huffcutt, 1994)
  9. 9. Frame-of-Reference Training • Proposed by Bernardin & Buckley (1981) in response to the disappointing results of RET • Essentially an extension of PDT, but incorporates a practice and feedback session (Woehr & Huffcutt, 1994) • Involves categorizing behaviors into appropriate dimensions and correctly judging the effectiveness of those behaviors (Sulsky & Day, 1992; 1994)
  10. 10. FORT • Has emerged as the most popular approach for improving rating accuracy (Roch et al., 2011) • Meta-analytic effect sizes – Cohen’s d = .83 (Woehr & Huffcutt, 1994) – Cohen’s d = .50 (Roch et al., 2011)
  11. 11. Criticisms of FORT • Does not instruct raters on how to process behavior information with goal of remembering the behavior at a later time (Noonan & Sulsky, 2001) • May cause raters to see certain behaviors that were never exhibited (Noonan & Sulsky, 2001; Sulsky & Day, 1992) • Little attempt to measure the information processing that supposedly occurs during training (Arvey & Murphy, 1998) • Overreliance on standard videos of performance and student raters in contrived rating situations (Arvey & Murphy, 1998; Noonan & Sulsky, 2001)
  12. 12. Behavioral Observation Training • Emphasizes the accuracy of behavioral observations • Important when considering that raters often must observe performance in noisy environments where competing demands deplete cognitive resources (Noonan & Sulsky, 2001) • Typically involves note taking or keeping a diary (Woehr & Huffcutt, 1994)
  13. 13. BOT • Reduces rating errors (Bernardin & Walter, 1977; Latham, Wexley, & Pursell, 1975) • Leads to increased observational accuracy (Thornton & Zorich, 1980) • Significantly increases rating accuracy (Hedge & Kavanagh, 1988; Noonan & Sulsky, 2001; Pulakos, 1986) • Criticisms – Lack of agreement on what constitutes an observational training program (Noonan & Sulsky, 2001) – Note taking and diary keeping is likely impractical
  14. 14. Combinations of Rater Training Approaches • RET + FORT = no significant increase in rating accuracy (McIntyre et al., 1984; Pulakos, 1984) • RET + other approaches = no increase in rating accuracy (Smith, 1986) • FORT + BOT = no significant increase in rating accuracy beyond FORT alone (Noonan & Sulsky, 2001; Roch & O’Sullivan, 2003)
  15. 15. Summary of Rater Training Research • FORT has become the “go to” training – Although may have limited generalizability • RET is effective – At reducing rating accuracy • Practice and feedback appear to be important components of any successful rater training program (Borman, 2001; Latham, 1986; Smith, 1986) • Accumulation of empirical evidence suggests that rater training programs should be worthwhile interventions for improving ratings in organizations
  16. 16. However…. • Lack of widespread adoption of rater training programs in applied settings (Bernardin, Buckley, Tyler, & Wiese, 2001) – Time consuming and expensive to implement (Stamoulis & Hauenstein, 1993) – Developing target scores for computing rating accuracy indices is complex and time consuming (Bernardin et al., 2001; Ilgen & Favero, 1985) – May be insufficient due to low levels of user acceptance and political influence (Carroll & Schneier, 1982; Longnecker, Gioia, & Sims, 1987) – Has yet to be shown to generalize across jobs and members in organizations (Arvey & Murphy, 1998)
  17. 17. The Present Study • No scholarly evidence of the prevalence and types of rater training programs in organizations today • Some anecdotal evidence – TVA, JP Morgan Chase, Lucent Technologies, AT&T have adopted rater training programs (Levy, 2010) – Employers Resource Council (2008) – 46% of the 73 organizations surveyed provide rater training • Exploratory research question: Is rater training related to organizational performance?
  18. 18. Method • Procedure – Survey part of a larger data collection effort on current performance management practices (Gorman, Ray, Nugent, et al., 2012) – Recruited HR executives to complete survey • Directly e-mailing HR departments in Fortune 500 companies • Advertising on popular online business forums • Asking HR execs to forward survey to other HR execs • Participants • HR executives from 101 U.S. organizations • 88% report revenues of 1 million + dollars • 88% employ at least 100 employees • Largest percentage of the organizations were headquartered in the Southeastern U.S. (44%)
  19. 19. Measures • Rater Training – 8 items (e.g., Does your company train managers how to conduct performance appraisals?) • Performance appraisal system effectiveness – 1 item (Overall, how would you rate the effectiveness of your company’s performance appraisal system?) – 1 (extremely ineffective) to 5 (extremely effective) • Performance appraisal system fairness – 1 item (Overall, how would you rate the fairness of your company’s performance appraisal system?) – 1 (extremely unfair) to 5 (extremely fair) • Firm-level performance – 1 item (Approximately how much revenue does your company make annually?) – 1 (less than $1 million) to 4 (more than $100 million)
  20. 20. Results Do Organizations Conduct Rater Training? Response Train Managers Train Non-Managers Refresher/Recalibration Training Yes 77 31 50 No 24 70 19
  21. 21. Results Frequency of Rater Training Approaches Rater Training Approach Frequency Percent No training 24 23.76% Rater error training 13 12.87% Performance dimension training 23 22.77% Frame-of-reference training 31 30.69% Behavioral observation training 8 7.92% Other 2 1.98%
  22. 22. Results Who Conducts Rater Training? Training Conducted by Frequency Percent External consultant 2 2.63% Internal consultant 2 2.63% Human resource personnel 61 80.26% Department manager 6 7.89% Other 5 6.58%
  23. 23. Results Frequency of Rater Training Frequency of Rater Training Frequency Percent Less than one time a year 6 8.00% One time per year 28 37.33% Two times per year 13 17.33% Three times per year 0 0.00% Four times per year 3 4.00% As needed 25 33.33%
  24. 24. Exploratory Analyses • Control variable: – Company size • Performance appraisal systems that utilize managerial rater training were judged to be more effective (M = 3.70, SD = 1.51) than those that do not (M = 3.37, SD = 1.61), t(98) = 1.77, p < .05.
  25. 25. Exploratory Results • No significant difference in perceived fairness of performance appraisal system • Organizations that utilized managerial rater training generated higher revenue (M = 3.09, SD = 1.03) than those that did not (M = 2.71, SD = 1.04), t(98) = 3.07, p < .01. • Performance appraisal systems were perceived as significantly more legally defensible when the system included a rater training program, χ2(1, N = 101) = 4.13, p < .05.
  26. 26. Rater Training Focus and Perceived Performance Appraisal Effectiveness
  27. 27. Rater Training Focus and Company Revenue
  28. 28. Discussion • Rater training is alive and well – 76% of organizations surveyed utilize managerial rater training – 31% train non-managers – FORT (40%) and PDT (30%) most popular – Preliminary evidence that rater training is linked to firm-level performance
  29. 29. Discussion • Encouraging results – In contrast to the presumed scientist- practitioner gap in performance appraisal (Banks & Murphy, 1985; Bretz, Milkovich, & Read, 1992) – Evidence-based approaches are the predominant rater training methods in use today – Widespread adoption across many organizations and industries
  30. 30. Areas for Improvement • Only 22 of the 77 organizations that offer rater training have evaluated the training • Majority of rater training sessions are only offered either once per year or as needed
  31. 31. Limitations • Single source data • Links between rater training and firm performance are not causal • Training programs in practice may not contain all elements of what is described in the literature • Small number of organizations; may not be generalizable
  32. 32. Thank You! • If you would like a copy of the chapter, please e-mail me – gormanc@etsu.edu

×