A CRITICA...
satisfied or very satisfied with the final         complaint was that the EER was “not done as
outcomes of their own EER R...
differ in their levels of satisfaction with their   boss. In this case “good” is defined by level
own EER Ratings and Revi...
remaining 25% or so chose to interpret their        consider       incorporating      360-degree
Areas for Improvement pos...
perspectives on the employee. Many support         scales could generate. The supervisor’s task
“360-degree” type evaluati...
Upcoming SlideShare
Loading in...5

2006 Kilburg EER Short Form Article 5pgs


Published on

2006 EER Study Short form article. Unofficial research study on State's EER. Abridged article. Cleared on August 17, 2006, by PA/SCP, for publication.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

2006 Kilburg EER Short Form Article 5pgs

  1. 1. THE EER: IS IT TRANSFORMATIONAL? A PSYCHOLOGIST TURNED DIPLOMAT TAKES A CRITICAL LOOK AT THE CURRENT EER SYSTEM Don Kilburg, Ph.D. Transformational Diplomacy work at our missions comes to a grinding halt & the EER for one or two months a year – simply to deal In a key address at Georgetown with the business of evaluating ourselves, and University, Secretary of State Condoleezza apparently ineffectively at that. Are EERs Rice showcased “transformational really worth it? Do they truly support the diplomacy” (January 18, 2006). America goals of transformational diplomacy? needs a “diplomacy that not only reports about the world as it is, but seeks to change An Empirical Approach to the EER the world itself”, the Secretary advised. “We After seeing that no one had done any must transform old diplomatic institutions to empirical research on the EER, I created an serve new diplomatic purposes” and we must on-line survey to collect data on employees’ “prepare” and “challenge” our own diplomats experiences with it. I wanted to know what with “new expertise” and “new expectations.” factors go into getting a good EER and As both a diplomat and a psychologist, it advancing in the Foreign Service. I occurred to me that if we want to advance this hypothesized that one’s experience with the sort of diplomacy, we might need to take a EER is based on much more than just his or critical look at how we formally shape her actual work performance. Rather it is ourselves as a diplomatic force: the Employee based on the circumstances and approaches of Evaluation Report, or EER. Is it the Rater and Reviewer, arbitrary background transformational? features of the employee, the dynamics of the To answer this question, I conducted a EER process, and other extraneous factors. survey of over 600 Foreign Service My survey consisted of 80 questions, employees and their experiences with the mostly multiple-choice. The response rate EER. I began this project after observing a was remarkably high and remarkably common complaint about the EER system comprised of hires from the Diplomatic that suggests it is probably not Readiness Initiative (DRI). I will focus on the transformational. Namely, the system is DRI Generalists here. alleged to be poorly calibrated in its use of This article is a summary of a both praise and criticism. For example, many lengthier, 20-page research report. In the full employees are shown to “walk on water”, report I detail the methodology and a wider others are “damned with faint praise”, and range of findings. I utilized standard survey still others are just victims of unthinking techniques and statistical tools widely evaluators. A system that praises and accepted in the field of psychology. If you criticizes disproportionately would seem to would like a copy of the full report, please ineffectively categorize employees, resulting contact me at: in little transformation of the workforce for the better. Top 10 Key Findings The temptation is to say: “Who cares? 1) Much Seems Good about the Current ‘Corridor reputation’ is what drives the real EER System. transformation of the workforce.” This Most Generalists are satisfied with conclusion may be valid, but it begs another their EERs. A full 86% were either somewhat question. Why spend so much time and satisfied or very satisfied with the final energy on EERs if corridor reputation is what outcomes of their EER Ratings. A full 82% matters? Some estimate progress on other of these Generalists were either somewhat 1
  2. 2. satisfied or very satisfied with the final complaint was that the EER was “not done as outcomes of their own EER Reviews. quickly as it could/should have been” (28%). Further, most Generalists are satisfied Only 2% complained there was “too much with their bosses. Satisfaction with Raters ran criticism.” relatively high among the Generalists. Roughly 79% liked their Raters as bosses at 3) An Unsatisfactory EER System has least “somewhat” and most liked them “very Negative Consequences. much”. Only 21% reported liking their Raters Inaccurate counseling dates mean less as bosses “very little”. As much as 92% liked satisfaction with one’s own EER and with the their Raters as people at least “somewhat” and EER system. While the Generalists were most liked them “very much”. Only 8% quite satisfied with their own EER Ratings reported liking their bosses as people “very and Reviews overall, the more they reported little”. The figures were very similar for their counseling dates as inaccurate and/or Reviewers. their written counseling sessions as absent, the less likely they were to be as satisfied with 2) A Deeper Look Reveals an their Ratings, Reviews, and the EER system. Unsatisfactory EER System. Generalists are less satisfied with their Most Generalists are quite dissatisfied EERs and the EER system when they see with the EER system, despite being quite their bosses as low in proactiveness. The satisfied with their own Ratings and Reviews. Generalists viewing their Raters and A whopping 71% were either neutral about or Reviewers as low in proactiveness were less dissatisfied with the current EER system. The satisfied all around, in terms of their own bulk of the Generalists were either somewhat EER Ratings, EER Reviews, and the EER dissatisfied or very dissatisfied with the system in general. current EER system. Only 29% of the Generalists are less satisfied with their Generalists were satisfied with the current EERs and the EER system when they see EER system. their bosses as low in nurturant leadership. I Procedural regulations for completing measured the extent employees perceived EERs are very poorly followed and Raters their Raters as mentors or coaches and Reviewers are not seen as being “proactively nurturing professional proactive. Only about half of the Generalists development throughout the rating period.” reported their Raters/Reviewers to be In all cases, if employees viewed their Raters proactive in getting their EERs completed. as low in this nurturing factor, they also Only 27% reported that the counseling dates reported lower levels of satisfaction with their on their EERs were accurate and another 27% EER Ratings, Reviews, and with the EER reported their counseling dates did “not at all” system as a whole. correspond to any actual counseling dates. Lastly, dissatisfaction with the EER Only 44% reported getting a written system is related to employees’ career counseling statement, something that is in contentment. Those who had a higher level of theory supposed to document good dissatisfaction with the current EER system performance as much as bad. Lastly, one in had a lower level of interest in the Foreign four of the Generalists reported not getting a Service as a career. If this relationship is Work Requirements Statement on time. causal, dissatisfaction with the EER system The Generalists had many complaints may quite possibly play a part in FS about their EERs: two-thirds commented at resignations. length in the open-ended comment box at the end of the survey. The Generalists’ 4) Gender Matters. complaints were mainly about style, form, Females are less satisfied with their and timeliness. The most common complaint EERs. Though males and females did not was that the EER was “not well-written in differ significantly in their levels of style/form” (32%). The second most common satisfaction with the EER system, they did 2
  3. 3. differ in their levels of satisfaction with their boss. In this case “good” is defined by level own EER Ratings and Reviews. The mean of satisfaction one has with one’s EER. satisfaction level for EER Ratings among One’s likelihood of being more satisfied with males was 3.42; for females it was 3.22 (on a one’s EER as correlated with perceiving scale of 0 to 4). The mean satisfaction level being liked by one’s boss as a person also for EER Reviews among males was 3.43; for contributed to greater satisfaction with the females it was 3.18. EER system. In all statistical analyses of Females are less likely to be tenured EER satisfaction, satisfaction increased as on the first review. Females were reviewed perception of being liked as a person for tenure an average of 1.40 times before increased. getting tenured. In contrast males were reviewed an average of 1.28 times before 7) Your Supervisor’s Writing Style getting tenured. Matters. Rater and Reviewer writing styles are 5) Grade Matters. connected with employee tenure rates: the Newer officers are especially “story” format is the best. I compared dissatisfied with the EER system. Those who satisfaction levels between groups of were relatively new to the FS (with less than employees whose Raters and Reviewers had nine years in) were less satisfied with the written their EERs in “list” versus “story” current EER system than their more format. It has been said that the story format experienced counterparts (with more than is more powerful in that it provides a nine years in). That said, neither group was chronological narrative of events, as opposed very satisfied with the system. The mean to a mere list or inventory of level of satisfaction among the new Officers accomplishments. Those tenured officers was 1.7 on a satisfaction scale of 0 to 4 who had the story format Rating had an (where 2 is neutral), i.e. the new Officers average number of tenure reviews of 1.12, were mainly dissatisfied. The mean level compared with their list format counterparts among more experienced Officers was 2.12; who were tenured on average in 1.38 reviews. i.e. they were neutral. A similar pattern was found for Reviews. 6) Personal Liking Matters. 8) The Area for Improvement (AFI) Needs Being liked as a person means being Improvement. liked as an employee. Employees’ Over a third of the Generalists (35%) perceptions of how much their bosses liked did not think that their Areas for Improvement them as people and how much they liked were germane to their actual performance. them as employees were positively correlated. This would seem to imply there is a relatively Also, employees’ ratings of their bosses as large gap between Raters’ and their people positively correlated with their ratings subordinates’ perceptions of the subordinates’ of their bosses as bosses. In short, there is Areas for Improvement. strong evidence that people do not or cannot Generalists appear disempowered or at readily separate their feelings toward one least disinclined to respond to their AFI another as people and as employees. I.e., if comments. Few chose to explicitly disagree your boss likes you as a person, chances are with their Rater’s assessments: less than 2%. he/she will like you as a subordinate, and vice The bulk (40%) of the Generalists surveyed versa. Further, if your subordinate likes you reported that they chose not to respond to the as a person, chances are he/she will like you Area for Improvement at all in their EER as a boss, and vice versa. statement, preferring to ignore it or to wait Being liked as a person means getting until the next EER to address it. Another a better EER. One’s likelihood of getting a 34% chose to agree with the comments in good EER increases rapidly if one perceives their Areas for Improvement and to grant the he/she is liked on a personal level by his/her items as “something to work on”. The 3
  4. 4. remaining 25% or so chose to interpret their consider incorporating 360-degree Areas for Improvement positively, with a evaluations. “spin” or reframe of the item. Most Generalists would like How Generalists respond to AFI supervisors rated by subordinates. A comments is implicated in rate of tenure: spin remarkable 76% wanted to have evaluations is most rewarded. When asked about the of supervisors, by subordinates. That was the Area for Improvement, those who reported only solid agreement among the Generalists they “interpreted it positively, with a ‘spin’ or on the following 360-degree components. reframing of it” were tenured in the lowest Fifty percent wanted to have evaluations of average number of reviews (1.24), then those Americans by FSNs and/or LES employees. who “agreed with it explicitly, granting it as Forty-five percent wanted to have evaluations something to work on” (1.33), then those who of same-level peers by same-level peers. “did not address it, preferring to ignore it or Only 9% reported they wanted no additional wait until next EER” (1.42), and finally those types of evaluations within the 360-degree who “disagreed with it explicitly, offering a concept. counterargument” (1.50). Generalists are divided on whether quantitative measures should be added to the 9) EERs Are Negotiable. EER and what type. A small majority of 58% Requesting changes in EERs is percent wanted some type of quantitative connected to getting better EERs. It is clear measure added. Twenty-nine percent from comparing EER complaints before and supported scaled “grades” for employees after requests for changes, that employees along each of the six core competencies. have significantly fewer complaints about Twenty-seven percent supported percentile EERs in the end – an estimated 30% less. rankings. Fourteen percent supported The survey respondents collectively had 492 “within-the-person” rankings. Those who did various, reported complaints about their own not support quantitative measures were EER Ratings/Reviews before requesting largely concerned about possible “grade changes. After requesting changes, the inflation”. number of total complaints reported dropped to 349, a difference of 143 or 30%. In sum, it Implications & Concluding Remarks is safe to say that requesting changes from Though the current EER system your Rater/Reviewer can dramatically reduce results in EERs that employees are satisfied to complaints you have about your EER, and in receive, it hardly lends itself to turn probably influence your own transformational diplomacy. My research competitiveness vis-à-vis your peers. strongly confirms the hypothesis that one’s experience with the EER is based on the 10) What Generalists Want: Changes to circumstances and approaches of the Rater the EER System. and Reviewer, arbitrary background features Most Generalists would like a 360- of the employee, the dynamics of the EER degree employee evaluation system process, and other extraneous factors. considered. A whopping 92% of the I submit to you that there are structural Generalists reported that they would like at changes that can be made to the EER system least a “little bit” the Department to consider that would lead to a more transformed changing the EER system to incorporate 360- workforce. This would be a system wherein degree evaluations. The bulk of the promising employees would advance faster respondents (36%) “absolutely” wanted the than their mediocre counterparts and arbitrary Department to consider 360-degree factors affecting evaluation would be evaluations, 25% “very much”, and 20% “a minimized and thereby less influential. moderate amount”. Only 8% reported that For starters, we need an EER system they do “not at all” want the Department to that has both qualitative and quantitative components, as well as multi-dimensional 4
  5. 5. perspectives on the employee. Many support scales could generate. The supervisor’s task “360-degree” type evaluations (and the would then not be so complicated in crafting Department has to its credit begun initiating the perfect Area for Improvement for the some). One would think that supervisors and subordinate. It would be a matter of saying, subordinates alike could better improve if “I see that you got your lowest 360-degree they got more varied types of feedback from a score in X core precept, and here is what I wider range of employees surrounding them propose you do to raise that score.” in the organizational structure. Employees could then of course have the Modern psychology has long held that same, system-generated Areas for qualitative and quantitative components of Improvement across EERs, as they worked to evaluation each contribute critical pieces of address pervasive problems and hence to information that should be viewed jointly. better themselves. Since quantitative components do not involve Ultimately deciding what if anything writing skills, adding one to the EER could be we should do to improve the EER system a useful and efficient means of addressing depends on the answer to the question: what complaints regarding over-emphasis on is our goal? Do we really want to transform Rater/Reviewer writing styles. Concerns our workforce to carry out the work of about “grade inflation” could be mitigated by transformational diplomacy? Or do we want collecting data on average quantitative to continue shaping and reinforcing a “go evaluations given by Raters/Reviewers, in along to get along” workforce? As Secretary order to provide context. Though we should Rice has said, “We must transform old consider adding other measures to the EER, diplomatic institutions to serve new we should also reduce overall the depth of diplomatic purposes.” I submit to you that these measures, shifting instead to more transforming an old EER system is a key frequent, perhaps quarterly evaluations of a component to reaching that goal. smaller, yet wider scale. One way to both enhance and About the Author streamline the EER system could be to design Don Kilburg has been an FSO since 2003. He and implement a new computer program served in Mexico City and is moving onward which could be utilized by randomly selected to Santo Domingo with his wife Keely. He members of a 360-degree rating panel whose holds a doctorate in Experimental Social members confidentially enter both qualitative Psychology from DePaul University and a and quantitative information into secured, on- bachelor’s degree in Research Psychology line employee profiles, in a systematic from the University of Illinois. Before fashion, orchestrated by Human Resources coming into the Foreign Service, he was a sections. Such a digital system would be easy professor at Eastern Washington University to create and could advance our outdated, and more recently at Saint Olaf College. analog system significantly. Regardless of whether a new system should be computerized, we might consider invoking the well-established core precepts as a foundation for additional rating components, to highlight employee strengths within the individual. Prompting questions could be taken directly from the six core precepts to stimulate quantitative, evaluative responses in order to arrive at scaled ratings for employees, within the core precepts and overall. We might even systematically derive Areas for Improvement from the output new 5