THE EER: IS IT TRANSFORMATIONAL?
A PSYCHOLOGIST TURNED DIPLOMAT TAKES
A CRITICAL LOOK AT THE CURRENT EER SYSTEM
Don Kilburg, Ph.D.
Transformational Diplomacy work at our missions comes to a grinding halt
& the EER for one or two months a year – simply to deal
In a key address at Georgetown with the business of evaluating ourselves, and
University, Secretary of State Condoleezza apparently ineffectively at that. Are EERs
Rice showcased “transformational really worth it? Do they truly support the
diplomacy” (January 18, 2006). America goals of transformational diplomacy?
needs a “diplomacy that not only reports
about the world as it is, but seeks to change An Empirical Approach to the EER
the world itself”, the Secretary advised. “We After seeing that no one had done any
must transform old diplomatic institutions to empirical research on the EER, I created an
serve new diplomatic purposes” and we must on-line survey to collect data on employees’
“prepare” and “challenge” our own diplomats experiences with it. I wanted to know what
with “new expertise” and “new expectations.” factors go into getting a good EER and
As both a diplomat and a psychologist, it advancing in the Foreign Service. I
occurred to me that if we want to advance this hypothesized that one’s experience with the
sort of diplomacy, we might need to take a EER is based on much more than just his or
critical look at how we formally shape her actual work performance. Rather it is
ourselves as a diplomatic force: the Employee based on the circumstances and approaches of
Evaluation Report, or EER. Is it the Rater and Reviewer, arbitrary background
transformational? features of the employee, the dynamics of the
To answer this question, I conducted a EER process, and other extraneous factors.
survey of over 600 Foreign Service My survey consisted of 80 questions,
employees and their experiences with the mostly multiple-choice. The response rate
EER. I began this project after observing a was remarkably high and remarkably
common complaint about the EER system comprised of hires from the Diplomatic
that suggests it is probably not Readiness Initiative (DRI). I will focus on the
transformational. Namely, the system is DRI Generalists here.
alleged to be poorly calibrated in its use of This article is a summary of a
both praise and criticism. For example, many lengthier, 20-page research report. In the full
employees are shown to “walk on water”, report I detail the methodology and a wider
others are “damned with faint praise”, and range of findings. I utilized standard survey
still others are just victims of unthinking techniques and statistical tools widely
evaluators. A system that praises and accepted in the field of psychology. If you
criticizes disproportionately would seem to would like a copy of the full report, please
ineffectively categorize employees, resulting contact me at: firstname.lastname@example.org.
in little transformation of the workforce for
the better. Top 10 Key Findings
The temptation is to say: “Who cares? 1) Much Seems Good about the Current
‘Corridor reputation’ is what drives the real EER System.
transformation of the workforce.” This Most Generalists are satisfied with
conclusion may be valid, but it begs another their EERs. A full 86% were either somewhat
question. Why spend so much time and satisfied or very satisfied with the final
energy on EERs if corridor reputation is what outcomes of their EER Ratings. A full 82%
matters? Some estimate progress on other of these Generalists were either somewhat
satisfied or very satisfied with the final complaint was that the EER was “not done as
outcomes of their own EER Reviews. quickly as it could/should have been” (28%).
Further, most Generalists are satisfied Only 2% complained there was “too much
with their bosses. Satisfaction with Raters ran criticism.”
relatively high among the Generalists.
Roughly 79% liked their Raters as bosses at 3) An Unsatisfactory EER System has
least “somewhat” and most liked them “very Negative Consequences.
much”. Only 21% reported liking their Raters Inaccurate counseling dates mean less
as bosses “very little”. As much as 92% liked satisfaction with one’s own EER and with the
their Raters as people at least “somewhat” and EER system. While the Generalists were
most liked them “very much”. Only 8% quite satisfied with their own EER Ratings
reported liking their bosses as people “very and Reviews overall, the more they reported
little”. The figures were very similar for their counseling dates as inaccurate and/or
Reviewers. their written counseling sessions as absent,
the less likely they were to be as satisfied with
2) A Deeper Look Reveals an their Ratings, Reviews, and the EER system.
Unsatisfactory EER System. Generalists are less satisfied with their
Most Generalists are quite dissatisfied EERs and the EER system when they see
with the EER system, despite being quite their bosses as low in proactiveness. The
satisfied with their own Ratings and Reviews. Generalists viewing their Raters and
A whopping 71% were either neutral about or Reviewers as low in proactiveness were less
dissatisfied with the current EER system. The satisfied all around, in terms of their own
bulk of the Generalists were either somewhat EER Ratings, EER Reviews, and the EER
dissatisfied or very dissatisfied with the system in general.
current EER system. Only 29% of the Generalists are less satisfied with their
Generalists were satisfied with the current EERs and the EER system when they see
EER system. their bosses as low in nurturant leadership. I
Procedural regulations for completing measured the extent employees perceived
EERs are very poorly followed and Raters their Raters as mentors or coaches
and Reviewers are not seen as being “proactively nurturing professional
proactive. Only about half of the Generalists development throughout the rating period.”
reported their Raters/Reviewers to be In all cases, if employees viewed their Raters
proactive in getting their EERs completed. as low in this nurturing factor, they also
Only 27% reported that the counseling dates reported lower levels of satisfaction with their
on their EERs were accurate and another 27% EER Ratings, Reviews, and with the EER
reported their counseling dates did “not at all” system as a whole.
correspond to any actual counseling dates. Lastly, dissatisfaction with the EER
Only 44% reported getting a written system is related to employees’ career
counseling statement, something that is in contentment. Those who had a higher level of
theory supposed to document good dissatisfaction with the current EER system
performance as much as bad. Lastly, one in had a lower level of interest in the Foreign
four of the Generalists reported not getting a Service as a career. If this relationship is
Work Requirements Statement on time. causal, dissatisfaction with the EER system
The Generalists had many complaints may quite possibly play a part in FS
about their EERs: two-thirds commented at resignations.
length in the open-ended comment box at the
end of the survey. The Generalists’ 4) Gender Matters.
complaints were mainly about style, form, Females are less satisfied with their
and timeliness. The most common complaint EERs. Though males and females did not
was that the EER was “not well-written in differ significantly in their levels of
style/form” (32%). The second most common satisfaction with the EER system, they did
differ in their levels of satisfaction with their boss. In this case “good” is defined by level
own EER Ratings and Reviews. The mean of satisfaction one has with one’s EER.
satisfaction level for EER Ratings among One’s likelihood of being more satisfied with
males was 3.42; for females it was 3.22 (on a one’s EER as correlated with perceiving
scale of 0 to 4). The mean satisfaction level being liked by one’s boss as a person also
for EER Reviews among males was 3.43; for contributed to greater satisfaction with the
females it was 3.18. EER system. In all statistical analyses of
Females are less likely to be tenured EER satisfaction, satisfaction increased as
on the first review. Females were reviewed perception of being liked as a person
for tenure an average of 1.40 times before increased.
getting tenured. In contrast males were
reviewed an average of 1.28 times before 7) Your Supervisor’s Writing Style
getting tenured. Matters.
Rater and Reviewer writing styles are
5) Grade Matters. connected with employee tenure rates: the
Newer officers are especially “story” format is the best. I compared
dissatisfied with the EER system. Those who satisfaction levels between groups of
were relatively new to the FS (with less than employees whose Raters and Reviewers had
nine years in) were less satisfied with the written their EERs in “list” versus “story”
current EER system than their more format. It has been said that the story format
experienced counterparts (with more than is more powerful in that it provides a
nine years in). That said, neither group was chronological narrative of events, as opposed
very satisfied with the system. The mean to a mere list or inventory of
level of satisfaction among the new Officers accomplishments. Those tenured officers
was 1.7 on a satisfaction scale of 0 to 4 who had the story format Rating had an
(where 2 is neutral), i.e. the new Officers average number of tenure reviews of 1.12,
were mainly dissatisfied. The mean level compared with their list format counterparts
among more experienced Officers was 2.12; who were tenured on average in 1.38 reviews.
i.e. they were neutral. A similar pattern was found for Reviews.
6) Personal Liking Matters. 8) The Area for Improvement (AFI) Needs
Being liked as a person means being Improvement.
liked as an employee. Employees’ Over a third of the Generalists (35%)
perceptions of how much their bosses liked did not think that their Areas for Improvement
them as people and how much they liked were germane to their actual performance.
them as employees were positively correlated. This would seem to imply there is a relatively
Also, employees’ ratings of their bosses as large gap between Raters’ and their
people positively correlated with their ratings subordinates’ perceptions of the subordinates’
of their bosses as bosses. In short, there is Areas for Improvement.
strong evidence that people do not or cannot Generalists appear disempowered or at
readily separate their feelings toward one least disinclined to respond to their AFI
another as people and as employees. I.e., if comments. Few chose to explicitly disagree
your boss likes you as a person, chances are with their Rater’s assessments: less than 2%.
he/she will like you as a subordinate, and vice The bulk (40%) of the Generalists surveyed
versa. Further, if your subordinate likes you reported that they chose not to respond to the
as a person, chances are he/she will like you Area for Improvement at all in their EER
as a boss, and vice versa. statement, preferring to ignore it or to wait
Being liked as a person means getting until the next EER to address it. Another
a better EER. One’s likelihood of getting a 34% chose to agree with the comments in
good EER increases rapidly if one perceives their Areas for Improvement and to grant the
he/she is liked on a personal level by his/her items as “something to work on”. The
remaining 25% or so chose to interpret their consider incorporating 360-degree
Areas for Improvement positively, with a evaluations.
“spin” or reframe of the item. Most Generalists would like
How Generalists respond to AFI supervisors rated by subordinates. A
comments is implicated in rate of tenure: spin remarkable 76% wanted to have evaluations
is most rewarded. When asked about the of supervisors, by subordinates. That was the
Area for Improvement, those who reported only solid agreement among the Generalists
they “interpreted it positively, with a ‘spin’ or on the following 360-degree components.
reframing of it” were tenured in the lowest Fifty percent wanted to have evaluations of
average number of reviews (1.24), then those Americans by FSNs and/or LES employees.
who “agreed with it explicitly, granting it as Forty-five percent wanted to have evaluations
something to work on” (1.33), then those who of same-level peers by same-level peers.
“did not address it, preferring to ignore it or Only 9% reported they wanted no additional
wait until next EER” (1.42), and finally those types of evaluations within the 360-degree
who “disagreed with it explicitly, offering a concept.
counterargument” (1.50). Generalists are divided on whether
quantitative measures should be added to the
9) EERs Are Negotiable. EER and what type. A small majority of 58%
Requesting changes in EERs is percent wanted some type of quantitative
connected to getting better EERs. It is clear measure added. Twenty-nine percent
from comparing EER complaints before and supported scaled “grades” for employees
after requests for changes, that employees along each of the six core competencies.
have significantly fewer complaints about Twenty-seven percent supported percentile
EERs in the end – an estimated 30% less. rankings. Fourteen percent supported
The survey respondents collectively had 492 “within-the-person” rankings. Those who did
various, reported complaints about their own not support quantitative measures were
EER Ratings/Reviews before requesting largely concerned about possible “grade
changes. After requesting changes, the inflation”.
number of total complaints reported dropped
to 349, a difference of 143 or 30%. In sum, it Implications & Concluding Remarks
is safe to say that requesting changes from Though the current EER system
your Rater/Reviewer can dramatically reduce results in EERs that employees are satisfied to
complaints you have about your EER, and in receive, it hardly lends itself to
turn probably influence your own transformational diplomacy. My research
competitiveness vis-à-vis your peers. strongly confirms the hypothesis that one’s
experience with the EER is based on the
10) What Generalists Want: Changes to circumstances and approaches of the Rater
the EER System. and Reviewer, arbitrary background features
Most Generalists would like a 360- of the employee, the dynamics of the EER
degree employee evaluation system process, and other extraneous factors.
considered. A whopping 92% of the I submit to you that there are structural
Generalists reported that they would like at changes that can be made to the EER system
least a “little bit” the Department to consider that would lead to a more transformed
changing the EER system to incorporate 360- workforce. This would be a system wherein
degree evaluations. The bulk of the promising employees would advance faster
respondents (36%) “absolutely” wanted the than their mediocre counterparts and arbitrary
Department to consider 360-degree factors affecting evaluation would be
evaluations, 25% “very much”, and 20% “a minimized and thereby less influential.
moderate amount”. Only 8% reported that For starters, we need an EER system
they do “not at all” want the Department to that has both qualitative and quantitative
components, as well as multi-dimensional
perspectives on the employee. Many support scales could generate. The supervisor’s task
“360-degree” type evaluations (and the would then not be so complicated in crafting
Department has to its credit begun initiating the perfect Area for Improvement for the
some). One would think that supervisors and subordinate. It would be a matter of saying,
subordinates alike could better improve if “I see that you got your lowest 360-degree
they got more varied types of feedback from a score in X core precept, and here is what I
wider range of employees surrounding them propose you do to raise that score.”
in the organizational structure. Employees could then of course have the
Modern psychology has long held that same, system-generated Areas for
qualitative and quantitative components of Improvement across EERs, as they worked to
evaluation each contribute critical pieces of address pervasive problems and hence to
information that should be viewed jointly. better themselves.
Since quantitative components do not involve Ultimately deciding what if anything
writing skills, adding one to the EER could be we should do to improve the EER system
a useful and efficient means of addressing depends on the answer to the question: what
complaints regarding over-emphasis on is our goal? Do we really want to transform
Rater/Reviewer writing styles. Concerns our workforce to carry out the work of
about “grade inflation” could be mitigated by transformational diplomacy? Or do we want
collecting data on average quantitative to continue shaping and reinforcing a “go
evaluations given by Raters/Reviewers, in along to get along” workforce? As Secretary
order to provide context. Though we should Rice has said, “We must transform old
consider adding other measures to the EER, diplomatic institutions to serve new
we should also reduce overall the depth of diplomatic purposes.” I submit to you that
these measures, shifting instead to more transforming an old EER system is a key
frequent, perhaps quarterly evaluations of a component to reaching that goal.
smaller, yet wider scale.
One way to both enhance and About the Author
streamline the EER system could be to design Don Kilburg has been an FSO since 2003. He
and implement a new computer program served in Mexico City and is moving onward
which could be utilized by randomly selected to Santo Domingo with his wife Keely. He
members of a 360-degree rating panel whose holds a doctorate in Experimental Social
members confidentially enter both qualitative Psychology from DePaul University and a
and quantitative information into secured, on- bachelor’s degree in Research Psychology
line employee profiles, in a systematic from the University of Illinois. Before
fashion, orchestrated by Human Resources coming into the Foreign Service, he was a
sections. Such a digital system would be easy professor at Eastern Washington University
to create and could advance our outdated, and more recently at Saint Olaf College.
analog system significantly.
Regardless of whether a new system
should be computerized, we might consider
invoking the well-established core precepts as
a foundation for additional rating
components, to highlight employee strengths
within the individual. Prompting questions
could be taken directly from the six core
precepts to stimulate quantitative, evaluative
responses in order to arrive at scaled ratings
for employees, within the core precepts and
We might even systematically derive
Areas for Improvement from the output new