Evaluating the Work of Applied Animal Behaviorists as Applied.docx

Evaluating the Work of Applied Animal Behaviorists as Applied
Behavior Analysis
Jenna M. Gray and James W. Diller
Eastern Connecticut State University
Within the field of behavior analysis, there is a divide between
human and nonhuman
animal research and application. Although behavior-analytic
techniques could be used
with nonhuman animals, to become a board-certified behavior
analyst (BCBA), non-
human animal practice is not allowed. Animal behavior
professionals (e.g., certified
applied animal behaviorists, certified professional dog trainers,
animal behavior con-
sultants) and BCBAs completed a 31-item survey to evaluate the
extent to which their
professional activities meet the definition of applied behavior
analysis and the extent to
which they are consistent with the Behavior Analyst
Certification Board (BACB) task
list. Although there are differences in the techniques being
implemented, the survey
yielded more similarities than differences in the work of these
two groups of profes-
sionals. Both animal behavior professionals and BCBAs are
engaging in work that
meets the standards of the BACB’s task list and the definition
of applied behavior
analysis.

Keywords: basic and applied research, board-certified behavior
analyst, certified
applied animal behaviorist, certified professional dog trainers,
International Association
of Animal Behavior Consultants
In 1958, the Journal of the Experimental
Analysis of Behavior was founded (Laties,
2008), marking the formalization of behavior
analysis. Since its inception, behavior analysis
has grown into a sophisticated and well-
developed discipline. The experimental analysis
of behavior (EAB) and applied behavior analy-
sis (ABA) are empirical branches of this field,
supported by the underlying philosophy of be-
haviorism. EAB uses data to identify behavioral
principles within a laboratory setting, frequently
with nonhuman animal subjects (Mace &
Critchfield, 2010). ABA is the practice of uti-
lizing principles of behavior to solve socially
significant problems (Baer, Wolf, & Risley,
1968, 1987). Although distinct, each area of
research is heavily influenced by the other, and
they share common assumptions via their
shared philosophy of behaviorism and their sta-
tus as science.
Although less than 10% of psychological
journals have dealt with animal behavior, the
contribution that animal research has made in
the field of behavior analysis is astounding (Lat-
tal, 2001). Many behavior-analytic techniques
were identified using nonhuman animals before
being implemented with humans. For instance,
Skinner’s pigeons are known both within be-

havior analysis and in the general population.
Without the research Skinner conducted on
nonhuman animals, the field of ABA may never
have developed basic techniques such as rein-
forcement, punishment, and generalization
(Dymond, Roche, & Barnes-Holmes, 2003).
Thus, the nonhuman animal roots of the science
of behavior analysis directly inform its applica-
tion.
Despite the historical demonstration of be-
havioral principles using nonhuman subjects,
there is a divide in the application of behavior-
analytic techniques between practitioners who
work with humans and those who work with
nonhuman animals. On the human side, we have
This article was published Online First May 19, 2016.
Jenna M. Gray and James W. Diller, Department of
Psychology, Eastern Connecticut State University.
Portions of these data were presented at the 12th annual
meeting of the Connecticut Association for Behavior Anal-
ysis, New Haven, Connecticut held on March 3 and 4, 2016.
Correspondence concerning this article should be ad-
dressed to James W. Diller, Department of Psychology,
Eastern Connecticut State University, 83 Windham Street,
Willimantic, CT 06226. E-mail: [email protected]
T
hi
s
do

cu
m
en
t
is
co
py
ri
gh
te
d
by
th
e
A
m
er
ic
an
P
sy
ch
ol
og

ic
al
A
ss
oc
ia
ti
on
or
on
e
of
it
s
al
li
ed
pu
bl
is
he
rs
.
T

hi
s
ar
ti
cl
e
is
in
te
nd
ed
so
le
ly
fo
r
th
e
pe
rs
on
al
us
e

of
th
e
in
di
vi
du
al
us
er
an
d
is
no
t
to
be
di
ss
em
in
at
ed
br

oa
dl
y.
Behavior Analysis: Research and Practice © 2016 American
Psychological Association
2017, Vol. 17, No. 1, 33– 41 2372-9414/17/$12.00
http://dx.doi.org/10.1037/bar0000041
33
mailto:[email protected]
http://dx.doi.org/10.1037/bar0000041
board-certified behavior analysts (BCBAs),
who are governed by the Behavior Analysis
Certification Board (BACB). The BACB sets
the requirements for the BCBA credential via its
task list (BACB, 2012) to ensure consistency
within the training of behavior analysts.
On the nonhuman side of behavioral problem
solving, we have a variety of professionals,
including applied animal behaviorists and cer-
tified dog trainers. The purpose of the present
study is to evaluate (a) the extent to which the
work of these animal behavior professionals is
consistent with the definition of applied behav-
ior analysis (cf. Baer et al. 1968, 1987), (b) the
extent to which the animal training commun-
ity’s practice reflects elements of the BACB’s
task list, and (c) the extent to which the tasks
performed by animal behavior professionals are
similar to work done by BCBAs.

To obtain certification within the field of be-
havior analysis, the BACB specifies that an
individual must meet certain requirements be-
yond classroom instruction. For instance, she or
he may have to complete 1,500 hours of super-
vised independent fieldwork in behavior analy-
sis. Appropriate clients to gain this experience
are defined as “any persons for whom behavior-
analytic services are appropriate” (BACB,
2015). By stating that clients must be persons,
work with nonhuman animals, even if that work
is behavior-analytic, does not count toward cer-
tification requirements. Thus, people who work
with nonhuman animals can obtain an alterna-
tive credential, certified applied animal behav-
iorist (CAAB), or become a certified profes-
sional dog trainer. To obtain the CAAB
certification, a person must demonstrate profi-
ciency in (a) the theoretical ideologies of animal
behavior, (b) the practice of performing exper-
imental research on the actions of animals either
in a laboratory setting or in a real-world envi-
ronment, (c) the use of animal behavioral tech-
niques to solve problems, and (d) the distribu-
tion of correct information regarding animal
behavior by teaching and research (Applied An-
imal Behaviorists, 2015).
Both BCBA and CAAB apply research-based
techniques. Additionally, both BCBAs and
CAABs have the primary goal of using those
procedures to solve significant behavioral prob-
lems. If an individual can effectively use differ-
ential reinforcement to resolve a dog’s problem
behavior, it seems logical that he or she could

also implement an equivalent procedure with a
human being. That ability to be able to apply
knowledge in multiple populations (e.g., human
and canine) is a goal within the field of behavior
analysis (Baer et al., 1968, 1987). In a discus-
sion of such cross-species application, Bailey
and Burch (2006) caution that “dogs are a dif-
ferent species from children with autism, so you
will need species-specific training” (pp. 75–76).
Such species-specific training might include in-
formation about the phylogenic history of the
organism and build skills associated with iden-
tifying relevant species-specific stimuli (e.g.,
reinforcers). Although the procedures used with
human and nonhuman animals might differ—
your typically developing college students
would probably be baffled if you used a clicker
when they responded correctly in class, while
your border collie might respond with zeal—
but the foundational principles (e.g., reinforce-
ment) are the same.
Given that there are similarities in the applied
goals of CAABs and BCBAs, this study is an
evaluation of the extent to which the practice of
animal behavior professionals is consistent with
the practice of BCBAs. This analysis was de-
signed to evaluate the prevalence of techniques
used and, to a lesser extent, the principles that
underlie them. An electronic survey was distrib-
uted to CAABs, certified dog trainers, and
BCBAs. The survey consisted of questions
based on the seven dimensions of applied be-
havior analysis (Baer et al., 1968, 1987) and
items on the current version of the BCBA task

list (BACB, 2012). This survey was designed to
evaluate the extent to which the practice of
these professionals is consistent, as well as
where it differs, to situate the practice of applied
behavior analysis among related fields.
Method
Participants
Surveys were distributed electronically to an-
imal behavior professionals and BCBAs. Ani-
mal behavior professionals is an inclusive term
encompassing CAABs, certified professional
dog trainers, and also International Association
of Animal Behavior consultants. These partici-
pants were found using directories available on
animal behavior professional websites.
34 GRAY AND DILLER
T
hi
s
do
cu
m
en
t
is
co

py
ri
gh
te
d
by
th
e
A
m
er
ic
an
P
sy
ch
ol
og
ic
al
A
ss
oc
ia

ti
on
or
on
e
of
it
s
al
li
ed
pu
bl
is
he
rs
.
T
hi
s
ar
ti
cl
e

is
in
te
nd
ed
so
le
ly
fo
r
th
e
pe
rs
on
al
us
e
of
th
e
in
di
vi

du
al
us
er
an
d
is
no
t
to
be
di
ss
em
in
at
ed
br
oa
dl
y.
In total, 145 animal behavior professionals
were contacted via email solicitation by the first

author. Of the 145 potential participants, 75
responded, yielding a response rate of 52%,
which is quite high for electronic survey re-
search (cf. Coughlin et al., 2011). This sample
was predominately female (n � 67) and Cau-
casian (n � 70) with a mean age of 49.16 years
(SD � 10.95). The mean for number of years in
their field was 16.60 (SD � 9.75).
A total of 55 behavior analysts completed this
survey: 4 with the Board Certified Assistant
Behavior Analyst (BCaBA) credential, 43 with
the BCBA credential, and 8 with the Board Cer-
tified Behavior Analyst-Doctoral (BCBA-D) cre-
dential. These participants were recruited via
email solicitation using the mailing list of a
state professional organization in the north-
eastern United States and by direct solicita-
tion by the second author. This sample con-
sisted of 45 females and 10 males and was
predominately Caucasian (n � 50). The mean
age of BCBAs was 38.11 (SD � 11.38), with
an average of 12.66 years (SD � 9.39) work-
ing in the field.
Survey
A 31-item survey was delivered electroni-
cally to participants via email solicitation.
Twenty-four items were based on the BCBA
task list. An example of such an item is, “How
often in your daily work do you plan for un-
wanted effects of your intervention?” This sur-
vey was based on the requirements within the
BACB task list. Twenty-four items describing
behavioral principles, techniques, and elements

of the ethical code were included. The survey
items were selected by the authors in an attempt
to develop a set of questions that cover the
broad spectrum of behavior-analytic practices.
However, items related to verbal behavior were
excluded because one set of professionals deals
exclusively with nonverbal clients (i.e., nonhu-
man animals). Other questions within the sur-
vey were derived from the seven dimensions of
behavior as stated by Baer et al. (1968, 1987).
For instance, the item “How often in your daily
work do you deal with problems that are so-
cially important?” originated from the applied
dimension of applied behavior analysis. The
survey consisted of a 5-point Likert-type scale
in which 1 � never, 2 � sometimes, 3 � neu-
tral, 4 � often, and 5 � very often.
For analysis and interpretation, the questions
were divided into three groups: principles and
techniques, research, and social validity. The
category of principles and techniques consisted
of survey items regarding behavioral principles
(e.g., reinforcement, punishment) and tech-
niques that could be implemented with a variety
of clients, including shaping, errorless learning,
conditioned reinforcement, and so forth. The
research category was composed of items con-
cerning research strategies and procedures, such
as organizing data and exercising control over a
particular behavior. The social validity category
consisted of questions that were concerned with
how these individuals are affecting society or
interacting with other people; items in this sec-
tion asked about consideration of client prefer-

ences and arranging for orderly termination of
services.
In addition to asking the participants about
the type of work they perform, demographic
information was also collected, including age,
gender, ethnicity, and number of years working
in their particular field. The participants solic-
ited via the state professional organization mail-
ing list were instructed to specify their creden-
tial (BCBA-D, BCBA, BCaBA, other, or none,
to filter out individuals without a relevant cre-
dential who may have responded [e.g., stu-
dents]). All procedures were approved by our
university’s institutional review board.
Results
The frequencies of answers to each item for
both animal behavior professionals and BCBAs
are presented in Table 1. The table contains the
percentage of responses in each category, so it
is possible to compare the responses between
the unequal groups. For instance, for the ques-
tion “How often in your daily work do you use
reinforcement or reward?” 70 (95.9%) of the
animal behavior professionals and 49 BCBAs
(69.1%) indicated that they use reinforcement
very often.
Chi-square tests for independence were used
to evaluate if there were differences between the
distribution of responses across the categories
(i.e., never to very often) between the behavior
analysts and the animal behavior professionals.
Table 2 presents the results of the chi-square

35EVALUATING THE WORK OF ANIMAL BEHAVIORISTS
T
hi
s
do
cu
m
en
t
is
co
py
ri
gh
te
d
by
th
e
A
m
er
ic

an
P
sy
ch
ol
og
ic
al
A
ss
oc
ia
ti
on
or
on
e
of
it
s
al
li
ed
pu

bl
is
he
rs
.
T
hi
s
ar
ti
cl
e
is
in
te
nd
ed
so
le
ly
fo
r
th
e

pe
rs
on
al
us
e
of
th
e
in
di
vi
du
al
us
er
an
d
is
no
t
to
be
di

ss
em
in
at
ed
br
oa
dl
y.
Table 1
Percent of Survey Responses in Each Category
Animal behavior
professionals
Board-certified behavior
analysts
1 2 3 4 5 1 2 3 4 5
How often in your daily work do you
Principles and techniques
Measure frequency (rate, duration, etc.) of a certain
behavior? 5.3 9.3 20.0 22.7 42.7 0 1.8 9.1 20.0 69.1
Use reinforcement or reward? 0 1.4 0 2.7 95.9 0 0 1.8 9.1 89.1
Use punishment? 10.8 63.5 13.5 8.1 4.1 9.1 63.6 18.2 3.6 5.5
Use shaping? 0 8.0 10.7 26.7 54.7 0 3.6 16.4 25.5 54.5

Use extinction? 1.3 30.7 16.0 28.0 24.0 1.8 9.1 20.0 21.8 47.3
Use errorless learning procedures? 11.3 12.7 21.1 23.9 31.0 5.5
7.3 10.9 27.3 49.1
Use response-independent (time-based) schedules of
reinforcement (i.e., noncontingent reinforcement)? 12.0 36.0
18.7 16.0 17.3 3.6 12.7 23.6 36.4 23.6
Use other conditioned reinforcement systems (e.g.,
clickers)? 2.7 4.0 10.7 14.7 68.0 21.8 16.4 18.2 23.6 20.0
Use procedures that arise from a specific and
identifiable theoretical base rather than being a set
of packages or tricks? 0 1.4 6.8 23.0 68.9 0 3.7 1.9 27.8 66.7
Deal with behavior that can be measured? 1.3 5.3 12.0 24.0 57.3
0 0 1.8 12.7 85.5
Program for maintenance of a behavior (ensuring the
behavior change will last once the intervention has
ended)? 1.3 0 10.7 32.0 56.0 0 3.7 13.0 33.3 50.0
Define behavior in observable and measurable terms? 0 1.4 5.4
24.3 68.9 0 0 5.5 16.4 78.2
State intervention goals in observable and measurable
terms? 2.7 5.4 12.2 31.1 48.6 0 3.6 3.6 29.1 63.6
Consider biological/medical variables that may be
affecting the client? 0 0 6.9 15.3 77.8 1.8 1.8 12.7 29.1 54.5
Research
Review and interpret research literature? 2.7 27.0 17.6 24.3
28.4 1.8 21.8 30.9 34.5 10.9
Design data collection systems? 20.0 24.0 28.0 14.7 13.3 0 7.3
21.8 32.7 38.2

Plot data that you have collected? 33.8 37.8 13.5 9.5 5.4 0 5.5
9.1 34.5 50.9
Interpret data you have collected? 12.2 10.8 21.6 23.0 32.4 0
3.6 3.6 29.1 63.6
Organize, analyze, and interpret observed data? 8.0 6.7 18.7
29.3 37.3 0 5.5 3.6 27.3 63.6
Arrange independent variables to demonstrate their
effects on dependent variables? 20.3 14.9 16.2 20.3 28.4 1.9
16.7 16.7 27.8 37.0
Procedures or techniques produce strong results? 0 0 2.7 44.0
53.3 0 0 5.5 60.0 34.5
Exercise control over behavior? (how often can you
control a behavior to occur or not) 0 4.1 14.9 36.5 44.6 1.8 3.6
18.2 45.5 30.9
Describe your procedures well enough so that
someone else can replicate or copy your
procedure? 0 2.7 4.0 36.0 57.3 0 1.8 10.9 27.3 60.0
Evaluate the effectiveness of the behavioral program? 0 0 8.2
28.8 63.0 0 1.8 7.3 36.4 54.5
Social validity
Evaluate the accuracy or reliability of your
measurement procedures? 10.7 24.0 22.7 26.7 16.0 1.8 14.5 34.5
32.7 16.4
Deal with problems that are socially important? 2.7 1.3 9.3 33.3
53.3 0 0 0 12.7 87.3
Plan for any possible unwanted effects of your
intervention? 1.4 8.1 9.5 28.4 52.7 1.8 10.9 20.0 34.5 32.7
Explain concepts using nontechnical language? 0 0 5.3 14.7
80.0 0 3.6 7.3 21.8 67.3

Procedures result in behaviors occurring in new
environments and continuing after the formal
treatments have ended? 0 6.8 9.6 50.7 32.9 0 0 23.6 40.0 36.4
Arrange for the orderly termination of services when
they are no longer required? 6.8 16.2 20.3 16.2 40.5 3.6 20.0
45.5 18.2 12.7
Select intervention strategies based on client
preferences? 2.7 13.7 15.1 37.0 31.5 0 0 21.8 41.8 36.4
36 GRAY AND DILLER
T
hi
s
do
cu
m
en
t
is
co
py
ri
gh
te
d

by
th
e
A
m
er
ic
an
P
sy
ch
ol
og
ic
al
A
ss
oc
ia
ti
on
or
on
e

of
it
s
al
li
ed
pu
bl
is
he
rs
.
T
hi
s
ar
ti
cl
e
is
in
te
nd
ed

so
le
ly
fo
r
th
e
pe
rs
on
al
us
e
of
th
e
in
di
vi
du
al
us
er
an

d
is
no
t
to
be
di
ss
em
in
at
ed
br
oa
dl
y.
analysis for each item, grouped by category. Of
the 31 survey items, only 14 items yielded sig-
nificant results when a chi-square test was con-
ducted comparing the groups, suggesting that
these two groups of professionals responded in
a similar way for the majority of the items.
The category regarding behavioral principles
and techniques contained 14 survey items. Of

those, significant differences in response pat-
terns between groups were found for five of the
questions. For example, for the survey item
asking, “How often in your daily work do you
use other conditioned reinforcement systems
(e.g., clickers)?” a significant effect was found,
�2(4, n � 130) � 34.07, p � .01, with animal
behavior professionals indicating they are more
Table 2
Chi-Square Values for Survey Items
Survey question n
Chi-square
value
How often in your daily work do you/your
Principles and techniques
Measure frequency (rate, duration, etc.) of a certain behavior?
130 12.52�
Use reinforcement or reward? 128 4.55
Use punishment? 129 1.69
Use shaping? 130 1.79
Use extinction? 130 12.75�
Use errorless learning procedures? 126 6.77
Use response-independent (time-based) schedules of
reinforcement
(i.e., noncontingent reinforcement)? 130 15.55��
Use other conditioned reinforcement systems (e.g., clickers)?

130 34.07��
Use procedures that arise from a specific and identifiable
theoretical
base rather than being a set of packages or tricks? 128 2.65
Deal with behavior that can be measured? 130 13.66��
Program for maintenance of a behavior (ensuring the behavior
change
will last once the intervention has ended)? 129 3.87
Define behavior in observable and measurable terms? 129 2.07
State intervention goals in observable and measurable terms?
129 5.72
Consider biological/medical variables that may be affecting the
client? 127 9.01
Research
Review and interpret research literature? 129 8.62
Design data collection systems? 130 29.58��
Plot data that you have collected? 129 69.07��
Interpret data you have collected? 129 23.28��
Organize, analyze, and interpret observed data? 130 14.88��
Arrange independent variables to demonstrate their effects on
dependent variables? 128 10.02�
Procedures or techniques produce strong results? 130 4.71
Exercise control over behavior? (how often can you control a
behavior to occur or not) 129 3.73
Describe your procedures well enough so that someone else can

replicate or copy your procedure? 130 3.07
Evaluate the effectiveness of the behavioral program? 128 2.31
Social validity
Evaluate the accuracy or reliability of your measurement
procedures? 130 7.03
Deal with problems that are socially important? 130 18.21��
Plan for any possible unwanted effects of your intervention?
129 6.06
Explain concepts using nontechnical language? 130 4.53
Procedures result in behaviors occurring in new environments
and
continuing after the formal treatments have ended? 128 8.62�
Arrange for the orderly termination of services when they are
no
longer required? 129 15.85��
Select intervention strategies based on client preferences? 128
10.24�
� p � .05. �� p � .01.
T
hi
s
do
cu
m

en
t
is
co
py
ri
gh
te
d
by
th
e
A
m
er
ic
an
P
sy
ch
ol
og
ic
al

A
ss
oc
ia
ti
on
or
on
e
of
it
s
al
li
ed
pu
bl
is
he
rs
.
T
hi
s

ar
ti
cl
e
is
in
te
nd
ed
so
le
ly
fo
r
th
e
pe
rs
on
al
us
e
of
th

e
in
di
vi
du
al
us
er
an
d
is
no
t
to
be
di
ss
em
in
at
ed
br
oa
dl

y.
likely to partake in this task than BCBAs. How-
ever, of the 14 questions, nine of the survey
items were not significant (64.3%), indicating
that the groups are using these principles and
techniques equally. No significant differences
were observed for the survey items regarding
the use of reinforcement or reward, �2(4, n �
128) � 4.55, p � .21, or punishment, �2(4, n �
129) � 1.69, p � .79. Additionally, for the
survey item asking, “How often in your daily
work do you use procedures that arise from a
specific and identifiable theoretical base rather
than being a set of packages or tricks?” no
significant effect was found, �2(4, n � 128) �
2.65, p � .45. Overall, animal behavior profes-
sionals and BCBAs are more alike than differ-
ent in their responses to this set of questions.
Of the 10 items regarding research, 5 dem-
onstrated a statistically significant difference
between the distributions of responses when a
chi-square test was conducted. For the survey
item “How often in your daily work do you
design data collection systems?” a significant
effect was found, �2(4, n � 130) � 29.58, p �
.01, with BCBAs reporting that they are more
likely to engage in this task. Additionally, a
significant effect was also found for the survey
item asking, “How often in your daily work do
you organize, analyze, and interpret observed
data?” �2(4, n � 130) � 14.88, p � .005, with

BCBAs performing this task more often than
animal behavior professionals. There was no
significant difference for the survey item re-
garding “How often in your daily work do you
review and interpret research literature?” �2(4,
n � 129) � 8.62, p � .07, or for the item asking
about exercising control over a behavior, �2(4,
n � 129) � 3.73, p � .44. Overall, the groups
were split within this category, with only 50%
of the items regarding research strategies differ-
ing between these participants.
For the category concerning social validity,
four of the seven items yielded statistically sig-
nificant chi-square tests. For the question ask-
ing, “How often in your daily work do you
arrange for the orderly termination of services
when they are no longer required?” a significant
effect was found, �2(4, n � 129) � 15.85, p �
.003, with the animal behavior professionals
planning for termination more often than the
BCBAs. Of the 55 BCBAs, only 7 reported
planning for terminations of their services very
often, whereas of the 74 animal behavior pro-
fessionals, 30 reported planning for termination
of their services very often. The questions that
did not produce significant results include ex-
plaining concepts using nontechnical language,
�2(4, n � 130) � 4.53, p � .21, and planning
for any possible unwanted effects of interven-
tions, �2(4, n � 129) � 6.06, p � .20. Again,
there were more similarities than differences in
the reported practice of BCBAs and animal
behavior professionals.

Discussion
The results of this survey reveal that there is
a lot of overlap between BCBAs’ work tasks
and those of animal behavior professionals.
Overall, the results show that of 31 items to
assess the work these professionals are engag-
ing in, 17 questions had no significant differ-
ence in their endorsement, suggesting that ani-
mal behavior professionals and BCBAs are
engaging in similar tasks as they carry out their
work.
Within the survey items, there were several
interesting and possibly concerning results.
For instance, the distribution of responses for
the question asking, “How often in your daily
work do you use procedures that arise from a
specific and identifiable theoretical base
rather than being a set of packages or tricks?”
was not significantly different for the two
groups. It is interesting that both animal be-
havior professionals and BCBAs both con-
sider their procedures to be from a theoretical
base, suggesting a similarity of theory be-
tween the two fields. A potentially concerning
result was found with the question “How of-
ten in your daily work do you arrange for the
orderly termination of services when they are
no longer required?” This survey item re-
sulted in a significant effect, with the animal
behavior professionals endorsing this item
more than the BCBAs. Of 55 BCBAs, only 7
reported planning for termination of services
very often, whereas 30 of 74 animal behavior
professionals reported planning for termina-

tion of services very often. This is particularly
worrying because the BACB requires the or-
derly termination of services when they are
no longer required as an element of their task
list (BACB, 2012). It is plausible that several
of the BCBAs were involved in early inter-
vention for children diagnosed with autism
38 GRAY AND DILLER
T
hi
s
do
cu
m
en
t
is
co
py
ri
gh
te
d
by
th
e

A
m
er
ic
an
P
sy
ch
ol
og
ic
al
A
ss
oc
ia
ti
on
or
on
e
of
it
s

al
li
ed
pu
bl
is
he
rs
.
T
hi
s
ar
ti
cl
e
is
in
te
nd
ed
so
le
ly

fo
r
th
e
pe
rs
on
al
us
e
of
th
e
in
di
vi
du
al
us
er
an
d
is
no

t
to
be
di
ss
em
in
at
ed
br
oa
dl
y.
spectrum disorder, in which the intervention
does not have a clear termination point. Still,
planning for the termination of services seem
like important components of behavior-
analytic practice.
The results regarding reinforcement and
punishment usage were noteworthy as well.
There was no significant difference found for
either of these techniques, suggesting that
both animal behavior professionals and
BCBAs are utilizing these behavioral proce-
dures. However, the intriguing aspect of these

results was the fact that 96% of animal be-
havior professionals and 89% of BCBAs im-
plemented reinforcement very often. For pun-
ishment, less than 1% of our respondents
reported using this technique very often.
These results highlight the parallel between
professions with an emphasis on reinforce-
ment and simultaneous reluctance to engage
in punishment, which is consistent with the
ethical guidelines put forth by the BACB.
Several possible reasons could account for
the majority of overlap between behavior-
change techniques for nonhuman and human
animals. First, it is plausible that many individ-
uals, regardless of profession, are using behav-
ior-analytic techniques. Behavioral techniques
may have been disseminated to the point that a
variety of people are implementing these pro-
cedures, even without the explicit endorsement
or knowledge of behavior analysts. Since both
animal behaviorists and BCBAs have the com-
mon objective of altering behavior of their cli-
ents, it is not surprising that these professionals
are engaging in similar, evidence-based prac-
tices to reach this goal.
As with all studies, there are limitations
within the present analysis. The sample size of
75 animal behavior professionals and 55
BCBAs is relatively small, but our response rate
was high, where we could measure it. Addition-
ally, both samples were composed mostly of
Caucasian women; this limited demographic
may actually represent the composition of both
professions. However, the mean work experi-

ence of both groups was similar to each other,
which excludes the limitation of one group be-
ing more experienced than the other. Further-
more, the survey designed in this study con-
sisted of only 31 items. There are many
different techniques these professionals could
be implementing, and a brief survey is not ex-
haustive.
Future research should replicate with the in-
tent to increase generality by using a broader
sample of behavior-change professionals who
work with human and nonhuman animals. Ad-
ditional research may also ask questions focus-
ing on a different set of elements of practice.
The questions within this study were derived
from the BCBA task list and the seven dimen-
sions of applied behavior analysis (Baer et al.,
1968, 1987). A different collection of questions
addressing other behavioral techniques may
provide a more detailed analysis of this partic-
ular issue. Another suggestion for future re-
search would be to use different methods to
figure out which techniques and procedures
practitioners are using. For example, direct ob-
servation of practice may yield a more reliable
and valid view of techniques that are employed
than self-report methodologies.
The results of the present study demonstrate
that animal behavior professionals and BCBAs
are using similar techniques to achieve their
goals, and they suggest that the theory and
knowledge within each discipline may be sim-
ilar. With this information, there are at least

three possible routes the field of behavior anal-
ysis could take. The first option is for the field to
continue as it is, with the behavior-analytic
community failing to acknowledge these animal
behavior professionals as participating in the
behavior-analytic tradition. Another option
could be to develop a training program for an-
imal behavior professionals interested in both
ABA and nonhuman animals to allow people to
get experience in ABA while still working with
animals as their main clients. This type of in-
teraction could be akin to translational research,
where application informs the search for basic
principles, and basic principles potentially yield
more effective intervention (e.g., Mace &
Critchfield, 2010). Here, instead of going be-
tween true basic research and application, there
could be a cross-species continuum, which
would potentially enhance the practice in both
professions.
A final option would be to integrate these two
professions in some manner. To combine the
ABA and animal training, open communication
must be established between animal behavior
professionals and BCBAs. One way for integra-
tion to take place is for BCBAs to attend con-
T
hi
s
do

oa
dl
y.
ferences with emphasis on training of nonhu-
man animals and for behavior professionals to
attend BCBA conferences. To make this more
likely, coordination should occur so that each
group offers relevant continuing education ex-
periences to attract professionals from the other
discipline. Another way to integrate the fields is
for practitioners in both disciplines to read re-
search literature from the other discipline. By
attending each other’s conferences and reading
research, collaborations could emerge, strength-
ening the integration of these professions. A
main goal of this article is to start a dialogue
between these two fields, potentially increasing
the scope of behavior analysis and providing a
mutually beneficial exchange of research and
professional experiences. The application of
ABA to nonhuman animals potentially strength-
ens behavior analysis because it would increase
the diversity of application and skills within the
field. Incorporating these animal behavior pro-
fessionals would expand knowledge within the
field and broaden the impact of behavior anal-
ysis, a recently stated goal of prominent behav-
ior analysts (cf. Friman, 2014; Normand, 2014;
Reed, 2014; Schlinger, 2014). An integration of
these two areas of expertise would potentially
expand the impact of behavior analysis, which
may help to support its long-term survival.

However, before any formal integration can be
achieved, more research is necessary to clarify
the skills needed within each profession and to
identify the extent of cross-species generality of
principles and procedures.
Even though no formal assimilation has oc-
curred between these two disciplines, there is
significant work and research being conducted
that uses a combination of behavior-analytic
procedures to solve socially significant issues
by training certain nonhuman animals (e.g., Pol-
ing, Weetjens, Cox, Beyene, & Sully, 2010).
This body of practical research has successfully
utilized behavioral principles, specifically rein-
forcement, to train giant African pouched rats to
detect landmines by smell (Mahoney et al.,
2014). The Hero Rats, as they are now known,
have cleared upward of 70,000 explosive de-
vices, benefiting more than 900,000 people
(Apopo Organization, 2015). This area of re-
search demonstrates the potential that is possi-
ble if these two fields collaborate.
References
Apopo Organization. (2015). Apopo Organization
impact results. Retrieved from https://www.apopo
.org/en/about/results/impact
Applied Animal Behaviorists. (2015). Animal Be-
havior Society Certification Program for applied
animal behaviorists. Retrieved from http://www
.animalbehaviorsociety.org/web/downloads/
CAAB-ACAAB-ProgramRequirements.pdf

Baer, D. M., Wolf, M. M., & Risley, T. R. (1968).
Some current dimensions of applied behavior anal-
ysis. Journal of Applied Behavior Analysis, 1, 91–
97. http://dx.doi.org/10.1901/jaba.1968.1-91
Baer, D. M., Wolf, M. M., & Risley, T. R. (1987).
Some still-current dimensions of applied behavior
analysis. Journal of Applied Behavior Analysis,
20, 313–327. http://dx.doi.org/10.1901/jaba.1987
.20-313
Bailey, J., & Burch, M. (2006). How to think like a
behavior analyst. New York, NY: Routledge.
Behavior Analyst Certification Board. (BACB).
(2012). BACB fourth ed. task list. Retrieved from
http://bacb.com/wp-content/uploads/2015/05/
BACB_Fourth_Edition_Task_List.pdf
Behavior Analyst Certification Board. (BACB).
(2015). Experience Standards. Retrieved from
http://bacb.com/wp-content/uploads/2015/08/
150824-experience-standards-english.pdf
Coughlin, S. S., Aliaga, P., Barth, S., Eber, S., Maillard,
S., Mahan, C. M., . . . Williams, M. (2011). The
effectiveness of a monetary incentive on response
rates in a survey of recent U.S. veterans. Survey
Practice, 4. Retrieved from http://surveypractice.org/
index.php/SurveyPractice/issue/view/25
Dymond, S., Roche, B., & Barnes-Holmes, D.
(2003). The continuity strategy, human behavior,
and behavior analysis. The Psychological Record,
53, 333–347.

Friman, P. C. (2014). Publishing in journals outside
the box: Attaining mainstream prominence re-
quires demonstrations of mainstream relevance.
The Behavior Analyst, 37, 73–76. http://dx.doi.org/
10.1007/s40614-014-0014-1
Laties, V. G. (2008). The Journal of the Experimen-
tal Analysis of Behavior at fifty. Journal of the
Experimental Analysis of Behavior, 89, 95–109.
http://dx.doi.org/10.1901/jeab.2008.89-95
Lattal, K. A. (2001). The human side of animal
behavior. The Behavior Analyst, 24, 147–161. Re-
trieved from http://www.ncbi.nlm.nih.gov/pmc/
articles/PMC2731511/
Mace, F. C., & Critchfield, T. S. (2010). Transla-
tional research in behavior analysis: Historical tra-
ditions and imperative for the future. Journal of the
Experimental Analysis of Behavior, 93, 293–312.
40 GRAY AND DILLER
T
hi
s
do
cu
m
en
t

is
co
py
ri
gh
te
d
by
th
e
A
m
er
ic
an
P
sy
ch
ol
og
ic
al
A
ss

oc
ia
ti
on
or
on
e
of
it
s
al
li
ed
pu
bl
is
he
rs
.
T
hi
s
ar
ti

cl
e
is
in
te
nd
ed
so
le
ly
fo
r
th
e
pe
rs
on
al
us
e
of
th
e
in

di
vi
du
al
us
er
an
d
is
no
t
to
be
di
ss
em
in
at
ed
br
oa
dl
y.
https://www.apopo.org/en/about/results/impact

https://www.apopo.org/en/about/results/impact
http://www.animalbehaviorsociety.org/web/downloads/CAAB-
ACAAB-Program
ACAAB-Program
ACAAB-Program
http://dx.doi.org/10.1901/jaba.1968.1-91
http://bacb.com/wp-
content/uploads/2015/05/BACB_Fourth_Edition_Task_List.pdf
http://bacb.com/wp-
content/uploads/2015/05/BACB_Fourth_Edition_Task_List.pdf
http://bacb.com/wp-content/uploads/2015/08/150824-
experience-standards-english.pdf
http://bacb.com/wp-content/uploads/2015/08/150824-
experience-standards-english.pdf
http://surveypractice.org/index.php/SurveyPractice/issue/view/2
5
http://surveypractice.org/index.php/SurveyPractice/issue/view/2
5
http://dx.doi.org/10.1007/s40614-014-0014-1
http://dx.doi.org/10.1007/s40614-014-0014-1
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2731511/
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2731511/
Mahoney, A., Lalonde, K., Edwards, T., Cox, C.,
Weetjens, B., & Poling, A. (2014). Landmine-
detection rats: An evaluation of reinforcement pro-
cedures under simulated operational conditions.
Journal of the Experimental Analysis of Behavior,
101, 450 – 456. http://dx.doi.org/10.1002/jeab.83

Normand, M. P. (2014). Opening Skinner’s box: An
introduction. The Behavior Analyst, 37, 67– 68.
http://dx.doi.org/10.1007/s40614-014-0016-z
Poling, A., Weetjens, B. J., Cox, C., Beyene, N. W.,
& Sully, A. (2010). Using giant African pouched
rats (Cricetomys gambianus) to detect landmines.
The Psychological Record, 60, 715–728.
Reed, D. D. (2014). Determining how, when, and
whether you should publish outside the box: Sober
advice for early career behavior analysts. The Be-
havior Analyst, 37, 83– 86. http://dx.doi.org/10
.1007/s40614-014-0012-3
Schlinger, H. D. (2014). Publishing outside the box:
Unforeseen dividends of talking to strangers. The
Behavior Analyst, 37, 77– 81. http://dx.doi.org/10
.1007/s40614-014-0010-5
Received December 26, 2015
Revision received April 5, 2016
Accepted April 13, 2016 �
T
hi
s
do
cu
m

y.
http://dx.doi.org/10.1002/jeab.83
http://dx.doi.org/10.1007/s40614-014-0016-z
http://dx.doi.org/10.1007/s40614-014-0012-3
http://dx.doi.org/10.1007/s40614-014-0012-3
http://dx.doi.org/10.1007/s40614-014-0010-5
http://dx.doi.org/10.1007/s40614-014-0010-5Evaluating the
Work of Applied Animal Behaviorists as Applied Behavior
AnalysisMethodParticipantsSurveyResultsDiscussionReferences
Journal of Operations Management 22 (2004) 209–217
Metrics and performance measurement in operations
management:
dealing with the metrics maze
Steven A. Melnyk∗ , Douglas M. Stewart, Morgan Swink
Department of Marketing and Supply Chain Management, The
Eli Broad Graduate School of Management,
Michigan State University, East Lansing, MI 48824-1039, USA
Accepted 16 January 2004
Abstract
Metrics provide essential links between strategy, execution, and
ultimate value creation. Changing competitive dynamics
are placing heavy demands on conventional metrics systems,
and creating stresses throughout firms and their supply chains.
Research has not kept pace with these new demands in an
environment where it is no longer sufficient to simply let

metrics
evolve over time—we must learn how to proactively design and
manage them. The intent of this paper is to convey the
importance and need for metrics-related research. An outline of
the important characteristics of the metrics research topic is
provided. Specifically, we address the functions of metrics;
their focus and tense; their operational and strategic contexts;
as
well as discuss the distinction between metrics, metrics sets and
metrics systems. Some initial theoretical grounding for the
research topic is provided through agency theory. We conclude
with a discussion of the intent and process of the special issue,
and introduction of the associated articles.
© 2004 Elsevier B.V. All rights reserved.
Keywords:Metrics; Performance measurement; Operations
management
1. The metrics challenge
One of the most powerful management disciplines,
the one that more than any other keeps people fo-
cused and pulling in the same direction, is to make
an organization’s purposes tangible. Managers do
this by translating the organization’s mission—what
it, particularly, exists to do—into a set of goals and
performance measures that make success concrete
for everyone. This is the real bottom line for every
organization—whether it’s a business or a school or
a hospital. Its executives must answer the question,
∗ Corresponding author. Tel.:+1-517-353-6381.
E-mail address:[email protected] (S.A. Melnyk).
“Given our mission, how is our performance going
to be defined?” (Magretta and Stone, 2002, p. 129)

The quote fromMagretta and Stone (2002)sug-
gests that metrics and performance measurement are
the critical elements in translating an organization’s
mission, or strategy, into reality. Metrics and strat-
egy are tightly and inevitably linked to each other.
Strategy without metrics is useless; metrics without a
strategy are meaningless. The importance of metrics
has been long recognized. Manufacturing and man-
agement consultant Oliver Wight almost 30 years ago
offered the oft-repeated maxim, “You get what you
inspect, not what you expect.” Every firm, every ac-
tivity, every worker needs metrics. Metrics fulfill the
fundamental activities of measuring (evaluating how
0272-6963/$ – see front matter © 2004 Elsevier B.V. All rights
reserved.
doi:10.1016/j.jom.2004.01.004
210 S.A. Melnyk et al. / Journal of Operations Management 22
(2004) 209–217
we arc doing), educating (since what we measure is
what is important; what we measure indicates how we
intend to deliver value to our customers), and direct-
ing (potential problems are flagged by the size of the
gaps between the metrics and the standard). Yet, per-
formance measurement continues to present a chal-
lenge to operations managers as well as researchers
of operations management. Operating metrics are of-
ten poorly understood and guidelines for the use of
metrics arc often poorly articulated.
As a focus of research, little attention has been de-

voted to this topic within the field of operations man-
agement. A great deal of what we currently know
about metrics comes from the managerial literature
(e.g.,Brown, 1996; Cooke, 2001; Dixon et al., 1990;
Kaydos, 1999; Ling and Goddard, 1988; Lynch and
Cross, 1995; Maskell, 1991; Melnyk and Christensen,
2000; Melnyk et al., in press; Smith, 2000; Williams,
2001). While there are numerous examples of the use
of various metrics, there are relatively few studies
in operations management that have focused on the
development, implementation, management, use and
effects of metrics within either the operations man-
agement system or the supply chain. Nascent exam-
ples can be found in the research ofBeaumon (1999),
Leong and Ward (1995), Neely (1998), Neely et al.
(1994, 1995), andNew and Szwejczewski (1995).
We should point out that topic of metrics as dis-
cussed by managers differs from the topic of mea-
surement as typically discussed by academics. This is
primarily a byproduct of different priorities between
these groups. The academic is concerned with defin-
ing, adapting and validating measures to address spe-
cific research questions. The time required to develop
and collect the measures is of less importance than
the validity and generalizability of the results beyond
the original context. Managers face far greater time
pressures, and are less concerned about generalizabil-
ity. They are generally more than willing to use a
“good enough” measure if it can provide useful in-
formation quickly. However, as long as the difference
in priorities is recognized there are undoubtedly many
lessons academic measurement experts can contribute
to managers’ understanding of metrics.
Recent indicators suggest that metrics and perfor-

mance measurement are receiving more attention.
In 1999, the Education and Research Foundation of
APICS commissioned a research program dealing
with measuring supply chain performance. The 2002
POMS National Conference included a special session
focusing on performance measurement. In late 2002,
KPMG in conjunction with the University of Illinois
at Champagne undertook a major research initiative
aimed at funding and encouraging research in perfor-
mance measurement (to the tune of US$ 2.8 million).
Finally, the January 2003Harvard Business Review
case study focused on the miscues and disincentives
created by poorly thought out performance measure-
ment systems (Kerr, 2003). Why the increasing inter-
est? We believe the answer is in the business environ-
ment faced by today’s operations managers. Today’s
environment is characterized by: (1) “never satisfied”
customers (McKenna, 1997); (2) the need to manage
the “total” supply chain, rather than only internal
factors; (3) shrinking product life cycles; (4) more
(but not necessarily better) data; and (5) an increasing
number of alternatives. These dynamics make static
metrics systems obsolete, and call for new perfor-
mance measures and metrics approaches that go be-
yond simple reporting to create means for identifying
improvement opportunities and anticipating potential
problems. Further, metrics are now seen as an im-
portant means by which priorities are communicated
within the firm and across the supply chain. Metrics
misalignment is thought to be a primary source of in-
efficiency and disruption in supply chain interactions.
Given this environment, the research challenge is
to better understand the roles and impacts of metrics
in operating systems, and using this knowledge to de-

sign metrics systems and guidelines that provide clar-
ity of purpose, real-time feedback and predictive data,
and insights into opportunities for improvement. In
addition, these new metrics systems need to be flex-
ible in recognizing and responding to changing de-
mands placed on the operating system due to product
churn, heterogeneous customer requirements, as well
as changes in operating inputs, resources, and perfor-
mance over time.
By way of introducing this special issue on perfor-
mance measures and operating metrics, in the remain-
ing sections of this article we:
• Identify the defining elements and different types
of metrics.
• Position metrics within the operations management
research environment.
S.A. Melnyk et al. / Journal of Operations Management 22
(2004) 209–217 211
• Identify the special research challenges associated
with metrics.
• Introduce the articles that comprise the special is-
sue.
Ultimately, the goal of this special issue is to direct,
shape, and encourage research into this very important
topic area.
1.1. Defining metrics—an overview

A metric is a verifiable measure, stated in either
quantitative or qualitative terms and defined with re-
spect to a reference point. Ideally, metrics are consis-
tent with how the operation delivers value to its cus-
tomers as stated in meaningful terms.
This definition identifies several critical elements.
First, a metric should be verifiable, that is, it should
be based on an agreed upon set of data and a
well-understood and well-documented process for
converting this data into the measure. Given the data
and the process, independent sources should be able
to arrive at the same metric value. Second, metrics
are measures. They capture characteristics or out-
comes in a numerical or nominal form. In order to
interpret meaning from a metric, however, it must be
compared to a reference point. The reference point
acts as a basis of comparison, and can be an absolute
standard or an internally or externally developed stan-
dard. Standards can be based on past metric values or
based on metric values for a comparable process (e.g.,
a “benchmark”). Zero defects would be an absolute
standard, for example, as would be 100% utilization.
An operating budget is an internally developed stan-
dard, whereas environmental performance might be
compared to external standards published by the Envi-
ronmental Protection Agency (EPA). Because metrics
are expressed relative to some reference point, they
encourage comparison by the users of the metric or by
external parties (as in the case of ISO 9000 auditors).
It is generally desirable that a metric be expressed
in meaningful terms. If metrics are to be effective, they
must be understood—they must make sense to the per-
son using the metrics. In addition, metrics should be

value-based. That is, a metric should be linked to how
the operation delivers value to its targeted customers.
Naturally, not all metrics will be directly related to
customer value. Metrics may also be related to the
values of other stakeholders in the process. For exam-
ple, worker safety-oriented metrics are important, but
indirectly related to customer value.
2. The fundamental need for metrics
Metrics provide data refinement. As the volume of
inputs increases, through greater span of control or
growing complexity of an operation, data manage-
ment becomes increasingly difficult. Metrics provide
a means of distilling the volume of the data while si-
multaneously increasing its information richness. Op-
erations need these functions in order to operate ef-
fectively and efficiently on a day-to-day basis.
Finally, metrics exist as tools for people. Ultimately,
the actions people take and the decisions they make
determine the degree and nature of value that an op-
eration creates. These actions and decisions can be
greatly influenced by metrics.
Metrics provide the following three basic functions:
• Control: Metrics enable managers and workers to
evaluate and control the performance of the re-
sources for which they are responsible.
• Communication: Metrics communicate perfor-
mance not only to internal workers and managers
for purposes of control, but to external stakeholders
for other purposes as well (e.g., Wall Street, the

EPA or to a bank). Many times stakeholders and
users of metrics do not understand the workings
and processes of a firm or operation, nor do they
need to. Well-designed and communicated metrics
provide the user a sense of knowing what needs
to be done without necessarily requiring him/her
to understand the intricacies of related processes.
Poorly developed or implemented metrics can lead
to frustration, conflict, and confusion.
• Improvement: Metrics identify gaps (between per-
formance and expectation) that ideally point the way
for intervention and improvement. The size of then
gap and the direction of the gap (positive or nega-
tive) provide information and feedback that can be
used to identify productive process adjustments or
other actions.
There are dynamic tensions inherent in requiring
one system to perform multiple functions. One such
tension stems from the desire to change metrics in re-
(2004) 209–217
sponse to new strategic priorities, and the desire to
maintain metrics to allow comparison of performance
over time. This tension will dictate the metrics life
cycle. Moreover, as the metrics reflect underlying pri-
orities and decisions, metrics-related stress between
various parties is often simply the first indicator of un-
observed, unresolved conflicts between the customer,
strategy, and operations of the firm.

3. A metrics typology
One source of complexity regarding the study of
metrics is the variety of different types of metrics that
researchers and managers encounter. We suggest that
various metrics can be readily classified according
to two primary attributes:metrics focusand metrics
tense. Metrics focus pertains to the resource that is the
focus of the metric. Generally, metrics report data in
either financial (monetary) or operational (e.g., oper-
ational details such as lead times, inventory levels or
setup times) terms. Financial metrics define the perti-
nent elements in terms of monetary resource equiva-
lents, whereas operational metrics tend to define ele-
ments in terms of other resources (e.g., time, people)
or outputs (e.g., physical units, defects). The second
attribute, metrics tense, refers the how the metrics are
intended to be used. Metrics can be used to both to
judge outcome performance and to predict future per-
formance. An outcome-oriented use of a metric im-
plicitly assumes that the problems and lessons uncov-
ered from a study of past outcomes can be applied to
current situations. That is, by studying the past, we
can improve the present. In general, managers who
monitor and reward activities and associated person-
nel use metrics in an outcome-based way. Executives
Scheme 1. Metrics typology.
often use metrics in this way. Many of the cost-based
metrics encountered in firms belong to this category.
Similarly, many of the accounting based information
systems observed in many firms typically generate
outcome-based metrics.
In contrast, a predictive use of a metric is aimed at

increasing the chances of achieving a certain objective
or goal. Predictive metrics are associated to aspects of
the process that will result in the outcomes of inter-
est. If our interest is in reducing lead time, then we
might assess metrics such as distance covered by the
process, setup times, and number of steps in the pro-
cess. Reductions in one or more of these areas should
be reflected in reductions in lead time. An emphasis
on identifying and using metrics in a predictive way
is relatively new. Predictive metrics are appropriate
when the interest is in preventing the occurrence of
problems, rather than correcting them.
Combining of these two metrics attributes provides
four distinct types of metrics: financial/outcome,
financial/predictive, operational/outcome, and oper-
ational/predictive (Scheme 1). These different cat-
egories appeal to different groups within the firm.
Top managers, for example, are typically most in-
terested in financial/outcome. In contrast, operations
managers and workers are most likely interested in
operational/predictive or operational/outcome metrics
since these two sets pertain to the processes that these
managers must manage and change.
4. Levels of metrics
The term, “metrics” is often used to refer to one of
three different constructs: (1) theindividual metrics;
(2) the metrics sets; and (3) the overallperformance
measurement systems. These terms are often used in-
(2004) 209–217 213

terchangeably, thus contributing to the confusion. We
suggest that recognition of the different levels of met-
rics and their interactions is important for the research
and design of metrics.
At the highest level, theperformance measure-
ment systemlevel integrates. That is, it is respon-
sible for coordinating metrics across the various
functions and for aligning the metrics from the
strategic (top management) to the operational (shop
floor/purchasing/execution) levels. For every activity,
product, function, or relationship, multiple metrics
can be developed and implemented. The challenge
is to design a structure to the metrics (i.e., grouping
them together) and extracting an overall sense of per-
formance from them (i.e., being able to address the
question of “Overall, how well are we doing?”).
Several different approaches have been proposed
for developing such an integrative system. These in-
clude: (1) thebalanced scorecard, as presented by
Kaplan and Norton (1992, 1996, 2001)and elaborated
on by others (e.g.,Ittner and Larcker, 1998); (2) the
strategic profit impact model(otherwise known as the
Dupont model (Lambert and Burduroglu, 2000); and
(3) thetheory of constraints(TOC) measurement sys-
tem (Lockamy III and Spencer, 1998; Smith, 2000).
Each of these major systems has strengths and weak-
nesses. For example, the balanced scorecard excels at
its ability to force top management to recognize that
multiple activities must be carried out for corporate
success and the management and monitoring of these
activities must be balanced. The strategic profit im-
pact model provides the operations manager with a
mechanism, whereby operational improvements such

as reductions in inventory—changes of interest to the
operating personnel—can be translated into its impact
on financial performance, changes of interest primar-
ily to top management. Finally, the TOC approach is
attractive because of its ability to simplify the perfor-
mance measurement problem and reduce the number
of measures and areas actively and continuously mon-
itored by top management.
Theperformance measurement systemis ultimately
responsible for maintainingalignment and coordi-
nation. Alignment deals with the maintenance of
consistency between the strategic goals and metrics
as plans are implemented and restated as they move
from the strategic through the tactical and operational
stages of the planning process. Alignment attempts to
ensure that at every stage that the objectives set at the
higher levels are consistent with and supported by the
metrics and activities of the lower levels. In contrast,
coordination recognizes the presence of interdepen-
dency between processes, activities or functions.
Coordination deals with the degree to which the met-
rics in various related areas are consistent with each
other and are supportive of each other. Coordination
strives to reduce potential conflict that can occur
when one area focuses on maximizing uptime (by
avoiding setup and running large batches) and another
focuses on quality and flexibility. Coordination tries
to maintain an equivalence of activities, goals, and
purpose across departments, groups, activities and
processes.
To date, most research and managerial attention has
focused onperformance measurement systemsor on
individual metrics. Melnyk et al. (in press)suggest

that these two levels are not sufficient by themselves.
There exists another metrics construct—themetrics
set. The metric set consists of the metrics assigned by
a higher level of management to direct, motivate and
evaluate a single person in charge of a specific activity,
process, area, or function. The metrics set is critical
because it is often the relevant unit of analysis, and
because the scope and complexity of an individual’s
metrics set can be viewed as a load imposed upon that
person’s finite mental capacity.
These three levels of metrics are linked. At the base
is theindividual metric, the building block.Individual
metrics are aggregated to form variousmetrics sets.
Each set directs, guides, and regulates an individual’s
activities in support of strategic objectives. Coordinat-
ing and managing the development of the variousindi-
vidual metricsand themetrics setsis theperformance
measurement system.
4.1. Positioning metrics within the research
environment
There has long been recognition of metrics and its
importance within the operations management field.
Wickham Skinner in 1974 identified simplistic perfor-
mance evaluation as being one of the major causes for
factories getting into trouble (Skinner, 1974). Subse-
quently, Hill (1999) recognized the role and impact
of performance measures and performance measure-
ment systems in his studies of manufacturing strategy.
(2004) 209–217

In these and other studies, metrics are often viewed
as being part of the infrastructure or environment in
which manufacturing must operate.
However, while we have recognized the role of met-
rics as an influencing factor, there is still a need to posi-
tion the topic of metrics within a theoretical context—a
framework that gives metrics a central role. One such
theoretical framework isagency theory.
Agency theory applies to the study of problems aris-
ing when one party, the principal, delegates work to
another party, the agent (Eisenhardt, 1989a; Lassar and
Kerr, 1996). The unit of analysis is the metaphor of
a contract between the agent and the principal. Prior
studies using agency theory as the theoretical frame-
work have used “coordination efforts” (Celly and
Frazier, 1996), “control” (Anderson and Oliver, 1987),
and “management” (McMillan, 1990) as the unit of
analysis.
There are numerous factors and variables that in-
fluence the most efficient “contract” in the dyadic
relationship between a principal and agent. These
include the information systems (Eisenhardt, 1989a),
outcome uncertainty (Eisenhardt, 1989a), risk aver-
sion (Anderson and Oliver, 1987; Celly and Frazier,
1996), programmability (Eisenhardt, 1989a), and the
relationship length (Eisenhardt, 1989a; Celly and
Frazier, 1996).
Within operations management, agency theory
has been used to study such topics as decentralized
cross-functional decision-making (Kouvelis, 2000),
group technology (Beh-Arieh, 1999), international

manufacturing (Change, 1999), scheduling (Gu, 1997;
Kim, 1996), and inventory management (Allen, 1997).
Yet, what makes agency theory so attractive is that
the recognition that in most organizations the concept
of a contract as a motivating and control mechanism
is not really appropriate. Rather, the contract is re-
placed by the metric (Austin, 1996). It is the metric
that motivates and directs; it is the metric that en-
ables principals to manage and direct the activities
of their various agents. The development, selection,
use, and refinement of metrics becomes a major con-
cern of both principals and agents. Consequently,
agency theory provides a potentially interesting and
useful theoretical context for operations management
researchers to analyze this critical topic.
Dependency theory (Pfeffer and Salancik, 1978)
might also be seen as a potentially fruitful lens through
which to view the role of metrics in operations man-
agement. This theory states that the degree of interde-
pendence and the nature of interactions among func-
tional specialists within an organization are influenced
by the nature of the collective task they seek to accom-
plish. In dynamic environments involving rapid prod-
uct change and high degrees of heterogeneity in cus-
tomer requests, agents responsible for different func-
tional aspects of order taking, processing, and fulfill-
ment become more and more dependent on each other
for information necessary to complete their respec-
tive tasks. This theory has implications for the design
of metrics systems. For example, questions such as
“How should metrics reflect the interdependencies of
different functional areas?” could be posed. And fur-
ther, “How should the rotation or change in metrics be
associated to the dynamics of demands placed on the

operating system?” These types of research questions
start to get at the coordination attributes of higher or-
der performance measurement systems raised earlier
in this article.
We have in several places noted that metrics provide
a vital linkage between intended strategies and actual
execution. This notion harkens to theories of strategic
fit. Skinner (1969)offered one of the earliest opera-
tions management perspectives on strategic fit as he
argued the need for strategic fit between manufactur-
ing goals and decisions and other functional and cor-
porate strategies.Wheelwright (1984)further defined
the notion of strategic fit, stating the need for consis-
tency between operations strategy and business strat-
egy, other functional strategies, and the competitive
environment, respectively. While the need for strate-
gic fit is recognized, how to go about achieving it has
received less attention in the operations strategy liter-
ature. Taking strategy fit theory as the frame of ref-
erence for the study of metrics might lead to insights
into the role of metrics in achieving fit, and again, the
implications of strategy fit for the design of metrics
systems.
An information processing perspective (Galbraith,
1973) offers yet another potentially rewarding way to
look at metrics. Presumably, a richer “metrics set” cre-
ates the basis for richer communications among de-
cision makers, workers, strategy representatives, and
customers of a process. However, there may be limits
to the organization’s (as well as individuals’) ability
to process larger sets of metrics, and increasing num-

(2004) 209–217 215
bers of metrics could lead to greater conflict in the
implied priorities, as well as greater equivocality re-
garding future actions. Given this apparent trade-off
between metrics set richness and complexity, an in-
formation processing theoretical view could stimu-
late research into questions regarding the optimal size
of a metrics set, or perhaps the optimal combination
of outcome and predictive metrics included in the
set.
Linkage research in services, beginning with the
work of Schneider et al. (1980)and more recently re-
viewed in Pugh et al. (2002)has focused on validat-
ing and quantifying perceived relationships between
internal actions of a firm (such as particular human
resources practices) and important strategic outcomes
(such as customer satisfaction and profitability). This
research may be particularly applicable to addressing
such questions as “How does one derive a predictive
metric?” and “Is a perceived predictive metric truly
predictive?” It may also help in illuminating the dif-
ferences between the metrics system as conceived by
management and its actual structure. This will be par-
ticularly relevant in understanding the relationships
between metrics that are not mathematically derived,
such as when metrics interact through a lens of altered
behavior.
4.2. Overview of the special issue
As can be seen from the preceding discussion,
metrics and performance measurement is inherently a
complex topic. There is a need for research intended

to better structure and to appropriately simplify the
analysis of this topic. These and other factors formed
the motivation for this special issue.
The announcement for this special was framed in
very broad terms. Both theoretical and empirical pa-
pers, as well as rigorous case studies, were invited.
Cross-functional studies were particularly encouraged.
Suggested topics for the special issue included:
• Assessing the impact of operating/predictive met-
rics on system performance.
• Evaluating the relationship between financial and
operating metrics.
• Measuring performance within the supply chain
environment—practices, challenges, problems and
opportunities.
• Assessing consistency of metrics, both within the
set of metrics being used and between the metrics
and corporate strategy (and the potential effects at-
tributable to consistency or the lack of consistency
within the metrics).
• Implementing performance measurement systems.
• Changing performance measurement system or
metrics over time.
• Measuring performance of product/process design
processes.
• Integrating environmental issues into the perfor-
mance measurement systems.

• Integrating metrics with the real or perceived reward
structure.
• Assessing performance measurement and metrics
within services and manufacturing settings.
Papers that were submitted for the special issue were
subjected to an initial review by the co-editors to assess
the compatibility of the topic addressed by the paper
with the theme and focus of the special issue. Papers
that did not pass this initial screen were, at the authors’
discretion, forwarded to the editor of theJournal of
Operations Managementto be included in the normal
review process for the journal. Of 10 papers submitted,
8 passed this initial review. These eight papers were
then subject to a normal double-blind review process
with three being accepted for publication.
The first paper is “An exploratory study of perfor-
mance measurement systems and relationships with
performance results” by James Evans. Evans uses the
metrics framework provided by the Malcolm Baldrige
Award, which is similar to that of the balanced score-
card to conduct an empirical investigation of the re-
lationship between the scope or types of metrics used
and customer, financial and market performance. The
paper addresses issues relating to both metrics and the
metrics system.
The second paper by Shinn Sun is “Assessing joint
maintenance shops in the Taiwanese army using data
envelopment analysis”. This paper is application ori-
ented, and in it Sun develops a DEA-based perfor-
mance assessment tool to evaluate the performance
of multiple similar functional units, and identify op-

portunities for improvement. The tool is applied to
a collection of maintenance shops to demonstrate its
usefulness. Sun’s tool directly addresses two of the
functions of metrics: control through setting objective
(2004) 209–217
standards and assuring comparability of the results;
and improvement through identifying areas of relative
inefficiency. Indirectly, the model supports the third
function of communication in that it identifies simi-
lar but more efficient functional units to communicate
with.
The final paper “Perceptual measures of perfor-
mance: fact or fiction?” is from Mikko Ketokivi and
Roger Schroeder. In it the authors investigate the reli-
ability and validity of perceptual measures of perfor-
mance. They introduce and empirically demonstrate
a method for using multi-informant survey data, and
conclude that the reliability and validity of such an ap-
proach is satisfactory. This has implications for prac-
tice in situations where perceptual data is the best or
only source of assessment, and perhaps more impor-
tantly for future research on the linkage between met-
rics attributes and firm performance.
4.3. Concluding comments
The intent of our discussion has been to first con-
vince the reader of the importance of metrics as a topic
and the need for an increased understanding of metrics
and their role in the firm, and to provide some orga-

nization to our current understanding of the topic. We
have suggested an outline of what we see as important
characteristics by which the research space can be or-
ganized, and provided some initial theoretical ground-
ing for this research in agency theory, dependency the-
ory, strategic fit theory, information processing theory,
and linkage research. Taken in the greater context of
the special issue, it should suggest many profitable av-
enues of inquiry to follow. It is our hope that it will
serve to inspire other researchers to contribute to our
understanding of this very important topic.
Acknowledgements
All papers contained in this special issue were sub-
ject to an exhaustive review process. Critical to the
success of this review process were the various re-
viewers who gave extensively of their time and in-
sights. We would like to acknowledge the following re-
viewers: Joe Biggs, California Polytechnic State Uni-
versity; Ken Boyer, Michigan State University; David
Collier, Ohio State University; Kevin J. Dooley, Ari-
zona State University; Janet L. Hartley, Bowling Green
State University; Nancy Lea Hyer, Vanderbilt Univer-
sity; Jay Jayaram, University of Oregon; Chris Mc-
Dermott, Rensselaer Polytechnic Institute; Linda G.
Sprague, CEIBS; Srinivas Talluri, Michigan State Uni-
versity; Shawnee Vickery, Michigan State University;
D.B. Waggoner, Cambridge University; Ravi Behara,
Florida Atlantic University; Rohit Verma, University
of Utah; Anthony Ross, Michigan State University;
Daniel Krause, Arizona State University.
References

Allen, D.S., 1997. A multi-sector inventory model. Journal of
Economic Behavior and Organization 32 (1), 55–87.
Anderson, E., Oliver, R.L., 1987. Perspectives on behavior-
based
versus outcome-based salesforce control systems. Journal of
Marketing 51, 76–88.
Austin, R.B., 1996. Measuring and Managing Performance in
Organizations. Dorset House Publishing, New York, NY.
Beaumon, B.M., 1999. Measuring supply chain performance.
International Journal of Operations and Production Management
19 (3).
Beh-Arieh, D., 1999. Information analysis in a distributed
dynamic
group technology method. International Journal of Production
Economics 60–61, 427–432.
Brown, M.G., 1996. Keeping Score: Using the Right Metrics to
Drive World-Class Performance. Quality Resources, New York,
NY.
Celly, K.S., Frazier, G.L., 1996. Outcome-based and behavior-
based coordination efforts in channel relationships. Journal of
Marketing Research 33, 200–210.
Change, E., 1999. Control in multinational corporations
(MNCs):
the case of Korean manufacturing subsidiaries. Journal of
Management 25 (4).
Cooke, J.A., 2001. Metrics systems. Logistics Management and
Distribution Report 40 (10), 45–49.

Dixon, J.R., Nanni Jr., Alfred, J., Vollmann, T.E., 1990. The
New
Performance Challenge: Measuring Operations for World-Class
Competition. Dow Jones–Irwin, Homewood, IL.
Eisenhardt, K.M., 1989a. Agency theory: an assessment and
review.
The Academy of Management Review 14 (1), 57–74.
Galbraith, J.R., 1973. Designing Complex Organizations.
Addison-Wesley, Reading, MA.
Gu, P., 1997. Bidding-based process planning and scheduling in
a multi-agent system. Computers and Industrial Engineering
32 (2), 477–496.
Hill, T., 1999. Manufacturing Strategy: Text and Cases.
McGraw-Hill, Burr Ridge, IL.
Ittner, C.D., Larcker, D.F., 1998. Innovations in performance
measurement: trends and research implications. Journal of
Management Accounting Research 10, 205–238.
Kaplan, R.S., Norton, D., 1992. The balanced scorecard—
measures that drive performance. Harvard Business Review,
January–February, pp. 71–79.
(2004) 209–217 217
Kaplan, R.S., Norton, D., 1996. The Balanced Scorecard.
Harvard
Business School Press, Cambridge, MA.

Kaplan, R.S., Norton, D.P., 2001. The Strategy-Focused
Organization: How Balanced Scorecard Companies Thrive in
the New Business Environment. Harvard Business School Press,
Boston, MA.
Kaydos, W., 1999. Operational Performance Measurement:
Increasing Total Productivity. St. Lucie Press, Boca Raton, FL.
Kerr, S., 2003. The best-laid incentive plans. Harvard Business
Review, January, pp. 27–37.
Kim, K.H., 1996. A distributed scheduling and shop floor
control
method. Computers and Industrial Engineering 31 (3–4), 583–
586.
Kouvelis, P., 2000. Decentralizing cross-functional decisions:
coordination through internal markets. Management Science 46
(8).
Lambert, D.M., Burduroglu, R., 2000. Measuring and selling the
value of logistics. International Journal of Logistics Research
11 (1), 1–17.
Lassar, W.M., Kerr, J.L., 1996. Strategy and control in
supplier–distributor relationships: an agency perspective.
Strategic Management Journal 17, 613–632.
Leong, G.K., Ward, P.T., 1995. The six Ps of manufacturing
strategy. International Journal of Operations and Production
Management 15 (12), 32–45.
Ling, RC., Goddard, W.E., 1988. Orchestrating Success:
Improve
Control of the Business with Sales and Operations Planning.
Wiley, New York, NY.

Lockamy III, A., Spencer, M.S., 1998. Performance
measurement
in a theory of constraints environment. International Journal of
Production Research 36 (8), 2045–2060.
Lynch, R.L., Cross, K.F., 1995. Measure Up! How to Measure
Corporate Performance. Blackwell, Malden, MA.
Magretta, J., Stone, N., 2002. What Management is: How it
Works
and Why it’s Everyone’s Business. Free Press, New York,
NY.
Maskell, B.H., 1991. Performance Measurement for World Class
Manufacturing. Productivity Press, Cambridge, MA.
McKenna, R., 1997. Real Time: Preparing for the Age of the
Never
Satisfied Customer. Harvard Business School Press, Boston,
MA.
McMillan, J., 1990. Managing suppliers: incentive systems in
Japanese and U.S. industry. California Management Review
32 (4), 38–55.
Melnyk, S.A., Christensen, R.T., 2000. Back to Basics: Your
Guide
to Manufacturing Excellence. St. Lucie Press, Boca Raton, FL.
Melnyk, S.A., Stewart, D.L., Calantone, Roger J., Speier, C.
Metrics and the Supply Chain: An Exploratory Study. APICS,
Alexandria, VA, in press.
Neely, A., 1998. The Performance Measurement Revolution:
Why

Now and What Next? International Journal of Operations and
Production Management 18 (9–10).
Neely, A., Gregory, M., Platts, K., 1995. Performance
measurement
system design. International Journal of Operations and
Production Management 4, 80–116.
Neely, A., Mills, J., Platts, K., Gregory, M., Richards, H., 1994.
Realizing strategy through measurement. International Journal
of Operations and Production Management 14 (3), 140–152.
New, C.C., Szwejczewski, M., 1995. Performance measurement
and the focused factory: empirical evidence. International Jour-
nal of Operations and Production Management 15 (4), 63–79.
Pfeffer, J., Salancik, G.R., 1978. The External Control of
Organizations: A Resource Dependence Perspective. Harper &
Row, New York.
Pugh, S.D., Dietz, J., Wiley, J.W., Brooks, S.M., 2002. Driving
service effectiveness through employee–customer linkages.
Academy of Management Executive 16 (4), 73–84.
Schneider, B., Parkington, J.J., Buxton, V.M., 1980. Employee
and customer perceptions of service in banks. Administrative
Science Quarterly 25 (2), 252–267.
Skinner, W., 1969. Manufacturing—missing link in corporate
strategy. Harvard Business Review, May–June, pp. 136–145.
Skinner, W., 1974. The decline, fall, and renewal of
manufacturing
plants. Harvard Business Review, May–June.
Smith, D., 2000. The Measurement Nightmare: How the Theory

of Constraints can Resolve Conflicting Strategies, Policies, and
Measures. St. Lucie Press, Boca Raton, FL.
Wheelwright, S.C., 1984. Manufacturing strategy: defining the
missing link. Strategic Management Journal 5, 77–91.
Williams, B.R., 2001. Is your supply chain competitive? How
do
you know? In: 2001 APICS Conference, San Antonio, TX,
October.
Metrics and performance measurement in operations
management: dealing with the metrics mazeThe metrics
challengeDefining metrics-an overviewThe fundamental need
for metricsA metrics typologyLevels of metricsPositioning
metrics within the research environmentOverview of the special
issueConcluding commentsAcknowledgementsReferences

Evaluating the Work of Applied Animal Behaviorists as Applied.docx

Recommended

Recommended

More Related Content

Similar to Evaluating the Work of Applied Animal Behaviorists as Applied.docx

Similar to Evaluating the Work of Applied Animal Behaviorists as Applied.docx (17)

More from SANSKAR20

More from SANSKAR20 (20)

Recently uploaded

Recently uploaded (20)

Evaluating the Work of Applied Animal Behaviorists as Applied.docx