Academic Paper Writing Service http://StudyHub.vip/A-School-Psychology-Program-S-Initial-E 👈

1. Psyrhology in zyxwvutsrqponmlkji
the Schools zyxwvutsrqpo
1981, l8. 60-66 zyxwvutsrqponm
A SCHOOL PSYCHOLOGY PROGRAM’S INITIAL EVALUATION
OF ITS COMPETENCY-BASED TRAINING MODEL
STEVEN I. zyxwvuts
PFEIFFER
Northern Arizona University
This study describes a school psychology program’s first attempt at evaluating its
competency-based training model. Two competencieswere selected for investigation:
psychological report writing and consultation. A set of videotaped simulation and
situational response tests were developed,and a group of first-year schoolpsychology
students who had completed the program’s assessment and consultation sequence
were compared to newly admitted graduate students. Results confirmed the training
program’s success in teaching report writing and generating behavioral recommen-
dations during consultation, but found zyxwv
no difference in interpersonal effectiveness
between the two groups. Although support was found for the feasibilityof conducting
a program evaluation, questions were raised regarding the practicality of such an
enterprise. Suggestions are made to deal with the problem of the expenseand tremen-
dous energy required to evaluate a CBTP.
The demand for accountability and educational reform has stimulated the develop-
ment of competency-based training programs. (CBTP). Unfortunately, there exists very
little empirical support to indicate that the CBTPs do make a difference(Mowder, 1979).
Tied to this problem is the issue that very little has been written on how to design or im-
plement an evaluation of a CBTP.
Reilly, Barclay, and Culbertson (1977) discussed a number of issues facing
competency-based training. They pointed out that performance-based training programs
have two Achilles’ heels: (a) the sticky problem of measuring the competencies, and (b)
validating that these performances make a significant difference in pupil behavior.
Mowder (1979) recently added fuel to the fire by pointing out that “the proliferation of
competency statements does not necessarily capture the various roles of a practicing
school psychologist” (p. 700).
The discipline of school psychology seems to lend itself most logically to a CBTP
model. The professional practice of psychology in the schools requires a large number of
skills and competencies for which a training program would want to ensure adequate
coverage. National organizations such as NASP and APA encourage such program
evaluation. For example, one section in NASP’s Standards for Training Programs in
School Psychology (1978) reads: “The program (shall) carry out a well-defined plan for
evaluating the school psychologists it prepares and using the results in the study, develop-
ment and improvement of its program’’ (p. 22).
School psychology as a profession has been experiencing dramatic growth during
the past ten years. A number of new and divergent roles have been conceptualized for
practitioners, going beyond the traditional clinical/psychometric model (Bardon zy
&
Bennett, 1974; Hunter & Lambert, 1974). Trends toward the future, influenced as much
by legislation, judicial action, and consumer needs as by innovative professional direc-
tions, point to a broadening role that includes: serving all children in the schools, func-
tioning to a larger extent on interprofessional teams, becoming involved in curriculum
planning and evaluation, and supervising paraprofessionals (Pfeiffer, Note 4). Given this
extensive projection of activities and roles, it is apparent that school psychology training
Requests for reprints should be sent to Steven I. Pfeiffer, Dept. of Psychology, Northern Arizona Univer-
sity, Flagstaff, A 2 86011.
60

2. Program Evaluation zyxw
61 zy
programs need to determine how effectively they are preparing their graduates for the
positions of the future.
Even after school psychology training programs delineate what competencies its
students will master, the question of how to evaluate these skills still exists. A number of
competencies are difficult to measure objectively, yet are almost universally agreed upon
as extremely important for the successful functioning of a psychologist. For example,
how can a training program reliably assess whether a student will “relate effectively” to
parents or school personnel, or demonstrate “professional sensitivity” to a client’s par-
ticular sociocultural background? Although these process and higher order integrative
skills are, perhaps, psychometrically elusive, a creative program evaluation should per-
mit assessment of these important competencies.
The present evaluation was conducted by a school psychology program to determine
the feasibility of evaluating its CBTP. It was hoped that information could be obtained to
determine which skills might need more thorough coverage during the training sequence,
and also to obtain a more accurate picture of how successfully the program was
“packaging” its graduates, A final objective was to set the stage for the development of
an ongoing program evaluation system (Stufflebeam, 1968).
The school psychology program that conducted the evaluation was a two-year, z
60
semester hour master’s degree program, quite similar to the majority of nondoctoral
programs in terms of course work, required semester hours, and training objectives listed
in the Directory zyxwvu
of School Psychology Training Programs in the United States and
Canada (Brown zyxwvut
& Lindstrom, 1977). An initial set of competencies was developed dur-
ing 1971-1972, prior to the opening of the University. (See Gavilan, et al., 1977, for an
excellent description of this early phase in the development of a CBTP.)
The school psychology program then refined the detailed set of statements of both
its prerequisite and exit competencies (Anderson & Smith, Note 1; Pfeiffer, Note 3).
Upon completion of training, the graduates wereexpected to demonstrate competency in
six broad areas: (1) observational strategies, (2) psychoeducational assessment/report
writing, (3) behavioral change strategies/counseling, (4) in-service and staff develop-
ment, zyxwvut
(5) program evaluation, and (6) consultation.
For the purpose of the initial project, a decision was made to examine only two of
the six exit competencies. The two skill areas that were selected were “Report Writing”
and “Consultation.” The report writing competency specified, “Assessing social,
emotional, intellectual, sensory-motor and perceptual-language behavior with standard-
ized psychological and psychoeducational instruments, integrating the various informa-
tion into a meaningful written report, and formulating appropriate recommendations”
(Anderson & Smith, Note zyxwvu
I).
The competency dealing with consultation read, “Consult with parents, teachers,
school administrators and community/agency representatives; demonstrating a high
level of facilitative behavior and giving specific suggestions to alleviate the problems”
(Pfeiffer, Note 3).The three objectivesof the present study were to: (a) assess in a reliable
way each student’s performance in psychological report writing and consultation, (b)
present as an example a methodology for evaluating a CBTP, and (c) determine whether
an ongoing competency-based evaluation system would be a feasible and practical
endeavor for a training program to adopt.

3. 62 zyxwvutsr
Psychology in the Schools, January. zyxw
1981, Vol. 18. No. 1.
METHOD
Subjects zyxwvuts
Two groups of students were used in this study: the total group of first-year school
psychology students (N= zyxwv
1I), and the pool of recently accepted candidates in the school
psychologyprogram who would be beginning their studiesin the followingsemester,and
were in the vicinity of the University at the time of the evaluation zy
(N-10).
All first-year students had successfully completedthree core courses in Assessment
and two core courses in Behavioral Counseling zyxw
& Consultation. These five courses in-
cluded practicum experienceboth io the public schoolsand at the University’s Learning
DisabilitiesClinic. None of the newly accepted (control)studentshad taken any graduate
courses prior to the study, and had zyxwv
no prior experience nor training in psychological
testing or consultation.The two groups were strikingly similar in age and socioeconomic
status, as well as on the criteria that the training program uses for admissions decisions
(undergraduate grade-point average, graduate record examination scores, and letters of
recommendation).Since the two groups seem to represent a random sample of the pop-
ulation of students who both apply and qualify for admission to the school psychology
program, the internal validity of the design was fairly satisfactory.
Instrumentation
Report Writing Scales. Two instruments were developed specifically to rate the
quality of psychological reports. The Psychological Report Rating Scale consistsof five
dimensions: language of the report/terminology, organization of the report, focus of the
report, recommendations, and overall diagnostic conclusions/summary, with each
dimensionon a five-pointscale(Pfeiffer& Anthony, 1978).’The scale was found to have
high interrater reliability (39) when used in a pilot study at the University Learning
Disabilities Clinic.
The second report writing instrument secures a global estimate of the
“meaningfulness” of the report from the perspectiveof a parent or teacher.The Teacher-
Parent Rating Form requires the respondent to read over the psychological report and
then “Rate the report as to how meaningful it would be,” on a seven-point scale from,
“the report is not meaningful, and does not describethe child clearly,” to “the report is
extremely meaningful, and very accurately describes the child.” Twelve sets of parents
obtained an interrater reliability coefficientof .84 using the TPR, while twelve teachers
secured a .89 coefficient with the TPR.
Consultation Scales. A number of writers have discussed various models of con-
sultation (e.g. Lambert, 1974;Meyers, 1973),and recent studieshave attempted to com-
pare the relative effectivenessof differentapproaches,such as behavioral vs. processcon-
sultation (Jason & Ferone, 1978). This study viewed both the behavioral and
process/interpersonal elements as important. Successfulconsultation was defined as in-
cluding not only problem identification,feedback,specificinstructions,praise, and plans,
but also the creation of a supportiverelationship. For this reason, the present program
evaluation looked at both interpersonal effectiveness and the quality of behavioral
recommendations offered during a consultation session.
Interpersonaleffectivenesswas evaluated using the Truax scale of genuineness/self-
congruence(Truax & Carkhuff, 1967).A pilot study by Blair (Note 2) describes in detail
the theoretical and psychometric rationale for the choice of this particular five-point
scale.
‘A copy of the Psychological Report Rating Scale can be obtained from the author.

4. Program Evaluation 63 zy
The behavioral aspect of consultation was evaluated using a newly developed obser-
vational scale (BehavioralRecommendations), which consisted of a five-point rating of
the consultant’s recommendations, ranging from, “the school psychologist did not offer
any specific suggestion(s), plan(s), or recommendation(s),” to “the school psychologist
offered one or more specific suggestion(s), plan(s), or recommendation(s).”
Procedure
Report WritingSimulation. A fictitious case was assembled based upon information
from the University’s Learning Disabilities Clinic. The case consisted of a reason for
referral, classroom and behavioral observations, medical, developmental, and family
history, summary of family interview, and test protocols of a WISC-R, zy
B-G,
figure
drawing, ITPA, PIAT, Sentence Completion, five subtests of the Detroit, and four TAT
stories. Each student was to review the case, and then “put together a comprehensive
psychological report.” The instructions for the incoming (control) students were slightly
modified, to account for their lack of experience with analyzing test material: “We un-
derstand that you may have no training in either interpreting test data or writing a
report; that is O.K.! Just try to put the various information together as best as you can
into a meaningful report. Include not only a statement on findings, but also a summary
and set of recommendations. You will not be graded on this task. The school psychology
program is simply trying to determine how well incoming students can write psy-
chological reports, without prior training.” The task was untimed, but studentsgenerally
needed no more than 1%to 2 hours to complete the report.
The 21 completed psychological reports were coded and typed to control for
anonymity and the influence of handwriting. Two doctoral-level, certified school psy-
chologists were asked to assist in rating the reports. The two judges reviewed with the
author the Psychological Report Rating Scale, discussing all of the dimensions, clarify-
ing questions, the five-point scale, and both independently and as a group practicing
rating two reports. The two judges then rated the 21 psychological reports, with both
raters evaluating 10 of the same reports as a check on interjudge reliability.
Six parents and four elementary school teachers also were asked to review the 21
psychological reports, rating each on the Teacher-Parent Rating Form. As with the
judges used for the Psychological Report Rating Scale, the parents and teachers met in
two groups to discuss and try out the rating form they would be using. zyx
An index of inter-
judge reliability was similarly secured with the parents and teachers.
ConsultationSimulation. A 20-minute appointment was scheduled for each student
with a trained graduate assistant at the Learning Disabilities Clinic. Students were told
beforehand that they would be requested to role-play a school psychologist during a con-
ference with a person enacting a third-grade teacher. Prior to the sessions, a standard
script (see Blair, Note 2) had been developed and extensively practiced and memorized
by a research assistant. The simulated consultation was held in a private room where
videotape equipment was unobtrusively located in one corner. Each student was
presented verbally with the problem situation, involving the teacher and one of her male
students. The taped simulations were shown to two doctoral-level counseling psy-
chologists who independently reviewedand rated them on the Interpersonal Efectiveness
variable, with both judges evaluating 10 of the same videotapes. Two doctoral-level
behavioral psychologists also reviewed the videotapes, rating each on the Behavioral
Recommendations scale. As with the other instruments, an assessment of interrater
reliability was made.

5. 64 zyxwvutsrq
Psychology in the Schools, January, 1981, Vol. 18, No. zyx
1. zyx
RESULTS
Reliability of the Instruments
All four of the instruments yielded acceptable interrater reliability coefficients:
Report Rating Scale=.88; Teacher-Parent Rating Form= .91 for the teachersand .87 for
the parents; Interpersonal Effectiveness(genuineness)=.74; and Behavioral Recommen-
dations=.92.
Report Writing
Results clearly indicatethat the presently enrolled school psychology students were
much more competentthan newly enrolledstudentsin writingpsychologicalreports. The
reports written by the first-year students werejudged stronger zyx
on both technical aspects
(PsychologicalReport Rating Scale), t( 19)=11.12,p<.Ol, and inoverall meaningfulness
(Teacher-ParentRating Form), t(19)=7.21, zyxw
p<.Ol.
Consultation
The findings with regard to performance in consultation were not as clear-cut as
with report writing. First-year students were judged to be more competent than newly
enrolled students in providing specific behavioral suggestions, t(19)=3.89, p<.Ol, but
there was no differencebetween the two groups in interpersonaleffectiveness,t( 19)=S3.
Intercorrelations among the four dependent variables for both first-year and in-
coming students were run. None of the variables was significantlyrelated to any other
among the incoming students, as would be expected. In contrast, the zyx
two features of
report writing (Psychological Report Rating Scale: Teacher-Parent Rating) and con-
sultation (BehavioralRecommendations: Interpersonal Efectiveness) were highly cor-
related among first-year students.As expected,the technical aspectsof the psychological
reports and their overall rating on meaningfulness produced a significant positive cor-
relation (r=.84,p < .01). Contrary to program expectations,the independent ratings for
Behavioral Recommendations and Interpersonal Effectiveness were negatively cor-
related among first-year students (r=-.73, p<.05), suggesting that for the presently
enrolled students interpersonal effectiveness and the generation of specific behavioral
recommendations were incompatible features of their skills as consultants.
DISCUSSION
The present pilot study demonstrated the feasibilityof evaluatinga CBTP. A set of
reliable instruments were developed that provided a school psychology program with
useful informationon how well two exit competencieswerebeing mastered.The program
was able to make modificationsin existing training components and emphases based on
sound feedback.
With regard to the two competencies that were investigated, it is clear that the
school psychologyprogram was doing a finejob of training its students in psychological
report writing and generatingspecificbehavioral recommendationsduring consultation.
The area that did seem to get adequatecoverage,based upon the present evaluation,was
training studentsto be more “genuine” (i.e., facilitate a more supportive, open interper-
sonal encounter during consultation). In fact, one might speculate that the training
program’s behavioral emphasis differentially reinforced a strong didactic approach to
consultation, at the cost of not providing students with a humanistic framework within
which to operate.
The present program evaluation was limited in scope in at least two ways. First,
although the evaluation provided the one particular school psychology program under

6. Program Evaluation zyxw
65 zy
review with useful data, the results cannot be generalized safely to other training
programs (i.e., the external validity of a program evaluation is typically lacking). And,
second, since the evaluation was the training program’s first attempt at examining how
well its students were mastering core competencies, only two dimensions wereexamined.
Future research would need to evaluate the many competencies not looked at in the pres-
ent study. Additionally, an effort should be made to validate the present system by hav-
ing the program go into the field and see how effectively its graduates are performing on
the same criteria.
Although the present exploratory study illustrated the feasibility of evaluating a
CBTP, the practicality of such an enterprise remains questionable. The development, im-
plementation, and rating of simulation and situation response tests were extremely time-
consuming, expensive, and laborious. The need for a CBTP to continually develop new
simulations to ensure their effectiveness in measuring student competence (and not
simply “learning to perform for the test”) demands a major program commitment to test
development that most, if not all, training programs do not have the luxury of affording.
One strategy to assist in evaluating the competencies of CBTPs might be the
national development of a simulations-system assessment package. A representative
group of school psychology trainers, practitioners, students, and consumers of school
psychological servicescould be given the charge of: (a) compiling an exhaustive listing of
competencies, (b) designing appropriate and psychometrically sound simulation tech-
niques (such as Engin zyxwvu
& Klein, 1975; or Gallessich, 1974, describe), (c) making them
readily available to training programs, and (d) continuously gathering feedback to
evaluate and modify the simulations.
This author believes that trainers can, and in fact are ethically responsible to, ar-
ticulate in clear, operational terms the competencies that their graduates need to attain,
no matter how elusive or seemingly complex particular roles may appear. The present
study empirically demonstrated that competencies can be evaluated. However, programs
will face the problem of the impracticality of designing and implementing meaningful
ongoing competency-based evaluations. This dilemma needs to be addressed by training
programs that wish to make the CBTP model work. zyxw
REFERENCE
NOTES
1. ANDERSON,
R., & SMITH,
D. C. School psychology handbook (First revision). Unpublished manuscript,
Florida International University, 1975.
2. BLAIR,
J. zyxwvutsrqpo
A process evaluation of the consultation component zyxw
of the competency-based school psy-
chology program at F.I.U.Unpublished manuscript, Florida International University, 1978.
3. PFEIFFER,
S. I. School psychology handbook 11. (Second revision). Unpublished manuscript, Florida
International University, 1977.
4. PFEIFFER,
S. I. The multidisciplinary team and nondiscriminatory assessment. Manuscriptsubmitted for
publication, 1980.
5. PFEIFFER,
S. I., & ANTHONY,
L. The development ofa psychological report rating scale. Unpublished
manuscript, Florida International University, 1978.
REFERENCES
AMERICAN
PSYCHOLOGICAL
ASSOCIATION.
Accreditation procedures and criteria. Washington, D.C., 1973.
BARDON,
J., & BENNETT,
V. School psychology. Englewood Cliffs, NJ: Prentice-Hall, 1974.
BROWN,
D. T., & LINDSTROM,
J. P. Directory ofschool psychology trainingprograms in the United States
ENGIN,
A. W., & KLEIN,
E. The effectiveness of a simulation technique as an integral part of a school psy-
and Canada. Stratford, CT: National Association of School Psychologists, 1977.
chology training program. Journal of School Psychology, 1975, 13, 171-184.

7. 66 Psychology in the Schools, January, 1981, Vol. 18, zyx
No. zy
1 zyxw
GALLESSICH,
J. zyxwvutsrqpo
Training the school psychologist for consultation. Journal of School Psychology, zy
1974, 12,
GAVILAN,
M., SMITH, D. C., RYAN,
C., zyxwvut
& ANDERSON,
R. Evaluation of a performance-based program in
HUNTER,
C., & LAMBERT,
N. M. Needs assessment activities in school psychology program development.
JASON, L. A,, & FERONE,
L. Behavioral versus process consultation interventions in school settings.
LAMBERT,
N. A. A school-based consultation model for school psychological services. Journal of School
MOWDER,
B. A. Training school psychologists: The issue of competency-based education. Professional
NATIONAL
ASSOCIATION
OF SCHOOL
PSYCHOLOGISTS.
Standardsfor trainingprograms in schoolpsychology,
1978.
REILLY,
D., BARCLAY,
J., & CULBERTSON,
F. The current status of competency-based training, Part I:
Validity, reliability, logistical and ethical issues. Journal of School Psychology, 1977, 15, 68-74.
STUFFLEBEAM,
0. L. Evaluation (
I
S enlightenmentfor decision making. Columbus, OH: Evaluation Center,
Ohio State University, 1968.
TKUAX,
C. B., & CARKHUFF,
R. R. Toward efective counseling and psychotherapy: Training & practice.
Chicago, 1L: Aldine, 1967.
138-149.
school psychology. The School Psychology Digest, 1977, 6, 37-50.
Journal of School Psychology, 1974, 12, 130-137.
American Journal of Community Psychology, 1978, 6, 531-543.
Psychology, 1973, 11, 5-15.
Psychology, 1979, 10, 697-102.