by John E. Westfall, James W. Peltier and Joseph Sheehan
ince passage of the federal No Child Left NCLB mandates rigorous improvement planning
Behind Act (NCLB) Act, the majority of for schools that do not meet threshold scores in
principals in public schools have embraced reading and math. Significantly, schools are required
some type of quality improvement planning for to use research based approaches in their school
increasing student achievement.1 improvement planning, pointing the attention of
Their motivation is due partially to the fact that principals and school district administrators toward
relevant research literature.
Despite the undeniable need to use school quality
improvement planning mandated by NCLB, most
In 50 Words principals have been slow to adopt current and
Or Less sophisticated quality practices, particularly when
compared to administrators in the public and pri-
vate sectors of social services and healthcare.
• School administrators have been slow to adopt quality
One indication of this lagging adoption is the
practices. late entry of the education sector into the Malcolm
Baldrige National Quality Award. Only three school
• Best practice research led to use of an enhanced logic districts have received the award since the segment’s
entry, compared to 10 service businesses over the
model for quality planning in a large Wisconsin school
same period. Clearly, we need more conceptual and
district. empirical research on which principals can build
their quality planning initiatives.
• The model resulted in a three-phase initiative to The school improvement literature clusters in
three areas, yet all are substantially prescriptive
improve curriculum and change classroom practices.
approaches.2, 3 In most cases, principals adopt—or
I AUGUST 2005 I 55
in some cases purchase—an approach, handing highly prescriptive initiative for quality improvement.
down the “what” and “how” of quality improve- A districtwide discussion of quality improve-
ment to the teaching staff. ment started in December 2003 and concentrated on
Unfortunately, the literature is replete with exam- best practices from the effect-size research of school
ples of how top-down approaches for quality im- improvement, a common index for measuring stu-
provement have failed to produce desired learning dent achievement in quasi-experimental designs.
outcomes. It brings to mind the expression, “To a This effect-size literature documents about 8,000
child with a hammer, everything is a nail,” in that quasi-experimental designs from the past 40 years.
many principals see only prescriptive rather than Each study reports the measurable impact or
vision sharing approaches to quality improvement. effect-size a specific change in school practice had
There are, however, a small but growing number on student achievement. R.J. Marzano’s meta-
of thoughtful educators advocating more empow- analysis reviews these major studies.4
erment based models. These models promote col- Effect-size is the difference between the experimen-
laboration, mutual understandings and researched tal and control group means divided by an estimate
based consensus about what quality education is. of the population standard deviation. Therefore, it
One vision sharing approach with broad applica- expresses the differences between group means in a
tions across public education is called the enhanced standardized or Z score structure. (See the outer ring
logic model for quality planning (see Table 1). of Figure 1 for examples of effect-sizes.)
A larger effect-size of 1.0 or 1σ is equal to about
District Planning Example 30 percentile points toward the middle of the distri-
This case study examines the Sheboygan Area bution on standardized tests. In other words, imple-
School District (SASD) in Wisconsin, a district of menting a larger effect-size practice in a school
about 10,000 students and 840 teachers. In many would likely raise the average score of students
ways, SASD is a microcosm of school districts across scoring at the 50th percentile to approximately the
the United States and thus a good case for testing 80th percentile. Nevertheless, most studies yielded
quality improvement strategies and methods. smaller (~0.2) or even negative effect-sizes.
SASD has relatively large minority and economi-
Enhanced Logic Model
cally disadvantaged populations. Student achieve-
ment scores districtwide, while near the state The process for developing and deploying best
average, have remained flat. Teachers’ philosophies practices into the classroom was set into an enhanced
toward teaching vary extensively, yet these differing logic model. A logic model is a visual representation
approaches are accepted within the fiercely indepen- of a quality improvement process that explicitly links
dent structure of schools and classrooms. Because of outcomes (desired changes in student cognition, atti-
this diverse environment, it was unlikely staff (or tudes or behavior) with resources, activities and
students and parents) would voluntarily adopt a products.5
Enhanced Logic Model for Quality Planning
Inputs Methods and activities Outputs Outcomes Indicators Interventions
Stakeholders and The research-pyramid Quality improvement cycle Explicitly identifying what Making certain each student Arranging in advance what to do
research literature approach, collaborative for student achievement. a student should know, knows, feels or does the when a student doesn’t know, feel
(see Figure 1). techniques used for building feel or do by subject area explicit outcome by using or do the explicit outcome by using
Districtwide and school
understandings and and grade level for valid common assessments school hour interventions, in
based action plans.
consensus from increasing student across subject areas and contrast to after school or summer
the ground up . achievement. grade levels. ones.
I AUGUST 2005 I www.asq.org
The enhanced logic model adds indicators and Inputs for Student Achievement
interventions. Although logic models are increas- And Educational Quality
ingly used within other service disciplines, princi-
pals and other school management decision mak-
ers have not adopted them.
Initiating a districtwide improvement campaign viab
in January 2004, an ad hoc steering committee
employed the enhanced logic model for quality
go ent for
planning in Table 1 to detail a quality planning
schedule, which included a set of larger effect-size
or quality improvement strategies as inputs to the
Inputs or resources included staff, planning time
l e a i ni
and the quality improvement literature for schools
st r s
(see Figure 1). This literature was used to prompt
group discussions and surveys. Journal articles about
d e er
the quality strategies were distributed to all stake-
eo s u lop
holder groups prior to the discussions. in f hi lt y
F ac u dev
These discussions included four major groups gs
t a ff
k i ll s
and centered on the five larger effect-sizes listed
in Figure 1, which became the prime inputs to the
quality improvement plan. Thus, the plan was
built through consensus of what would constitute
quality education within and between groups.
The consensus strategies included a guaranteed
and viable curriculum, or one in which students staff’s level of understanding regarding best practices
learn essential content and are given the time they and a wealth of other organization specific matters. As
need to learn it. Rigorous goals and assessments such, no single set of methods and activities exists that
for learning include a depth of essential content, matches or is appropriate for all scenarios. Instead,
not simply coverage of topics, and common dis- the philosophy adhered to for any quality improve-
trictwide assessments useful in guiding such ment project should be organization specific.
depth of instruction. Within SASD, these considerations had to pro-
Faculty support and staff development comprise vide multiple opportunities for staff engagement
an empowerment based approach, providing need- and maximum facilitation for consensus building.
ed resources to teachers. Students’ use of higher The process had to be completely open, inclusive
order thinking skills, such as reasoning and analy- and transparent. The ad hoc steering committee
sis, are classroom practices that build cognitive skills chose the research-pyramid approach as a frame-
around the essential content. Differentiated instruc- work of methods for meeting these various goals.6
tion and effective feedback demonstrate teaching The research-pyramid approach (see sidebar on
differing students in different ways to motivate p. 60 for more information) is a mixed methods
them and increase individual achievement. design using qualitative and quantitative tech-
niques and rooted in the belief that extensive
Methods and Activities exploratory or qualitative research must first be
The approach used for any specific quality conducted and foundations established before
improvement initiative depends on such factors as more generalizable specific quantitative/descrip-
the organization’s environment and culture, the tive research efforts can be undertaken.7, 8
I AUGUST 2005 I 57
Stages of Collaboration by Stakeholder
Stakeholders Discussions Survey Focus Survey Prioritize best Align priorities with Draft quality
groups practices resources improvement cycle
and action plans
Dates January April May May May June June
● ● ● ● ● ●
● ● ● ● ● ●
● ● ● ● ●
● ● ● ● ●
● ● ●
● All group members.
● Selected representatives.
surveys were conducted with each group using the
The qualitative stage is designed to:
five larger effect-sizes as prompts. These examined
• Obtain insight from key publics within and
quality practices teachers should continue, enhance
outside the organization.
or add vs. practices teachers should reduce or cut.
• Generate a deeper understanding of impor-
As shown in Table 2, the series included:
tant quality issues.
• Initial informal discussions.
• Identify question content areas to include in
• Six districtwide focus groups with 80 educa-
the survey stage.
These publics, represented by key informants,
• Focus groups with all teachers at each school.
are believed to have insight critical to furthering
• A qualitative survey of all staff.
the data collection process and are queried for
• A quantitative survey of educational leaders.
understanding the above goals. Such conversations
This collaborative data collection process took
are additionally useful because they provide staff
about 1,450 hours of staff time over the spring
the opportunity to participate in the quality initia-
semester and represented less than two hours per
tive and build consensus within the organization.
teacher. Moreover, the actual quality plan re-
In this way, the process becomes a social as well as
arranged underutilized time so teachers could
research activity. Focus groups are also crucial for
work on curriculum improvement teams. Each
identifying the most important quality issues to
teacher received 16 hours per semester for working
address in the subsequent quantitative research.
on curriculum teams, which represented an eight-
The qualitative stage provides the foundation for
fold return on the time invested in planning.
the descriptive survey. The survey results may be
A qualitative survey of all staff resulted in about
used for establishing baselines that can be general-
500 respondents and 270 pages of transcript con-
ized to the overall population. Moreover, findings
taining about 6,000 comments. A content analysis
are valuable for developing effective and efficient
reduced the transcript to a set of 242 classroom
long-term quality improvement plans that have
practices staff suggested be continued, enhanced,
initial buy-in from stakeholders.
added, reduced or cut.
In practice, a series of group discussions and
I AUGUST 2005 I www.asq.org
A quantitative survey of educational leaders rules or principles).
examined about 70 classroom practices. The input • Unpacking is a set of techniques used to artic-
variables (classroom practices) were regressed ulate and ensure the alignment or validity of
against the output variable, “I believe these the standards, benchmarks and proficiency
strategies/practices will greatly improve student criteria.
achievement.” The results of this analysis confirmed • Common classroom assessments are valid
strong predictive validity with an R2 of 0.833. measures of the essential benchmarks and
Educational leaders then recommended an explicit skills.
set of classroom practices for increasing student In fact, these concepts are similar to those used
achievement. in other quality models, although the educational
Next, a representative districtwide team of 25 terms may be quite different.
took all results and reduced them to a set of 76
Example With Outcomes
action priorities (enhance, add, reduce or cut). The
Now a concrete example. One of about 36 Wis-
priorities revolved around best practices, which
consin eighth grade math standards says, “Students
were incorporated into the quality improvement
will use reasoning abilities.” Previously, the math
staff unpacked the standard by drafting bench-
Outputs marks, which included one saying students will
The primary output or product from this collab- demonstrate estimating skills in recognizing the
oration was the SASD quality improvement cycle reasonability of an answer and defend their work
for student achievement, a three-phase initiative in a variety of ways—for example, written, oral or
for continuously improving curriculum and chang- charts.
ing classroom practices. The phases of the cycle are Staff is now articulating proficiency criteria and
linear, but tasks within each phase are not neces- developing rubrics or scoring guides for measuring
sarily sequential. Staff consensus viewed comple- student performance against the proficiency. Next,
tion of the cycle as likely to result in large increases they will develop assessments to determine each
in student achievement. student’s level of proficiency as measured against
The SASD quality improvement cycle for stu- the rubric. Finally, staff repeats this cycle of curric-
dent achievement includes steps to: ular development for each standard (there are
1. Align standards and benchmarks and inte- about 500), grade level and subject.
grate them into classroom practices. These are not quick fixes. Increasing student test
2. Align essential benchmarks and skills, big ideas results through meaningful school improvement
(also known as essential questions), proficiency initiatives takes at least several years. However, the
levels and common classroom assessments most recent state testing results (currently embar-
throughout the district and integrate them into goed) show SASD increases across most grades
instructional practices. and subject areas. Significantly, in Report on the No
3. Create or revise course content and create or Child Left Behind (NCLB) Act, Year 3, the Center on
revise standards based report cards. Education Policy concluded SASD seemed to be
Some definitions are important at this point: producing positive achievement results, particular-
• Academic standards are issued by each state ly in narrowing the gap between differing student
and include what students should know and groups.9
be able to do and what schools are expected to Additionally, in its first year of implementation
teach. the district has shown encouraging results by
• Benchmarks translate standards into develop- meeting all its initially targeted outcomes. Subject
mentally appropriate grade-level learning out- area and grade level teams have conducted their
comes. self-assessments and begun executing their indi-
• Proficiency is an acceptable level of student vidual quality improvement action plans.
performance based on criteria (guidelines, Staff and parents have responded positively,
I AUGUST 2005 I 59
The research-pyramid approach Research Tools to improve public relations with the
has been used successfully within community and provide a positive
hundreds of organizations and image of the United Way as an
• A community VIP steering
across most industry segments. effective and responsive organiza-
The following is an example of its tion.
• Key informant interviews and
use within the nonprofit sector. The results were communicated
focus groups with educators,
to all key publics including the local
clergy, funded agencies and
media. Meetings were also held to
non-funded agencies, health-
The social services problems in
convey the findings to key con-
care professionals, minorities,
Sheboygan County, WI, are com-
stituency groups. (Additional infor-
social services personnel, gov-
plex. There are many client groups
mation about the study, including
ernment/law enforcement per-
and service providers and a vast
reports, are available online at
sonnel, area business leaders
array of community perceptions on
needs and problems.
• Questionnaires sent to a large- Outcomes
A community needs assessment
scale sample taken from key
was conducted to gather as much The allocations committee of
information from a broad range of the United Way has since directed
possible perspectives. This approach funding toward programs that
Use of Findings
ensured all perspectives were repre- address the highest priority
The needs identified and priori-
sented to the greatest extent possi- needs. The committee has begun
tized in the study were used to set
ble, and the collection of valid data a system with all funded agencies
the framework for funding alloca-
would provide better information for of outcome measurement with
decision making. semiannual reporting. All future
The Sheboygan Area United Way funding will be based on the
wanted information with which to needs assessment and the out-
It was essential the publics
guide funding and other decisions, come measurement reporting.
understood the United Way was
but an important secondary effect Furthermore, in the most recent
interested in their perspectives and
of the community needs assess- campaign following the study,
needs, would be listening to their
ment process was to develop good donations increased about 20%
concerns, would be involving them
relationships with the major stake- from the previous year. This was
in the process and, most important,
holders. To begin, stakeholder especially noteworthy in a year in
would act upon the data once it
groups were identified and classi- which donations across the United
fied into three general segments: States were flat to declining.
Thus, the effect of the communi-
donors, providers and clients. ty needs assessment process was
I AUGUST 2005 I www.asq.org
with about 420 teachers (50%) having volunteered 2. G.D. Borman, G.M. Hewes, L.T. Overman and Shelly
to work on districtwide quality improvement Brown, “Comprehensive School Reform and Achievement:
A Meta-Analysis,” Review of Educational Research, Vol. 73,
teams. This is clear demand based evidence of ini-
No. 2, 2003, pp. 125-230.
tial success, deployment and engagement across
3. R.J. Marzano, A New Era of School Reform: Going Where
staff and parents.
the Research Takes Us, McRel, 2000.
Of special note, there were no additional costs or
time for the planning or initial implementation.
5. Logic Model Development Guide: Logic Models To Bring
Besides, the district has made a financial commit- Together Planning, Evaluation & Action, W.K. Kellogg Foun-
ment to quality improvement. In future years, as dation, 2001.
needs are more clearly identified, resources will be 6. Information on the research-pyramid approach can be
allocated through the district’s budgeting process. found at www.research-pyramid.us.
It is at the intermediate and long-term stages 7. J.E. Westfall, Relationship Marketing in Educational
that increases in student test scores are expected. Administration: An Empirical Study, doctoral dissertation,
Cardinal Stritch University, UMI: AAT 3078590, 2003.
As the district continues to work through the cycle,
8. J.W. Peltier, J.E. Westfall and T.L. Ainscough, “Using the
it will build its student achievement and other
Pyramid Approach To Teach Marketing Research,” Business
databases, using these to longitudinally model aca-
Education Forum, Vol. 55, No. 4, 2001, pp. 40-43.
demic growth by subject area and grade level and
9. Report on the No Child Left Behind (NCLB) Act, Year 3,
thus target resources as needed.
The superintendent, principals and teachers con-
tinue monitoring and adjusting the quality improve-
JOHN E. WESTFALL is coordinator of assessment for the
ment action plans each semester. Department and
grade level teams continue analyzing the results Sheboygan Area School District in Wisconsin. He previ-
from agreed on common assessments to adjust ously held product development and research positions with
instruction and provide students with meaningful the Kohler Co. and BlueCross & BlueShield. Westfall
feedback. earned a doctorate in education and marketing from Car-
Finally, building-level teams continue articulat- dinal Stritch University, Milwaukee. He is a member of
ing a targeted sequencing of interventions for stu-
ASQ and serves on Quality Progress’ editorial review
dents who are not achieving or need to enhance
their learning. Comparatively, this process is simi-
lar to those used in healthcare by physicians and JAMES W. PELTIER is a professor of marketing at the
administrators when collaboratively drafting com- University of Wisconsin-Whitewater. He earned a doctor-
mon diagnostic and treatment protocols for quality ate in marketing from the University of Wisconsin-
The following conclusions are currently support-
JOSEPH SHEEHAN is superintendent of the Sheboygan
Area School District, Sheboygan, WI. He earned a doctor-
• The quality improvement model can be repli-
cated by other school districts. ate in education from Marquette University in Milwaukee.
• There is evidence broad collaboration between
groups helps build a quality focused environ-
• Initial incremental costs are low.
• The initiative deserves its place in the quality
If you would like to comment on this article,
REFERENCES AND NOTES
please post your remarks on the Quality Progress
1. Katherine Ryan, “Shaping Educational Accountability Discussion Board at www.asq.org, or e-mail
Systems,” American Journal of Evaluation, Vol. 23, No. 4, 2002,
them to firstname.lastname@example.org.
I AUGUST 2005 I 61