Final Class Presentation - CSDE - QR Team

1. An Analysis of Quality Review
Alignment with Selected
Governance Models
Connecticut Quality Review Team
Jamie Alter, Gabriella Barbosa, Ranjana Reddy, Ryan Thorpe, William David Williams
April 8, 2013
1

2. Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review
with experimentalism
• Challenges in designing an experimentalist Quality Review
system
• Examination of an unresolved question: Should CSDE use
Quality Review for school accountability?
2

3. The team supported the Connecticut State Department of Education
by designing standards, tools, and a strategic process for piloting
Quality Review.
Created first iteration of
QR standards, tools, and Conducted focus groups Strategically recruited
process based on with CSDE to solicit districts to participate in
research and effective feedback pilot
practices
Supported training of
Solicited feedback on Designed Reviewer and reviewers and
QR from district leader School Guides orientation for schools
and district personnel
3

4. During the QR process, reviewers will assess schools against four
overarching standards.
Overview of Standard Rationale
• High quality schools have an overall vision focused on a
1. Create and monitor a
pathway to college and career readiness for all students that
vision and theory of
has generated a thoughtful theory of action, comprehensive
action focused on student
data gathering, and careful plans for implementing the theory
learning
of change.
• High quality schools evaluate and select appropriate
2. Develop and implement strategies, plans, instructional programs, and curriculum for
a rigorous instructional their circumstances to support college and career readiness
program that meets the for all students.
learning needs of all • Similarly, they organize themselves so that school leaders
students are helping to facilitate continuous improvement in
educator effectiveness at meeting learning goals.
• High quality schools promote a positive learning
3. Create a school culture environment and engage families and the community in
and climate that enables continuous improvement.
all students to learn
4. Align management and • High quality schools continuously monitor and improve
operations to facilitate management practices to ensure the are effectively serving
achievement of student student learning as the school’s primary objective
learning goals and
steadily improve 4

5. These standards are aligned to 17 indicators of school practice.
Standard Indicator (To what extent do schools…)
1.1 Create a Shared Vision for Student Learning
Student Learning-
Centered Culture
1.2 Develop Theory of Action and Goals
1.3 Target Student Needs
1.4 Assess and Monitor Progress
Continuously- 2.1 Develop Rigorous Curriculum
Improving
Instruction
2.2 Build Instructional Capacity
2.3 Develop Teacher Leadership
2.4 Promote Evidence-based Instruction

6. These standards are aligned to 17 indicators of school practice.
Standard Indicator (To what extent do schools…)
3.1 Establish Welcoming Environment
Student and
Family Investment
3.2 Engage Teachers and Staff
3.3 Engage Students
3.4 Communicate with and Engage Families
3.5 Build Community and Family Partnerships
Operational 4.1 Align Resources and Policies to Goals
Alignment to
Goals 4.2 Monitor and Revise Operations
4.3 Manage Talent
4.4 Develop and Pursue Innovative Solutions

7. The Quality Review process involves pre-review data collection, the
2-day onsite review, and post-review feedback.
Pre-review Review Post-review
Description: Reviewers and schools prepare During a two-day school visit, Reviewers provide verbal and
for the formal process of data reviewers collect and synthesize written feedback to schools
collection by gathering evidence from a variety of based on evidence collected
documents, examining school sources to gain a full picture of during the quality review and
data, self-evaluating, and school performance against the facilitate a structured debrief
setting logistical parameters. set of standards for effective and reflection on results.
schools.
Key steps: • Principal Orientation • Classroom Observations • Presentation of Findings
• School Self-Evaluation • Constituent Interviews • Goal Setting and Planning
• Conversation with District • Observation of Teacher Teams • Rating and Scoring
Staff • Reviewer Check-ins • Written Report
• Planning Conversation Among • Principal Debriefs • Verification
Reviewers • Reconsideration, if necessary
• Planning Conversation with
School Leader

8. The pilot will include two teams of reviewers conducting quality
reviews in 4 East Hartford schools in April/May.
School 1
REVIEWER TEAMS SCHOOLS
•
Reviewer
Include at least 1 • Four elementary
state and 1 district Team 1 schools; identified
reviewer by CSDE as Focus
School 2 Schools
• Trained on the QR
standards and • Located in Alliance
process by CSDE District
vendor
• District has 61%
• Have experience as FRL, 47% proficient
educators and/or
School 3
reading, 64%
educational leaders Reviewer proficient math
Team 2 (Grade 3 CMT)
School 4

9. The QR pilot is designed to accomplish three key objectives.
• Reviewers, districts, and schools will provide feedback on the pilot tools,
To troubleshoot and process, and reports.
improve the • This feedback will be used to revise the QR system and to inform
standards, tools, and recommendations for roll-out.
process
• East Hartford will have an opportunity to test-drive quality review.
To increase • The CSDE will conduct outreach sessions with other Alliance Districts to
encourage more districts to engage in Quality Review in future years.
stakeholder support
for Quality Review
• State reviewers will be trained to conduct reviews, evaluate schools, write
reports, provide verbal feedback to schools, and to ensure inter-rater
To build capacity reliability through internal training.
within the state • The CSDE will develop internal procedures for collecting and analyzing data
from quality reviews and managing the pre- and post- review elements of the
process.
• Personnel from East Hartford will gain experience as co-reviewers.
9

10. Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review
with experimentalism
• Challenges in designing an experimentalist Quality Review
system
• Examination of an unresolved question: Should CSDE use
Quality Review for school accountability?
10

11. CT QR was designed to inform an “experimentalist” problem-solving
process at the school level, by providing schools with targeted
feedback on areas of improvement
Clarify and Translate
Visions and Strategy
SCHOOL
Communicate and Link PROBLEM Enhance Strategic
Strategic Objectives SOLVING FOR Feedback and Learning
and Measures IMPROVEMENT
Plan, Set Targets and
Align Strategic Initiatives
11

12. CT QR will ensure that expertise flows from local learning to the
center, and that the state, in turn, will accumulate and share local
learning across schools.
SCHOOL SCHOOL
State Knowledge
Base of Effective
Practices
SCHOOL SCHOOL

13. Other specific design features of CT QR were also intentionally linked
to experimentalism.
Feature of CT QR How it Meets an Experimentalist Objective
The system is built around standards that
exemplify effective schools, while being broad
Strategic, Open-Ended
enough to generate fruitful conversation and
Standards
learning among schools operating in different
contexts.
Processes such as self-evaluation, principal
check-in, and goal setting meetings
Involvement of School
empower local school leaders to be a critical
Throughout Process
part of monitoring and adjusting their own
performance.
Experimentalism allows for a diverse group of
Involvement of Other stakeholders to be included as part of an
Stakeholders accountability system, including those usually
left out of the conversation like parents and
the community. 13

14. Standard 4: Operational Alignment to Goals
Indicator 4.2: Monitor and Revise Operations
• Minimal process. School does not have in place any coherent system for regularly
monitoring the effects of school operations and management systems on student learning
Minimal or outcomes.
no evidence
• Limited process. School has a plan for monitoring effectiveness and efficiency of
Progressing operations and management systems for their effects on student learning outcomes, but
toward inconsistently uses the information learned from this monitoring.
standard
• Strategic process. School effectively and strategically plans and executes systems to
regularly monitor and revise effectiveness and efficiency of operations and management
Meets systems to improve their impact on student learning outcomes.
standard
• Strategic process and continuous improvement. School effectively and strategically
plans and executes systems to regularly monitor and revise effectiveness and efficiency of
Exceeds operations and management systems to improve their impact on student learning
standard outcomes.
14

15. Other specific design features of CT QR were also intentionally linked
to experimentalism.
Feature of CT QR How it Meets an Experimentalist Objective
The system is built around standards that
exemplify effective schools, while being broad
Strategic, Open-Ended
enough to generate fruitful conversation and
Standards
learning among schools operating in different
contexts.
Processes such as self-evaluation, principal
check-in, and goal setting meetings
Involvement of School
empower local school leaders to be a critical
Throughout Process
part of monitoring and adjusting their own
performance.
Experimentalism allows for a diverse group of
Involvement of Other stakeholders to be included as part of an
Stakeholders accountability system, including those usually
left out of the conversation like parents and
the community. 15

16. Self-Review ensures the QR process is collaborative, and empowers
stakeholders by helping them engage in a structured process of
reflection on strengths and weaknesses.
Plan:
• What are our goals?
• How will we achieve them?
• What support do we need?
Reflect: Execute:
• How do we put our plan into
• How are we doing?
action?
• How do we know?
• What more needs to happen
• How did we get here?
to make the plan a reality?
Monitor: Collaborate:
• What progress are we • With whom do we need to
making toward our goals? partner to support our
• How well are we aligning goals?
resources to achieving our • Who needs to know about
goals? our goals and progress?
16

17. Other specific design features of CT QR were also intentionally linked
to experimentalism.
Feature of CT QR How it Meets an Experimentalist Objective
The system is built around standards that
exemplify effective schools, while being broad
Strategic, Open-Ended
enough to generate fruitful conversation and
Standards
learning among schools operating in different
contexts.
Processes such as self-evaluation, principal
check-in, and goal setting meetings
Involvement of School
empower local school leaders to be a critical
Throughout Process
part of monitoring and adjusting their own
performance.
Experimentalism allows for a diverse group of
Involvement of Other stakeholders to be included as part of an
Stakeholders accountability system, including those usually
left out of the conversation like parents and
students. 17

18. Constituent conversations allow reviewers to gain information about
stakeholders’ perspectives and experiences with the school across
multiple indicators.
Focus Group with Teachers
Focus Group with Parents
Focus Group with Students
18

19. Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review
with experimentalism
• Challenges in designing an experimentalist Quality Review
system
• Examination of an unresolved question: Should CSDE use
Quality Review for school accountability?
19

20. Tension #1: An experimentalist QR relies on autonomy at the school
level, but autonomy is noticeably constrained.
EXPERIMENTALIST BUREAUCRATIC
PROPOSAL REALITY
Schools should have the Schools have limited
discretion to develop flexibility in allocating
context-specific strategies financial resources and
to close gaps identified by guiding talent
the QR system management constrain
their ability to make
changes advocated by QR
RESOLUTION
By working with the state and districts to design and implement QR, we
ultimately hope to build buy-in for necessary structural changes at the school
level. Essentially, states and districts will see that schools have internal capacity
to drive change and that they need additional autonomy to facilitate
improvement through the QR process.

21. Tension #1 Resolution: Indicator 4.3 Managing Talent was adapted to
acknowledge school-level constraints around talent management.
Original Indicator Adapted Indicator
Develop and implement a strategy for Develop and implement a strategy for
managing human capital, including marshaling resources to manage human
identifying, recruiting, mentoring, capital, including identifying, recruiting,
providing career-enhancing mentoring, providing career-enhancing
opportunities for, and retaining the opportunities for, and retaining the most
most effective teachers and assigning effective teachers and assigning teachers
teachers to areas where their skills to areas where their skills best match
best match student learning needs. student learning needs, given each
individual school’s contextual
constraints.
21

22. Tension #2: An experimentalist QR uses broad standards that are not
prescriptive, but stakeholders want clear expectations.
EXPERIMENTALIST BUREAUCRATIC
PROPOSAL REALITY
CPRL proposed a rubric with The CSDE and many school
intentionally broad standards and district personnel believe
to encourage a dialogue that providing specific
between reviewers and standards with concrete
school personnel and to resist expectations for acceptable
the tool’s use as a compliance evidence ensures the process
mechanism. is fair and best communicates
expectations to stakeholders.
RESOLUTION
The version of QR being piloted includes a rubric with broad, experimentalist
standards and an evidence document with narrative-style vignettes as
examples of what "well-developed" looks like for each indicator.

23. Tension #2 Resolution: Revised QR includes evidence document with
vignette-like narratives to provide examples of “well-developed.”
Original version included only indicators Adapted version added vignette-style narratives
“The school effectively conveys high expectations to
Indicator 1.1 students and parents that result in a deep commitment to
Create a widely shared vision of the the school’s vision. Success is celebrated during “Honors
progress students are expected to Night” where teachers, parents and students are rewarded
for exemplary work. The school provides parents with
make while at the school to get and continuous information on how well their children are
stay on track to graduate on time learning via report cards, progress reports and Engrade, an
and become college and career- on-line reporting system. In an effort to better support
students and effectively work with families, the school
ready and foster a deeply embedded administers a student survey that aids the school to better
culture of high expectations understand the circumstances of different families and
committed to realizing that vision. differentiate support accordingly. Guidance counselors
work closely with families and actively assist in the
completion of high school applications ensuring that
students and families make good choices when selecting
high schools. The emphasis on high standards and open
exchange of information results in an atmosphere of care
and support. Consequently, parents report appreciating the
high expectations for attendance, good citizenship and
academic effort within a highly inclusive and welcoming
environment.”

24. Agenda
• Overview of Connecticut Quality Review project
• Design choices made to align Connecticut Quality Review
with experimentalism
• Challenges in designing an experimentalist Quality Review
system
• Examination of an unresolved question: Should CSDE use
Quality Review for school accountability?
24

25. The CSDE faces a decision about whether to use the results of Quality
Review as part of its system for school accountability.
School ratings on the qualitative indicators would
be integrated with the quantitative metrics
Factor in School
currently used (state test scores and graduation
Accountability
rates) to classify schools and to determine which
schools require intervention.
Schools would continue to be classified based on
Purely only quantitative metrics; Quality Review ratings
Developmental will be provided only to school and district leaders,
who will be encouraged to use the information to
drive their own improvement.

26. Performance management suggests using QR for Accountability
whereas professionalism would use QR purely developmentally.
QR as a Factor in
Performance
Management
School Experimentalism
Accountability
QR as a Purely
Professionalism Experimentalism
Developmental Tool

27. The CSDE’s decision about how to use Quality Review is closely related
to its theory of action for school improvement.
If the CSDE believes… Then…
Systems of school Schools are likely to
accountability Teachers and school respond to QR for
should…. leaders…. stakes by… QR should be…
• be fair, valid, include • are motivated by fair • taking the standards
both leading and and valid more seriously and Factor in
lagging indicators, assessments of working harder to student
and provide schools performance that are authentically accountability
with actionable public and result in improve
information sanctions and
interventions
• help the state • are professionals • “gaming” the quality
identify schools that who, when provided review by prepping Purely
need the most with information for the review and Developmental
support and autonomy, are performing on the
motivated to day of the site visit
improve without
sanctions

28. Discussion teams
Team 1 Dara, Michael, Rohan
Team 2 Seo Yun, Tom, Zahreen, Jason
Team 3 Jill, Sana, Brian, Jason
Team 4 Regina, Alia, Matthew
Team 5 New Haven

29. Discussion Role Play
• Take a point of view on whether Quality Review should be
part of Connecticut’s school accountability system and why
Task • If so, how would this work?
– Publically reported
– Factor in classifying schools
– Factor in determining whether schools are closed or turned
around
• Split into groups of three or four and take the following roles:
Roles – Person who is youngest will represent a Teacher in the School
– Person who is second youngest will represent the Leader of the
School
– Person who is third youngest will represent the District
Superintendent
– Person who is the oldest will represent the State Commissioner
• Each party should advocate for the use quality review that is
best from the assigned perspective.
• Note that each entity likely has a different viewpoint
regarding the purpose for conducting of quality review as
well as which use of quality review best serves the stated
purpose.

30. As you discuss whether QR should be a factor in school
accountability, consider which use of QR is most likely to meet each
of QR’s primary objectives.
Part of Purely
Acct. Dev. Explanation
Meets primary objectives
Improve accountability
system
Create a common
language around school
quality
Build capacity in districts
and state
Build support among
educators and the public
Encourage and enable
authentic school
improvement
Likely Unlikely
Source: Interviews with districts and vendors.
30

31. Report out.
Factor in Accountability? Purely Developmental?
• To come • To come
Teacher
• To come • To come
Principal
• To come • To come
Superintendent
• To come • To come
State
Commissioner

32. Appendix

33. NCLB exposed poor performance, but is blunt and limited;
Connecticut is developing a richer, more nuanced system.
NCLB A richer system
Use status-based Use rich accountability and
accountability and tiered classification to enable
identification of “schools in and motivate schools to
need of improvement” to improve
motivate schools to improve Support low performing schools
in the process of strategic
planning and intervention
Give targeted low performing
schools the flexibility and
resources they need to
improve
Provide all schools with tools
and information needed to
facilitate improvement
Impose sanctions if schools do Impose sanctions if schools do
not improve not improve
33

34. Qualitative Review could be integrated into Connecticut’s new
accountability system for all schools.
Quantitative Factors
64 < SPI < 88; 64 < SPI < 88;
SPI > 88;
SPI < 64 did not meet met annual
met state goals
annual targets targets
Well
Transition Progressing Excelling Excelling
Developed
Qualitative Factors
Proficient Transition Transition Progressing Excelling
Developing Review Transition Transition Progressing
Under-
Review Review Transition Transition
developed
34

35. Qualitative Review could be integrated into Connecticut’s new
accountability system as an additional component for schools on the
cusp.
Excelling
Schools on the cusp
of Excelling
Schools receive QR for
Progressing next three years
Schools on the cusp If score Proficient or
of Progressing Well-Developed, will
be classified in the
Transition higher category even
if quantitative data
Schools on the cusp falls short by slight
of Transition margin.
Review
35

36. Qualitative Review could be integrated into Connecticut’s new
accountability system as an additional requirement for exiting Focus
and Turnaround status.
Meet subgroup SPI
and grad rate
Lowest subgroup targets
SPI or grad rates Focus Status Achieve Proficient
or Well-Developed
QR Score
Meet SPI and grad
Lowest SPI for rate targets
“all students”
Turnaround Status Achieve Proficient
or Well-Developed
QR Score
36