This document outlines a developmental evaluation approach used by the Bill and Melinda Gates Foundation's Community Partnerships Portfolio to understand system change in the postsecondary field. It describes the current state of the postsecondary system and the goals and outcomes of the initiative. A developmental evaluation was used because the initiative was complex, innovative and untested. Key lessons included that developmental evaluation looks different than other types of evaluation by engaging partners, developing tools for multiple uses, and refining tools over time. The evaluation aimed to develop understanding of what the system change model might look like through a reflective process.

1. Using
Developmental
Evaluation
to Learn about
System Change in the
Postsecondary Field
EERS: 35th Annual Conference
OMG Center for Collaborative Learning
May 1, 2012

2. Bill and Melinda Gates Foundation’s
Community Partnerships Portfolio
CLIP sitesPPS sites

3. Bill and Melinda Gates Foundation’s
Community Partnerships Portfolio: Theory of Change
Current State
Actions to Drive
Change
System
Outcomes
Student
Outcomes
Limited awareness of/concern
about low completion rates
Limited knowledge of potential
solutions to low postsecondary
success (PS) rates
Lack of knowledge about
college-going among students
and parents/guardians
Limited alignment among K-
12, PSE, youth development,
and workforce systems,
especially around
standards/curriculum
Limited relationships among
key stakeholders to drive to a
common goal
Institutional policies &
practices create obstacles to
student success
Increased Academic
Readiness for College work
Increased College Knowledge
Increased Enrollment Intensity
Increased Persistence
Increased Completion
•% of students completing
credential
•% of CC transfer students
completing 4-yr degrees
Increased placement in jobs
with labor market value
1. Community commitment to
achieving PS goals,
especially among key
leaders and institutions
2. Community continuously
measures progress towards
a set PS goal, uses this
information to drive change
and publicly reports
progress.
3. Sustainable structures are
in place for community to
plan, coordinate and
execute strategies that
increase PS
4. Relevant stakeholders
adopt and implement
supportive and effective PS
policies and practices
1. Enhance key PS
stakeholders’ commitment
and action to drive PS change
2. Develop infrastructure and
mechanisms to use
qualitative and quantitative
data to drive PS decision‐
making on an ongoing basis
3. Build sustainable
partnerships with clear
leadership and appropriate
roles and responsibilities to
drive the change agenda
4. Identify and address
necessary policy and practice
changes to align, coordinate
and scale up postsecondary
success pathways and
supports

4. 4 System Outcomes
• Enhanced commitment
• Sustainable partnership structures
• Use of data to drive decision‐making
• Adoption and implementation of effective
policies and practices

5. Why developmental evaluation?
• Formative evaluation
– Is the model being followed?
• Summative evaluation
– Is the model successful?
• Developmental evaluation
– What might the model look like?

6. Why did we use developmental evaluation?
• Complex, innovative, and un-tested
initiative
• Goal of understanding, informing, and
developing a model of system change

7. Lessons
• We need to think differently about the
evaluator’s role
• Developmental evaluation looks
fundamentally different from other types of
evaluation

8. Redefining the evaluator’s role
Bill and
Melinda
Gates
Foundation
MDC
National
League of
Cities (NLC)
OMG Center
Raleigh, NC
Brownsville,
TX
Amarillo, TX
Mesa, AZ
New York,
NY
Riverside,
CA
San
Francisco,
CA

9. 1. Engage multiple partners and perspectives
in developing evaluation tools
• Involve partners in tool/instrument development
• Engage in ongoing communications with
partners
• Be open to partner feedback about evaluation
methods, and willing to change

10. 2. Develop tools for multiple uses
• Evaluation
• Capacity Building
• Technical Assistance
• Planning

11. 3. Refine and revisit tools
• Structure reflection points into evaluation
plan
• Be prepared to drop and add
methodologies as you learn more about
the work

12. Example: Developing understanding through
developmental evaluation

13. Example: Developing understanding through
developmental evaluation

14. Questions for the field
• What are the steps evaluators can take to
ensure that investments in developmental
evaluation lead to learning that informs future
investments?
• How can evaluators work with intermediaries
and grantees to ensure that the
developmental evaluation approach and
intent is commonly understood, practiced,
and valued?

15. Contact
• Justin Piff: justin@omgcenter.org
• Sarah Singer Quast: sarah@omgcenter.org