64968.PDF

2
ACKNOWLEDGMENTS
Rosalind Fielder, M.A., M.S. (L.I.S.)
Assistant Professor of Library & Information Services and Reference & Instruction Librarian, Acting Library
Instruction Coordinator, Selector and Liaison for Political Science, Law and Military Science, and Criminal
Justice University Library at Chicago State University
Timothy Jones-Yelvington
Independent Consultant, currently completing the Masters of Education in Youth Development at the
University Of Illinois at Chicago College of Education
CONTRIBUTING STAFF
Lisa Marie Pickens, Independent Consultant – Facilitation, Research and Evaluation
Illinois Department of Human Services
Layla Suleiman Gonzalez, Chief Results Officer
Donors Forum
Valerie S. Lies, President and CEO
Delia Coleman, Director, Public Policy
Laurel O’Sullivan, Vice President, Public Policy
Laura Zumdahl, Vice President, Nonprofit Services
Donors Forum thanks the Boeing Corporation for supporting this work.
ABOUT DONORS FORUM
Donors Forum is the hub of Illinois’ philanthropic and nonprofit community, and the only statewide
association of funders and nonprofits of all kinds, as well as their advisors. Together, we connect
and collaborate, discover and share ideas and practices, and protect and promote our missions.
Join us as we leverage our collective power to improve lives and strengthen communities:
http://www.donorsforum.org.
208 South LaSalle Street
Suite 1540
Chicago, IL 60604
Phone: 312.578.0090
Toll-free: 888.578.0090
Library
Phone: 312.578.0175
TTY: 312.578.0159
Email: info@donorsforum.org

3
DEFINING, NOT REINVENTING, THE WHEEL: A PILOT STUDY ON
YOUTH DEVELOPMENT OUTCOMES AND BUDGETING RESULTS
EXECUTIVE SUMMARY
This report can be used as a follow up to the report in 2012 by American University and Indiana University
report, Budgeting for Results: Key Issues for Concern, commissioned on behalf of Donors Forum.
1
The
goal of this current report is to contribute to the implementation of BFR by modeling a process to engage
providers in outcomes development. The 2012 study was national in scope; in contrast, this study is local.
It took two state programs (Community Youth Services and TeenREACH) and engaged a small cohort of
providers within those programs who responded to a survey and then participated in two focus groups in
Springfield and Chicago. This report also outlines reasonable next steps and is a starting place for bridging
agency outcomes indicators and on-the-ground provider activities. Most importantly, it indicates how
developing outcomes can be a complex process, requiring a sustained, long-term, and broad level of
contact with providers and experts field by field, program by program.
CRITICAL QUESTIONS
• What capacity do providers need in order to be successful under BFR?
• What kind of outcomes training needs are there from providers (and the state agency)?
• What is a beginning point for creating evidence-informed outcomes (not outputs) for the state to begin
to measure that both providers and state practitioners can agree upon?
KEY FINDINGS
On outcomes:
• The field literature reveals 5 outcome models that can provide a useful starting point from which
the State and providers can develop outcomes for youth development programs. Providers not only
need human capital, funding and technology resources in order to be successful in BFR, they need
guidance from the State on what success looks like for positive youth development.
• Providers are familiar with outcomes and want to use them but the patchwork delivery system
is a significant barrier which creates outputs rather than outcomes.
• Providers are not only familiar with developments in the field and evidence-informed best practices,
they are eager for the State to use one of these already present models to define these programs
so they have greater clarity around where they (and their activities) fit within them.
On capacity building for BFR:
• Concerns persist (from both providers and agency staff) about the State’s internal capacity
to identify, collect and manage outcomes data.
KEY THEMES IN THE RECOMMENDATIONS
For Government: recommendations focus on landscape assessment, data collection and the
need to invest in capacity building for both its contracted providers and IL DHS.
For Philanthropy: recommendations encourage foundations to invest in capacity building for
grantees and share their program measurement expertise with government.
For Providers: recommendations encourage nonprofit organizations to use BFR as an opportunity
for continued strategic growth and impact for their communities.
WHERE TO GO FROM HERE
The ideal relationship between nonprofit providers, philanthropy and government is one that is fair, efficient,
accountable, responsive and transparent – all the goals of Budgeting for Results. While recognizing the
limitations of this initial pilot study, we hope the State, and its nonprofit and foundation partners, can adopt
a more robust study that can ground the service delivery landscape in best practices, gather useful internal
and external data, and create evidence-informed outcomes for the best use of public resources
DOWNLOAD THE 2012 BFR REPORT HERE.

4
TABLE OF CONTENTS
INTRODUCTION
....................................................................................................................................
5

Purpose
of
the
Report
...........................................................................................................................
6

IL
DHS
Youth
Programs
.........................................................................................................................
7

METHODOLOGY
......................................................................................................................................
8

LESSIONS
LEARNED
.................................................................................................................................
9

Key
Findings
From
The
Positive
Youth
Development
Literaure
Review
...............................................
9

Key
Findings
From
Interviews
With
IL
DHS
Staff
.................................................................................
15

PROVIDERS
ON
PYD
MODELS,
OUTCOMES
&
CAPACITY
BUILDING
........................................................
16

Defining
PYD
.......................................................................................................................................
16

Reaching
Consensus
on
PYD
Outcomes
..............................................................................................
18

How
Providers
Define
PYD
Outcomes
................................................................................................
21

The
Data
Providers
Currently
Collect
..................................................................................................
23

How
Providers
Collect
Outcomes
.......................................................................................................
24

How
Providers
Use
Outcome
Data
.....................................................................................................
26

Developing
Outcomes:
Provider
Concerns
.........................................................................................
27

PROVIDERS
ON
BUDGETING
FOR
RESULTS
&
CAPACITY
BUILDING
........................................................
31

Awareness
...........................................................................................................................................
31

Program
Implications
of
BFR
...............................................................................................................
32

Capacity
Building
Needed
by
Providers
..............................................................................................
33

Illinois
Implementation
Capacity
and
Provider
Concers
.....................................................................
34

STAKEHOLDER
RECOMMENDATIONS
....................................................................................................
36

For
Goverment
....................................................................................................................................
36

For
Philanthropy
.................................................................................................................................
37

For
Providers
.......................................................................................................................................
38

Where
Do
We
Go
From
Here?
............................................................................................................
38

APPENDIX
A–
Positive
Youth
Development
Literature
Review
and
References
........................................
39

APPENDIX
B
–Staff
Interview
Protocol
...................................................................................................
47

APPENDIX
C
–
Key
Themes
From
Staff
Interviews
...................................................................................
48

APPENDIX
D
–
Youth
Provider
Survey
&
Focus
Group
Protocol
...............................................................
53

APPENDIX
E
–
Aggregated
Data
From
The
Youth
Provider
Surveys
..........................................................
54

APPENDIX
F
–
Youth
Provider
Focus
Group
Materials
.............................................................................
93

APPENDIX
G
–
Forum
For
Youth
Investment
With
The
National
Collaboration
For
Youth
.........................
94

APPENDIX
H
–
Youth
Provider
Focus
Group
Data
..................................................................................
103

5
INTRODUCTION
The state of Illinois’ Department of Human Services is one of the largest financial supporters of youth
service provision in the state and as such is also deemed responsible for the quality of said services.
The state has historically used a traditional and incremental approach to budgeting for organizational
contracts in which the previous year’s budget was used as a baseline for developing the budget for
next year. Budgeting for Results (BFR) is a performance based budgeting approach which theoretically
establishes state priorities and connects funding to outcomes for each of those priorities. Put another way,
resources are allocated based on how effectively a program or service provider achieves established goals
and objectives as opposed to being based on historical funding levels.
Donors Forum has been monitoring BFR since it was enacted in 2010 by Governor Quinn. Since then, in
public testimony, research and its direct outreach to officials, Donors Forum has consistently advocated
for a prolonged conversation between state government, philanthropy and nonprofits as a way to strengthen
public decision-making at large, and Budgeting for Results, in particular. If done well, BFR can be a
strategic guide to help Illinois ensure more responsible management of the existing partnerships between
state government, philanthropy, and the nonprofits that deliver many of the services Illinois relies on.
If not done well, the fabric of service delivery (which is already tearing due to continued fiscal stress)
will be irreparably damaged.
This current report can be seen as a follow up to Donors Forum’s 2012 report, Budgeting for Results:
Key Issues of Concern, which was produced in cooperation with researchers from The American University
and Indiana University. This report endeavored to help identify best practices and common challenges from
other states as a basis for recommendations for implementing performance-based budgeting in Illinois. It
offered Six Key Features of Sound Performance-based Budgeting:
1. Government-wide, Multi-year Strategic Planning
2. Agencies Develop Operating Plans to Achieve Government-wide Goals
3. Valid and Meaningful Performance Measures for Agencies and Programs
4. Resource Allocations are Aligned with Goals
5. Independent Collection and Transparent Reporting of Outcomes
6. Outcomes Measures Inform Goals, Objectives, Allocations, and Operations
One year after Donors Forum released that research, it is good to report that the state is making steady
progress with its strategic planning and the development of state agency operating plans; they have created
Chief Results Officers in each agency and are beginning to build a portal for outcomes data to be collected.
It has yet to begin allocating resources in alignment with goals. We hope this current report would be
useful for Illinois state government, and the BFR implementation team, as it continues to create valid and
meaningful performance measures (#3), collect and transparently report on goals and outcomes (#5), and
eventually use these chosen outcomes to inform the future, strategic planning of programs (#6) – and thus
budgets.
Ultimately, Donors Forum’s goal is to help BFR become a useful tool to strengthen the operating
environment for nonprofits engaged with the state.

6
INTRODUCTION
PURPOSE OF THE REPORT
In early 2011, as the discussions about BFR implementation were beginning, Donors Forum and
designated leaders of the Illinois Department of Human Services decided to partner and pilot a process
for engaging providers around outcomes development, review evidence-informed practices from the field
so outcomes are grounded in practice, and help the state discover an efficient process by which to engage
their providers around the complicated work of measuring diverse human services for this heightened
environment of results-based budgeting.
These were the critical questions and assumptions we hoped to explore with this small pilot:
What do nonprofit providers need in order to be successful under BFR?
Assumption: nonprofit providers will need a great deal of capacity building support.
What kind of capacity would be expected from nonprofit providers in this new era of outcomes
measurement?
Assumption: nonprofit providers will need extensive training in outcomes definition, collection and
analysis, as well as increased infrastructure support to manage higher demand for results.
How can the state effectively engage with practitioners and philanthropy around developing
these outcomes?
Assumption: the state needs to do ‘more’ to engage providers in the implementation process.
In the course of our pilot, we have learned lessons (particularly around the dynamics of community
inclusion and reflexivity), answers have been given to our questions, and some of our assumptions
have been challenged – or given a deeper context, (particularly around the question of whose capacity
needs to be built.)
On the recommendation of IL DHS, Youth Service Providers were chosen as a pilot focus group.
Knowing that attempting this effort with all Youth Service Providers was too ambitious and unwieldy
given a 3 month time constraint, this pilot process was to be limited to a subsection of Youth Service
Providers that are a part of IL DHS’ Division of Family and Community Services, specifically Community
Youth Services (CYS) which is a program of the Bureau of Positive Youth Development (BPYD),
Teen Responsibility, Education, Attainment, Caring and Hope (TeenREACH) and Comprehensive
Community-Based-Youth Services (CCBYS), which are programs of Youth Intervention Services.
The pilot was to incorporate an environmental assessment of the Positive Youth Development field
with a focus on outcomes and indicators; garner input from and engage with IL DHS staff and program
providers about outcomes and indicators, and culminate in a final report that will guide IL DHS in
adopting jointly agreed upon outcomes and indicators for their youth development programs.
Additionally, it would help identify areas for capacity building for the providers field, to strengthen
overall service delivery and measurement. It is hoped that this process could be mirrored in other
program areas, and with support from the foundation community which has invested heavily in
several program areas. This report represents what has been learned during the pilot process

7
INTRODUCTION
IL DHS YOUTH PROGRAMS
The Bureau of Positive Youth Development (BPYD) is comprised of programs that address primarily
prevention. They share these characteristics:
• Community-based services
• Use similar approaches (e.g. after-school program)
• Target similar age groups (e.g. youth)
• Target multiple domains (e.g. youth, parents)
• Target universal and selected populations
Community Youth Services (CYS) —a program of BPYD-- is described as offering “….innovative programs
in two hundred diverse communities and counties for youth, young adults and families. CYS programs are
aimed at reducing and preventing juvenile delinquency” (IL DHS Website). CYS works through community
committees to address, evaluate and resolve the needs of the youth, families (the community) in the area
in which they are working.
Youth Intervention Services (YIS) strives to offer prevention, diversion, intervention, and treatment
services targeting youth to support families in crisis; prevent juvenile delinquency; encourage academic
achievement; and to divert youth at risk of involvement in the child welfare, juvenile justice, and correctional
systems (IL DHS Website). TeenREACH and CCBYS are housed under Youth Intervention Services.
TeenREACH programs, services and activities are provided during non-school hours to youth ages 6-17,
when youth are most likely to get into trouble. Core Service areas include: improving educational
performance; life skills education; parental involvement; recreation; sports and cultural/artistic activities;
and positive adult mentors and service learning (IL DHS Website).
Comprehensive Community Based Youth Services’ (CCBYS) goals are to provide comprehensive and
community-based individualized services to at-risk youth and their families to achieve family stabilization
and reunification, thereby diverting or minimizing youth contact with the juvenile justice and/or child welfare
systems. The mandatory core population, ages 11 - 17, are any minors who meet the following criteria:
• Youth who are absent from home without consent of parent, guardian or custodian
• Youth who are beyond the control of their parents/guardians or custodians in circumstances
which constitute a substantial or immediate danger to the minor's physical safety
• Youth, who after being taken into limited custody and offered interim crisis intervention
services, refuse to return home after the minor and his or her parents, guardians or custodians,
cannot agree to an arrangement for an alternative voluntary residential placement or to the
continuation of such placement
• Lockouts: Minors, 11 - 17 years of age, whose parent or caregiver has denied the child
access to the home and has refused or failed to make provisions for another living arrangement.
The CCBYS Program provides a continuum of services statewide to youth in high risk situations. A
24-hour crisis intervention response system is available in emergency situations for referrals
from youth, parents/guardians, police, courts, schools, Safe Place, and the Department of Children
and Family Services (DCFS). In addition to the CCBYS mandated programming, other discretionary
services appropriate to the youth may also be provided. CCBYS was included in the survey, but was
not included in the focus group discussions because this programming is focused specifically on
at-risk youth, unlike the description of CYS and TeenREACH.

8
METHODOLOGY
This project had three phases, with one phase building upon the last. In the first phase,
we began by conducting a literature review of the Positive Youth Development field to
help us prepare for our engagement with the Youth Service Providers, and to ground
our approach in the field’s evidence-informed practices.
In the second phase, we conducted in-depth interviews (see Appendix A & B) with IL DHS Staff
who managed the major subset of youth development programs to be included in the pilot: CYS,
TeenREACH and CCBYS. The goal of these interviews was to better understand how the state
is currently developing outcomes within these program areas; better understand the individual
programs under their purview; and to benefit from the guidance of staff on how best to engage
Youth Service Providers.
In the third phase, engagement with Youth Service Providers was multi-layered as well.
We developed and fielded a survey to all Youth Service Providers (see Appendix C & D) in CYS,
TeenREACH, and CCBYS. This survey explored ways in which providers defined Positive Youth
Development; the types of outcomes providers currently track; and the measurement and tracking
tools they use. While the overall pilot was constrained by time, conducting a youth provider survey
provided the perfect opportunity to collect a breadth of information that could inform both the focus
group discussions (designed for greater exploration) and recommendations for the next phase of
the project. Based on youth provider lists provided by IL DHS staff, the survey was sent to
323 providers from TeenREACH, CYS and CCBYS; it received about 100 bounce backs from
bad email addresses, and had a response rate of 11% (24 respondents). Some key organizations
were not included in the original youth service provider list provided to us by IL DHS Staff and
based on this exclusion strongly we suggest these organizations be engaged in the next phase
of this project.
Coupled with the literature review and our discussions with IL DHS staff, as well as our internal team
discussion, this data allowed us to develop contextualizing materials (See Appendix E) for focus group
participants to review prior to group sessions. This layered approach primed the pump for their thinking
about BFR, outcomes, tracking methods, etc. and allowed for an extremely candid and rich set of
discussions with both downstate service providers and Chicago-based providers (See Appendix F).
Because they represent providers not engaged directly in prevention work and most focused on
utilizing Positive Youth Development principles in their programming, only CYS and TeenREACH
service providers were selected for participation in the focus groups.
Upon entering this project we asked whether youth providers and the state agency shared the
same vision of their programs. A shared vision between providers and the State would also lead one
to infer a shared understanding of anticipated program outcomes and program success indicators.

9
LESSONS LEARNED
KEY FINDINGS FROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW
HISTORY AND THEORETICAL UNDERPINNINGS
From the literature review (See Full Literature Review in Appendix A), we know that PYD has
about a 20 year history as a field, with its beginnings tied both to sound youth worker and practitioner
experience-based knowledge and practice. PYD is also grounded in the ecological model of human
development and developmental systems theory which views the various layers of influence in
young people’s lives, i.e., family, community, schools, social networks, and the broader society/culture,
as interconnected and reciprocally influential. It is important to note that both IL DHS programs
like CYS and TeenREACH incorporate this PYD principle fully into their programming and recognize
the interconnectedness and reciprocal nature of young people’s influence on their environments, thus
making those two programs ideal for this effort.
WHAT IS PYD?
Practitioners see PYD as a framework that appreciates youth as resources to be developed, not
as problems to be solved, or risks to be mitigated; PYD is also a set of principles, a philosophy or
programmatic strategy, which can be applied to all types of youth programs with the holistic approach
of developing youth. Thus it would be important for IL DHS to determine which of their programs is
using PYD as a framework differing from those programs using it as a programmatic strategy or philosophy
in order to properly align PYD outcomes. In other words, those programs that use a PYD framework
should be held accountable for that framework and the agreed to outcomes — which may look a bit
different from programs using a PYD strategy in which a program may be using elements
of a positive youth development framework.
THE RELATIONSHIP BETWEEN PREVENTION AND PYD
Karen Pittman’s statement (“Problem free is not fully prepared”) is apropos as we think about the
relationship between prevention and PYD. The relationship between prevention frameworks and
PYD is not antagonistic. The notion of tension may arise more from the isolated ways in which funding
is allotted and dispersed rather than from any fundamental ideological differences. Ongoing research
demonstrates that, in practice, PYD researchers, practitioners and advocates understand prevention
and PYD frameworks to be complimentary and interrelated. This relationship should be incorporated
into thinking about IL DHS youth programming. Ideally, there should be a process for identifying
general evidence-informed PYD principles and outcomes used by all programs that would be
appropriate to apply across programs, and then to denote specific intervention-based outcomes
that make sense for the prevention programs. It would be important that PYD and prevention not be
seen as in conflict, but as complimentary. For instance, avoiding pregnancy or drugs as a teen is not
the same as having developed the competencies to do more than survive, but thrive. The two
behaviors are related, but not the same.

10
LESSONS LEARNED
While the literature review was not exhaustive, it is clear that there are a wide range of PYD Asset
models available for use by state program administrators, service providers, and youth development
funders. These models present domains in which young people (all of whom already possess assets
or potential assets) develop skills and competencies. An asset may be important take away from
the literature review is that PYD outcomes might be best understood as domains for ongoing
growth/developments as opposed to a goal unto itself. Because we primed the pump in focus group
discussions by achieving consensus on broad outcomes domains (see the work by the National
Collaboration for Youth (NCY) and the Forum for Youth Investment (FYI) in Appendix A), the overlap
between the major PYD Asset Frameworks potentially provide IL DHS with a starting place to pick up
the next phase of the discussion with youth providers about agreed to outcomes.
These models may be used as possible checklists for gauging whether a program has
laid the appropriate groundwork to promote PYD.
MODEL 1
The Search Institute, a youth development intermediary institution with a lengthy track record
of providing research, technical assistance and training to youth development organizations,
developed a comprehensive list of 40 developmental assets critical for youth to thrive.
These developmental were divided into two groups, external and internal assets
(40 Developmental Assets, 1997, 2007). Within each domain, the assets were further
categorized within one of four sub-domains. To provide an example of how assets were
grouped within sub-domains, the external sub-domain “support” included a caring
neighborhood and school climate, and positive family communication, among others.
The Search Institute has had success in motivating community-wide initiatives around
these 40 assets (Hamilton, et al. 2004). Their website and other materials provide an
array of resources for putting youth development principles into practice.
External assets (those provided by programs, families and communities) included:
• Support
• Empowerment
• Constructive Use Of Time
• Boundaries And Expectations
Internal assets (those possessed and further developed by youth themselves) included:
• Commitment To Learning
• Positive Values
• Social Competencies
• Positive Identity

11
LESSONS LEARNED
MODEL 2
The Committee on Community-Level Programs for Youth, an initiative of the National
Academy of Sciences, has also developed a list of developmental assets. This list had
similarities with the Search Institute’s, and divided assets into one of four domains:
• Physical Development
• Intellectual Development
• Social Development
• Psychological and Emotional Development
Examples of physical development included good health habits and risk management;
examples of intellectual development included school success and knowledge of essential
vocational skills; examples of social development included a sense of connectedness with
parents and peers; and examples of psychological and emotional development (the most
extensive category) included emotional self regulation, conflict resolution and coping skills.
The Committee on Community Level Programs for Youth further elaborated on this asset
framework with a list of eight attributes necessary for community settings seeking to promote
the development of these assets, making their framework especially useful for organizations
engaged in program development and evaluation. (National Research Council, 2002). These
eight community attributes included:
• Physical And Psychological Safety
• Appropriate Structure
• Supportive Relationships
• Opportunities To Belong
• Positive Social Norms
• Support For Efficacy And Mattering
• Opportunities For Skill Building
• Integration Of Family, School And Community Efforts

12
LESSONS LEARNED
MODEL 3
The “Five C’s,” originally developed by Karen Pittman, the ‘5 C’s’ have provided a simple and
memorable method for identifying PYD outcomes. The “C’s” are Competence, Character,
Connections, Confidence and Contribution (Hamilton, et al., 2004, Lerner, et al., 2005a). In some
formulations, Contribution has been replaced with “caring.” While the Search Institute and the
Committee on Community Level Programs for Youth’s frameworks are useful for providing a
more detailed breakdown of the assets needed for young people to thrive, they may be unwieldy
for quickly communicating the goals of PYD to communities, funders, or other stakeholders.
Contribution (or Caring) refers to a young person’s empathy and commitment to
“give back” to others.
Competence refers to the various skills needed to achieve one’s goals and adapt to
diverse contexts.
Confidence is the self-assurance required to pursue one’s objectives.
Character is a young person’s commitment to moral and ethical principles.
Connections are healthy social relationships with adults and peers.
MODEL 4
The National Collaboration for Youth (NCY), a coalition of national agencies committed
to PYD, has worked with the Forum for Youth Investment (FYI) through their “Ready by 21”
initiative to develop a set of outcomes for PYD programs. These outcomes were categorized
within five developmental domains – Thriving, Connecting, Leading, Learning and Working.
Each domain contains three to four outcomes. For instance, outcomes within the “leading”
domain included community connectedness, social responsibility, and leadership development.
A major advantage of the NCY/FYI evaluation resource is that each outcome is illustrated
by specific indicators of progress, as well as suggestions of concrete measurement tools
culled from NCY’s members and other youth organizations. These are tools that organizations
(including state programs) may use to measure their progress (The Forum for Youth
Investment with the National Collaboration for Youth Research Group, 2012). Although
evaluations based in youth outcomes and supported by empirical evidence, rather than
program quality, are challenging to implement within a PYD framework, tools are
nonetheless available to help practitioners track their outcomes.

13
LESSONS LEARNED
MODEL 5
Youth-led participatory evaluation is one of the most progressive evaluation frameworks
within the youth development field. In a participatory evaluation process, young people
identify their own goals and metrics for achievement - a process that serves the dual
purpose of further developing their skills, competencies and engagement in programming,
while simultaneously evaluating the success of the program in which they are participating.
After reviewing the literature to identify youth-oriented evaluation models, and finding
none, Kim Sabo-Flores (2008) elaborated the youth participatory evaluation approach.
Sabo-Flores detailed concrete processes youth development practitioners may use to
facilitate participatory evaluation. Participatory evaluation metrics are necessarily open-ended
at the onset, which may make them unsuitable for organization structures whose funders
expect concrete metrics in advance of funding (e.g., the State of Illinois.) Yet elements of
participatory evaluation may still be implemented in specific youth-led campaigns and
projects within organizational structures that practice more traditional forms of evaluation.

14
LESSONS LEARNED
KEY FINDINGS FROM THE POSITIVE YOUTH DEVELOPENT LITERATURE REVIEW
PYD EVALUATION & PROGRAM QUALITY
There are a number of research studies that show high quality PYD programming results in positive
outcomes. (See Appendix A) And we know from the literature that it would be challenging for Illinois youth
service agencies to conduct their own evaluations of their PYD programs with the level of rigor necessary to
show direct correlations between their programs and positive outcomes. IL DHS is presented with a unique
opportunity to support youth providers to effectively track agreed upon outcomes and it would be incumbent
upon the State (or a third-party intermediary) to conduct an evaluation tracking outcomes in an effort to
better under both the impact of programming and thus the impact of funding.
There are a range of tools that organizations may use to measure program quality including the Committee
on Community Level Programs for Youth’s list of features that promote PYD (National Research Council,
2002), the Rochester Evaluation of Asset Development for Youth (Sabaratnam & Klein, 2006), and the
Youth Program Quality Assessment, a widely employed tool disseminated by the David P. Weikert
Center for Youth Program Quality.
EXAMPLES OF PYD ORGANIZATIONS
It may not be necessary for the State of Illinois to reinvent the wheel when it comes to PYD outcomes and
measurement. The wheel has already been set in motion. Positive Youth Development (PYD) emerged from
the work of academics and youth development intermediary organizations (i.e., Karen Pittman’s Forum for
Youth Investment) and was informed by the work of grassroots youth workers. Some youth organizations
have since been influenced to explicitly reference PYD principles in their outreach materials, reports and/or
evaluations, and otherwise more actively self-identify as PYD organizations. Who are leaders in PYD
practice that Illinois can engage or model? (What follows is not meant to be exhaustive.)
PYD organizations include the STEM education organization Project Exploration, the 4-H Youth
Development Organization (a national leader in PYD evaluation efforts) and the Boys and Girls Club of
America. Currently, the Chicago antiviolence organization BUILD is working with Stacey S. Horn from
University of Illinois at Chicago, with support from the Polk Brothers Foundation, to comprehensively
implement PYD. It is important to note that PYD does not necessarily mandate a particular approach to
programming, but is rather a set of asset-based principles that may be applied in a range of different
programs. All of the programs listed above incorporate PYD philosophies and values (all have some level
of focus on holistic development, i.e. promoting assets and working across two or more aspects of the
developmental system such as school/family/ community), and offer a decent mix of size/constituency/focus
area. In Catalano, et al’s (1998) seminal evaluation of the positive impact of PYD approaches, organizations
were categorized as PYD based upon whether their work contributed to young people’s developmental
assets, and the extent to which they functioned across multiple contexts (i.e. family/school/community), not
whether they self-identified as PYD organizations. This study included a number of entities whose primary
identifications were as prevention or mentoring organizations (although organizations that targeted only
specific “at risk” groups were not eligible for inclusion).

15
LESSONS LEARNED
KEY FINDINGS FROM INTERVIEWS WITH IL DHS STAFF
These are summary conclusions from our conversations with the staff from the Bureau for
Positive Youth Development (where CYS is housed) and Youth Prevention Services (which
houses TeenREACH.) (See Appendix C for details.)
INTERNAL COMPETENCIES
Key Finding: The staff has long histories at IDHS. Both have practitioner experience with
evidence-based strategies, logic models, and best practices; however, resource constraints seem to
be affecting their ability to implement these types of practices in their current positions and divisions.
INTERNAL CAPACITY & BFR IMPLEMENTATION CONCERNS
Key Finding: Staff shares provider concerns about how BFR will impact federal funding, lack of funding
for capacity building, data collection methods and a potential threat to General Revenue Funding.
INTERNAL CAPACITY TO COLLECT AND ANALYZE DATA
Key Finding: Staff from both programs shares concerns that IL DHS does not have the capacity
to collect or analyze the type of data that would be required by BFR.
PROGRAM CONCERNS
Key Finding: There is some internal disagreement about the relationship between CYS and
TeenREACH, program definitions, and program outcomes.
PROVIDER ABILITY TO MEET BFR EXPECTATIONS
Key Finding: TeenREACH staff anticipates their providers are much more prepared to achieve
standards for BFR than does CYS staff.
DETERMINING HIGH-PERFORMING PROGRAMS
Key Finding: Staff point to differing modes of describing quality and high-performance with
each program. It is unclear whether there is an evidence-informed rubric to define quality.
DETERMINING POOR PERFORMING PROGRAMS
Key Finding: While staff describes poor performance in different ways for each program, what is
clear is that the assessment of ‘poor’ seems rather subjective, rather than based on evidence-informed
strategies found in field literature or research. It is unclear whether there is a rubric to define quality.
PROGRAM DATA COLLECTION
Key Finding: In both CYS and TeenREACH quantitative and qualitative data are being collected,
however it is mostly outputs, not outcomes.

16
PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING
DEFINING PYD
Key Finding: The majority of both survey and focus group respondents understood the
concept of Positive Youth Development in ways that correspond to the definition in the
literature review, as either a framework or strategy or principle or philosophy and directly
connected to the work of their organizations.
We asked the youth providers to define PYD and they either responded with a definition very similar
to the definition in the literature review, or they identified some specific component of PYD as identified
in the literature review, or shared specific frameworks as their definition of PYD. Many of the definitions
incorporated the young person’s environment as part of the equation. The responses were a combination
of defining PYD as a framework or strategy, principle and philosophy, yet the important thing to build
upon was the universal recognition of PYD and its connection to the work of these providers.
Some of the most interesting definitions from the survey include:
• Positive youth development is a framework that fosters internal and external protective
factors to moderate risk factors and enhances optimal physical, cognitive, social and
emotional development.
• Positive youth development is efforts (activities/interventions) to support youth in achieving
their dreams and becoming contributing members of their community. It emphasizes the
importance of focusing on youths’ strengths and providing opportunities to help others
through service.
• PYD emphasizes the importance of focusing on youths’ strengths instead of their risk factors
to ensure that all youth grow up to become contributing adults." Many researchers have
looked at this and have created frameworks around it. It relates to our programming because
much of our prevention programming is emphasizing strengths and building capacities in
order to mitigate risk factors that exist in their communities, their homes, or even their
physical predisposition.
• Positive Youth Development is a process which prepares young people to meet the
challenges of adolescence and adulthood through a coordinated, progressive series of activities
and experiences which help them to become socially, morally, emotionally, physically, and
cognitively competent.
• PYD is a policy perspective that emphasizes providing services and opportunities to
support all young people in developing a sense of competence, usefulness, belonging
and empowerment.
However, when thinking about cautions or concerns related to a shared definition of PYD, focus
group participants thought it would be important to make sure there was complete clarity and
alignment around language both among themselves and with the state, and that language should
be related to social/emotional development would be included, as well as concrete achievement
indictors (i.e., related to academic achievement.)

17
WHERE DOES DELINQUENCY FIT?
Key Finding: Despite consensus among youth providers about the definition of delinquency,
a question remains whether there is consensus within IL DHS about the definition of delinquency
and how it fits within the context of their positive youth development programs.
Since the term ‘delinquency’ does not fit into Positive Youth Development models and frameworks as
observed in the literature (neither is it used in the description of the Bureau of Positive Youth Development
nor in the description of TeenREACH program), it was interesting to discover that CYS was described as
“…reducing and preventing juvenile delinquency” and was similarly described in Youth Intervention Services.
When we coupled this with one staff person raising concern there was no agreed upon IL DHS definition of
delinquency it seemed worth investigating whether there was consensus among the providers.
Twenty-one providers responded with a definition of delinquency. Eighteen providers offer definitions that
involved young people’s unwillingness to adhere to rules (such as not attending school, no fighting, etc),
engaging in criminal active and/or involved with law enforcement or juvenile justice system, or locked out
by parents or refusing to return home. One of the eighteen providers shared that they did not use this term
in their programming but then went on to define it in terms similar to others.
Three providers offered very different responses to this question. Two offered indictments of the
circumstances in which the young people find themselves as the definition of delinquency.
• Youth who are unable to live at poverty levels and are in dire needs of social and emotional supports.
Youth who struggle to meet the challenges of everyday life.
• Youth without support systems.
The other provider instead of offering a definition of delinquency shared an approach to their work.
We don't use the word Delinquency on a regular basis. We choose to focus
on the whole youth, and address problems and needs as they occur.
There appears to be consensus among 18 of the 21 about the definition of delinquency or
about the behaviors that constitute delinquency. Among at least 4 providers, this is not language
they use in their programming.

18
REACHING CONSENSUS ON PYD OUTCOMES
Key Finding: Among survey and focus group participants, providers were very engaged around
coming to a common agreement about PYD outcomes, and there was broad consensus among
them about broad outcome domains (Learning, Working, Connecting, Leading and Thriving) –
despite the difficulty posed in collecting data for them. We think their engagement can be
attributed, in part, to the staff level they represented (i.e., Senior and Executive level), as well
as their familiarity with evidence-informed practices and field literature.
The participants described the State’s previous focus on outcomes as being more about outputs and
numbers; they felt it was important for the State to begin to focus on critical areas related to outcomes.
Another participant felt it was time to shift from the subjective and focus on real hard data. It was felt
that Out of School Time (OST) was still nebulous as a program area, that OST funders wanted to
see outcomes outlined in a plan, and that going in that direction (i.e., toward more rigorous outcomes
measurement) made sense.
In the guided portion of the focus group, we utilized the broad outcome domains (Learning, Working,
Connecting, Leading and Thriving)
2
to determine if we could reach consensus. Because we had heard
from participants during the unguided portion of the focus group about the types of outcomes that
should be included, we asked participants not to get stuck on the specific title of the domain, but to
suspend judgment instead and share what items would fit under the broad outcome domain headings
and simply allow us to use the specific title as a place holder. Because there was general agreement
about these broad domains, perhaps it would be most important for the State, coupled with providers, to
identify next level program outcomes and then measures that would correlate to these broad domains.
Under the broad outcome domain of LEARNING providers grouped the following possible outcomes:
• Demonstrate engagement in Academic/Educational Success/Learning
• Achieve School success (i.e., grades)
• Graduate middle school
Because there was general agreement about these broad domains,
perhaps it would be most important for the State, coupled with providers,
to identify next level program outcomes and then measures that would
correlate to these broad domains.
2 As outlined in A Shared vision for youth: Common Outcomes and Indicators

19
However, providers also pointed out that, in order to measure those things accurately, they (and the
State) needed to engage in Longitudinal Research; they (and the State) needed to define the kind of
academic engagement that would be appropriate for their particular programs to measure and collect
data for. For instance, providers questioned whether it was fair to be held to a standard for youth
achieving academic success when they saw that role being played firmly by schools and teachers.
Rather, they offered program outcomes that measure ‘engagement’ in learning and educational
achievement. One provider who said they have a very strong connection still finds it difficult get
academic achievement data from the schools and trying to obtain such data from parents is
still difficult.
Providers defined program outcomes for LEADING AS:
• Demonstration of civic mindedness, engagement, responsibilities, empowerment, or altruism
• Participation in a service learning project
• Involvement of youth in program planning and implementation
• Involvement of youth in their community
• Developing critical thinking skills
• Demonstrating self-control & respecting others opinion
• Building critical soft skills
• Building character
• You have students dong well academically-but they are still jerks- [they need] character building
Building a sense of empowerment and developing values were also potential program outcomes that
measured LEADING, but it also lacked sufficient data to support it as a worthwhile measure for the
providers. In one participant’s words “Developing a sense of values [important]... [but] this leads to
trouble...Research says about 1/3 of youth not responding.” Another said, “A lot of [our] work is
empowerment and not as much the language related to the materials or survey. [It’s] about having youth
involved in the planning of the programs.” But despite the difficulty in measurement, providers still think
outcomes on developing leadership and character skills is valuable, citing instances of students with
academic or hard skills achievement but lacking appropriate socialization or the requisite soft skills a
prospective employer would need.
When asked if WORKING was a relevant domain providers were less sure of how to measure this
kind of success, citing potential barriers such as how their organizations often achieves stabilizing
one family in the community but the family moves up and out, only to be replaced by less stable families –
and the process begins again. Another participant described financial literacy as a part of the equation and
lamented the dissolution of home economics in the schools and wondered how youth were learning about
money management. Another provider drew a strong link between education and economics/economic
development and felt that these both had real implications for the lives of the youth and families in their
programs. These barriers, created by structural inequity and policymaking outside these organizations’
spheres of influence, would presumably require long term solutions and measurement.

20
Providers were surer of program outcomes under CONNECTING - but again pointed out how difficult
it would be to collect indicators. They described these as:
• Demonstration of pro-social engagement
• Demonstration of peer-to-peer, and adult, connectedness
• Maintaining or demonstrating engagement with Caregiver, parent, or community
Demonstrating the difficulty of establishing core outcomes for diverse populations, providers pointed
out that sometimes CONNECTING engagement would look different for nontraditional families, and
for non-filial relationships (e.g., “What about mentor/adult engagement?”). One provider also asked
about the limits of data collection: “In eCornerstone – how do you track these nuances – how do you
track talking with parents for 15 minutes?”
If THRIVING is a form of resiliency, providers arrived at the following consensus about what that looked like,
though, again, there were nuances to their conclusions. Some participants felt that youth could demonstrate
resiliency by being successful; others felt that they could demonstrate it by their ability to see the future
beyond the age of 19/20 years of age; and still others felt youth were demonstrating thriving by setting
goals for themselves. Several providers mentioned that a sense of spirituality or a sense of hope was a
part of THRIVING, although they felt it was avoided due to the separation of church and state.
Some participants felt that youth could demonstrate resiliency by being
successful; others felt that they could demonstrate it by their ability to see
the future beyond the age of 19/20 years of age; and still others felt youth
were demonstrating thriving by setting goals for themselves.
There are challenges to across the board measurement of a program domain under THRIVING. One
provider asked, ‘Where do protective factors having an impact on positive youth outcomes come in?’
When it came to establishing an indicator about safety and violence, another provider shared that they
were training staff to recognize adverse childhood experiences and mapping out a plan to support youth.
It is provider opinion that THRIVING is difficult to measure - but initially the training itself is an outcome.
However, they believe they would be able to see the long-term measureable impact over time from such
a training and intervention. Several participants shared that thriving could be nurtured, supported or
facilitated with the right programming and the right staff. Another provider talked about their use of the
YPQA Assessment Tool which allows them to explore if their staff were creating opportunities for youth
to be more reflective. The tool helps staff with successful implementation of such practices which support
THRIVING. There was also a strong sense that THRIVING and RESILLIENCY was a key outcome to
incorporate into Budgeting for Results.

21
By using a standard methodology from the field that providers were familiar with and encouraging
the focus group members not to get stuck on the name of the broad outcome domain, but rather,
to think about which outcomes might fit under each domain, we were able to encourage participants
to reach consensus at the broadest level. Providers suggested the next step might be to think about
the individual outcomes under each broad domain as well as the measure that would be used to track
said outcome. These broad domains fit quite nicely with the type of outcomes shared by the survey
respondents as well, offering a shared place for the State to continue the conversation with providers.
HOW PROVIDERS DEFINE PYD OUTCOMES
Key Finding: Providers have a baseline understanding of outcomes and performance
measurement that could provide the State with a foundation for future conversations with
them about which outcomes might be appropriate for their programs.
We thought it would be useful to understand how providers defined outcomes. We found that all of
the survey respondents showed an understanding of outcomes (ranging from the general to the specific).
For example, one provider defined outcomes as a measurement tool for measuring their effectiveness.
10 of the remaining 19 respondents offered generic definition of outcomes and the other nine offered
examples of program specific indicators of outcomes.
Here are examples of general outcome definitions:
• Outcomes are specific short and long-term results/changes for individuals, groups, communities,
etc. stemming from programs or services.
• Based on the logic models that we develop, there are theories that form the basis for activities
within programs. These in turn are tied to outputs, which could be things such as attendance in
programming activities, # of participants that attend workshops, etc. These outputs affect what
we expect to be the results of the program activities otherwise known as the outcomes. Since
we have programming across the continuum of prevention to treatment, each of the logic models
reflect outcomes that are expected within those programs. Outcomes for youth prevention
reflects things such as improvement in knowledge of substance use/abuse and the effects,
improved attendance at school, improved peer relationships, ability to define a post-secondary
plan, ability to create a safety plan for themselves and within their communities, decreases in
suspensions and expulsions, decreases in physical fighting among peers, improvements in
family functioning, improvements in time management and organization skills.
• Youth Services define outcomes as means to determine the level of success in achieving the
overall goals and objectives of the program, the changes that are expected to occur as a result
of services provide.
But some respondents also provided more program-specific outcomes:
• Measurable deduction of juvenile arrests and recidivism from a restorative prospective
(i.e. station adjustments, peer court/juries). Measurable decreases in youth going to Juvenile
Detention Center. Decreases in expulsions, and suspensions. Increase in accountability to
community by offender. Increase in community based restorative initiatives and practices.
• Reduced occurrences of drug use and new arrests.
• Outcomes are youth transitioning to independent living after completing and "working"
their life plan.

22
HOW PROVIDERS DEFINE PYD OUTCOMES
Focus group participants categorized outcomes in the following three groups:
1. PYD-specific outcomes
2. Prevention-specific outcomes (which were more focused on reduction of behaviors)
3. Combination of PYD and Prevention outcomes
Drilling down further, providers gave these examples of PYD-specific outcomes:
• Academic, life skills, attitude toward education, more and improved and more career
choices, recreation, cultural and social progress, health and nutritional meals.
• Overall, in our entire program: academic retention and achievement; development of
career aspiration/choices;- development of soft jobs skills; increase decision-making and
problem-solving skills; increase anger-management and conflict resolution skills; increase
choices in favor of physical fitness and nutrition; increase engagement with the community.
Prevention-specific outcomes:
• Reduced occurrences of drug use and new arrests
• Increased abstinence, decreased legal involvement
• The number of youth who are returned to their home rather than become wards of the state
PYD/Prevention outcomes:
• Reduction in juvenile delinquent behaviors. Reduction in youth entering Juvenile Detention.
Reduction in expulsions, suspensions from school. Increase in positive youth development.
Increase in positive recreational, cultural, and educational activities and employment opportunities
for youth. Increase in alternatives to suspensions, expulsions, and juvenile detention.
• Reduce childhood obesity, reduce the achievement gap, increase kindergarten readiness,
increase school attendance, increase academic achievement, reduce school-based behavioral
incidents, reduce gang involvement, reduce juvenile recidivism, foster safer communities,
increase child safety, etc.
• Decrease in the number of youth going into DCFS care. Decrease in the number of youth
sentenced to DJJ. Increase in the number of families remaining intact.
• Youth engagement.
• Academic achievement.
It is important to note that the survey respondents included CCBYS providers who have much more of a
prevention focus. But the fact that there were many respondents who offered both PYD and Prevention
outcomes again makes the case for a complimentary relationship between PYD and Prevention; it implies
that it is possible, even across youth service providers who focus on Prevention, to incorporate PYD
outcomes. It is also possible that the difference between generic and specific programmatic outcomes
was the result of how we phrased the question which could have been interpreted differently by different
providers. The important takeaway is that there is a baseline understanding of outcomes and performance
measurement that can provide the State with a foundation for future conversations with providers about
what these outcomes might be for their programs.

23
THE DATA PROVIDERS CURRENTLY COLLECT
Key Finding: Youth development providers are collecting common program outcomes data that
could provide potentially useful program outcome categories for the State in the following asset
domains: Learning, Thriving, Connecting, Working, and Leading. This would be an area of deeper
exploration.
In our survey, we wanted to discover which types of outcomes being collected corresponded with a PYD
framework. 17 of the respondents said they were collecting their own outcome data.
We could identify that the majority of providers surveyed (13 of the 17 providers) were collecting some
data that correlated to Learning outcomes (i.e., academic achievement via collecting grades, GPA,
on-time promotion data, etc.) One provider reported using the Developmental Assets Profile developed
by the Search Institute which is correlated to another learning outcome—Engagement in Learning.
There is a baseline understanding of outcomes and performance measurement
that can provide the State with a foundation for future conversations with
providers about what these outcomes might be for their programs.
9 of the 17 were collecting outcomes which would be grouped under the Thriving domain.
Some of the measures providers report using includes DAP (Search), the Illinois Youth Survey,
YASI, and Lionquest Substance Abuse/Use survey.
2 of the providers responded that they collected outcome data that would correspond to the
Connecting domain using measures such as the DAP and Me and My World (Search).
4 of the providers shared outcomes that relate to the domain of Working (i.e., using employability
assessments, participation in job readiness programs, and transitioning to independent living were
specific indicators) but specific measure were not identified.
And 2 providers shared outcomes that fall into the Leading domain, with 1 specifically identifying
youth participation in leadership development activities as one of the outcomes, but again not identifying
a specific measure for this outcome.

24
HOW PROVIDERS COLLECT OUTCOMES
Key Finding: While providers are collecting data, they either do so across different platforms or
while trying to overcome significant barriers. Because of these barriers there may be little to no
practical connection between provider identified outcomes and those required by the State.
We explored with providers which programs they were using to house the outcome data they were
collecting. 23 providers responded; 3 providers reported manually storing data, a process which most
likely translated into keeping hard copies.
The programs that are most familiar are Excel, Access, Google Data Share, SPSS and Word.
eCornerstone is the database used by TeenREACH Providers which was developed by the state,
COPA and CitySpan are City of Chicago Department of Family and Support Databases, and Efforts to
Outcomes is a commercially developed Case Management System. The remaining database systems
are unfamiliar and there is no additional information on them at this time. The breakdown for the other
electronic programs is as follows:

25
HOW PROVIDERS COLLECT OUTCOMES
While 22 providers responded that they collected their own outcome data, 9 of those 22 identified
the following barriers to data collection:
• No staff, or not enough staff,
available to enter the data.
• Redundant or duplicative efforts due
to not having the right database to
house their own data; a mismatch
in the various databases resulting in
an inability to export their data to the
state system, thus requiring double
or sometime triple data entry; the
need to assess the data from
multiple internal sources and then
having to input the same data into
multiple systems: i.e., city, state,
federal and funders, resulting in a
serious drain on staff resources.
• Difficulty obtaining consent to access needed client data. For example, providers reported
that gathering real-time school data is next to impossible because the long and arduous
authorization CPS process. As a result, providers often rely on, and try to get reports,
progress reports, etc., from program participants or their guardians directly.
• Lack of guidance and consensus from the State on outcomes, related metrics, and
requisite help for providers in connecting their programs to the appropriate outcomes.
Because of these of barriers, the last barrier in particular, there may be a disconnect
between outcomes identified by providers and those required by the State.
Alignment — of data, databases, outcomes, and program asset models —
between the State and its providers is a critical key to success.
What does such a disconnect look like? For the sake of illustration we can borrow from the State’s
own presentation about its outcomes development process (see Illustration 1). While the State is
making commendable progress developing its internal process for creating state, agency, and
program outcomes and indicators, what is missing is a bridge between Agency Program Indicators
and programmatic activities providers are currently engaging in and, presumably, the outcomes those
activities create. Alignment – of data, databases, outcomes, and program asset models – between
the State and its providers is a critical key to success.

26
HOW PROVIDERS USE OUTCOME DATA
Key Finding: Providers use program results for program improvement – but more could be
done to support providers’ capacity to analyze their own findings. However, it remains unclear
how the State uses program outcomes provided by practitioners.
In some detail, 17 providers shared what they are learning from their outcome data. Most of what
they learned had to do with the positive impacts of the programs on students (either through improved
grades, increased graduation rates, more youth remaining at homes with the parents, less interaction
with DOJ, improved attitudes toward family, school, etc.) To a lesser degree, providers had less
guidance on their impact on parents and, even less, on teachers.
• We learn from the data how our outcome is, compared to our goal, e.g., last fiscal year we
have 27 youth in senior class, all of them graduated, and 25 of them (92%) entered
post-secondary education.
• Because of the afterschool programming, 89% of parents agree that their child has learned
new knowledge or skills; 87% agree that their child gets along better with other students;
79% report helping their child with homework at least weekly; 87% report that they are talking
to their child about school at least weekly; and 86% report that the afterschool program helped
them learn more ways to support their child’s education.
• Teachers feel more efficacious, students are spending more time on task, and disciplinary
referrals are decreasing.
Some providers report learning what is not working as well or where they need to put more
effort to ensure better outcomes.
• Ways in which programs need to be modified, what students we are impacting the most.
• We are learning that our programs are significantly increasing the high school graduation
rate (i.e., our program graduates youth at approximately 90%, compared to general school graduation
rate for our partner high school of approximately 40%), but our high school graduates
are not entering or staying in college at nearly as high a rate.
• I am learning that maybe we aren't doing a good enough job on academics, as far as looking
at their report cards. It also shows the days missing from school.
• Our students on average are maintaining 3.0 or higher GPA and have a very low school
absence which is positive. While 89% of our youth report that they will obtain a 4 year college degree,
44% of students are unsure or think they probably cannot afford to attend college using financial aid,
scholarships, or their family’s resources. Staff are targeting the 44% as a data point
to improve this year. Staff are also part of a collaborative evaluation group and, in meetings
where program staff compare data across agencies, they have learned from other agencies
who are getting better results in some areas. Staff have incorporated these ideas into program
improvement plans (for instance we are restructuring our tutoring services based on what
we’ve learned in collaboration meetings)
Several providers noted that if they are to know the long-term effects of their programs, they have
a greater need for more longitudinal data on students no longer in the system or those who are
no longer being tracked.
• Difficulty collecting outcome data once youth no longer is receiving services. Funding does
not provide means for long term follow-up.
• The programs we are trying to measure only have collected data up to a point, or
have not been collecting long enough.

27
HOW PROVIDERS USE OUTCOME DATA
The difficult social context in which program participants are living has serious implications for
providers’ programs and makes thinking about outcomes more complex.
• Youth are having difficulties remaining drug free.
• Lack of quality education and a living wage job are the main challenges for youth and adults.
When providers were asked how they were analyzing the data, most responded with answers
that fell into one of four categories. Some focused on the type of software program they were using
for analyzing the data (e.g., SPSS, eCornerstone, etc.); others focused on the frequency of the analysis –
whether it was conducted quarterly or yearly; others focused on certain indicators, such as the
success, changes or improvements that would be needed; and still others focused on who was
involved in either conducting the analysis or reviewing the analysis. In these cases, the responsibility
for analysis fell to a range of staff types: front line, manager or mid-level management (in one case,
clients were involved in reviewing the outcome data and giving input.)
At least 17 of the providers talked about using the data for improving their programs and planning,
while only 6 shared that they using the data for reporting, marketing or funding purposes. While it is
probably safe to say that all of the providers use this data to report back to funders and/or donors
with the goal of receiving funding or renewed funding, it did not appear to be the primary focus for
this particular set of respondents.
Four providers shared that there were barriers to analyzing their own outcome data, though all
conducted some level of analysis. Chief among these barriers were human resource limitations
(in some cases, the staff responsible for analysis also carried other responsibilities); available
technology failing to be conducive to the type of analysis staff wanted to conduct; or the absence
of sufficient funding to purchase appropriate technology that would make data collection and
analysis easier and shareable with multiple entities. Most significantly, a provider again raised
the concern that lack of an agreed upon set of defined outcomes and metrics was also a barrier
to data analysis.
DEVELOPING OUTCOMES: PROVIDER CONCERNS
Key Finding: Focus group participants felt there were enough PYD models in existence
and that neither they nor the state needed to start from scratch, but felt that the model
selected needed to be developmental in nature. In addition, providers felt that these
impacts needed to be thought of as longitudinal and that the state and other stakeholders
(the youth themselves, the continuum of staff at their agencies, parents, agency boards,
funders, and the business sector) would need to be willing to support ongoing training to
achieve “Agreement in minimal training for youth development Core Competencies,” and
provide the type of resources to make outcomes development (and, by extension,
Budgeting for Results) successful.
While providers were able to reach consensus within and across focus groups, they shared a
range of cautions and concerns when it came to addressing how the State should develop
agreed upon outcomes.

28
Both groups reiterated that there were enough models out there (i.e., Search Institute, etc) that they
did not need to start from scratch and they would be happy for the State to pick one model and then
they could work it through. The other important thing shared by the providers which seems important
to emphasize is that whatever model is selected must be developmental in nature and, as explained
by one provider, “The tool must be sensitive to the intersection of all of these assets/domains.”
Providers further emphasized the importance of working as a group to determine which outcomes were
directly correlated to, or aligned with, their specific programming because there were programmatic
differences (in terms of audiences, goals, etc.) They felt that this conversation about the desired
outcomes for youth needed to include a wide range of voices – in particular, youth, line staff, and other
PYD stakeholders. The focus group participants spoke of the fact that many stakeholders support
outcomes and outcomes development should not be the sole burden of the youth provider. They spoke
specifically of the youth themselves being a major stakeholder and there being a need to incorporate their
voices into these discussions, possibly through the TeenREACH State Youth Advisory Council. Youth
workers were another voice that needed to be added to the discussion so that the full continuum of the
youth provision sector was included, not just those in the administrative roles.
But while these responses are thoughtful and act as a testament to the dedication of these providers,
it is hard to imagine the state having the current capacity or time (given the BFR implementation
schedule) for such a bottom-up process for the creation and implementation of program outcomes
across hundreds of programs and with thousands of communities.
Focus group participants also thought it was important to think about outcomes from a longer view
and that some outcomes would be more longitudinal, requiring tracking beyond the time youth spend
in programs. One provider noted that “Change of behavior can occur but what about sustainability of
change over time, beyond the life of our program?” Another agreed and said it would be helpful to
“connect early childhood to college.” Similar concerns were raised by survey respondents.
Providers were equally concerned about efficiency and streamlining, wanting programs
to be seen, managed, and operated without silos:
• CYS and TeenREACH are silo[ed] from each other. [There’s a] huge disconnect.
• Two different state departments [are] working on youth outcomes--but extremely different.
• IL DHS’ [and] DCFS’ youth programs - these entities should come together and see where
here is duplication. We have been trying to do this on our own when there should be one unified
youth grant.
• Silo issue is huge--but [there] aren’t enough resources to deal with housing [or] placement.
Provider participants identified a number of ways thinking about different programs in relationship to
one another, as well as how their joint impact and role in outcomes could be helpful. For instance, in
the words of one provider:
“In certain IL DHS divisions’ [prevention models]
that are good, [the] State should take a look at what
worked internally and learn and use that.”

29
Other recommendations to explore what is happening with shared outcomes in other program
areas included:
• Looking at “Early childhood programs could be helpful in tracking [outcomes] over time.
• Designing support groups with smaller organizations in different areas and supporting
collaboration.
• [The State] should look at NWEA-MAP (long ranging-elementary to high school
educational tool) [that] has benchmarks, [measures] growth over time, [and] breaks each
domain down.
Providers are adamant that training has to be a key component for success. They also think there has
to be ongoing sustained training as BFR becomes more integrated as a state practice, not just upon
initial roll-out. Particularly, they supported the need for outcomes training “at the individual vendor
level” not just for capacity purposes, but to achieve “Agreement in minimal training for youth
development Core Competencies.” Ongoing support for BFR means joining policy and practice in
order to create sustained buy in among practitioners and prevent the all-too familiar practice of merely
adding to a nonprofit provider’s already strained capacity. Participants believe that in order for this to
work there will have to be buy-in at all levels within the provider agency (including youth, parents, line
staff, administrative staff, and board,) that training and on-going support would have to exist, and that
tasks associated with outcomes measurement could not be added randomly to someone’s current job.
For providers, there would need to be internal commitment at the provider agency, aligned with external
support from the state, other funders and even the business sector.
Ongoing support for BFR means joining policy and practice in order to create
sustained buy in among practitioners and prevent the all-too familiar practice
of merely adding to a nonprofit provider’s already strained capacity.
Quality control and assurance were also important components to the discussion of outcomes data.
Providers felt that it was not enough that the data was being entered, but that systems would be in
place to assure quality in data entry and analysis. They particularly focused on using a model of peer
to peer sharing and review in order to assure not just the quality of data but the accountability of it – pointing
to a sense of their own strength as a field and, perhaps more cynically, of their distrust of the State’s capacity
to collect (and leverage) ‘clean’ data. They raised such questions as “What about quality control? Who is
checking data quality, perhaps establishing a peer review process?” Providers also suggested that
establishing a normal quality control/ compliance process which reflects accountability would be the first
step; the next step would be quality assurance which would shape program quality and implementation.
Again, one participant suggested that incorporating peer to peer site visits might be a way to develop quality
assurance across programs. There was also the mention of cohort-based support as providers attempted
to tackle program evaluation. They outlined that having an hour-long site visit to review data did not go deep
enough. For instance, if one site was ranking low on a specific metric perhaps that site could be paired with
another provider agency that was doing well on that specific metric in order to create an additional level of
peer, or cohort, support. This was felt to be an innovative way to get a bigger ‘bang for the buck’ (or, return
on investment) for the youth. Providers held up the Maternal Infant Early Childhood Home Visiting
Program as an example of a cohort-based support model the State might explore.

30
For focus group participants a system for data collection would share the
following characteristics: streamlined; single depository; and come with a
‘friendly’ tracking system.
Perceived lack of cohesion or alignment in technology is another area of concern for survey and
focus group participants (see How Providers Collect Data). For focus group participants a system for
data collection would share the following characteristics: streamlined; single depository; and come with
a ‘friendly’ tracking system. In some instances providers and the State have partnered by using common
data platforms but the results have been mixed. “The ETO Database; there was intentionality but it still
does not work that way,” said one provider. Another added, “ECornerstone was introduced 8 years ago;
the glitches took 2-3 years to iron out. [We’ve] only been using it for about 5 years.” Another strong
recommendation from providers was for the state to “Match technology to outcomes to make sure we get
useful reports,” indicating a perceived deficiency and pointing to a need for outcomes to feed a loop of
constant program improvement, thus improving baseline results for the state. In other words, outcomes
shouldn’t just be for the state alone (in an upward direction) but should also have an impact ‘on the ground.’

31
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BULDING
AWARENESS
Key Finding: While descriptions varied, providers were generally aware of BFR as a
state-sponsored strategy. These responses suggest that while there is a continued
need to make sure all providers understand BFR and how it works, there is enough of
a basic understanding of BFR to build upon. These responses further suggest that
providers support increased accountability and want to use outcomes strategically in
their programming, but all will need assistance in doing so. This might be a crucial role
for a partnership between the state and the philanthropic community.
In the survey, 18 providers expressed a baseline understanding of BFR as a method to tie
resource allocations to measureable outcomes:
• [BFR is] tying money to outcomes.
• Governor Quinn signed into law Budgeting For Results (BFR), an historic spending reform
act requiring the state of Illinois to institute a results-based budgeting process that will nd the
automatic funding of programs. By requiring the State to live within its means and focus on
performance, BFR will transform the way that state officials, legislators – and the public –
prioritize, think about and implement the State’s budget. Going forward, the State will fund
only those programs that can demonstrate effectiveness and help the State achieve its
stated outcomes and goals.
• A strategic alternative to incremental budgeting, which resources are allocated based
on how effectively programs or service achieves established goals and objectives.
• From my understanding Budgeting for Results have something to do with the budget
and having available dollars to the most significant programs and activities.
One provider effectively connected the dots between funding and meeting program goals,
objectives, and outcomes:
“Have not had any formal training on BFR. Although we do our
very best to maximize every dollar using volunteers, in kind
contributions, it's hard to budget for results when the payments
you rely on are severely delayed. It's hard to expect the maximum
results under these circumstances when every program that
yields the best results relies on a steady cash flow.”
Few providers (2) had no knowledge of BFR.
During the focus groups, when we prompted providers to think about Budgeting for Results (BFR)
in a big picture, not only were most providers aware of BFR, their awareness emerged in two different
(yet related) themes: Efficiency vs. Collaboration. Springfield providers concentrated on outcomes
development in order to prevent duplicating services or processes that were already in place; they
really wanted to try and streamline the processes related to outcomes and service provision, as well
as incorporate economic development and connecting youth with employment in the discussion of
outcomes. The Chicago-based group was much more focused on collaborating with each other to
improve their practice related to outcomes; they were also concerned that smaller groups had not been
included in the focus groups because they felt true innovation comes from smaller organizations; they were
also advocates for long-terms capacity supports be put in place, and extremely concerned that they would
be held accountable for academic outcomes that even school systems are unable to meet.

32
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BUILDING
PROGRAM IMPLICATIONS OF BFR
Key Finding: Providers and the State share similar expectations of BFR to be
meaningful to their programs, indicating the possibility of shared vision and
goal-setting as implementation continues.
When asked, provider responses (20 in total) about the program implications of BFR for their
organizations can be categorized in one of four ways:
• Demonstrating Effectiveness/Accountability
• Strategic Prioritization
• Technology Improvement
• Uncertain or None
Most providers (10) said BFR would require them to Demonstrate Effectiveness/Accountability
(emphasis added to show shared values between the State and providers):
• We must be able to demonstrate successful program outcomes to secure and maintain
funding to services.
• It is good to show taxpayers that their money is being used for important projects
as I have seen a lot of waste in the past. Funded agencies need to be accountable for
how they use money. All non profits should have an outcomes measurement process. If not,
they should not be funded (or provide[d] funds to create one).
Four providers described implications that related to Strategic Prioritization:
• Sometimes leadership uses it to reduce in areas that are not a priority or preference. If you use
BFR there will be less of that.
• BFR for our organization would require a thorough analysis of each department's budget in
comparison to the produced outcomes and then an agency-wide budget -outcome analysis
to find out which programs and services are most cost-effective and which are least effective.
We would then need to use that data to decide which programs and/or services to
continue, to expand, or to discontinue.
• To prioritize and focus on quality service that will bring about long-term results.
Three providers shared that the implications for their organization were related to Technology
Improvements.
• More resources to build up our technology capacity and capability.
• The results are based on the data we collect. So we have to have very good data
collection methods.
• All non profits should have an outcomes measurement process. If not, they should
not be funded (or provide[d] funds to create one).
For six providers the implications were uncertain or unknown and only 2 of those said they were
unfamiliar with BFR or had not been educated about BFR. One provider curiously responded that
BFR had no implications for their organization.

33
CAPACITY BUILDING NEEDED BY PROVIDERS
Key Finding: Not surprisingly, in order to meet the requirements of Budgeting for
Results successfully, what providers need most are resources – human capital,
training, and funding – in order to build (or expand) their ability to be effective.
In the survey, providers ranked their needs as follows:
• Resources (people, funding or technology)
• Training (general and specific)
• Access to Institutional Data
• Agreed upon Outcomes/Metrics
Resources was the most often identified support with 5 providers identifying personnel
supports, 4 identifying funding support for any number of needs and 7 identifying technology
needs (emphasis added):
• An improvement in our obtaining professionals and programs to support our life skills
i.e. anti-drug/alcohol education for the students.
• Our organization would require funding for and/or an assigned evaluator who has experience
in strategic financial planning and operations to help determine whether the data we are
collecting is accurately measuring what we desire and, if it isn't what data we should collect,
help us develop the tools to collect the data if we do not already have them, and assist the
PQI department and department leaders in analyzing the data that is collected.
• We find that ETO is a good tool for our collecting data to evaluate our outcome in programs.
However, we are only using ETO in certain programs, we need further assistance,
technology and financial, to implement it in other programs in the agency.
6 providers mentioned the need for training as a necessary support if they were to be successful
in meeting the outcomes of BFR. Such training would help staff understand BFR or would be specific
to technology or outcomes.
• Staff training to understand the concept in order to provide quality service that delivers
the intended results.
• Training for our admin team and understanding of the process.
• Training on managing data.
• Again, I can only imagine having the software and the education to implement BFR.
2 providers mentioned the need for agreed upon outcomes and metrics in order for their
organization to successfully implement BFR. This is a recurring theme through the survey data.
• Hoping that BFR will be in line with current research/thinking around outcomes from
existing sources and not created from scratch as that could lead to a separate outcomes
process for organizations and create inefficiencies. For example, there is a lot of work being
done around social emotional learning - these concepts should be used in BFR because
most organizations that have an outcomes process will also use these concepts.
• As previously described and additionally training, technical assistance in constructing
appropriate outcomes, metrics, and sufficient technology.
2 providers mentioned access to institutional data or more cooperation from the parents, schools
and CPS to get data and as having organizational implications.

34
ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS
Key Finding: Though the focus group size was small, providers were positively engaged
around the topic of outcomes. However, they were extremely vocal in their concerns about
BFR and the state’s capacity to implement such a complex process. The providers remain
concerned that BFR will be another “flavor of the month” that will fade with time and be used
as mechanism to deplete already lean resources to support successful development of youth.
While sharing that they were grateful for this pilot, and to hear the State’s admission they do not
have the expertise in this area and are willing to partner with them, providers are still concerned
about the State’s limitations in understanding outcomes. (See also IL DHS Staff Discussion)
• You find only one person at the state level to work with who you talk with about outcomes, etc.
• Everybody at the state level needs to know more than we do, more than the agencies.
• Appreciate [the] state admitting it does not have expertise and will partner with the providers
and others on this.
From the providers’ perspective (a point of view that would likely differ from the State’s in rather
significant ways) BFR would require an increase in funding to their programs to be successful in
response to BR. They have serious concerns and distrust about the State’s commitment to such
funding, given that Illinois has already demonstrated an inability to meet its financial obligations.
• Especially given that the base amount to us is $75 thousand versus federal programs which
get $250 thousand.
• [We are] expected to input data and [it’s] not reflected in budget. Connect tasks to funding.
• State must make in-roads to punctuate federal dollars. In early childhood, federal dollars are
creating hubs regionally and nationally. Nothing like this [is] happening in PYD at the national
level. This ices out some of the smaller orgs who can be very innovative. [It] must intentionally
include smaller and larger organizations.
• For profit businesses need to be brought to the table, they have a stake in this too.
• We must monitor pennies. State [funding] is not worth the paper it is written on. My board is
weary, our partners are weary. [The] State can change its mind. Grants do not cover interest
when we've had to take out loans to keep things going in between when monies come.
• The state cancelled some of our [funding and] rolled our program into another area and then
came back and asked us for a report when the program was no longer in our hands.
Repeating a refrain heard often, providers voiced concern that BFR was the “new flavor of the
month” and that the State changes ideas, concepts, and plans quickly (perhaps too quickly). They
were skeptical to the idea that BFR would be handled differently. One provider asked, “How much
of this is short-term as the State shifts to the health consortium as its model?” Another observed,
“TeenREACH was zeroed out, [then] brought back. Things change too much.”

35
ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS
Of more concern is the distrust some providers have of the State’s motivations surrounding
BFR. Some feel that BFR is a Trojan horse to further hollow out an already lean service delivery
system. A provider said, “Theoretically [I have] no problem, but [I’m] concerned that it will be used as
a way to hire the lowest quality, that it’s not being tied to quality or ongoing improvement.” Another
added, “Money must come along with this or we are disinvesting in youth.” Another preferred to think
about BFR’s potential to prompt cooperation and collaboration, rather than competition:
“BFR would be great if not connected to the budget.
We are all on the same team; let's elevate the reflection,
talk with one another, and innovate when tied to money.
Money creates competition.”
Acknowledging the small size of both the survey and focus group samples, we want to encourage
the State not to ignore these concerns about gaps in capacity. The State’s lack of staffing, planning
and resource capacity, as well as its uneven practices and processes, is well documented in annual
Inspector General audit reports, public testimony at legislative hearings and committees, as well as
external evaluations (such as the 2010 Pew Center on the States’ Grading the States report on Illinois
government performance.) Facing, and overcoming, distrust between strategic partners will not be
easy, and would probably require some additional change management and internal culture work
on the part of IL DHS (as well as some rather significant changes to the State’s financial processes.)
However, greater intentional and more transparent communication about the State’s
commitment to BFR, as well as to a consistent evidence-informed methodology for
implementation, could balance this skepticism.

36
STAKEHOLDER RECOMMENDATIONS
FOR GOVERNMENT
CONDUCT EXTERNAL FIELD REVIEW
There is a need to conduct a more exhaustive research into and compilation of the major activities
in Chicago and Illinois, particularly public-private partnerships, around Positive Youth Development
Outcomes and Indicators in order inform the State’s goals. But while there is evidently a City level
project to assess PYD outcomes, as well as a long-established State level project with the Afterschool
Partnership, there could be a more coordinated effort to compile lessons and best practices, as well
as to avoid duplication and build synergy.
CONDUCT INTERNAL AUDIT
Assess their current youth programming offerings across units to eliminate duplication and
redundancy.
Review programs within their own divisions where outcomes and indicators have been
successfully implemented and use those as models to guide those developed in the Youth
Development division.
Invest in program evaluation.
CREATE TRAINING PROTOCOL
The State should consider supporting sustained and ongoing training as BFR becomes more
integrated as a state practice, not just upon initial roll-out. Particularly, they should consider
supporting the need for training “at the individual vendor level” not just for capacity purposes
but to achieve “Agreement in minimal training for youth development Core Competencies.”
Identify and partner with organizations to provide ongoing capacity building support.
PARTICIPATE IN STAKEHOLDER ENGAGEMENT
Incorporate the voices of youth (i.e., TeenREACH State Advisory Council), organizations
that support youth (i.e., Illinois Collaboration on Youth-ICOY) and other key stakeholders
in the next phase of the process.
Identify opportunities for partnership with philanthropic and corporate stakeholders in
evidence-informed outcomes development and/or capacity development to strengthen the
PYD delivery system.
BASE OUTCOMES ON EVIDENCE-INFORMED PRACTICES AND RESEARCH
Identify an evidence-informed and best practice model for PYD programs and outcomes
(e.g., from the Search Institute, etc.), particularly as the providers are familiar with the range
of models available. Leverage the expertise of foundations and practitioners in this process.
Complete the process of obtaining consensus on the broad PYD outcome domains, move to
work with providers on specific outcomes within those domains and finalize the measures that
would be utilized across Youth Provider Services; subsequently, the State could take the additional
step of determining which additional Prevention Outcomes would be needed.

37
STAKEHOLDER RECOMMENDATIONS
FOR GOVERNMENT
Survey providers on which outcomes providers collect internally (which may be different from
State requirements) to reduce duplication of efforts, establish shared benchmarks, and provide
an opportunity for all providers to benefit from the experience of peers.
Survey providers on which measures they use to determine outcomes to reduce duplication of efforts
and provide an opportunity for all providers to benefit from the experience of peers.
OFFER BFR INFORMATIONAL SESSIONS
Consider offering an informational workshop and materials on BFR so all providers (particularly smaller
organizations) have a full understanding and opportunity to ask questions and get clarification on BFR.
Identify and partner with strategic intermediary organizations to achieve outreach goals.
STANDARDIZE AND MODERNIZE TECHNOLOGY
Assess current technology (or methods) used by Youth Providers to see whether there is a ‘match’
between State and provider tools and the outcomes and indicators they’re required to provide.
Survey providers about the database systems (or methods) they are using to house and analyze
their own data.
Reduce duplication of efforts by streamlining data submission processes and providing opportunities
for peer sharing.
Conduct a needs assessment and cost analysis to determine which upgrades or innovations would be
required to bring agencies on technological par with one another in order to track and analyze agreed
upon outcome data.
FOR PHILANTHROPY
Consider the types of capacity building needed by grantees (i.e., human capital, program development
and measurement) to build and expand their ability to effectively meet the rigorous requirements of
Budgeting for Results.
Consider supporting ongoing training and technical assistance for smaller organizations as BFR
becomes more integrated as a state practice, not just upon initial roll-out. In particular, consider
supporting the need for training “at the individual vendor level” not just for capacity purposes but to
achieve “Agreement in minimal training for youth development Core Competencies.”
Consider ways to help align philanthropic best practices in PYD with State outcomes, in order to
develop a unified set of outcomes and measures for greater grant making strategic impact (as well
as to avoid unnecessary conflicts, duplications or redundancy experienced by grantees of both
foundations and the State.)

64968.PDF

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (20)

Similar to 64968.PDF

Similar to 64968.PDF (20)

64968.PDF