Download to read offline
![19
PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING
REACHING CONSENSUS ON PYD OUTCOMES
However, providers also pointed out that, in order to measure those things accurately, they (and the
State) needed to engage in Longitudinal Research; they (and the State) needed to define the kind of
academic engagement that would be appropriate for their particular programs to measure and collect
data for. For instance, providers questioned whether it was fair to be held to a standard for youth
achieving academic success when they saw that role being played firmly by schools and teachers.
Rather, they offered program outcomes that measure ‘engagement’ in learning and educational
achievement. One provider who said they have a very strong connection still finds it difficult get
academic achievement data from the schools and trying to obtain such data from parents is
still difficult.
Providers defined program outcomes for LEADING AS:
• Demonstration of civic mindedness, engagement, responsibilities, empowerment, or altruism
• Participation in a service learning project
• Involvement of youth in program planning and implementation
• Involvement of youth in their community
• Developing critical thinking skills
• Demonstrating self-control & respecting others opinion
• Building critical soft skills
• Building character
• You have students dong well academically-but they are still jerks- [they need] character building
Building a sense of empowerment and developing values were also potential program outcomes that
measured LEADING, but it also lacked sufficient data to support it as a worthwhile measure for the
providers. In one participant’s words “Developing a sense of values [important]... [but] this leads to
trouble...Research says about 1/3 of youth not responding.” Another said, “A lot of [our] work is
empowerment and not as much the language related to the materials or survey. [It’s] about having youth
involved in the planning of the programs.” But despite the difficulty in measurement, providers still think
outcomes on developing leadership and character skills is valuable, citing instances of students with
academic or hard skills achievement but lacking appropriate socialization or the requisite soft skills a
prospective employer would need.
When asked if WORKING was a relevant domain providers were less sure of how to measure this
kind of success, citing potential barriers such as how their organizations often achieves stabilizing
one family in the community but the family moves up and out, only to be replaced by less stable families –
and the process begins again. Another participant described financial literacy as a part of the equation and
lamented the dissolution of home economics in the schools and wondered how youth were learning about
money management. Another provider drew a strong link between education and economics/economic
development and felt that these both had real implications for the lives of the youth and families in their
programs. These barriers, created by structural inequity and policymaking outside these organizations’
spheres of influence, would presumably require long term solutions and measurement.](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-19-320.jpg)
![28
PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING
DEVELOPING OUTCOMES: PROVIDER CONCERNS
Both groups reiterated that there were enough models out there (i.e., Search Institute, etc) that they
did not need to start from scratch and they would be happy for the State to pick one model and then
they could work it through. The other important thing shared by the providers which seems important
to emphasize is that whatever model is selected must be developmental in nature and, as explained
by one provider, “The tool must be sensitive to the intersection of all of these assets/domains.”
Providers further emphasized the importance of working as a group to determine which outcomes were
directly correlated to, or aligned with, their specific programming because there were programmatic
differences (in terms of audiences, goals, etc.) They felt that this conversation about the desired
outcomes for youth needed to include a wide range of voices – in particular, youth, line staff, and other
PYD stakeholders. The focus group participants spoke of the fact that many stakeholders support
outcomes and outcomes development should not be the sole burden of the youth provider. They spoke
specifically of the youth themselves being a major stakeholder and there being a need to incorporate their
voices into these discussions, possibly through the TeenREACH State Youth Advisory Council. Youth
workers were another voice that needed to be added to the discussion so that the full continuum of the
youth provision sector was included, not just those in the administrative roles.
But while these responses are thoughtful and act as a testament to the dedication of these providers,
it is hard to imagine the state having the current capacity or time (given the BFR implementation
schedule) for such a bottom-up process for the creation and implementation of program outcomes
across hundreds of programs and with thousands of communities.
Focus group participants also thought it was important to think about outcomes from a longer view
and that some outcomes would be more longitudinal, requiring tracking beyond the time youth spend
in programs. One provider noted that “Change of behavior can occur but what about sustainability of
change over time, beyond the life of our program?” Another agreed and said it would be helpful to
“connect early childhood to college.” Similar concerns were raised by survey respondents.
Providers were equally concerned about efficiency and streamlining, wanting programs
to be seen, managed, and operated without silos:
• CYS and TeenREACH are silo[ed] from each other. [There’s a] huge disconnect.
• Two different state departments [are] working on youth outcomes--but extremely different.
• IL DHS’ [and] DCFS’ youth programs - these entities should come together and see where
here is duplication. We have been trying to do this on our own when there should be one unified
youth grant.
• Silo issue is huge--but [there] aren’t enough resources to deal with housing [or] placement.
Provider participants identified a number of ways thinking about different programs in relationship to
one another, as well as how their joint impact and role in outcomes could be helpful. For instance, in
the words of one provider:
“In certain IL DHS divisions’ [prevention models]
that are good, [the] State should take a look at what
worked internally and learn and use that.”](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-28-320.jpg)
![29
PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING
DEVELOPING OUTCOMES: PROVIDER CONCERNS
Other recommendations to explore what is happening with shared outcomes in other program
areas included:
• Looking at “Early childhood programs could be helpful in tracking [outcomes] over time.
• Designing support groups with smaller organizations in different areas and supporting
collaboration.
• [The State] should look at NWEA-MAP (long ranging-elementary to high school
educational tool) [that] has benchmarks, [measures] growth over time, [and] breaks each
domain down.
Providers are adamant that training has to be a key component for success. They also think there has
to be ongoing sustained training as BFR becomes more integrated as a state practice, not just upon
initial roll-out. Particularly, they supported the need for outcomes training “at the individual vendor
level” not just for capacity purposes, but to achieve “Agreement in minimal training for youth
development Core Competencies.” Ongoing support for BFR means joining policy and practice in
order to create sustained buy in among practitioners and prevent the all-too familiar practice of merely
adding to a nonprofit provider’s already strained capacity. Participants believe that in order for this to
work there will have to be buy-in at all levels within the provider agency (including youth, parents, line
staff, administrative staff, and board,) that training and on-going support would have to exist, and that
tasks associated with outcomes measurement could not be added randomly to someone’s current job.
For providers, there would need to be internal commitment at the provider agency, aligned with external
support from the state, other funders and even the business sector.
Ongoing support for BFR means joining policy and practice in order to create
sustained buy in among practitioners and prevent the all-too familiar practice
of merely adding to a nonprofit provider’s already strained capacity.
Quality control and assurance were also important components to the discussion of outcomes data.
Providers felt that it was not enough that the data was being entered, but that systems would be in
place to assure quality in data entry and analysis. They particularly focused on using a model of peer
to peer sharing and review in order to assure not just the quality of data but the accountability of it – pointing
to a sense of their own strength as a field and, perhaps more cynically, of their distrust of the State’s capacity
to collect (and leverage) ‘clean’ data. They raised such questions as “What about quality control? Who is
checking data quality, perhaps establishing a peer review process?” Providers also suggested that
establishing a normal quality control/ compliance process which reflects accountability would be the first
step; the next step would be quality assurance which would shape program quality and implementation.
Again, one participant suggested that incorporating peer to peer site visits might be a way to develop quality
assurance across programs. There was also the mention of cohort-based support as providers attempted
to tackle program evaluation. They outlined that having an hour-long site visit to review data did not go deep
enough. For instance, if one site was ranking low on a specific metric perhaps that site could be paired with
another provider agency that was doing well on that specific metric in order to create an additional level of
peer, or cohort, support. This was felt to be an innovative way to get a bigger ‘bang for the buck’ (or, return
on investment) for the youth. Providers held up the Maternal Infant Early Childhood Home Visiting
Program as an example of a cohort-based support model the State might explore.](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-29-320.jpg)
![30
PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING
DEVELOPING OUTCOMES: PROVIDER CONCERNS
For focus group participants a system for data collection would share the
following characteristics: streamlined; single depository; and come with a
‘friendly’ tracking system.
Perceived lack of cohesion or alignment in technology is another area of concern for survey and
focus group participants (see How Providers Collect Data). For focus group participants a system for
data collection would share the following characteristics: streamlined; single depository; and come with
a ‘friendly’ tracking system. In some instances providers and the State have partnered by using common
data platforms but the results have been mixed. “The ETO Database; there was intentionality but it still
does not work that way,” said one provider. Another added, “ECornerstone was introduced 8 years ago;
the glitches took 2-3 years to iron out. [We’ve] only been using it for about 5 years.” Another strong
recommendation from providers was for the state to “Match technology to outcomes to make sure we get
useful reports,” indicating a perceived deficiency and pointing to a need for outcomes to feed a loop of
constant program improvement, thus improving baseline results for the state. In other words, outcomes
shouldn’t just be for the state alone (in an upward direction) but should also have an impact ‘on the ground.’](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-30-320.jpg)
![31
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BULDING
AWARENESS
Key Finding: While descriptions varied, providers were generally aware of BFR as a
state-sponsored strategy. These responses suggest that while there is a continued
need to make sure all providers understand BFR and how it works, there is enough of
a basic understanding of BFR to build upon. These responses further suggest that
providers support increased accountability and want to use outcomes strategically in
their programming, but all will need assistance in doing so. This might be a crucial role
for a partnership between the state and the philanthropic community.
In the survey, 18 providers expressed a baseline understanding of BFR as a method to tie
resource allocations to measureable outcomes:
• [BFR is] tying money to outcomes.
• Governor Quinn signed into law Budgeting For Results (BFR), an historic spending reform
act requiring the state of Illinois to institute a results-based budgeting process that will nd the
automatic funding of programs. By requiring the State to live within its means and focus on
performance, BFR will transform the way that state officials, legislators – and the public –
prioritize, think about and implement the State’s budget. Going forward, the State will fund
only those programs that can demonstrate effectiveness and help the State achieve its
stated outcomes and goals.
• A strategic alternative to incremental budgeting, which resources are allocated based
on how effectively programs or service achieves established goals and objectives.
• From my understanding Budgeting for Results have something to do with the budget
and having available dollars to the most significant programs and activities.
One provider effectively connected the dots between funding and meeting program goals,
objectives, and outcomes:
“Have not had any formal training on BFR. Although we do our
very best to maximize every dollar using volunteers, in kind
contributions, it's hard to budget for results when the payments
you rely on are severely delayed. It's hard to expect the maximum
results under these circumstances when every program that
yields the best results relies on a steady cash flow.”
Few providers (2) had no knowledge of BFR.
During the focus groups, when we prompted providers to think about Budgeting for Results (BFR)
in a big picture, not only were most providers aware of BFR, their awareness emerged in two different
(yet related) themes: Efficiency vs. Collaboration. Springfield providers concentrated on outcomes
development in order to prevent duplicating services or processes that were already in place; they
really wanted to try and streamline the processes related to outcomes and service provision, as well
as incorporate economic development and connecting youth with employment in the discussion of
outcomes. The Chicago-based group was much more focused on collaborating with each other to
improve their practice related to outcomes; they were also concerned that smaller groups had not been
included in the focus groups because they felt true innovation comes from smaller organizations; they were
also advocates for long-terms capacity supports be put in place, and extremely concerned that they would
be held accountable for academic outcomes that even school systems are unable to meet.](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-31-320.jpg)
![32
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BUILDING
PROGRAM IMPLICATIONS OF BFR
Key Finding: Providers and the State share similar expectations of BFR to be
meaningful to their programs, indicating the possibility of shared vision and
goal-setting as implementation continues.
When asked, provider responses (20 in total) about the program implications of BFR for their
organizations can be categorized in one of four ways:
• Demonstrating Effectiveness/Accountability
• Strategic Prioritization
• Technology Improvement
• Uncertain or None
Most providers (10) said BFR would require them to Demonstrate Effectiveness/Accountability
(emphasis added to show shared values between the State and providers):
• We must be able to demonstrate successful program outcomes to secure and maintain
funding to services.
• It is good to show taxpayers that their money is being used for important projects
as I have seen a lot of waste in the past. Funded agencies need to be accountable for
how they use money. All non profits should have an outcomes measurement process. If not,
they should not be funded (or provide[d] funds to create one).
Four providers described implications that related to Strategic Prioritization:
• Sometimes leadership uses it to reduce in areas that are not a priority or preference. If you use
BFR there will be less of that.
• BFR for our organization would require a thorough analysis of each department's budget in
comparison to the produced outcomes and then an agency-wide budget -outcome analysis
to find out which programs and services are most cost-effective and which are least effective.
We would then need to use that data to decide which programs and/or services to
continue, to expand, or to discontinue.
• To prioritize and focus on quality service that will bring about long-term results.
Three providers shared that the implications for their organization were related to Technology
Improvements.
• More resources to build up our technology capacity and capability.
• The results are based on the data we collect. So we have to have very good data
collection methods.
• All non profits should have an outcomes measurement process. If not, they should
not be funded (or provide[d] funds to create one).
For six providers the implications were uncertain or unknown and only 2 of those said they were
unfamiliar with BFR or had not been educated about BFR. One provider curiously responded that
BFR had no implications for their organization.](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-32-320.jpg)
![34
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BUILDING
ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS
Key Finding: Though the focus group size was small, providers were positively engaged
around the topic of outcomes. However, they were extremely vocal in their concerns about
BFR and the state’s capacity to implement such a complex process. The providers remain
concerned that BFR will be another “flavor of the month” that will fade with time and be used
as mechanism to deplete already lean resources to support successful development of youth.
While sharing that they were grateful for this pilot, and to hear the State’s admission they do not
have the expertise in this area and are willing to partner with them, providers are still concerned
about the State’s limitations in understanding outcomes. (See also IL DHS Staff Discussion)
• You find only one person at the state level to work with who you talk with about outcomes, etc.
• Everybody at the state level needs to know more than we do, more than the agencies.
• Appreciate [the] state admitting it does not have expertise and will partner with the providers
and others on this.
From the providers’ perspective (a point of view that would likely differ from the State’s in rather
significant ways) BFR would require an increase in funding to their programs to be successful in
response to BR. They have serious concerns and distrust about the State’s commitment to such
funding, given that Illinois has already demonstrated an inability to meet its financial obligations.
• Especially given that the base amount to us is $75 thousand versus federal programs which
get $250 thousand.
• [We are] expected to input data and [it’s] not reflected in budget. Connect tasks to funding.
• State must make in-roads to punctuate federal dollars. In early childhood, federal dollars are
creating hubs regionally and nationally. Nothing like this [is] happening in PYD at the national
level. This ices out some of the smaller orgs who can be very innovative. [It] must intentionally
include smaller and larger organizations.
• For profit businesses need to be brought to the table, they have a stake in this too.
• We must monitor pennies. State [funding] is not worth the paper it is written on. My board is
weary, our partners are weary. [The] State can change its mind. Grants do not cover interest
when we've had to take out loans to keep things going in between when monies come.
• The state cancelled some of our [funding and] rolled our program into another area and then
came back and asked us for a report when the program was no longer in our hands.
Repeating a refrain heard often, providers voiced concern that BFR was the “new flavor of the
month” and that the State changes ideas, concepts, and plans quickly (perhaps too quickly). They
were skeptical to the idea that BFR would be handled differently. One provider asked, “How much
of this is short-term as the State shifts to the health consortium as its model?” Another observed,
“TeenREACH was zeroed out, [then] brought back. Things change too much.”](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-34-320.jpg)
![35
PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BUILDING
ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS
Of more concern is the distrust some providers have of the State’s motivations surrounding
BFR. Some feel that BFR is a Trojan horse to further hollow out an already lean service delivery
system. A provider said, “Theoretically [I have] no problem, but [I’m] concerned that it will be used as
a way to hire the lowest quality, that it’s not being tied to quality or ongoing improvement.” Another
added, “Money must come along with this or we are disinvesting in youth.” Another preferred to think
about BFR’s potential to prompt cooperation and collaboration, rather than competition:
“BFR would be great if not connected to the budget.
We are all on the same team; let's elevate the reflection,
talk with one another, and innovate when tied to money.
Money creates competition.”
Acknowledging the small size of both the survey and focus group samples, we want to encourage
the State not to ignore these concerns about gaps in capacity. The State’s lack of staffing, planning
and resource capacity, as well as its uneven practices and processes, is well documented in annual
Inspector General audit reports, public testimony at legislative hearings and committees, as well as
external evaluations (such as the 2010 Pew Center on the States’ Grading the States report on Illinois
government performance.) Facing, and overcoming, distrust between strategic partners will not be
easy, and would probably require some additional change management and internal culture work
on the part of IL DHS (as well as some rather significant changes to the State’s financial processes.)
However, greater intentional and more transparent communication about the State’s
commitment to BFR, as well as to a consistent evidence-informed methodology for
implementation, could balance this skepticism.](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-35-320.jpg)
![103
APPENDIX H – YOUTH PROVIDER FOCUS GROUP DATA
Focus Group Data
Springfield Group 1/16/2013
DEVELOPING A SHARED DEFINITION OF POSITIVE YOUTH DEVELOPMENT
Reactions to the Definitions Shared/Reactions to the survey data/Additions you would add/
Concerns/Cautions
• Lot of [our] work is empowerment and not as much the language related to the materials or
survey...about having youth involved in the planning of the programs
• "Empowerment" loosely stated--what is empowerment really? Living wage job, going to college.
Youth are saying help me navigate through my goals
• Protective factors having an impact on positive. Outcomes...where do they come in at? Issue of
strides--time span of change
• Positive outcomes--economic/quality of life (more + with school)...economic is education and
economic development
DEVELOPING A SHARED DEFINITION AND SET OF OUTCOMES
Reactions to the definitions of developmental domain, outcomes, indicator, measure/Reactions to the
survey data in comparison to the definitions/What is missing? Developmental domains, Outcomes,
indicators, measures/What guidance would you offer to the state around outcomes?/ Concerns/Cautions
• We are more interested in school success/absence of delinquency (resiliency)--being more
important to connect outcomes/vague stuff...Materials were mostly was about resiliency
stuff...volunteering/community service measures
• Too often we deal with outputs/#s--as opposed to critical areas--that is what we ought to be doing
• Programs are unique across--same tool (across all sites)--reduction of arrest/# of days and
involvement/grades/more they attend
• Shift from subjective measures to real hard data
• Change of behavior can occur-but what about sustainability of change over time--beyond the
life of our program--long terms effects of programs
• Band-Aid effect/hot new term impact--but what about the family context--are giving enough
to get long term effects?
• What about family resiliency?
• What about community? What about environmental consequences?
• Question of resources--to do enough at Human Services--must be realistic about expectations
• CYS--goals is to impact community/youth/family/
• BFR--what should programs be doing or not doing?
• Influence behavior change that is documented
• TeenREACH--parent/community/youth component--not sure if that happens across all programs
• Record # of youth showing up--staff having to up their professional development to measure
outcomes...CYS-TeenREACH silo(ed) from each other--Huge disconnect
Guided portion of the discussion to reach consensus on the developmental domains for the
outcomes:
Academic/Educational Success/Learning
School success
Graduate middle schools
Longitudinal Research](https://image.slidesharecdn.com/c47f48e4-26d4-4fd6-b3a0-ffb4cef63722-150319133814-conversion-gate01/85/64968-PDF-103-320.jpg)
64968.PDF
- 2. 2 ACKNOWLEDGMENTS Rosalind Fielder, M.A.,M.S. (L.I.S.) Assistant Professor of Library & Information Services and Reference & Instruction Librarian, Acting Library Instruction Coordinator, Selector and Liaison for Political Science, Law and Military Science, and Criminal Justice University Library at Chicago State University Timothy Jones-Yelvington Independent Consultant, currently completing the Masters of Education in Youth Development at the University Of Illinois at Chicago College of Education CONTRIBUTING STAFF Lisa Marie Pickens, Independent Consultant – Facilitation, Research and Evaluation Illinois Department of Human Services Layla Suleiman Gonzalez, Chief Results Officer Donors Forum Valerie S. Lies, President and CEO Delia Coleman, Director, Public Policy Laurel O’Sullivan, Vice President, Public Policy Laura Zumdahl, Vice President, Nonprofit Services Donors Forum thanks the Boeing Corporation for supporting this work. ABOUT DONORS FORUM Donors Forum is the hub of Illinois’ philanthropic and nonprofit community, and the only statewide association of funders and nonprofits of all kinds, as well as their advisors. Together, we connect and collaborate, discover and share ideas and practices, and protect and promote our missions. Join us as we leverage our collective power to improve lives and strengthen communities: http://www.donorsforum.org. 208 South LaSalle Street Suite 1540 Chicago, IL 60604 Phone: 312.578.0090 Toll-free: 888.578.0090 Library Phone: 312.578.0175 TTY: 312.578.0159 Email: info@donorsforum.org
- 3. 3 DEFINING, NOT REINVENTING,THE WHEEL: A PILOT STUDY ON YOUTH DEVELOPMENT OUTCOMES AND BUDGETING RESULTS EXECUTIVE SUMMARY This report can be used as a follow up to the report in 2012 by American University and Indiana University report, Budgeting for Results: Key Issues for Concern, commissioned on behalf of Donors Forum. 1 The goal of this current report is to contribute to the implementation of BFR by modeling a process to engage providers in outcomes development. The 2012 study was national in scope; in contrast, this study is local. It took two state programs (Community Youth Services and TeenREACH) and engaged a small cohort of providers within those programs who responded to a survey and then participated in two focus groups in Springfield and Chicago. This report also outlines reasonable next steps and is a starting place for bridging agency outcomes indicators and on-the-ground provider activities. Most importantly, it indicates how developing outcomes can be a complex process, requiring a sustained, long-term, and broad level of contact with providers and experts field by field, program by program. CRITICAL QUESTIONS • What capacity do providers need in order to be successful under BFR? • What kind of outcomes training needs are there from providers (and the state agency)? • What is a beginning point for creating evidence-informed outcomes (not outputs) for the state to begin to measure that both providers and state practitioners can agree upon? KEY FINDINGS On outcomes: • The field literature reveals 5 outcome models that can provide a useful starting point from which the State and providers can develop outcomes for youth development programs. Providers not only need human capital, funding and technology resources in order to be successful in BFR, they need guidance from the State on what success looks like for positive youth development. • Providers are familiar with outcomes and want to use them but the patchwork delivery system is a significant barrier which creates outputs rather than outcomes. • Providers are not only familiar with developments in the field and evidence-informed best practices, they are eager for the State to use one of these already present models to define these programs so they have greater clarity around where they (and their activities) fit within them. On capacity building for BFR: • Concerns persist (from both providers and agency staff) about the State’s internal capacity to identify, collect and manage outcomes data. KEY THEMES IN THE RECOMMENDATIONS For Government: recommendations focus on landscape assessment, data collection and the need to invest in capacity building for both its contracted providers and IL DHS. For Philanthropy: recommendations encourage foundations to invest in capacity building for grantees and share their program measurement expertise with government. For Providers: recommendations encourage nonprofit organizations to use BFR as an opportunity for continued strategic growth and impact for their communities. WHERE TO GO FROM HERE The ideal relationship between nonprofit providers, philanthropy and government is one that is fair, efficient, accountable, responsive and transparent – all the goals of Budgeting for Results. While recognizing the limitations of this initial pilot study, we hope the State, and its nonprofit and foundation partners, can adopt a more robust study that can ground the service delivery landscape in best practices, gather useful internal and external data, and create evidence-informed outcomes for the best use of public resources DOWNLOAD THE 2012 BFR REPORT HERE.
- 4. 4 TABLE OF CONTENTS INTRODUCTION .................................................................................................................................... 5 Purpose of the Report ........................................................................................................................... 6 IL DHS Youth Programs ......................................................................................................................... 7 METHODOLOGY ...................................................................................................................................... 8 LESSIONS LEARNED ................................................................................................................................. 9 Key Findings From The Positive Youth Development Literaure Review ............................................... 9 Key Findings From Interviews With IL DHS Staff ................................................................................. 15 PROVIDERS ON PYD MODELS, OUTCOMES & CAPACITY BUILDING ........................................................ 16 Defining PYD ....................................................................................................................................... 16 Reaching Consensus on PYD Outcomes .............................................................................................. 18 How Providers Define PYD Outcomes ................................................................................................ 21 The Data Providers Currently Collect .................................................................................................. 23 How Providers Collect Outcomes ....................................................................................................... 24 How Providers Use Outcome Data ..................................................................................................... 26 Developing Outcomes: Provider Concerns ......................................................................................... 27 PROVIDERS ON BUDGETING FOR RESULTS & CAPACITY BUILDING ........................................................ 31 Awareness ........................................................................................................................................... 31 Program Implications of BFR ............................................................................................................... 32 Capacity Building Needed by Providers .............................................................................................. 33 Illinois Implementation Capacity and Provider Concers ..................................................................... 34 STAKEHOLDER RECOMMENDATIONS .................................................................................................... 36 For Goverment .................................................................................................................................... 36 For Philanthropy ................................................................................................................................. 37 For Providers ....................................................................................................................................... 38 Where Do We Go From Here? ............................................................................................................ 38 APPENDIX A– Positive Youth Development Literature Review and References ........................................ 39 APPENDIX B –Staff Interview Protocol ................................................................................................... 47 APPENDIX C – Key Themes From Staff Interviews ................................................................................... 48 APPENDIX D – Youth Provider Survey & Focus Group Protocol ............................................................... 53 APPENDIX E – Aggregated Data From The Youth Provider Surveys .......................................................... 54 APPENDIX F – Youth Provider Focus Group Materials ............................................................................. 93 APPENDIX G – Forum For Youth Investment With The National Collaboration For Youth ......................... 94 APPENDIX H – Youth Provider Focus Group Data .................................................................................. 103
- 5. 5 INTRODUCTION The state ofIllinois’ Department of Human Services is one of the largest financial supporters of youth service provision in the state and as such is also deemed responsible for the quality of said services. The state has historically used a traditional and incremental approach to budgeting for organizational contracts in which the previous year’s budget was used as a baseline for developing the budget for next year. Budgeting for Results (BFR) is a performance based budgeting approach which theoretically establishes state priorities and connects funding to outcomes for each of those priorities. Put another way, resources are allocated based on how effectively a program or service provider achieves established goals and objectives as opposed to being based on historical funding levels. Donors Forum has been monitoring BFR since it was enacted in 2010 by Governor Quinn. Since then, in public testimony, research and its direct outreach to officials, Donors Forum has consistently advocated for a prolonged conversation between state government, philanthropy and nonprofits as a way to strengthen public decision-making at large, and Budgeting for Results, in particular. If done well, BFR can be a strategic guide to help Illinois ensure more responsible management of the existing partnerships between state government, philanthropy, and the nonprofits that deliver many of the services Illinois relies on. If not done well, the fabric of service delivery (which is already tearing due to continued fiscal stress) will be irreparably damaged. This current report can be seen as a follow up to Donors Forum’s 2012 report, Budgeting for Results: Key Issues of Concern, which was produced in cooperation with researchers from The American University and Indiana University. This report endeavored to help identify best practices and common challenges from other states as a basis for recommendations for implementing performance-based budgeting in Illinois. It offered Six Key Features of Sound Performance-based Budgeting: 1. Government-wide, Multi-year Strategic Planning 2. Agencies Develop Operating Plans to Achieve Government-wide Goals 3. Valid and Meaningful Performance Measures for Agencies and Programs 4. Resource Allocations are Aligned with Goals 5. Independent Collection and Transparent Reporting of Outcomes 6. Outcomes Measures Inform Goals, Objectives, Allocations, and Operations One year after Donors Forum released that research, it is good to report that the state is making steady progress with its strategic planning and the development of state agency operating plans; they have created Chief Results Officers in each agency and are beginning to build a portal for outcomes data to be collected. It has yet to begin allocating resources in alignment with goals. We hope this current report would be useful for Illinois state government, and the BFR implementation team, as it continues to create valid and meaningful performance measures (#3), collect and transparently report on goals and outcomes (#5), and eventually use these chosen outcomes to inform the future, strategic planning of programs (#6) – and thus budgets. Ultimately, Donors Forum’s goal is to help BFR become a useful tool to strengthen the operating environment for nonprofits engaged with the state.
- 6. 6 INTRODUCTION PURPOSE OF THEREPORT In early 2011, as the discussions about BFR implementation were beginning, Donors Forum and designated leaders of the Illinois Department of Human Services decided to partner and pilot a process for engaging providers around outcomes development, review evidence-informed practices from the field so outcomes are grounded in practice, and help the state discover an efficient process by which to engage their providers around the complicated work of measuring diverse human services for this heightened environment of results-based budgeting. These were the critical questions and assumptions we hoped to explore with this small pilot: What do nonprofit providers need in order to be successful under BFR? Assumption: nonprofit providers will need a great deal of capacity building support. What kind of capacity would be expected from nonprofit providers in this new era of outcomes measurement? Assumption: nonprofit providers will need extensive training in outcomes definition, collection and analysis, as well as increased infrastructure support to manage higher demand for results. How can the state effectively engage with practitioners and philanthropy around developing these outcomes? Assumption: the state needs to do ‘more’ to engage providers in the implementation process. In the course of our pilot, we have learned lessons (particularly around the dynamics of community inclusion and reflexivity), answers have been given to our questions, and some of our assumptions have been challenged – or given a deeper context, (particularly around the question of whose capacity needs to be built.) On the recommendation of IL DHS, Youth Service Providers were chosen as a pilot focus group. Knowing that attempting this effort with all Youth Service Providers was too ambitious and unwieldy given a 3 month time constraint, this pilot process was to be limited to a subsection of Youth Service Providers that are a part of IL DHS’ Division of Family and Community Services, specifically Community Youth Services (CYS) which is a program of the Bureau of Positive Youth Development (BPYD), Teen Responsibility, Education, Attainment, Caring and Hope (TeenREACH) and Comprehensive Community-Based-Youth Services (CCBYS), which are programs of Youth Intervention Services. The pilot was to incorporate an environmental assessment of the Positive Youth Development field with a focus on outcomes and indicators; garner input from and engage with IL DHS staff and program providers about outcomes and indicators, and culminate in a final report that will guide IL DHS in adopting jointly agreed upon outcomes and indicators for their youth development programs. Additionally, it would help identify areas for capacity building for the providers field, to strengthen overall service delivery and measurement. It is hoped that this process could be mirrored in other program areas, and with support from the foundation community which has invested heavily in several program areas. This report represents what has been learned during the pilot process
- 7. 7 INTRODUCTION IL DHS YOUTHPROGRAMS The Bureau of Positive Youth Development (BPYD) is comprised of programs that address primarily prevention. They share these characteristics: • Community-based services • Use similar approaches (e.g. after-school program) • Target similar age groups (e.g. youth) • Target multiple domains (e.g. youth, parents) • Target universal and selected populations Community Youth Services (CYS) —a program of BPYD-- is described as offering “….innovative programs in two hundred diverse communities and counties for youth, young adults and families. CYS programs are aimed at reducing and preventing juvenile delinquency” (IL DHS Website). CYS works through community committees to address, evaluate and resolve the needs of the youth, families (the community) in the area in which they are working. Youth Intervention Services (YIS) strives to offer prevention, diversion, intervention, and treatment services targeting youth to support families in crisis; prevent juvenile delinquency; encourage academic achievement; and to divert youth at risk of involvement in the child welfare, juvenile justice, and correctional systems (IL DHS Website). TeenREACH and CCBYS are housed under Youth Intervention Services. TeenREACH programs, services and activities are provided during non-school hours to youth ages 6-17, when youth are most likely to get into trouble. Core Service areas include: improving educational performance; life skills education; parental involvement; recreation; sports and cultural/artistic activities; and positive adult mentors and service learning (IL DHS Website). Comprehensive Community Based Youth Services’ (CCBYS) goals are to provide comprehensive and community-based individualized services to at-risk youth and their families to achieve family stabilization and reunification, thereby diverting or minimizing youth contact with the juvenile justice and/or child welfare systems. The mandatory core population, ages 11 - 17, are any minors who meet the following criteria: • Youth who are absent from home without consent of parent, guardian or custodian • Youth who are beyond the control of their parents/guardians or custodians in circumstances which constitute a substantial or immediate danger to the minor's physical safety • Youth, who after being taken into limited custody and offered interim crisis intervention services, refuse to return home after the minor and his or her parents, guardians or custodians, cannot agree to an arrangement for an alternative voluntary residential placement or to the continuation of such placement • Lockouts: Minors, 11 - 17 years of age, whose parent or caregiver has denied the child access to the home and has refused or failed to make provisions for another living arrangement. The CCBYS Program provides a continuum of services statewide to youth in high risk situations. A 24-hour crisis intervention response system is available in emergency situations for referrals from youth, parents/guardians, police, courts, schools, Safe Place, and the Department of Children and Family Services (DCFS). In addition to the CCBYS mandated programming, other discretionary services appropriate to the youth may also be provided. CCBYS was included in the survey, but was not included in the focus group discussions because this programming is focused specifically on at-risk youth, unlike the description of CYS and TeenREACH.
- 8. 8 METHODOLOGY This project hadthree phases, with one phase building upon the last. In the first phase, we began by conducting a literature review of the Positive Youth Development field to help us prepare for our engagement with the Youth Service Providers, and to ground our approach in the field’s evidence-informed practices. In the second phase, we conducted in-depth interviews (see Appendix A & B) with IL DHS Staff who managed the major subset of youth development programs to be included in the pilot: CYS, TeenREACH and CCBYS. The goal of these interviews was to better understand how the state is currently developing outcomes within these program areas; better understand the individual programs under their purview; and to benefit from the guidance of staff on how best to engage Youth Service Providers. In the third phase, engagement with Youth Service Providers was multi-layered as well. We developed and fielded a survey to all Youth Service Providers (see Appendix C & D) in CYS, TeenREACH, and CCBYS. This survey explored ways in which providers defined Positive Youth Development; the types of outcomes providers currently track; and the measurement and tracking tools they use. While the overall pilot was constrained by time, conducting a youth provider survey provided the perfect opportunity to collect a breadth of information that could inform both the focus group discussions (designed for greater exploration) and recommendations for the next phase of the project. Based on youth provider lists provided by IL DHS staff, the survey was sent to 323 providers from TeenREACH, CYS and CCBYS; it received about 100 bounce backs from bad email addresses, and had a response rate of 11% (24 respondents). Some key organizations were not included in the original youth service provider list provided to us by IL DHS Staff and based on this exclusion strongly we suggest these organizations be engaged in the next phase of this project. Coupled with the literature review and our discussions with IL DHS staff, as well as our internal team discussion, this data allowed us to develop contextualizing materials (See Appendix E) for focus group participants to review prior to group sessions. This layered approach primed the pump for their thinking about BFR, outcomes, tracking methods, etc. and allowed for an extremely candid and rich set of discussions with both downstate service providers and Chicago-based providers (See Appendix F). Because they represent providers not engaged directly in prevention work and most focused on utilizing Positive Youth Development principles in their programming, only CYS and TeenREACH service providers were selected for participation in the focus groups. Upon entering this project we asked whether youth providers and the state agency shared the same vision of their programs. A shared vision between providers and the State would also lead one to infer a shared understanding of anticipated program outcomes and program success indicators.
- 9. 9 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW HISTORY AND THEORETICAL UNDERPINNINGS From the literature review (See Full Literature Review in Appendix A), we know that PYD has about a 20 year history as a field, with its beginnings tied both to sound youth worker and practitioner experience-based knowledge and practice. PYD is also grounded in the ecological model of human development and developmental systems theory which views the various layers of influence in young people’s lives, i.e., family, community, schools, social networks, and the broader society/culture, as interconnected and reciprocally influential. It is important to note that both IL DHS programs like CYS and TeenREACH incorporate this PYD principle fully into their programming and recognize the interconnectedness and reciprocal nature of young people’s influence on their environments, thus making those two programs ideal for this effort. WHAT IS PYD? Practitioners see PYD as a framework that appreciates youth as resources to be developed, not as problems to be solved, or risks to be mitigated; PYD is also a set of principles, a philosophy or programmatic strategy, which can be applied to all types of youth programs with the holistic approach of developing youth. Thus it would be important for IL DHS to determine which of their programs is using PYD as a framework differing from those programs using it as a programmatic strategy or philosophy in order to properly align PYD outcomes. In other words, those programs that use a PYD framework should be held accountable for that framework and the agreed to outcomes — which may look a bit different from programs using a PYD strategy in which a program may be using elements of a positive youth development framework. THE RELATIONSHIP BETWEEN PREVENTION AND PYD Karen Pittman’s statement (“Problem free is not fully prepared”) is apropos as we think about the relationship between prevention and PYD. The relationship between prevention frameworks and PYD is not antagonistic. The notion of tension may arise more from the isolated ways in which funding is allotted and dispersed rather than from any fundamental ideological differences. Ongoing research demonstrates that, in practice, PYD researchers, practitioners and advocates understand prevention and PYD frameworks to be complimentary and interrelated. This relationship should be incorporated into thinking about IL DHS youth programming. Ideally, there should be a process for identifying general evidence-informed PYD principles and outcomes used by all programs that would be appropriate to apply across programs, and then to denote specific intervention-based outcomes that make sense for the prevention programs. It would be important that PYD and prevention not be seen as in conflict, but as complimentary. For instance, avoiding pregnancy or drugs as a teen is not the same as having developed the competencies to do more than survive, but thrive. The two behaviors are related, but not the same.
- 10. 10 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW While the literature review was not exhaustive, it is clear that there are a wide range of PYD Asset models available for use by state program administrators, service providers, and youth development funders. These models present domains in which young people (all of whom already possess assets or potential assets) develop skills and competencies. An asset may be important take away from the literature review is that PYD outcomes might be best understood as domains for ongoing growth/developments as opposed to a goal unto itself. Because we primed the pump in focus group discussions by achieving consensus on broad outcomes domains (see the work by the National Collaboration for Youth (NCY) and the Forum for Youth Investment (FYI) in Appendix A), the overlap between the major PYD Asset Frameworks potentially provide IL DHS with a starting place to pick up the next phase of the discussion with youth providers about agreed to outcomes. These models may be used as possible checklists for gauging whether a program has laid the appropriate groundwork to promote PYD. MODEL 1 The Search Institute, a youth development intermediary institution with a lengthy track record of providing research, technical assistance and training to youth development organizations, developed a comprehensive list of 40 developmental assets critical for youth to thrive. These developmental were divided into two groups, external and internal assets (40 Developmental Assets, 1997, 2007). Within each domain, the assets were further categorized within one of four sub-domains. To provide an example of how assets were grouped within sub-domains, the external sub-domain “support” included a caring neighborhood and school climate, and positive family communication, among others. The Search Institute has had success in motivating community-wide initiatives around these 40 assets (Hamilton, et al. 2004). Their website and other materials provide an array of resources for putting youth development principles into practice. External assets (those provided by programs, families and communities) included: • Support • Empowerment • Constructive Use Of Time • Boundaries And Expectations Internal assets (those possessed and further developed by youth themselves) included: • Commitment To Learning • Positive Values • Social Competencies • Positive Identity
- 11. 11 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW MODEL 2 The Committee on Community-Level Programs for Youth, an initiative of the National Academy of Sciences, has also developed a list of developmental assets. This list had similarities with the Search Institute’s, and divided assets into one of four domains: • Physical Development • Intellectual Development • Social Development • Psychological and Emotional Development Examples of physical development included good health habits and risk management; examples of intellectual development included school success and knowledge of essential vocational skills; examples of social development included a sense of connectedness with parents and peers; and examples of psychological and emotional development (the most extensive category) included emotional self regulation, conflict resolution and coping skills. The Committee on Community Level Programs for Youth further elaborated on this asset framework with a list of eight attributes necessary for community settings seeking to promote the development of these assets, making their framework especially useful for organizations engaged in program development and evaluation. (National Research Council, 2002). These eight community attributes included: • Physical And Psychological Safety • Appropriate Structure • Supportive Relationships • Opportunities To Belong • Positive Social Norms • Support For Efficacy And Mattering • Opportunities For Skill Building • Integration Of Family, School And Community Efforts
- 12. 12 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW MODEL 3 The “Five C’s,” originally developed by Karen Pittman, the ‘5 C’s’ have provided a simple and memorable method for identifying PYD outcomes. The “C’s” are Competence, Character, Connections, Confidence and Contribution (Hamilton, et al., 2004, Lerner, et al., 2005a). In some formulations, Contribution has been replaced with “caring.” While the Search Institute and the Committee on Community Level Programs for Youth’s frameworks are useful for providing a more detailed breakdown of the assets needed for young people to thrive, they may be unwieldy for quickly communicating the goals of PYD to communities, funders, or other stakeholders. Contribution (or Caring) refers to a young person’s empathy and commitment to “give back” to others. Competence refers to the various skills needed to achieve one’s goals and adapt to diverse contexts. Confidence is the self-assurance required to pursue one’s objectives. Character is a young person’s commitment to moral and ethical principles. Connections are healthy social relationships with adults and peers. MODEL 4 The National Collaboration for Youth (NCY), a coalition of national agencies committed to PYD, has worked with the Forum for Youth Investment (FYI) through their “Ready by 21” initiative to develop a set of outcomes for PYD programs. These outcomes were categorized within five developmental domains – Thriving, Connecting, Leading, Learning and Working. Each domain contains three to four outcomes. For instance, outcomes within the “leading” domain included community connectedness, social responsibility, and leadership development. A major advantage of the NCY/FYI evaluation resource is that each outcome is illustrated by specific indicators of progress, as well as suggestions of concrete measurement tools culled from NCY’s members and other youth organizations. These are tools that organizations (including state programs) may use to measure their progress (The Forum for Youth Investment with the National Collaboration for Youth Research Group, 2012). Although evaluations based in youth outcomes and supported by empirical evidence, rather than program quality, are challenging to implement within a PYD framework, tools are nonetheless available to help practitioners track their outcomes.
- 13. 13 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW MODEL 5 Youth-led participatory evaluation is one of the most progressive evaluation frameworks within the youth development field. In a participatory evaluation process, young people identify their own goals and metrics for achievement - a process that serves the dual purpose of further developing their skills, competencies and engagement in programming, while simultaneously evaluating the success of the program in which they are participating. After reviewing the literature to identify youth-oriented evaluation models, and finding none, Kim Sabo-Flores (2008) elaborated the youth participatory evaluation approach. Sabo-Flores detailed concrete processes youth development practitioners may use to facilitate participatory evaluation. Participatory evaluation metrics are necessarily open-ended at the onset, which may make them unsuitable for organization structures whose funders expect concrete metrics in advance of funding (e.g., the State of Illinois.) Yet elements of participatory evaluation may still be implemented in specific youth-led campaigns and projects within organizational structures that practice more traditional forms of evaluation.
- 14. 14 LESSONS LEARNED KEY FINDINGSFROM THE POSITIVE YOUTH DEVELOPENT LITERATURE REVIEW PYD EVALUATION & PROGRAM QUALITY There are a number of research studies that show high quality PYD programming results in positive outcomes. (See Appendix A) And we know from the literature that it would be challenging for Illinois youth service agencies to conduct their own evaluations of their PYD programs with the level of rigor necessary to show direct correlations between their programs and positive outcomes. IL DHS is presented with a unique opportunity to support youth providers to effectively track agreed upon outcomes and it would be incumbent upon the State (or a third-party intermediary) to conduct an evaluation tracking outcomes in an effort to better under both the impact of programming and thus the impact of funding. There are a range of tools that organizations may use to measure program quality including the Committee on Community Level Programs for Youth’s list of features that promote PYD (National Research Council, 2002), the Rochester Evaluation of Asset Development for Youth (Sabaratnam & Klein, 2006), and the Youth Program Quality Assessment, a widely employed tool disseminated by the David P. Weikert Center for Youth Program Quality. EXAMPLES OF PYD ORGANIZATIONS It may not be necessary for the State of Illinois to reinvent the wheel when it comes to PYD outcomes and measurement. The wheel has already been set in motion. Positive Youth Development (PYD) emerged from the work of academics and youth development intermediary organizations (i.e., Karen Pittman’s Forum for Youth Investment) and was informed by the work of grassroots youth workers. Some youth organizations have since been influenced to explicitly reference PYD principles in their outreach materials, reports and/or evaluations, and otherwise more actively self-identify as PYD organizations. Who are leaders in PYD practice that Illinois can engage or model? (What follows is not meant to be exhaustive.) PYD organizations include the STEM education organization Project Exploration, the 4-H Youth Development Organization (a national leader in PYD evaluation efforts) and the Boys and Girls Club of America. Currently, the Chicago antiviolence organization BUILD is working with Stacey S. Horn from University of Illinois at Chicago, with support from the Polk Brothers Foundation, to comprehensively implement PYD. It is important to note that PYD does not necessarily mandate a particular approach to programming, but is rather a set of asset-based principles that may be applied in a range of different programs. All of the programs listed above incorporate PYD philosophies and values (all have some level of focus on holistic development, i.e. promoting assets and working across two or more aspects of the developmental system such as school/family/ community), and offer a decent mix of size/constituency/focus area. In Catalano, et al’s (1998) seminal evaluation of the positive impact of PYD approaches, organizations were categorized as PYD based upon whether their work contributed to young people’s developmental assets, and the extent to which they functioned across multiple contexts (i.e. family/school/community), not whether they self-identified as PYD organizations. This study included a number of entities whose primary identifications were as prevention or mentoring organizations (although organizations that targeted only specific “at risk” groups were not eligible for inclusion).
- 15. 15 LESSONS LEARNED KEY FINDINGSFROM INTERVIEWS WITH IL DHS STAFF These are summary conclusions from our conversations with the staff from the Bureau for Positive Youth Development (where CYS is housed) and Youth Prevention Services (which houses TeenREACH.) (See Appendix C for details.) INTERNAL COMPETENCIES Key Finding: The staff has long histories at IDHS. Both have practitioner experience with evidence-based strategies, logic models, and best practices; however, resource constraints seem to be affecting their ability to implement these types of practices in their current positions and divisions. INTERNAL CAPACITY & BFR IMPLEMENTATION CONCERNS Key Finding: Staff shares provider concerns about how BFR will impact federal funding, lack of funding for capacity building, data collection methods and a potential threat to General Revenue Funding. INTERNAL CAPACITY TO COLLECT AND ANALYZE DATA Key Finding: Staff from both programs shares concerns that IL DHS does not have the capacity to collect or analyze the type of data that would be required by BFR. PROGRAM CONCERNS Key Finding: There is some internal disagreement about the relationship between CYS and TeenREACH, program definitions, and program outcomes. PROVIDER ABILITY TO MEET BFR EXPECTATIONS Key Finding: TeenREACH staff anticipates their providers are much more prepared to achieve standards for BFR than does CYS staff. DETERMINING HIGH-PERFORMING PROGRAMS Key Finding: Staff point to differing modes of describing quality and high-performance with each program. It is unclear whether there is an evidence-informed rubric to define quality. DETERMINING POOR PERFORMING PROGRAMS Key Finding: While staff describes poor performance in different ways for each program, what is clear is that the assessment of ‘poor’ seems rather subjective, rather than based on evidence-informed strategies found in field literature or research. It is unclear whether there is a rubric to define quality. PROGRAM DATA COLLECTION Key Finding: In both CYS and TeenREACH quantitative and qualitative data are being collected, however it is mostly outputs, not outcomes.
- 16. 16 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING DEFINING PYD Key Finding: The majority of both survey and focus group respondents understood the concept of Positive Youth Development in ways that correspond to the definition in the literature review, as either a framework or strategy or principle or philosophy and directly connected to the work of their organizations. We asked the youth providers to define PYD and they either responded with a definition very similar to the definition in the literature review, or they identified some specific component of PYD as identified in the literature review, or shared specific frameworks as their definition of PYD. Many of the definitions incorporated the young person’s environment as part of the equation. The responses were a combination of defining PYD as a framework or strategy, principle and philosophy, yet the important thing to build upon was the universal recognition of PYD and its connection to the work of these providers. Some of the most interesting definitions from the survey include: • Positive youth development is a framework that fosters internal and external protective factors to moderate risk factors and enhances optimal physical, cognitive, social and emotional development. • Positive youth development is efforts (activities/interventions) to support youth in achieving their dreams and becoming contributing members of their community. It emphasizes the importance of focusing on youths’ strengths and providing opportunities to help others through service. • PYD emphasizes the importance of focusing on youths’ strengths instead of their risk factors to ensure that all youth grow up to become contributing adults." Many researchers have looked at this and have created frameworks around it. It relates to our programming because much of our prevention programming is emphasizing strengths and building capacities in order to mitigate risk factors that exist in their communities, their homes, or even their physical predisposition. • Positive Youth Development is a process which prepares young people to meet the challenges of adolescence and adulthood through a coordinated, progressive series of activities and experiences which help them to become socially, morally, emotionally, physically, and cognitively competent. • PYD is a policy perspective that emphasizes providing services and opportunities to support all young people in developing a sense of competence, usefulness, belonging and empowerment. However, when thinking about cautions or concerns related to a shared definition of PYD, focus group participants thought it would be important to make sure there was complete clarity and alignment around language both among themselves and with the state, and that language should be related to social/emotional development would be included, as well as concrete achievement indictors (i.e., related to academic achievement.)
- 17. 17 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING WHERE DOES DELINQUENCY FIT? Key Finding: Despite consensus among youth providers about the definition of delinquency, a question remains whether there is consensus within IL DHS about the definition of delinquency and how it fits within the context of their positive youth development programs. Since the term ‘delinquency’ does not fit into Positive Youth Development models and frameworks as observed in the literature (neither is it used in the description of the Bureau of Positive Youth Development nor in the description of TeenREACH program), it was interesting to discover that CYS was described as “…reducing and preventing juvenile delinquency” and was similarly described in Youth Intervention Services. When we coupled this with one staff person raising concern there was no agreed upon IL DHS definition of delinquency it seemed worth investigating whether there was consensus among the providers. Twenty-one providers responded with a definition of delinquency. Eighteen providers offer definitions that involved young people’s unwillingness to adhere to rules (such as not attending school, no fighting, etc), engaging in criminal active and/or involved with law enforcement or juvenile justice system, or locked out by parents or refusing to return home. One of the eighteen providers shared that they did not use this term in their programming but then went on to define it in terms similar to others. Three providers offered very different responses to this question. Two offered indictments of the circumstances in which the young people find themselves as the definition of delinquency. • Youth who are unable to live at poverty levels and are in dire needs of social and emotional supports. Youth who struggle to meet the challenges of everyday life. • Youth without support systems. The other provider instead of offering a definition of delinquency shared an approach to their work. We don't use the word Delinquency on a regular basis. We choose to focus on the whole youth, and address problems and needs as they occur. There appears to be consensus among 18 of the 21 about the definition of delinquency or about the behaviors that constitute delinquency. Among at least 4 providers, this is not language they use in their programming.
- 18. 18 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING REACHING CONSENSUS ON PYD OUTCOMES Key Finding: Among survey and focus group participants, providers were very engaged around coming to a common agreement about PYD outcomes, and there was broad consensus among them about broad outcome domains (Learning, Working, Connecting, Leading and Thriving) – despite the difficulty posed in collecting data for them. We think their engagement can be attributed, in part, to the staff level they represented (i.e., Senior and Executive level), as well as their familiarity with evidence-informed practices and field literature. The participants described the State’s previous focus on outcomes as being more about outputs and numbers; they felt it was important for the State to begin to focus on critical areas related to outcomes. Another participant felt it was time to shift from the subjective and focus on real hard data. It was felt that Out of School Time (OST) was still nebulous as a program area, that OST funders wanted to see outcomes outlined in a plan, and that going in that direction (i.e., toward more rigorous outcomes measurement) made sense. In the guided portion of the focus group, we utilized the broad outcome domains (Learning, Working, Connecting, Leading and Thriving) 2 to determine if we could reach consensus. Because we had heard from participants during the unguided portion of the focus group about the types of outcomes that should be included, we asked participants not to get stuck on the specific title of the domain, but to suspend judgment instead and share what items would fit under the broad outcome domain headings and simply allow us to use the specific title as a place holder. Because there was general agreement about these broad domains, perhaps it would be most important for the State, coupled with providers, to identify next level program outcomes and then measures that would correlate to these broad domains. Under the broad outcome domain of LEARNING providers grouped the following possible outcomes: • Demonstrate engagement in Academic/Educational Success/Learning • Achieve School success (i.e., grades) • Graduate middle school Because there was general agreement about these broad domains, perhaps it would be most important for the State, coupled with providers, to identify next level program outcomes and then measures that would correlate to these broad domains. 2 As outlined in A Shared vision for youth: Common Outcomes and Indicators
- 19. 19 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING REACHING CONSENSUS ON PYD OUTCOMES However, providers also pointed out that, in order to measure those things accurately, they (and the State) needed to engage in Longitudinal Research; they (and the State) needed to define the kind of academic engagement that would be appropriate for their particular programs to measure and collect data for. For instance, providers questioned whether it was fair to be held to a standard for youth achieving academic success when they saw that role being played firmly by schools and teachers. Rather, they offered program outcomes that measure ‘engagement’ in learning and educational achievement. One provider who said they have a very strong connection still finds it difficult get academic achievement data from the schools and trying to obtain such data from parents is still difficult. Providers defined program outcomes for LEADING AS: • Demonstration of civic mindedness, engagement, responsibilities, empowerment, or altruism • Participation in a service learning project • Involvement of youth in program planning and implementation • Involvement of youth in their community • Developing critical thinking skills • Demonstrating self-control & respecting others opinion • Building critical soft skills • Building character • You have students dong well academically-but they are still jerks- [they need] character building Building a sense of empowerment and developing values were also potential program outcomes that measured LEADING, but it also lacked sufficient data to support it as a worthwhile measure for the providers. In one participant’s words “Developing a sense of values [important]... [but] this leads to trouble...Research says about 1/3 of youth not responding.” Another said, “A lot of [our] work is empowerment and not as much the language related to the materials or survey. [It’s] about having youth involved in the planning of the programs.” But despite the difficulty in measurement, providers still think outcomes on developing leadership and character skills is valuable, citing instances of students with academic or hard skills achievement but lacking appropriate socialization or the requisite soft skills a prospective employer would need. When asked if WORKING was a relevant domain providers were less sure of how to measure this kind of success, citing potential barriers such as how their organizations often achieves stabilizing one family in the community but the family moves up and out, only to be replaced by less stable families – and the process begins again. Another participant described financial literacy as a part of the equation and lamented the dissolution of home economics in the schools and wondered how youth were learning about money management. Another provider drew a strong link between education and economics/economic development and felt that these both had real implications for the lives of the youth and families in their programs. These barriers, created by structural inequity and policymaking outside these organizations’ spheres of influence, would presumably require long term solutions and measurement.
- 20. 20 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING REACHING CONSENSUS ON PYD OUTCOMES Providers were surer of program outcomes under CONNECTING - but again pointed out how difficult it would be to collect indicators. They described these as: • Demonstration of pro-social engagement • Demonstration of peer-to-peer, and adult, connectedness • Maintaining or demonstrating engagement with Caregiver, parent, or community Demonstrating the difficulty of establishing core outcomes for diverse populations, providers pointed out that sometimes CONNECTING engagement would look different for nontraditional families, and for non-filial relationships (e.g., “What about mentor/adult engagement?”). One provider also asked about the limits of data collection: “In eCornerstone – how do you track these nuances – how do you track talking with parents for 15 minutes?” If THRIVING is a form of resiliency, providers arrived at the following consensus about what that looked like, though, again, there were nuances to their conclusions. Some participants felt that youth could demonstrate resiliency by being successful; others felt that they could demonstrate it by their ability to see the future beyond the age of 19/20 years of age; and still others felt youth were demonstrating thriving by setting goals for themselves. Several providers mentioned that a sense of spirituality or a sense of hope was a part of THRIVING, although they felt it was avoided due to the separation of church and state. Some participants felt that youth could demonstrate resiliency by being successful; others felt that they could demonstrate it by their ability to see the future beyond the age of 19/20 years of age; and still others felt youth were demonstrating thriving by setting goals for themselves. There are challenges to across the board measurement of a program domain under THRIVING. One provider asked, ‘Where do protective factors having an impact on positive youth outcomes come in?’ When it came to establishing an indicator about safety and violence, another provider shared that they were training staff to recognize adverse childhood experiences and mapping out a plan to support youth. It is provider opinion that THRIVING is difficult to measure - but initially the training itself is an outcome. However, they believe they would be able to see the long-term measureable impact over time from such a training and intervention. Several participants shared that thriving could be nurtured, supported or facilitated with the right programming and the right staff. Another provider talked about their use of the YPQA Assessment Tool which allows them to explore if their staff were creating opportunities for youth to be more reflective. The tool helps staff with successful implementation of such practices which support THRIVING. There was also a strong sense that THRIVING and RESILLIENCY was a key outcome to incorporate into Budgeting for Results.
- 21. 21 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING REACHING CONSENSUS ON PYD OUTCOMES By using a standard methodology from the field that providers were familiar with and encouraging the focus group members not to get stuck on the name of the broad outcome domain, but rather, to think about which outcomes might fit under each domain, we were able to encourage participants to reach consensus at the broadest level. Providers suggested the next step might be to think about the individual outcomes under each broad domain as well as the measure that would be used to track said outcome. These broad domains fit quite nicely with the type of outcomes shared by the survey respondents as well, offering a shared place for the State to continue the conversation with providers. HOW PROVIDERS DEFINE PYD OUTCOMES Key Finding: Providers have a baseline understanding of outcomes and performance measurement that could provide the State with a foundation for future conversations with them about which outcomes might be appropriate for their programs. We thought it would be useful to understand how providers defined outcomes. We found that all of the survey respondents showed an understanding of outcomes (ranging from the general to the specific). For example, one provider defined outcomes as a measurement tool for measuring their effectiveness. 10 of the remaining 19 respondents offered generic definition of outcomes and the other nine offered examples of program specific indicators of outcomes. Here are examples of general outcome definitions: • Outcomes are specific short and long-term results/changes for individuals, groups, communities, etc. stemming from programs or services. • Based on the logic models that we develop, there are theories that form the basis for activities within programs. These in turn are tied to outputs, which could be things such as attendance in programming activities, # of participants that attend workshops, etc. These outputs affect what we expect to be the results of the program activities otherwise known as the outcomes. Since we have programming across the continuum of prevention to treatment, each of the logic models reflect outcomes that are expected within those programs. Outcomes for youth prevention reflects things such as improvement in knowledge of substance use/abuse and the effects, improved attendance at school, improved peer relationships, ability to define a post-secondary plan, ability to create a safety plan for themselves and within their communities, decreases in suspensions and expulsions, decreases in physical fighting among peers, improvements in family functioning, improvements in time management and organization skills. • Youth Services define outcomes as means to determine the level of success in achieving the overall goals and objectives of the program, the changes that are expected to occur as a result of services provide. But some respondents also provided more program-specific outcomes: • Measurable deduction of juvenile arrests and recidivism from a restorative prospective (i.e. station adjustments, peer court/juries). Measurable decreases in youth going to Juvenile Detention Center. Decreases in expulsions, and suspensions. Increase in accountability to community by offender. Increase in community based restorative initiatives and practices. • Reduced occurrences of drug use and new arrests. • Outcomes are youth transitioning to independent living after completing and "working" their life plan.
- 22. 22 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING HOW PROVIDERS DEFINE PYD OUTCOMES Focus group participants categorized outcomes in the following three groups: 1. PYD-specific outcomes 2. Prevention-specific outcomes (which were more focused on reduction of behaviors) 3. Combination of PYD and Prevention outcomes Drilling down further, providers gave these examples of PYD-specific outcomes: • Academic, life skills, attitude toward education, more and improved and more career choices, recreation, cultural and social progress, health and nutritional meals. • Overall, in our entire program: academic retention and achievement; development of career aspiration/choices;- development of soft jobs skills; increase decision-making and problem-solving skills; increase anger-management and conflict resolution skills; increase choices in favor of physical fitness and nutrition; increase engagement with the community. Prevention-specific outcomes: • Reduced occurrences of drug use and new arrests • Increased abstinence, decreased legal involvement • The number of youth who are returned to their home rather than become wards of the state PYD/Prevention outcomes: • Reduction in juvenile delinquent behaviors. Reduction in youth entering Juvenile Detention. Reduction in expulsions, suspensions from school. Increase in positive youth development. Increase in positive recreational, cultural, and educational activities and employment opportunities for youth. Increase in alternatives to suspensions, expulsions, and juvenile detention. • Reduce childhood obesity, reduce the achievement gap, increase kindergarten readiness, increase school attendance, increase academic achievement, reduce school-based behavioral incidents, reduce gang involvement, reduce juvenile recidivism, foster safer communities, increase child safety, etc. • Decrease in the number of youth going into DCFS care. Decrease in the number of youth sentenced to DJJ. Increase in the number of families remaining intact. • Youth engagement. • Academic achievement. It is important to note that the survey respondents included CCBYS providers who have much more of a prevention focus. But the fact that there were many respondents who offered both PYD and Prevention outcomes again makes the case for a complimentary relationship between PYD and Prevention; it implies that it is possible, even across youth service providers who focus on Prevention, to incorporate PYD outcomes. It is also possible that the difference between generic and specific programmatic outcomes was the result of how we phrased the question which could have been interpreted differently by different providers. The important takeaway is that there is a baseline understanding of outcomes and performance measurement that can provide the State with a foundation for future conversations with providers about what these outcomes might be for their programs.
- 23. 23 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING THE DATA PROVIDERS CURRENTLY COLLECT Key Finding: Youth development providers are collecting common program outcomes data that could provide potentially useful program outcome categories for the State in the following asset domains: Learning, Thriving, Connecting, Working, and Leading. This would be an area of deeper exploration. In our survey, we wanted to discover which types of outcomes being collected corresponded with a PYD framework. 17 of the respondents said they were collecting their own outcome data. We could identify that the majority of providers surveyed (13 of the 17 providers) were collecting some data that correlated to Learning outcomes (i.e., academic achievement via collecting grades, GPA, on-time promotion data, etc.) One provider reported using the Developmental Assets Profile developed by the Search Institute which is correlated to another learning outcome—Engagement in Learning. There is a baseline understanding of outcomes and performance measurement that can provide the State with a foundation for future conversations with providers about what these outcomes might be for their programs. 9 of the 17 were collecting outcomes which would be grouped under the Thriving domain. Some of the measures providers report using includes DAP (Search), the Illinois Youth Survey, YASI, and Lionquest Substance Abuse/Use survey. 2 of the providers responded that they collected outcome data that would correspond to the Connecting domain using measures such as the DAP and Me and My World (Search). 4 of the providers shared outcomes that relate to the domain of Working (i.e., using employability assessments, participation in job readiness programs, and transitioning to independent living were specific indicators) but specific measure were not identified. And 2 providers shared outcomes that fall into the Leading domain, with 1 specifically identifying youth participation in leadership development activities as one of the outcomes, but again not identifying a specific measure for this outcome.
- 24. 24 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING HOW PROVIDERS COLLECT OUTCOMES Key Finding: While providers are collecting data, they either do so across different platforms or while trying to overcome significant barriers. Because of these barriers there may be little to no practical connection between provider identified outcomes and those required by the State. We explored with providers which programs they were using to house the outcome data they were collecting. 23 providers responded; 3 providers reported manually storing data, a process which most likely translated into keeping hard copies. The programs that are most familiar are Excel, Access, Google Data Share, SPSS and Word. eCornerstone is the database used by TeenREACH Providers which was developed by the state, COPA and CitySpan are City of Chicago Department of Family and Support Databases, and Efforts to Outcomes is a commercially developed Case Management System. The remaining database systems are unfamiliar and there is no additional information on them at this time. The breakdown for the other electronic programs is as follows:
- 25. 25 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING HOW PROVIDERS COLLECT OUTCOMES While 22 providers responded that they collected their own outcome data, 9 of those 22 identified the following barriers to data collection: • No staff, or not enough staff, available to enter the data. • Redundant or duplicative efforts due to not having the right database to house their own data; a mismatch in the various databases resulting in an inability to export their data to the state system, thus requiring double or sometime triple data entry; the need to assess the data from multiple internal sources and then having to input the same data into multiple systems: i.e., city, state, federal and funders, resulting in a serious drain on staff resources. • Difficulty obtaining consent to access needed client data. For example, providers reported that gathering real-time school data is next to impossible because the long and arduous authorization CPS process. As a result, providers often rely on, and try to get reports, progress reports, etc., from program participants or their guardians directly. • Lack of guidance and consensus from the State on outcomes, related metrics, and requisite help for providers in connecting their programs to the appropriate outcomes. Because of these of barriers, the last barrier in particular, there may be a disconnect between outcomes identified by providers and those required by the State. Alignment — of data, databases, outcomes, and program asset models — between the State and its providers is a critical key to success. What does such a disconnect look like? For the sake of illustration we can borrow from the State’s own presentation about its outcomes development process (see Illustration 1). While the State is making commendable progress developing its internal process for creating state, agency, and program outcomes and indicators, what is missing is a bridge between Agency Program Indicators and programmatic activities providers are currently engaging in and, presumably, the outcomes those activities create. Alignment – of data, databases, outcomes, and program asset models – between the State and its providers is a critical key to success.
- 26. 26 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING HOW PROVIDERS USE OUTCOME DATA Key Finding: Providers use program results for program improvement – but more could be done to support providers’ capacity to analyze their own findings. However, it remains unclear how the State uses program outcomes provided by practitioners. In some detail, 17 providers shared what they are learning from their outcome data. Most of what they learned had to do with the positive impacts of the programs on students (either through improved grades, increased graduation rates, more youth remaining at homes with the parents, less interaction with DOJ, improved attitudes toward family, school, etc.) To a lesser degree, providers had less guidance on their impact on parents and, even less, on teachers. • We learn from the data how our outcome is, compared to our goal, e.g., last fiscal year we have 27 youth in senior class, all of them graduated, and 25 of them (92%) entered post-secondary education. • Because of the afterschool programming, 89% of parents agree that their child has learned new knowledge or skills; 87% agree that their child gets along better with other students; 79% report helping their child with homework at least weekly; 87% report that they are talking to their child about school at least weekly; and 86% report that the afterschool program helped them learn more ways to support their child’s education. • Teachers feel more efficacious, students are spending more time on task, and disciplinary referrals are decreasing. Some providers report learning what is not working as well or where they need to put more effort to ensure better outcomes. • Ways in which programs need to be modified, what students we are impacting the most. • We are learning that our programs are significantly increasing the high school graduation rate (i.e., our program graduates youth at approximately 90%, compared to general school graduation rate for our partner high school of approximately 40%), but our high school graduates are not entering or staying in college at nearly as high a rate. • I am learning that maybe we aren't doing a good enough job on academics, as far as looking at their report cards. It also shows the days missing from school. • Our students on average are maintaining 3.0 or higher GPA and have a very low school absence which is positive. While 89% of our youth report that they will obtain a 4 year college degree, 44% of students are unsure or think they probably cannot afford to attend college using financial aid, scholarships, or their family’s resources. Staff are targeting the 44% as a data point to improve this year. Staff are also part of a collaborative evaluation group and, in meetings where program staff compare data across agencies, they have learned from other agencies who are getting better results in some areas. Staff have incorporated these ideas into program improvement plans (for instance we are restructuring our tutoring services based on what we’ve learned in collaboration meetings) Several providers noted that if they are to know the long-term effects of their programs, they have a greater need for more longitudinal data on students no longer in the system or those who are no longer being tracked. • Difficulty collecting outcome data once youth no longer is receiving services. Funding does not provide means for long term follow-up. • The programs we are trying to measure only have collected data up to a point, or have not been collecting long enough.
- 27. 27 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING HOW PROVIDERS USE OUTCOME DATA The difficult social context in which program participants are living has serious implications for providers’ programs and makes thinking about outcomes more complex. • Youth are having difficulties remaining drug free. • Lack of quality education and a living wage job are the main challenges for youth and adults. When providers were asked how they were analyzing the data, most responded with answers that fell into one of four categories. Some focused on the type of software program they were using for analyzing the data (e.g., SPSS, eCornerstone, etc.); others focused on the frequency of the analysis – whether it was conducted quarterly or yearly; others focused on certain indicators, such as the success, changes or improvements that would be needed; and still others focused on who was involved in either conducting the analysis or reviewing the analysis. In these cases, the responsibility for analysis fell to a range of staff types: front line, manager or mid-level management (in one case, clients were involved in reviewing the outcome data and giving input.) At least 17 of the providers talked about using the data for improving their programs and planning, while only 6 shared that they using the data for reporting, marketing or funding purposes. While it is probably safe to say that all of the providers use this data to report back to funders and/or donors with the goal of receiving funding or renewed funding, it did not appear to be the primary focus for this particular set of respondents. Four providers shared that there were barriers to analyzing their own outcome data, though all conducted some level of analysis. Chief among these barriers were human resource limitations (in some cases, the staff responsible for analysis also carried other responsibilities); available technology failing to be conducive to the type of analysis staff wanted to conduct; or the absence of sufficient funding to purchase appropriate technology that would make data collection and analysis easier and shareable with multiple entities. Most significantly, a provider again raised the concern that lack of an agreed upon set of defined outcomes and metrics was also a barrier to data analysis. DEVELOPING OUTCOMES: PROVIDER CONCERNS Key Finding: Focus group participants felt there were enough PYD models in existence and that neither they nor the state needed to start from scratch, but felt that the model selected needed to be developmental in nature. In addition, providers felt that these impacts needed to be thought of as longitudinal and that the state and other stakeholders (the youth themselves, the continuum of staff at their agencies, parents, agency boards, funders, and the business sector) would need to be willing to support ongoing training to achieve “Agreement in minimal training for youth development Core Competencies,” and provide the type of resources to make outcomes development (and, by extension, Budgeting for Results) successful. While providers were able to reach consensus within and across focus groups, they shared a range of cautions and concerns when it came to addressing how the State should develop agreed upon outcomes.
- 28. 28 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING DEVELOPING OUTCOMES: PROVIDER CONCERNS Both groups reiterated that there were enough models out there (i.e., Search Institute, etc) that they did not need to start from scratch and they would be happy for the State to pick one model and then they could work it through. The other important thing shared by the providers which seems important to emphasize is that whatever model is selected must be developmental in nature and, as explained by one provider, “The tool must be sensitive to the intersection of all of these assets/domains.” Providers further emphasized the importance of working as a group to determine which outcomes were directly correlated to, or aligned with, their specific programming because there were programmatic differences (in terms of audiences, goals, etc.) They felt that this conversation about the desired outcomes for youth needed to include a wide range of voices – in particular, youth, line staff, and other PYD stakeholders. The focus group participants spoke of the fact that many stakeholders support outcomes and outcomes development should not be the sole burden of the youth provider. They spoke specifically of the youth themselves being a major stakeholder and there being a need to incorporate their voices into these discussions, possibly through the TeenREACH State Youth Advisory Council. Youth workers were another voice that needed to be added to the discussion so that the full continuum of the youth provision sector was included, not just those in the administrative roles. But while these responses are thoughtful and act as a testament to the dedication of these providers, it is hard to imagine the state having the current capacity or time (given the BFR implementation schedule) for such a bottom-up process for the creation and implementation of program outcomes across hundreds of programs and with thousands of communities. Focus group participants also thought it was important to think about outcomes from a longer view and that some outcomes would be more longitudinal, requiring tracking beyond the time youth spend in programs. One provider noted that “Change of behavior can occur but what about sustainability of change over time, beyond the life of our program?” Another agreed and said it would be helpful to “connect early childhood to college.” Similar concerns were raised by survey respondents. Providers were equally concerned about efficiency and streamlining, wanting programs to be seen, managed, and operated without silos: • CYS and TeenREACH are silo[ed] from each other. [There’s a] huge disconnect. • Two different state departments [are] working on youth outcomes--but extremely different. • IL DHS’ [and] DCFS’ youth programs - these entities should come together and see where here is duplication. We have been trying to do this on our own when there should be one unified youth grant. • Silo issue is huge--but [there] aren’t enough resources to deal with housing [or] placement. Provider participants identified a number of ways thinking about different programs in relationship to one another, as well as how their joint impact and role in outcomes could be helpful. For instance, in the words of one provider: “In certain IL DHS divisions’ [prevention models] that are good, [the] State should take a look at what worked internally and learn and use that.”
- 29. 29 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING DEVELOPING OUTCOMES: PROVIDER CONCERNS Other recommendations to explore what is happening with shared outcomes in other program areas included: • Looking at “Early childhood programs could be helpful in tracking [outcomes] over time. • Designing support groups with smaller organizations in different areas and supporting collaboration. • [The State] should look at NWEA-MAP (long ranging-elementary to high school educational tool) [that] has benchmarks, [measures] growth over time, [and] breaks each domain down. Providers are adamant that training has to be a key component for success. They also think there has to be ongoing sustained training as BFR becomes more integrated as a state practice, not just upon initial roll-out. Particularly, they supported the need for outcomes training “at the individual vendor level” not just for capacity purposes, but to achieve “Agreement in minimal training for youth development Core Competencies.” Ongoing support for BFR means joining policy and practice in order to create sustained buy in among practitioners and prevent the all-too familiar practice of merely adding to a nonprofit provider’s already strained capacity. Participants believe that in order for this to work there will have to be buy-in at all levels within the provider agency (including youth, parents, line staff, administrative staff, and board,) that training and on-going support would have to exist, and that tasks associated with outcomes measurement could not be added randomly to someone’s current job. For providers, there would need to be internal commitment at the provider agency, aligned with external support from the state, other funders and even the business sector. Ongoing support for BFR means joining policy and practice in order to create sustained buy in among practitioners and prevent the all-too familiar practice of merely adding to a nonprofit provider’s already strained capacity. Quality control and assurance were also important components to the discussion of outcomes data. Providers felt that it was not enough that the data was being entered, but that systems would be in place to assure quality in data entry and analysis. They particularly focused on using a model of peer to peer sharing and review in order to assure not just the quality of data but the accountability of it – pointing to a sense of their own strength as a field and, perhaps more cynically, of their distrust of the State’s capacity to collect (and leverage) ‘clean’ data. They raised such questions as “What about quality control? Who is checking data quality, perhaps establishing a peer review process?” Providers also suggested that establishing a normal quality control/ compliance process which reflects accountability would be the first step; the next step would be quality assurance which would shape program quality and implementation. Again, one participant suggested that incorporating peer to peer site visits might be a way to develop quality assurance across programs. There was also the mention of cohort-based support as providers attempted to tackle program evaluation. They outlined that having an hour-long site visit to review data did not go deep enough. For instance, if one site was ranking low on a specific metric perhaps that site could be paired with another provider agency that was doing well on that specific metric in order to create an additional level of peer, or cohort, support. This was felt to be an innovative way to get a bigger ‘bang for the buck’ (or, return on investment) for the youth. Providers held up the Maternal Infant Early Childhood Home Visiting Program as an example of a cohort-based support model the State might explore.
- 30. 30 PROVIDERS ON PYDMODELS, OUTCOMES & CAPACITY BUILDING DEVELOPING OUTCOMES: PROVIDER CONCERNS For focus group participants a system for data collection would share the following characteristics: streamlined; single depository; and come with a ‘friendly’ tracking system. Perceived lack of cohesion or alignment in technology is another area of concern for survey and focus group participants (see How Providers Collect Data). For focus group participants a system for data collection would share the following characteristics: streamlined; single depository; and come with a ‘friendly’ tracking system. In some instances providers and the State have partnered by using common data platforms but the results have been mixed. “The ETO Database; there was intentionality but it still does not work that way,” said one provider. Another added, “ECornerstone was introduced 8 years ago; the glitches took 2-3 years to iron out. [We’ve] only been using it for about 5 years.” Another strong recommendation from providers was for the state to “Match technology to outcomes to make sure we get useful reports,” indicating a perceived deficiency and pointing to a need for outcomes to feed a loop of constant program improvement, thus improving baseline results for the state. In other words, outcomes shouldn’t just be for the state alone (in an upward direction) but should also have an impact ‘on the ground.’
- 31. 31 PROVIDERS ON BUDGETINGFOR RESULTS & CAPACITY BULDING AWARENESS Key Finding: While descriptions varied, providers were generally aware of BFR as a state-sponsored strategy. These responses suggest that while there is a continued need to make sure all providers understand BFR and how it works, there is enough of a basic understanding of BFR to build upon. These responses further suggest that providers support increased accountability and want to use outcomes strategically in their programming, but all will need assistance in doing so. This might be a crucial role for a partnership between the state and the philanthropic community. In the survey, 18 providers expressed a baseline understanding of BFR as a method to tie resource allocations to measureable outcomes: • [BFR is] tying money to outcomes. • Governor Quinn signed into law Budgeting For Results (BFR), an historic spending reform act requiring the state of Illinois to institute a results-based budgeting process that will nd the automatic funding of programs. By requiring the State to live within its means and focus on performance, BFR will transform the way that state officials, legislators – and the public – prioritize, think about and implement the State’s budget. Going forward, the State will fund only those programs that can demonstrate effectiveness and help the State achieve its stated outcomes and goals. • A strategic alternative to incremental budgeting, which resources are allocated based on how effectively programs or service achieves established goals and objectives. • From my understanding Budgeting for Results have something to do with the budget and having available dollars to the most significant programs and activities. One provider effectively connected the dots between funding and meeting program goals, objectives, and outcomes: “Have not had any formal training on BFR. Although we do our very best to maximize every dollar using volunteers, in kind contributions, it's hard to budget for results when the payments you rely on are severely delayed. It's hard to expect the maximum results under these circumstances when every program that yields the best results relies on a steady cash flow.” Few providers (2) had no knowledge of BFR. During the focus groups, when we prompted providers to think about Budgeting for Results (BFR) in a big picture, not only were most providers aware of BFR, their awareness emerged in two different (yet related) themes: Efficiency vs. Collaboration. Springfield providers concentrated on outcomes development in order to prevent duplicating services or processes that were already in place; they really wanted to try and streamline the processes related to outcomes and service provision, as well as incorporate economic development and connecting youth with employment in the discussion of outcomes. The Chicago-based group was much more focused on collaborating with each other to improve their practice related to outcomes; they were also concerned that smaller groups had not been included in the focus groups because they felt true innovation comes from smaller organizations; they were also advocates for long-terms capacity supports be put in place, and extremely concerned that they would be held accountable for academic outcomes that even school systems are unable to meet.
- 32. 32 PROVIDERS ON BUDGETINGFOR RESULTS & CAPACITY BUILDING PROGRAM IMPLICATIONS OF BFR Key Finding: Providers and the State share similar expectations of BFR to be meaningful to their programs, indicating the possibility of shared vision and goal-setting as implementation continues. When asked, provider responses (20 in total) about the program implications of BFR for their organizations can be categorized in one of four ways: • Demonstrating Effectiveness/Accountability • Strategic Prioritization • Technology Improvement • Uncertain or None Most providers (10) said BFR would require them to Demonstrate Effectiveness/Accountability (emphasis added to show shared values between the State and providers): • We must be able to demonstrate successful program outcomes to secure and maintain funding to services. • It is good to show taxpayers that their money is being used for important projects as I have seen a lot of waste in the past. Funded agencies need to be accountable for how they use money. All non profits should have an outcomes measurement process. If not, they should not be funded (or provide[d] funds to create one). Four providers described implications that related to Strategic Prioritization: • Sometimes leadership uses it to reduce in areas that are not a priority or preference. If you use BFR there will be less of that. • BFR for our organization would require a thorough analysis of each department's budget in comparison to the produced outcomes and then an agency-wide budget -outcome analysis to find out which programs and services are most cost-effective and which are least effective. We would then need to use that data to decide which programs and/or services to continue, to expand, or to discontinue. • To prioritize and focus on quality service that will bring about long-term results. Three providers shared that the implications for their organization were related to Technology Improvements. • More resources to build up our technology capacity and capability. • The results are based on the data we collect. So we have to have very good data collection methods. • All non profits should have an outcomes measurement process. If not, they should not be funded (or provide[d] funds to create one). For six providers the implications were uncertain or unknown and only 2 of those said they were unfamiliar with BFR or had not been educated about BFR. One provider curiously responded that BFR had no implications for their organization.
- 33. 33 PROVIDERS ON BUDGETINGFOR RESULTS & CAPACITY BUILDING CAPACITY BUILDING NEEDED BY PROVIDERS Key Finding: Not surprisingly, in order to meet the requirements of Budgeting for Results successfully, what providers need most are resources – human capital, training, and funding – in order to build (or expand) their ability to be effective. In the survey, providers ranked their needs as follows: • Resources (people, funding or technology) • Training (general and specific) • Access to Institutional Data • Agreed upon Outcomes/Metrics Resources was the most often identified support with 5 providers identifying personnel supports, 4 identifying funding support for any number of needs and 7 identifying technology needs (emphasis added): • An improvement in our obtaining professionals and programs to support our life skills i.e. anti-drug/alcohol education for the students. • Our organization would require funding for and/or an assigned evaluator who has experience in strategic financial planning and operations to help determine whether the data we are collecting is accurately measuring what we desire and, if it isn't what data we should collect, help us develop the tools to collect the data if we do not already have them, and assist the PQI department and department leaders in analyzing the data that is collected. • We find that ETO is a good tool for our collecting data to evaluate our outcome in programs. However, we are only using ETO in certain programs, we need further assistance, technology and financial, to implement it in other programs in the agency. 6 providers mentioned the need for training as a necessary support if they were to be successful in meeting the outcomes of BFR. Such training would help staff understand BFR or would be specific to technology or outcomes. • Staff training to understand the concept in order to provide quality service that delivers the intended results. • Training for our admin team and understanding of the process. • Training on managing data. • Again, I can only imagine having the software and the education to implement BFR. 2 providers mentioned the need for agreed upon outcomes and metrics in order for their organization to successfully implement BFR. This is a recurring theme through the survey data. • Hoping that BFR will be in line with current research/thinking around outcomes from existing sources and not created from scratch as that could lead to a separate outcomes process for organizations and create inefficiencies. For example, there is a lot of work being done around social emotional learning - these concepts should be used in BFR because most organizations that have an outcomes process will also use these concepts. • As previously described and additionally training, technical assistance in constructing appropriate outcomes, metrics, and sufficient technology. 2 providers mentioned access to institutional data or more cooperation from the parents, schools and CPS to get data and as having organizational implications.
- 34. 34 PROVIDERS ON BUDGETINGFOR RESULTS & CAPACITY BUILDING ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS Key Finding: Though the focus group size was small, providers were positively engaged around the topic of outcomes. However, they were extremely vocal in their concerns about BFR and the state’s capacity to implement such a complex process. The providers remain concerned that BFR will be another “flavor of the month” that will fade with time and be used as mechanism to deplete already lean resources to support successful development of youth. While sharing that they were grateful for this pilot, and to hear the State’s admission they do not have the expertise in this area and are willing to partner with them, providers are still concerned about the State’s limitations in understanding outcomes. (See also IL DHS Staff Discussion) • You find only one person at the state level to work with who you talk with about outcomes, etc. • Everybody at the state level needs to know more than we do, more than the agencies. • Appreciate [the] state admitting it does not have expertise and will partner with the providers and others on this. From the providers’ perspective (a point of view that would likely differ from the State’s in rather significant ways) BFR would require an increase in funding to their programs to be successful in response to BR. They have serious concerns and distrust about the State’s commitment to such funding, given that Illinois has already demonstrated an inability to meet its financial obligations. • Especially given that the base amount to us is $75 thousand versus federal programs which get $250 thousand. • [We are] expected to input data and [it’s] not reflected in budget. Connect tasks to funding. • State must make in-roads to punctuate federal dollars. In early childhood, federal dollars are creating hubs regionally and nationally. Nothing like this [is] happening in PYD at the national level. This ices out some of the smaller orgs who can be very innovative. [It] must intentionally include smaller and larger organizations. • For profit businesses need to be brought to the table, they have a stake in this too. • We must monitor pennies. State [funding] is not worth the paper it is written on. My board is weary, our partners are weary. [The] State can change its mind. Grants do not cover interest when we've had to take out loans to keep things going in between when monies come. • The state cancelled some of our [funding and] rolled our program into another area and then came back and asked us for a report when the program was no longer in our hands. Repeating a refrain heard often, providers voiced concern that BFR was the “new flavor of the month” and that the State changes ideas, concepts, and plans quickly (perhaps too quickly). They were skeptical to the idea that BFR would be handled differently. One provider asked, “How much of this is short-term as the State shifts to the health consortium as its model?” Another observed, “TeenREACH was zeroed out, [then] brought back. Things change too much.”
- 35. 35 PROVIDERS ON BUDGETINGFOR RESULTS & CAPACITY BUILDING ILLINOIS IMPLEMENTATION CAPACITY AND PROVIDER CONCERNS Of more concern is the distrust some providers have of the State’s motivations surrounding BFR. Some feel that BFR is a Trojan horse to further hollow out an already lean service delivery system. A provider said, “Theoretically [I have] no problem, but [I’m] concerned that it will be used as a way to hire the lowest quality, that it’s not being tied to quality or ongoing improvement.” Another added, “Money must come along with this or we are disinvesting in youth.” Another preferred to think about BFR’s potential to prompt cooperation and collaboration, rather than competition: “BFR would be great if not connected to the budget. We are all on the same team; let's elevate the reflection, talk with one another, and innovate when tied to money. Money creates competition.” Acknowledging the small size of both the survey and focus group samples, we want to encourage the State not to ignore these concerns about gaps in capacity. The State’s lack of staffing, planning and resource capacity, as well as its uneven practices and processes, is well documented in annual Inspector General audit reports, public testimony at legislative hearings and committees, as well as external evaluations (such as the 2010 Pew Center on the States’ Grading the States report on Illinois government performance.) Facing, and overcoming, distrust between strategic partners will not be easy, and would probably require some additional change management and internal culture work on the part of IL DHS (as well as some rather significant changes to the State’s financial processes.) However, greater intentional and more transparent communication about the State’s commitment to BFR, as well as to a consistent evidence-informed methodology for implementation, could balance this skepticism.
- 36. 36 STAKEHOLDER RECOMMENDATIONS FOR GOVERNMENT CONDUCTEXTERNAL FIELD REVIEW There is a need to conduct a more exhaustive research into and compilation of the major activities in Chicago and Illinois, particularly public-private partnerships, around Positive Youth Development Outcomes and Indicators in order inform the State’s goals. But while there is evidently a City level project to assess PYD outcomes, as well as a long-established State level project with the Afterschool Partnership, there could be a more coordinated effort to compile lessons and best practices, as well as to avoid duplication and build synergy. CONDUCT INTERNAL AUDIT Assess their current youth programming offerings across units to eliminate duplication and redundancy. Review programs within their own divisions where outcomes and indicators have been successfully implemented and use those as models to guide those developed in the Youth Development division. Invest in program evaluation. CREATE TRAINING PROTOCOL The State should consider supporting sustained and ongoing training as BFR becomes more integrated as a state practice, not just upon initial roll-out. Particularly, they should consider supporting the need for training “at the individual vendor level” not just for capacity purposes but to achieve “Agreement in minimal training for youth development Core Competencies.” Identify and partner with organizations to provide ongoing capacity building support. PARTICIPATE IN STAKEHOLDER ENGAGEMENT Incorporate the voices of youth (i.e., TeenREACH State Advisory Council), organizations that support youth (i.e., Illinois Collaboration on Youth-ICOY) and other key stakeholders in the next phase of the process. Identify opportunities for partnership with philanthropic and corporate stakeholders in evidence-informed outcomes development and/or capacity development to strengthen the PYD delivery system. BASE OUTCOMES ON EVIDENCE-INFORMED PRACTICES AND RESEARCH Identify an evidence-informed and best practice model for PYD programs and outcomes (e.g., from the Search Institute, etc.), particularly as the providers are familiar with the range of models available. Leverage the expertise of foundations and practitioners in this process. Complete the process of obtaining consensus on the broad PYD outcome domains, move to work with providers on specific outcomes within those domains and finalize the measures that would be utilized across Youth Provider Services; subsequently, the State could take the additional step of determining which additional Prevention Outcomes would be needed.
- 37. 37 STAKEHOLDER RECOMMENDATIONS FOR GOVERNMENT Surveyproviders on which outcomes providers collect internally (which may be different from State requirements) to reduce duplication of efforts, establish shared benchmarks, and provide an opportunity for all providers to benefit from the experience of peers. Survey providers on which measures they use to determine outcomes to reduce duplication of efforts and provide an opportunity for all providers to benefit from the experience of peers. OFFER BFR INFORMATIONAL SESSIONS Consider offering an informational workshop and materials on BFR so all providers (particularly smaller organizations) have a full understanding and opportunity to ask questions and get clarification on BFR. Identify and partner with strategic intermediary organizations to achieve outreach goals. STANDARDIZE AND MODERNIZE TECHNOLOGY Assess current technology (or methods) used by Youth Providers to see whether there is a ‘match’ between State and provider tools and the outcomes and indicators they’re required to provide. Survey providers about the database systems (or methods) they are using to house and analyze their own data. Reduce duplication of efforts by streamlining data submission processes and providing opportunities for peer sharing. Conduct a needs assessment and cost analysis to determine which upgrades or innovations would be required to bring agencies on technological par with one another in order to track and analyze agreed upon outcome data. FOR PHILANTHROPY Consider the types of capacity building needed by grantees (i.e., human capital, program development and measurement) to build and expand their ability to effectively meet the rigorous requirements of Budgeting for Results. Consider supporting ongoing training and technical assistance for smaller organizations as BFR becomes more integrated as a state practice, not just upon initial roll-out. In particular, consider supporting the need for training “at the individual vendor level” not just for capacity purposes but to achieve “Agreement in minimal training for youth development Core Competencies.” Consider ways to help align philanthropic best practices in PYD with State outcomes, in order to develop a unified set of outcomes and measures for greater grant making strategic impact (as well as to avoid unnecessary conflicts, duplications or redundancy experienced by grantees of both foundations and the State.)
- 38. 38 STAKEHOLDER RECOMMENDATIONS FOR PROVIDERS LeverageBFR as an opportunity to take stock of current resource barriers (i.e., technological, human capital, funding, etc.) to collection and analysis of outcome data; be prepared to outline those limitations in detailed and concrete ways; and plan strategically for future capacity growth. Offer more details about the types of outcome data they currently collect; how they analyze said data; and which additional data they would need to collect in order to meet program goals. Use BFR as a starting place within their own organizations (across all levels) to talk more about BFR, its implications for their organization, and ways to prepare the organization for this new strategic direction in measuring impact. WHERE DO WE GO FROM HERE? Commendably, the State has begun a complex and long-term performance management process that should help strengthen the strategic partnership between government, nonprofit providers, and philanthropy – with the public as the ultimate beneficiaries. We recognize the limitations of this initial pilot study. However, as this process moves forward, we hope the State, along with strategic partners, can adopt a scaled-up process for involving its constituencies, assessing the service delivery landscape, gathering data, and creating evidence-informed outcomes for the best uses of public resources.
- 39. 39 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES HISTORY AND DEFINITIONS Positive Youth Development (PYD) is a framework that views young people as resources to be developed rather than as problems or potential risks to be mitigated (Lerner, Almerigi, Theokas, & Lerner, 2005). PYD approaches did not emerge from thin air, but are informed by the experiences of youth workers and practitioners over a number of years (Benson, et al. 2006). The PYD framework began to coalesce in the 1990’s through the work of scholars such as Richard Lerner, Jacquelynne Eccles, Stephen F. Hamilton, and others, and intermediaries such as the Search Institute and Karen Pittman’s Forum for Youth Investment. Programs grounded in PYD principles work to foster opportunities for young people to develop their competencies, including social, emotional, intellectual and ethical skills, as well as a sense of connection and contribution to their schools and communities. PYD developed within the context of, and as an alternative to, prevention-based programs that identified young people based upon particular areas of risk (i.e. pregnancy, delinquency, drug use, etc.), then sought to alleviate potential harm (Benson, et al., 2006, Lerner et al., 2005b; Mahoney, Larson, Eccles, & Lord, 2005; Pittman, Irby, Tolman, Yohalem, & Ferber, 2003). In the early 1990’s, researcher and advocate Karen Pittman coined an expression that has since been used frequently to characterize PYD: “Problem free is not fully prepared (Pittman, et al., 2003). It is not sufficient for young people to simply avoid risky behaviors such as gang involvement, pregnancy or drug use. They must also develop the competencies necessary to thrive.” Critical functions of PYD programs include providing supportive structures for young people, and facilitating opportunities for young people to develop skills and competencies. (Pittman, et al., 2003). PYD has been alternately understood as a philosophy or set of principles, a process, and a programmatic strategy (Benson, et al., 2006; Hamilton, Hamilton, & Pittman, 2004; Pittman, et al., 2003). As a philosophy, PYD may inform a diversity of youth programs, with the common denominator of promoting holistic development for young people. As a process, PYD refers to individual young peoples’ developmental trajectories within a PYD framework. A PYD program would be one that is explicitly informed by a PYD philosophy, or seeks to catalyze PYD processes. Most PYD programs have been implemented in an after-school or out of school time (OST) context, although elements of PYD may also be implemented in school settings (Ziegler, 2004). PYD & DEVELOPMENTAL SYSTEMS THEORY Informed by the ecological model of human development (Brofenbrenner, 1979), one fundamental component of PYD principles are their grounding in developmental systems theory, which views the various layers of influence in young people’s lives, such as family, community, schools and the broader society/culture as interconnected (Benson, et al., 2006; Hamilton, et al., 2004; Lerner et al., 2005b; Mahoney et al., 2005; Pittman, et al., 2003). Young people’s families and communities exert a direct and powerful influence on their development, yet this relationship is not one-way. Relationships between young people and these systems are bidirectional (Lerner et al., 2005b) meaning that young people influence these systems just as they’re influenced by them, offering a great opportunity to develop young people’s contribution to the world around them. PYD practitioners should operate with some level of understanding of the role these various contexts play in young people’s lives. Indeed, the most effective programs may be those that prioritize the “integration of family, school and community efforts.” (Mahoney et al., 2005). Pittman, et al. (2003) have worked to elaborate the various components of community, defining communities not just as physical spaces, but as “people coming together, working toward common goals” in such arenas as civic and political life, education, health and human services, religious institutions, social spaces and others. PYD practitioners should seek whenever possible to directly involve family and community members in PYD processes, encourage young people’s positive contribution to their communities, and leverage community assets to support positive development for young people.
- 40. 40 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES PYD AND PREVENTION Although PYD and prevention frameworks are sometimes viewed as oppositional, due to the contrast between strengths-based and deficit-based approaches, in practice, most PYD researchers, practitioners and advocates understand them as complementary and interrelated (ACT for Youth; Pittman, et al., 2002). Increasingly, more prevention programs have begun to incorporate aspects of PYD, and multiple studies have indicated that PYD programs contribute to decreased negative outcomes and involvement in risky behavior. Additionally, prevention workers and researchers have begun to acknowledge that isolating particular risk factors or problem behaviors, without addressing young people’s broader environmental context, does not lead to effective prevention (Catalano, et al., 2002). A fundamental principle of PYD is universality, the understanding that all young people require opportunities to develop, not just those youth categorized as belonging to a particular group deemed to be “at risk.” Although an extreme understanding of universality might see any effort that targets a group of young people with specific attributes – i.e., gang-affiliated young people, or high school-aged mothers -- as antithetical to PYD, most PYD researchers and advocates do not ascribe to this position. Rather, universality is a principle applied in context, with the understanding that all young people have contextually-specific developmental needs (Hamilton, et al., 2004). Amongst these, some young people may need to be exposed to explicitly prevention-oriented curricula, alongside more positive opportunities, in order to thrive. Ultimately, most PYD advocates view tensions between prevention and PYD perspectives as arising not from any fundamental ideological difference, but rather from siloed federal funding streams directed toward particular prevention priorities. PYD AND OST PROGRAMS The rise of PYD has coincided with the growth of OST (out of school time), or after school programs. Communities and policy makers have increasingly recognized a need to address what young people do when they are not in school, and to provide all young people with productive activities. This has resulted in the establishment of intermediary institutions like the Afterschool Alliance, and policies like the Illinois After-school and Youth Development Program Act, which aim to expand access to after school programming for all young people, and improve program quality. Although closely related, PYD and OST programs should not be mistaken as synonymous. Many of the institutions supporting the OST sector have embraced PYD frameworks, and OST programs have shown themselves to be fruitful contexts for PYD (Benson, et al., 2006; Catalano, Berglund, Ryan, Lonczak, & Hawkins, 1998; Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002; Hamilton, et al., 2004; Hansen, Larson, & Dworkin, 2003; Lerner et al., 2005b; Mahoney et al., 2005; Merry, 2000; Pittman, et al., 2003). That said, not all after school programs support PYD. In addition to prevention, many current programs maintain a primary mission of closing the academic achievement gap through such focuses as extended learning time (ELT), STEM (science, technology, engineering, math) education, and college and career readiness. Although these content areas may certainly be shaped by PYD philosophies (Chi, Snow, Goldstein, Lee & Chung, 2010), they may not prioritize human development as heavily as academic achievement and workforce development.
- 41. 41 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES PYD AND YOUTH ENGAGEMENT PYD frameworks have always viewed youth as active participants, rather than passive recipients of services. Building upon this emphasis, current work in PYD has begun to more heavily emphasize the importance of youth leadership, engagement and contribution. Pittman, et al. (2003) expanded Pittman’s seminal catch phrase “problem free is not fully prepared” to “fully prepared is not fully engaged.” An emphasis on engagement means giving youth a voice not only in programmatic decision-making, design and implementation, but also providing opportunities for young people to become change makers in their communities through participation in such activities as service learning projects, community organizing and civic engagement, peer health outreach, peer tutoring, youth media production, and others. For agencies moving toward incorporating youth leadership into their organizational structure, Zeldin and Collura (2010) have prepared a useful guidebook on facilitating effective youth-adult partnerships, with concrete tools and activities for successful change management for becoming a more youth-led organization. PYD AND EXPLICIT VS. EMBEDDED LEARNING Another debate within the PYD field has taken place around the tension between explicit and embedded curriculum in youth programs. The Collaborative for Academic, Social and Emotional Learning has argued that programs should be S.A.F.E. – sequenced, active, focused, and explicit (Payton, et al., 2008). This last priority has emphasized that learning goals are most likely to be met when they are made explicit to the young people who participate in a program. Others have argued that the development of specific skills and competencies may be more subtly embedded within subject matter that is of immediate interest to young people (Deschenes, McDonald, & McLaughlin, 2004). To offer an example, an explicit STEM (science, technology, engineering, and math) program might advertise itself as a robotics club, whereas an example of embedded STEM content would be a hip hop arts program that incorporates physics lessons within the study of break dancing. Although some researchers have openly disagreed about whether explicit or embedded approaches are preferable, it is likely that most practitioners use a blend of approaches based upon the needs of their young people. BEYOND PYD – RACE, CLASS & CULTURE Some youth development practitioners and scholars have interrogated and expanded PYD to address race, culture and class-specific issues. Villaruel, Montero-Sieburth, Dunbar & Outley (2005) have argued that the individualistic developmental outcomes prioritized by traditional PYD ignore the cultural and community supports that are particularly critical for young people of color, calling for more research to better understand culturally-specific understandings of development. Ginwright and Camarotta (2002) have noted that while PYD has made a critical move away from deficit-based models that pathologize poor communities of color, PYD does not sufficiently acknowledge how racial and economic inequity disenfranchise young people. They have called for a model of “Social Justice Youth Development” wherein young people develop their analysis of systemic inequality and their community organizing skills alongside other developmental assets.
- 42. 42 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES DEVELOPING OUTCOMES FOR AND EVALUATING PYD Although a number of empirical studies have been, or are being, conducted to demonstrate that PYD- oriented programs lead to positive outcomes for young people (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2002; Lerner, et al., 2005a), the promotion of positive outcomes is still see by many public and private funders as more difficult to measure than the avoidance of behaviors deemed “antisocial” (Lerner, et al., 2005a). The evaluation of PYD approaches becomes even more challenging when initiatives seek to gauge the impact of their programming on the broader community, as well as upon the young people who are direct participants. Many have argued that PYD outcomes cannot be authoritatively evaluated without the application of rigorous experimental or quasi-experimental evaluation design, involving treatment and control groups and random sampling (Catalano, et al., 1998; Catalano, et al., 2002; Merry, 2000). However, such research methods are often beyond the capacity of most community-based youth organizations and therefore researchers have recommended that organizations focus on evaluating program quality and the point of service, rather than attempting to track the developmental outcomes of participants in their programs. This recommendation is made with the understanding that research has already proven that high-quality PYD programming results in positive outcomes. Major evaluative research studies that organizations may call upon in support of PYD programming include the 4-H Study of Positive Youth Development (Lerner, et al., 2005a) and Catalano, et al’s (1998) findings on evaluations of positive youth development programs. The 4-H project is a longitudinal study that has demonstrated that young people who participate in PYD programs show growth in the “five C’s.” The Catalano study was funded by the U.S. Department of Health and Human Services, and demonstrated the effectiveness of PYD approaches in promoting positive developmental outcomes through case studies of 25 high quality programs across the country incorporating rigorous empirical evaluations. A number of major models exist which present domains in which young people develop skills and competencies (which then contribute to further development). These asset frameworks were created by youth development academics and intermediary institutions, and are informed by research and practice. The language of “assets” originates in the work of Peter Benson and the Search Institute (Benson, 1997). An asset may be understood simultaneously as an input, a driver and an outcome of PYD-oriented activities. Young people already possess assets and/or potential assets upon which to build as they develop (inputs). These assets are further developed through young people’s participation in PYD programming (outcomes), while also driving developmental processes (drivers) (Hamilton, et al., 2004). Because PYD understands development as a process, not a goal, the methods and outcomes of PYD are one and the same. For the purpose of this report, recommended PYD outcomes may be better understood as domains for ongoing growth. These models may be used as possible checklists for gauging whether a program has laid the appropriate groundwork to promote PYD.
- 43. 43 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES MODEL 1: THE SEARCH INSTITUTE, a youth development intermediary institution with a lengthy track record of providing research, technical assistance and training to youth development organizations, developed a comprehensive list of 40 developmental assets that are understood as critical for youth to thrive. These were divided into external assets – those provided by programs, families and communities, and internal assets – those possessed and further developed by youth themselves (40 Developmental Assets, 1997, 2007). Within each domain, the assets were further categorized within one of four sub domains. External assets included: support; empowerment; constructive use of time, and boundaries and expectations. Internal assets were: commitment to learning; positive values; social competencies and positive identity. To provide an example of how assets were categorized within these sub domains, the external sub domain of “support” included a caring neighborhood and school climate, and positive family communication, among others. The Search Institute has had success in motivating community-wide initiatives around these 40 assets (Hamilton, et al. 2004). Their website and other materials provide an array of resources for putting youth development principles into practice. MODEL 2: THE COMMITTEE ON COMMUNITY-LEVEL PROGRAMS FOR YOUTH, an initiative of the National Academy of Sciences, has also developed a list of developmental assets. This list shared many similarities with the Search Institute’s, and divided assets into one of four domains: Physical Development; Intellectual Development; Social Development, and Psychological and Emotional Development. Examples of physical development included good health habits and risk management; examples of intellectual development included school success and knowledge of essential vocational skills; examples of social development included a sense of connectedness with parents and peers; and examples of psychological and emotional development (the most extensive category) included emotional self-regulation, conflict resolution and coping skills. The Committee on Community Level Programs for Youth further elaborated on this asset framework with a list of eight attributes necessary for community settings seeking to promote the development of these assets, making their framework especially useful for organizations engaged in program development and evaluation. These attributes included: physical and psychological safety; appropriate structure; supportive relationships; opportunities to belong; positive social norms; support for efficacy and mattering; opportunities for skill building; and integration of family, school and community efforts (National Research Council, 2002). MODEL 3: THE “FIVE C’S,” originally developed by Karen Pittman, have provided a simple and memorable method for identifying PYD outcomes. The “C’s” are competence, character, connections, confidence and contribution (Hamilton, et al., 2004, Lerner, et al., 2005a). In some formulations, Contribution has been replaced with “caring.” Both “contribution” and “caring” have referred to a young person’s empathy and commitment to “give back” to others. Competence refers to the various skills needed to achieve one’s goals and adapt to diverse contexts. Coupled with competence, Confidence is the self-assurance required to pursue one’s objectives. Character is a young person’s commitment to moral and ethical principles. Connections are healthy social relationships with adults and peers. While the Search Institute and the Committee on Community Level Programs for Youth’s frameworks are useful for providing a more detailed breakdown of the assets needed for young people to thrive, they may be unwieldy for quickly communicating the goals of PYD to communities, funders, or other stakeholders.
- 44. 44 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES MODEL 4: THE NATIONAL COLLABORATION FOR YOUTH (NCY), a coalition of national agencies committed to PYD, has worked with the Forum for Youth Investment (FYI) through their “Ready by 21” initiative to develop a set of outcomes for PYD programs to measure. These were categorized within five developmental domains – thriving, connecting, leading, learning and working. Each domain contains three to four outcomes. For instance, outcomes within the “leading” domain included community connectedness, social responsibility, and leadership development. A major advantage of the NCY/FYI evaluation resource is that each outcome was illustrated by specific indicators of progress, as well as suggestions of concrete measurement tools culled from NCY’s members and other youth organizations, which organizations may utilize to measure their achievement of these indicators (The Forum for Youth Investment with the National Collaboration for Youth Research Group, 2012). Although evaluations that are based in youth outcomes, rather than program quality, and that are supported by empirical evidence, are challenging to implement within a PYD framework, there are nonetheless tools available to help youth organizations track their outcomes. MODEL 5: YOUTH-LED PARTICIPATORY EVALUATION is one of the most progressive evaluation frameworks within the youth development field is. In a participatory evaluation process, young people identify their own goals and metrics for achieving them, a process that serves the dual purpose of further developing their skills, competencies and engagement in programming, while simultaneously evaluating the success of the program in which they are participants. After reviewing the literature to identify youth-oriented evaluation models, and finding none, Kim Sabo-Flores (2008) elaborated the youth participatory evaluation approach. Sabo-Flores detailed concrete processes youth development practitioners may use to facilitate participatory evaluation. Participatory evaluation metrics are necessarily open-ended at the onset, which may make them unsuitable for organizations whose funders expect concrete metrics in advance of funding. Yet elements of participatory evaluation may still be implemented in specific youth-led campaigns and projects within broader organizations that practice more traditional forms of evaluation. EXAMPLES OF PYD ORGANIZATIONS The construct of Positive Youth Development (PYD) emerged from the work of academics and youth development intermediary organizations (i.e., Karen Pittman’s Forum for Youth Investment), informed by the work of grassroots youth workers. Some youth organizations have since been influenced to explicitly reference PYD principles in their outreach materials, reports and/or evaluations, i.e., more actively self- identify as PYD organizations. Examples include the STEM Education organization Project Exploration, the 4-H Youth Development Organization (a national leader in PYD evaluation efforts) and the Boys and Girls Club of America. Currently, the Chicago antiviolence organization BUILD is working with Stacey S. Horn from University of Illinois at Chicago, with support from the Polk Brothers Foundation, to comprehensively implement PYD. It is important to note that PYD does not necessarily mandate a particular approach to programming, but is rather a set of asset-based principles that may be applied in a range of different programs. All of the programs listed above incorporate PYD philosophies and values (all have some level of focus on holistic development, i.e. promoting assets and working across two or more aspects of the developmental system such as school/family/ community), and offer a decent mix of size/constituency/focus area. As you recall in Catalano, et al’s (1998) seminal evaluation of the positive impact of PYD approaches, organizations were categorized as PYD based upon whether their work contributed to young people’s developmental assets, and the extent to which they functioned across multiple contexts (i.e., family/school/community), not whether they self-identified as PYD organizations. This study included a number of entities whose primary identifications were as prevention or mentoring organizations (although organizations that targeted only specific “at risk” groups were not eligible for inclusion).
- 45. 45 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES REFERENCES 40 Developmental Assets for Adolescents. (1997, 2007). Search Institute. Retrieved from http://www.search-institute.org/content/40-developmental-assets-adolescents-ages-12-18. ACT for Youth Upstate Center of Excellence. Positive Youth Development Resource Manual. Ithaca, NY: Cornell University Family Life Development Center. Benson, P. L. (1997). All kids are our kids: What communities must do to raise caring and responsible children and adolescents. San Francisco: Jossey-Bass. Benson, P. L., Scales, P. C., Hamilton, S. F., Semsa Jr., A, Hong, K. L., & Roehlkepartain, E. C. (2006). Positive youth development so far: Core hypotheses and their implications for policy and practice. Minneapolis, MN: Search Institute. Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press. Ginwright, S. & Cammarota, J. (2002). New Terrain in Youth Development: The Promise of a Social Justice Approach. Social Justice, 29, 82-95. Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H.S., & Hawkins, J.D. (1998). Positive Youth Development in the United States: Research Findings on Evaluations of Positive Youth Development Programs. Report prepared for U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation, and the National Institute for Child Health and Human Development. (Available at: http://aspe.os.dhhs.gov/hsp/positiveyouthdev99/) Catalano, R.F., Berglund, M.L., Ryan, J.A.M., Lonczak, H.S., & Hawkins, J.D. (2002). Positive Youth Development in the United States: Research Findings on Evaluations of Positive Youth Development Programs. The ANNALS of the American Academy of Political and Social Science, 591, 98-124). Chi, B., Snow, J. Z., Goldstein, D., Lee, S., & Chung, J. (2010). Project exploration 10-year retrospective program evaluation: Summative report. Berkeley, CA: The Center for Research Evaluation & Assessment, Lawrence Hall of Science. Deschenes, S., McDonald, M., & McLaughlin, M. Youth Organizations: From Principles to Practice. The Youth Development Handbook. S.F. Hamilton & M.A. Hamilton (Ed.). Thousand Oaks, CA: Sage. The Forum for Youth Investment with the National Collaboration for Youth Research Group. (2012). A Shared vision for youth: Common Outcomes and Indicators. Hamilton, S.F., Hamilton, M.A., & Pittman, K. Principles for Youth Development. The Youth Development Handbook. S.F. Hamilton & M.A. Hamilton (Ed.). Thousand Oaks, CA: Sage. Hansen, D. M., Larson, R. W., & Dworkin, J. B. (2003). What Adolescents Learn in Organized Youth Activities: A Survey of Self-Reported Developmental Experiences. Journal of Research on Adolescence, 13, 25-55.
- 46. 46 APPENDIX A –POSITIVE YOUTH DEVELOPMENT LITERATURE REVIEW AND REFERENCES Lerner, R.M., Lerner, J.V., Almerigi, J.B., Theokas, C., Phelps, E., Gestsdottir, S., …von Eye, A. Positive Youth Development, Participation in Community Youth Development Programs, and Community Contributions of Fifth-Grade Adolescents: Findings from the First Wave of the 4-H Study of Positive Youth Development. Journal of Early Adolescence, 25, 17-71. Lerner, R.M., Almerigi, J.B., Theokas, C., & Lerner, J.V. Positive Youth Development: A View of the Issues. Journal of Early Adolescence, 25, 10-16. Mahoney, J.L., Larson, R.W., Eccles, J.S., & Lord, H. (2005). Organized Activities as Development Contexts for Children and Adolescents. Organized Activities as Contexts of Development: Extracurricular Activities, After-School and Community Programs. J.L. Mahoney, R.W. Larson, & J.S. Eccles (Ed.). New York: Psychology Press. National Research Council and Institute of Medicine. (2002). Community programs to promote youth development. J. Eccles & J.A. Gootman. (Ed.). Board on Children, Youth, and Families, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press. Merry, S. (2000). Beyond home and school: The role of primary supports in youth development. Chicago, IL: Chapin Hall Center for Children. Payton, J., Weissberg, R. P., Durlak, J., Dymnicki, A. B., Taylor, R., Schellinger, K. B., Pachan, M. (2008). The positive impact of social and emotional learning for kindergarten to eighth-grade students. Chicago, IL: The Collaborative for Academic, Social and Emotional Learning. Pittman, K., Irby, M., Tolman, J., Yohalem, N., & Ferber, T. (2002). Preventing problems, promoting development, encouraging engagement: Competing priorities or inseparable goals? Washington, DC: Forum for Youth Investment. Sabaratnam, P. & Klein, J. D. (2006). Measuring Youth Development Outcomes for Community Program Evaluation and Quality Improvement: Findings from Dissemination of the Rochester Evaluation of Asset Development for Youth (READY) Tool. Rochester, NY: J Public Health Management Practice. Sabo-Flores, K. (2008). Youth participatory evaluation: Creative strategies for engaging young people in evaluation. San Francisco: Jossey Bass. Villaruel, F., Montero-Sieburth, M., Dunbar, C., & Outley, C. (2005). Dorothy, there is no yellow brick road: The paradox of community youth development approaches for Latino and African American urban youth. In J.L. Mahoney, R.W. Larson & J.S. Eccles (Eds.), Organized Activities as Contexts of Development (pp. 111-129). Mahwah, NJ: Erlbaum. Zeldin, S. & Collura, J. (2010). Being Y-AP savvy: A primer on creating and sustaining youth-adult partnerships. Ithaca, NY: ACT for Youth Center for Excellence.
- 47. 47 APPENDIX B –STAFF INTERVIEW PROTOCOL 1. Your Role at IL IDHS 2. Your thoughts and understanding of BFR 3. Structure of Youth Programs 4. Programs under Your Purview Which are stellar, stand out? Why? What are areas that are routine problems? 5. Data Indicators currently used? Outcomes currently used? What is collected? Where is it housed? How is it analyzed? 6. Thoughts about this project Engaging providers Pre-survey IDHS capacity to collect and analysis data Providers’ ability to meet BFR expectations 7. Items we should review for this project 8. Additional stakeholder we should talk with? 9. Additional thoughts?
- 48. 48 APPENDIX C –KEY THEMES FROM STAFF INTERVIEWS These are further contextualizing details from conversations with staff from the Bureau for Positive Youth Development (where CYS is housed) and Youth Prevention Services (which houses TeenREACH.) INTERNAL CAPACITY & BFR IMPLEMENTATION CONCERNS One staffer is in favor of looking at results, but raises the concern that federal programs don’t have much funding support to build the capacity for it. This was also a concern echoed in both the survey and focus group data. General revenue, which could provide funding, is the first to be eliminated when funding is cut at the state level. Staff believes grants should have some funding directed to capacity building and, if thinking about local evaluation capacity, making it centralized would also make it valuable. Providers are not used to doing evaluation and collecting data. If we hold providers responsible for implementation then it creates the potential for outcomes. Due to the loss of 52% of funding over time, one staff person describes performing her own internal ‘BFR’ process to look at overlaps, consolidate programs, and work collaboratively with her providers to save critical programs and eliminate those that were redundant or duplicative (either within her own area or another area of IDHS) and those not meeting the performance measures. She has worked with her providers to develop logic models for all but one of her program (TeenREACH) which she was working to complete in the near future. Staff is concerned about making sure she has the time to review the data herself and not to delegate to subordinate staff. For TeenREACH, staff feels that she is already collecting the types of data that would be needed for BFR, but some it is not in the best format. Staff also fears that the only overall measure that would fit across all youth programs is safety. The others are prevention and focused on different issues and concerns. Staff reiterated her concern about measures across the youth programs when there were no clear goals across the office, unit, or division. INTERNAL CAPACITY TO COLLECT AND ANALYZE DATA Staff concerns were, in part, due to their belief that it would be hard to achieve agreed upon outcomes when there were no agreed upon set of goals or definitions for the office, unit or division. CYS staff describes all data collection being outsourced and handled externally. Similarly, TeenREACH raises the question about IL DHS' internal capacity to do the work required for BFR. Over time the agency has experienced a reduction in funding up to 52% in its TeenREACH program. Training, evaluations (there has been no formal evaluation since 2005 and its first pilot evaluation was in 2000), and research and analysis have been eliminated. Moreover, the need to adapt and build out eCornerstone to meet the new data requirements, which come with BFR, has yet to be met. Building out the database would include building an interface on the agency side, as well as on the provider side; staff is aware that connecting IL DHS to provider data would require resources (i.e., people, time and funding.)
- 49. 49 APPENDIX C –KEY THEMES FROM STAFF INTERVIEWS PROGRAM CONCERNS Regarding CYS, staff are concerned that many of the Community Committees, which should focus on organizing, advocacy, and direct service, have become too direct service oriented and are losing their trademark participation from local residents and multi-sector representation. While the committees are either in process, new, or existing for at least 3 years, there does not seem to be standard criteria for what each of these committees mean. Neither does there seem to be complete clarity on what constitutes good outcomes for these efforts, nor does there seem to exist a process for determining what each level of development means, what happens at each stage of development, and how a subject moves from one stage to the next. It is believed that in the beginning CAP was operating as intended, i.e., Affiliates were organizing within the community, but practices changed as they began to expand, leading to less organizing and capacity building than the model intended. In prevention-oriented TeenREACH, in addition to providing participant outcomes in accordance with the goals and objectives specified for TeenREACH, in addition to academic outcomes, providers must also enter into ECornerstone the following data: unduplicated number and demographic characteristics of program participants, annual enrollment data; participant attendance data; hours of operation; program service activity. Prior to 2006, external evaluations of TeenREACH created the link between outputs and other outcomes, such as academic achievement. The staff describes providers being asked do x, y, and z and if they do those things then they are successful, yet providers are not held accountable for the results/outcomes that come as a results of x, y, and z. For example, while providers are held accountable for providing activities which relate to academic achievement and they are deemed successful if they have provided those activities, yet there is no accountability related to improved academic achievement. Staff differs in their opinion about CYS fitting with TeenREACH. While it is true that both programs include PYD elements even in their contracts, it is noted that Teen Reach is a targeted population and was created for TANF families, free lunch children, or single mothers with children who have been historically considered at risk, unlike CYS which is open to all youth, regardless of perceived risk level. There are also concerns about Chicago Area Project (CAP) and CAP Affiliates remaining true to the model of organizing, direct service and advocacy.
- 50. 50 APPENDIX C –KEY THEMES FROM STAFF INTERVIEWS PROVIDER ABILITY TO MEET BFR EXPECTATIONS Staff differed in their thoughts on the providers’ ability to meet BFR expectations. CYS staff felt it would be important to figure out how to support this new way of performance management, and providing ongoing support to providers. CYS saw CAP as being most likely able to meet the outcome expectations of BFR although they would need to be pushed. In contrast, TeenREACH staff saw their providers as much more capable of meeting the challenge of BFR because providers participated in their own mini-BFR and staff-initiated process of developing program logic models. As staff compiled the list of providers for the survey and for the focus groups, we asked them to provide additional info about the providers, such as their geographic location (rural, suburban, urban), size of organization (small, medium and large) and the provider’s readiness to take up outcomes (ready, probably ready, not ready). Readiness was defined as being familiar with outcomes and having previously presented outcomes to IL DHS staff. The breakdown is as follows: The rankings make clear that TeenREACH staff certainly see their providers as much more prepared to meet the expectations of BFR than do the CYS staff. DETERMINING HIGH & POOR PERFORMING PROGRAMS The goals of the Community Youth Services program are to: • Recruit community residents to develop and become involved in activities which promote positive youth development in targeted geographic areas • Identify community needs and concerns • Develop Community Committees to address these needs and concerns • Provide support for the Community Committees to develop and improve the political, economic and environmental conditions in the target communities • Educate Community Committees on available resources and to advocate on their behalf Exemplary performance in CYS is tied to effective community organizing, i.e., mobilizing people whether they are residents or advisory council members around a particular issue or concern, effectively engaging partners to implement community plans, and using community volunteers well. TeenREACH is prevention oriented. Different youth receive different services based on surveys completed each year by parents. These surveys reflect parents’ ideas on their children’s needs 6 core service areas: 1. Improving academic performance, 2. recreation, sports, and cultural and artistic activities, 3. positive adult mentors, 4. life skills education, 5. parental involvement, and 6. Service Learning. One child might spend more time in one activity than another one, depending on the need identified by the parent.
- 51. 51 APPENDIX C –KEY THEMES FROM STAFF INTERVIEWS DETERMINING HIGH & POOR PERFORMING PROGRAMS While CYS CAPS is considered stellar by staff, who believes providers are interested in talking about outcomes, there is a concern that there has been a real departure from the original community organizing model and provider organizations have become locked into direct service, too focused on funding and reporting outputs. CYS staff further described poor performing programs as those with staff incompetence, those attempting expansion when outcomes are not being met currently, in addition to being overly focused on direct services and not doing enough of the organizing work. There is even a question as to whether CAP should still be funded to prop up Affiliates until these questions are answered. TeenREACH staff described stellar programs as those that train staff, work well with youth, receive federal funding to do the same work, have developed internal systems, consistently receive United Way funding, avoid “calling every two weeks with problems,” and would be sustainable beyond IL DHS funding. Staff describe the organization Alternative (a CAP Affiliate), which is no longer funded for TeenREACH, yet who were more successful at the model than even CAP. Staff says many of the organizations doing TeenREACH well are schools, churches, and smaller organizations which need more help with systems, outcomes and data entry. As a result, more program data is more likely to be in a file folder than in an IL DHS system. But despite this, staff insists providers are doing good work which should be separate from their ability to manage data. In terms of TeenREACH, the staff felt they had eliminated poor performing programs in a previous round of internal cuts and felt that the remaining programs were doing a good job meeting performance indicators. Staff mentioned that CAP was funded to do TeenREACH and while it is done badly, there are questions whether CAP should be still funded to prop up Affiliates.
- 52. 52 APPENDIX C –KEY THEMES FROM STAFF INTERVIEWS PROGRAM DATA COLLECTION According to CYS staff and the materials they provided, IL DHS is collecting mostly quantitative data in three areas: Organizing, Direct Service, and Advocacy. For Organizing IL DHS collects information on: • Community committee members (i.e., the total number of meetings held, the total number of community committee members, etc.) • Committee Sector Representation (i.e., among providers, the number of sectors represented; number of members per sector, etc.) • Assessment Update/Outcome (which also asks if the assessment has been planned for the fiscal year, progress made to date, and if the assessment has been completed) • Milestone/Objective Update (where providers lists the milestone, the object and progress to data). Under Direct Service IL DHS collects: • Direct Service Activities Information (primary issue, direct service type, frequency of activity, duration, etc.); • Recipients' age and totals served by age • Recipients' Race • Linkage with other Organizations (including the total number of unduplicated volunteers, volunteer hours, etc.) And under Advocacy, IL DHS collects: • General information such as the primary issue, the outcome, strategy, progress to data etc.; and • Linkages with other organizations (which include progress to data, volunteer hours, name and role of partner(s)) CYS staff describe collecting mostly outputs (number of youth, number of committee members) but not hours of programming. However, they do collect volunteer hours to leverage funds. They have only starting collecting demographic data all of which are kept in a spreadsheet (Excel) and they manually count what is being implemented. Providers also share narrative data on successes, challenges, organizational milestones and objectives which are kept in a spreadsheet – but nothing is done with the data. There is no criterion for the 3 different categories of groups. TeenREACH collects mostly outputs which are referred to as performance measures and collected through ECornerstone: 1. Days Open; 2. Proposed vs. Actual Youth Attendance Hours; 3. Proposed vs. Actual Average Daily Attendance; 4. Actual Population Served; 5. Program Dosage-average # of days; 6. Cost per Youth per Hour 7. Youth Employment supported by a % of the grant
- 53. 53 APPENDIX D –YOUTH PROVIDER SURVEY & FOCUS GROUP PROTOCOL In an effort to go deeper based on the literature review, interviews with IL DHS Youth Program Staff, and the survey responses, we hosted 2 focus groups with providers from CYS and TeenREACH. There were three main goals of the focus group: 1) determine if there was consensus on the broader PYD outcome domains like those outlined in A Shared vision for youth: Common Outcomes and Indicators developed by The Forum for Youth Investment with the National Collaboration for Youth Research Group; 2) provide an opportunity for providers to share the types of supports they believed would be necessary for them to succeed in implementing BFR; and 3) to share their concerns about BFR and State’s ability to implement BFR successfully. The focus group with downstate youth providers was video-conferenced and, besides a minor technological glitch that caused a late start, the participants were extremely engaged and provided interesting insights. There were a total of 7 participants, representing 6 different organizations, 5 of whom were TeenREACH providers and 2 were CYS providers. Two of the TeenREACH provider organizations also provided CCBYS programming and one of the CYS provider organizations was also a CCBYS provider. It might have been useful to have a separate focus group with organizations that provided both CYS and CCBYS or TeenREACH and CCBYS services in an effort to understand how to reach consensus around PYD and prevention outcomes (given that they represent two different types of programming models provided by IL DHS, but all of whom will be eventually subject to BFR.) One note on the demography of the focus group: organizations represented were mid-sized to large (not the tiny storefront), thus we do not have a good profile of the smaller to mid-sized organizations. This would have required another fielding of the survey to fill in this information, which the timeline did not allow. Participants were mostly Executive and Senior management level (i.e., the main staff overseeing programs and/or program evaluation, not delivering the service which means they were more likely to be familiar and comfortable with the discussion and language related to outcomes.) In addition, due to an omission from the list provided by IL DHS staff, several organizations were not recruited for the survey or the focus groups. Therefore, it would be important to include these community voices in any next phase of this project. The focus group with Chicago-based providers, as in the downstate group, went well with actively engaged participants who seemed to enjoy being together and talking about the issues. There were 11 participants, representing 9 different organizations, with 9 being TeenREACH providers and one of the nine provided both TeenREACH and CCBYS programming services. The chart below represents the breakdown by provider type across the two focus groups and as unduplicated count. Upon review of the chart it is clear that TeenREACH was overrepresented among the focus group participants and upon first glance might lead one to think it limits the conclusions that can be drawn based on program type. The balance to this is that the pilot was structured to include both CYS and TeenREACH programs because of their full incorporation of PYD frameworks into their programming and because TeenREACH includes prevention elements and CCBYS to an even greater degree; it bolsters the position that it is possible think about PYD outcomes across a range of program types.
- 54. 54 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW WOULD YOU DEFINE DELINQUENCY AS IT RELATES TO YOUR PROGRAMMING? • Our youth program serves Chinese youth from immigrant families who are in school. Our definition for delinquency would be youth that violates the law, e.g. violates school rules or curfew laws, drug and substance abuse etc. • We don't use this term in our programming but do recognize it to mean youth that are not engaged in school or the larger social constructs. They are usually involved in criminal activity. • Delinquency would pertain to court or gang-involved youth, or youths involved in juvenile court diversion programs. • Students who find themselves outside the norms or expected behavior for their peers (crime, substance abuse, truancy, etc.) • Delinquency relates to a youth coming in contact with law enforcement • Avoidance of risky behaviors, engaged in positive youth development • We have a mental health juvenile justice program that provides linkage and referrals for youth exiting youth detention facilities. For that program, we would define delinquency as arrested by the police for failure to abide by laws and tried as such. Other behavioral concerns are usually integrated into this definition within the school context because of rule breaking and safety related concerns that emerge that result in suspensions. • Delinquency is the state of maladaptive development that hinders an individual from contributing socially and cognitively to one's family and communities. Delinquency is characterized by destructive behaviors that inhibit the growth of self and/or others. • Criminal offenses completed by juveniles. • Delinquency for our programming is defined when students don't/can't engage with the programs, staff and/or volunteers to the level that they are entering the juvenile justice system or are at the very least disengaged from community involvement for youth such as school, peer groups, involvement in illegal activities, etc. • Youth who are unable to live at poverty levels and are in dire needs of social and emotional supports. Youth who struggle to meet the challenges of everyday life. • Youth not showing up for classes, scheduled appointments, etc. • In our Teen-REACH afterschool program I would define delinquency as young people missing school, keeping a child home from school with makes a child a truant. Any youth under the age of 18 breaking the law, behaving in a disrespectful manner. • Principals and school staff feel that the after school programs reduce delinquency in the community. Law Enforcement agrees with this response to the program. • Conduct that is out of accord with accepted behavior (e.g., inappropriate behavior, disrespecting authority, fighting, cursing, problems with the law, etc.) • Youth who go to police department refusing to return home or are locked out by parents. • Youth without support systems, • Delinquency is defined in Youth Services program as an at-risk youth who is homeless, truant, runaway or has been locked out of their homes by their parent or guardian due to their ungovernable behavior. • We don't use the word Delinquency on a regular basis. We choose to focus on the whole youth, and address problems and needs as they occur. • Criminal activity carried out by a juvenile. • Youth at high risk of delinquent behaviors or those involved in the juvenile justice system.
- 55. 55 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW WOULD YOU DEFINE POSITIVE YOUTH DEVELOPMENT? • Positive youth development is defined by us is a holistic development of youth, not only academic +- excellence, but development in leadership skills and civic engagement • A process necessary to ensure that youth are able to make and execute decisions that lead to positive life outcomes; additionally to assist youth in developing the necessary skills to be prepared for adult life • Positive youth development is a framework that fosters internal and external protective factors to moderate risk factors and enhances optimal physical, cognitive, social and emotional development. • Giving young people every opportunity to discover and grow skills and talents in a supportive environment with positive role models and adult support • Positive youth development is efforts (activities/interventions) to support youth in achieving their dreams and becoming contributing members of their community. It emphasizes the importance of focusing on youths’ strengths and providing opportunities to help others through service. • Built upon YPQA, program quality pyramid, 40 developmental assets • NA • "PYD emphasizes the importance of focusing on youths’ strengths instead of their risk factors to ensure that all youth grow up to become contributing adults." Many researchers have looked at this and have created frameworks around it. It relates to our programming because much of our prevention programming is emphasizing strengths and building capacities in order to mitigate risk factors that exist in their communities, their homes, or even their physical predisposition. • Positive Youth Development is a process which prepares young people to meet the challenges of adolescence and adulthood through a coordinated, progressive series of activities and experiences which help them to become socially, morally, emotionally, physically, and cognitively competent. • Providing strength based framework to foster kids’ growth. • Positive Youth Development involves building on assets youth or communities possess to either prevent risky behavior or to intervene at an early stage in risky behavior among youth. • PYD is a tool best utilized to develop youth leadership, character and living skills to become productive and successful members of society • Youth making the necessary changes to improve decision making. • I was once a teenager and I cut school, talked back and broke the rules but I turned out ok. In the afterschool program we have six core components; academic assistance, life skills, mentoring, recreation, parental involvement and service learning. Every day we work on those core components other than the Parental involvement part. I would define positive youth development as making a positive difference in the youth’s life. • Youth showing positive behavior to school, other students and a positive attitude toward their role in the community. They also have a perception about their future success and how education plays an important part of the success. • Positive youth development refers to intentional efforts of other youth, adults, communities, government agencies, and schools to provide opportunities for youth to enhance their interests, skills, and abilities. PYD suggests that helping young people achieve their full potential is the best way to prevent them from engaging in risky behaviors. • Improve ability to achieve normal developmental milestones. • Youth that has access to support systems, home, school, friends, family, community, etc. • Positive Youth Development in Youth Service program is used to focus on the youth's strengths in order to provide the necessary assistance needed for youth to achieve his/her objectives and goals in order become productive individuals.
- 56. 56 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW WOULD YOU DEFINE POSITIVE YOUTH DEVELOPMENT? • We focus on the assets/skills/strengths of the child/youth, and we work to build on those. • PYD is a policy perspective that emphasizes providing services and opportunities to support all young people in developing a sense of competence, usefulness, belonging and empowerment. • Search Institute tells us the best way to achieve positive youth development is by following the 40 developmental assets, the building blocks of healthy development. However to define, Positive Youth Development is the facilitation, support, and nurturing of young people’s growth through dependence to interdependence, having strong support systems in place, encouraging their personal and social development and enabling them to have a voice, be active, engaged, creative and responsible for their own development, and to allow them influence and a place in their communities where their positive development can be recognized and appreciated. HOW WOULD YOU DEFINE OUTCOMES IN YOUR PROGRAMMING? • Outcome is gauged by youth graduating from high school, entering post-secondary education or involved in vocational skills training that can lead to sustainable permanent employment and self sufficiency • A change in behavior or attitude as well as accomplishing a possible life altering achievement such as graduating from high school or college • Outcomes are specific short and long-term results/changes for individuals, groups, communities, etc. stemming from programs or services. • Measurable and sustained growth in student behavior and attitude. • Outcomes are the end results of the inputs and outputs of the program/service intervention. • Youth committed to learning, positive character development traits and leadership skills, youth demonstrate life and work skills, program engagement • Outcomes are youth transitioning to independent living after completing and "working" their life plan. • Based on the logic models that we develop, there are theories that form the basis for activities within programs. These in turn are tied to outputs, which could be things such as attendance in programming activities, # of participants that attend workshops, etc. These outputs affect what we expect to be the results of the program activities otherwise known as the outcomes. Since we have programming across the continuum of prevention to treatment, each of the logic models reflect outcomes that are expected within those programs. Outcomes for youth prevention reflects things such as improvement in knowledge of substance use/abuse and the effects, improved attendance at school, improved peer relationships, ability to define a post-secondary plan, ability to create a safety plan for themselves and within their communities, decreases in suspensions and expulsions, decreases in physical fighting among peers, improvements in family functioning, improvements in time management and organization skills. • Outcomes are the durable, long-term impacts that programs have on youth. • Outcomes are measurement tools we utilize to measure our effectiveness. • Measuring area of impact brought about through the programming we provide. • Reduced occurrences of drug use and new arrests. • Our outcomes have been making sure the youth are safe at all times during out program, boosting their self-esteem and confidence. We may not make a difference in each youth’s life but we strive to make a difference in their lives. We have 70- 75 youth that attend the after school program and we feel that we are doing something right because they could be doing other things. • Students’ behavior both at school, in the community and with their families is superior to that of the students that do not attend the after school program and their grades improve. • Outcomes are related to the impact that our services have had on changing the behavior and lives of our clients.
- 57. 57 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW WOULD YOU DEFINE OUTCOMES IN YOUR PROGRAMMING? • The number of youth who are returned to their home rather than become wards of the state. • Good • Youth Services define outcomes as means to determine the level of success in achieving the overall goals and objectives of the program, the changes that are expected to occur as a result of services provide • For us, outcomes are behaviors that exhibit growth toward our goals. • Positive changes which occurs in program participants as a result of their participation in our programming. • Measurable deduction of juvenile arrests and recidivism from a restorative prospective (i.e. station adjustments, peer court/juries). . Measurable decreases in youth going to Juvenile Detention Center. Decreases in expulsions, and suspensions. Increase in accountability to community by offender. Increase in community based restorative initiatives and practices. • The benefit for participants during or after their involvement with a program PLEASE LIST THE MAIN OUTCOME AREAS RELATED TO YOUR PROGRAMMING AREA: • 1. High school diploma 2. Post-secondary education acceptance letter 3. employment letter • Academic outcomes Life outcomes Prevention outcomes • Reduce childhood obesity, reduce the achievement gap, increase kindergarten readiness, increase school attendance, increase academic achievement, reduce school-based behavioral incidents, reduce gang involvement, reduce juvenile recidivism, foster safer communities, increase child safety, etc. • Reduction of problem behaviors (substance abuse and pregnancy rates, mental health crises, juvenile arrests, etc.) • This varies on specific program interventions but would include: passing - onto the next grade/graduation; successfully living independently (for a year); substance abuse free (for a year); arrest free (for a year) • Youth committed to learning, positive character development traits and leadership skills, youth demonstrate life and work skills, program engagement • Outcomes are youth transitioning to independent living after completing and "working" their life plan. • Social emotional changes, academic data including behavior, grades, attendance, graduation, matriculation, etc., changes in Child and Adolescent Global Assessment of Functioning (CGAS) and Daily Living Activities (DLA). • Youth who complete high school prepared for secondary education and/or a career – youth who have the communication and problem-solving skills to conscientiously identify and address challenges in their communities -youth who have developed the behaviors and attitudes to become healthy, responsible, self-reliant productive citizens and community members • Number of youth going into DCFS care Number of youth sentenced to DJJ Number of families remaining intact • Youth engagement 2) Academic achievement • Increased abstinence, decreased legal involvement • One of our core components is Parental involvement, if we can't get them to come out, we offer free computer classes, resume building classes, adult sessions once a week from Southern Illinois Healthcare Foundation. Our staff is very active in the schools that the youth attend making sure they are in a caring school environment and if now we make a referral to another department in our agency. Our outcome area is to meet the needs of the youth that we serve and their families. • Academic, life skills, attitude toward education, more and improved and more career choices, recreation, cultural and social progress, health and nutritional meals. • Consumer behavior, school complaints, and overall improvement behavior
- 58. 58 Are you collecting yourown outcome data? If yes, what type of outcome data are you currently collecting? What methods are you using to collect this data? What program (or technology) are you using to store this data? Yes 1. report cards 2. High school diplomas 3. activities reconfs (including frequency and duration) 4. tutoring (including ACT) participation record followed up with ACT scores 1. Our own data base system (ACCESS) 2. Efforts to Outcome (ETO) 3. IWDS (Illinois Work- force Development System) 4. paper attendance record We use ACCESS and/ or EXCEL to store the data we collect Yes Academic. It is the easiest and least expensive. To a lesser degree We collect data on life outcomes. Gathering graduation rates Surveys Excel Yes Data relating to #4, above. Surveys, student-level school data, observation and assess- ment, etc. CitySpan, Voyager, COPA and state-required reporting systems. Yes Pre and Post tests, IYS data, student asset levels DAP survey, Surveys, IYS surveys Excel Yes Problems encountered by youth while receiving services Performance Improvement Quality activities including: surveys, client satisfaction surveys, focus groups Internal client infor- mation system - in development APPENDIX E – AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS PLEASE LIST THE MAIN OUTCOME AREAS RELATED TO YOUR PROGRAMMING AREA: • Number of youth who return home 2) Number of youth who become wards of the state 3) improvement in family relationships • Lower drug use o Fewer program participants will be involved in violent and dangerous activities b. Parents will improve their parenting skills c. youth will remain at home and in school d. truancy level among program participants will decline • Overall, in our entire program: - academic retention and achievement - development of career aspiration/choices - development of soft jobs skills - increase decision-making and problem-solving skills - increase anger-management and conflict resolution skills - increase choices in favor of physical fitness and nutrition - increase engagement with the community • Organize communities to address community needs; advocate for change(s) and programming to address needs; link and or deliver direct services to youth and families to address the need; and collect and uses data ( quarterly reports. indicators and outcomes ) to demonstrate efficacy of the program. • Reduction in juvenile delinquent behaviors. Reduction in youth entering Juvenile Detention. Reduction in expulsions, suspensions from school. Increase in positive youth development. Increase in positive recreational, cultural, and educational activities and employment opportunities for youth. Increase in alternatives to suspensions, expulsions, and juvenile detention. • Students improve school performance Students graduate from high school Students enroll in post-secondary education Improved social skills and interpersonal behavior
- 59. 59 Are you collecting yourown outcome data? If yes, what type of outcome data are you currently collecting? What methods are you using to collect this data? What program (or technology) are you using to store this data? Yes Data correlated to outcomes listed above, data sources include: youth surveys, staff assessments, school attendance, promotion rates, grades (not a primary data source), Employability Assessment, program attendance data, college accep- tance/attending Kidtrax, Employability Assess- ment, Survey Monkey, Excel Excel, maybe Access Yes Outcomes are youth transitioning to independent living after completing and "working" their life plan. Case file review, life plan review, team consultation. Excel and Word. Yes Social emotional Surveys (Me and My World-Search Institute), satis- faction surveys, and anger (MASI), substance abuse/use (Lionsquest), and trauma-specific (TRRB) sur- veys from teachers, parents, and students and school-transcript data such as grades, behavior data, at- tendance, etc. For many of our programs, youth and parents must consent to release data for analysis. We have staff obtain consents and assents and only collect and track youth who give us permission to look at the data. Staff administers pre & post social emotional survey at beginning of after school programming and end of school year. Additional surveys are col- lected from teachers, students, and parents at the end of programming activities or end of school year SPSS, Excel Spreadsheets, CitySpan Yes Academic data (attendance records, grades, standardized test scores) -data on what type of post-second- ary institutions that students enter after leaving our program -data on changes in youth’s attitudes, skills, and behavior -data on youth’s participation in leadership develop- ment activities, community service, advocacy, and policy-making Transcripts received from schools – program developed or curricula pre and post tests client satisfaction surveys for key partners (e.g. school administrators) Funder created or specified data- bases (e.g. CitySpan) -Microsoft Office software: Access and Excel Yes DJJ commitments, DCFS getting guardianship, family status at case closure. Caseworker enters data at case closure TIER and ECornerstone Yes Average daily attendance, school report cards, arrest records, etc. Manually APPENDIX E – AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS
- 60. 60 Are you collecting yourown outcome data? If yes, what type of outcome data are you currently collecting? What methods are you using to collect this data? What program (or technology) are you using to store this data? Yes Increase in positive behavior Increases in academics Increases in school attendance Decreases in substance use among teens, etc. Surveys Need Assessments Custom built data base to synthesize all data collected Yes Abstinence and legal involvement GAIN and GPRA Evaluator's SPSS Yes The Teen-REACH program is using ECornerstone. ECornerstone is a statewide management system that currently supports various youth services. The system was implemented in 1997, developed to integrate community based material and child health services provided to Illinois Department of Human Services customer and to report their health outcomes. I use ECornerstone to input the attendance, record the prog- ress reports and report cards. ECornerstone allows me to go in and make notes on the youth. For instance if one is put out of the center or I have had to speak to the youth on several occasions that is put in the system. You have a password so the information is not for public use. ECornerstone Yes academic and school attendance collection grades and atten- dance eCornerstone, and secure files Yes Ohio Scales and Child Behavior Checklist Parent/Caregiver reports, Teacher reports, and Counselor reports Excel Yes Counting number of clients, where they go to live, and if parents go to court for a lockout, score on the YAZI Simple counting methods, complete YAZI at time of initial intervention and at closing Excel Yes grades, high risk behaviors grades, entry to our high risk program Excel Yes a. participants risk level: the percentage of youth score of low, medium or high on the Youth Assessment Pre-screening which categorize specific issues or prob- lems that need to be addressed The Washington Model (National Youth Violence Pre- vention Center. This is required to utilize the Youth Assessment Screening which is used to develop Individualized Service Plan for each youth and family. A date base called eCornerstone which is a computerized tracking program use to collect and store the youth and family information. Yes self-records APPENDIX E – AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS
- 61. 61 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS IF YOU ARE NOT COLLECTING YOUR OWN OUTCOME DATA, WHAT ARE THE BARRIERS TO COLLECTING YOUR OWN DATA? • Technology, staffing, consent forms, state reporting systems that do not allow us to export data and require double data entry, etc. • Cumbersome process because of individual student level data collected and needs to be inputted from numerous data sources. Other funder requirements (city of Chicago, state of Illinois, federal departments) have their own databases so we are sometimes double/tripling data efforts. • We do collect our own but some of the data comes back from other web-based entry locations and some require consents to even obtain data. Access to school data is contingent on consents/assents (FERPA) and CPS fulfilling data requests in a timely and complete manner. This last school year, the data requests that were sent did not come back quickly nor completely which does not allow for appropriate analysis. Real-time school data is harder to obtain in general, which would be most useful for staff. • No technology or sufficient resources to secure the database we need to accomplish this. • Other data is subjective in nature and indirectly considered as an indication of outcomes. • Need to find way to tie in programs with questions • Information is being collected • Though we do collect it, time and personnel are an issue in gathering and inputting the data. • Need to first establish more defined outcomes and related metrics and the technology to track. • Collecting information from other sources makes it difficult to determine outcomes because the information can be sporadic and inaccessible. WHAT ARE YOU LEARNING FROM YOUR OUTCOME DATA? (PLEASE GIVE AN EXAMPLE.) • We learn from the data how our outcome is compared to our goal. e.g. last fiscal year we have 27 youth in senior class, all of them graduated and 25 of them (92%) entered postsecondary education • The graduation rates of our participants as well as their attitude toward family, school, themselves, etc. • We learn how many students • That developmental asset levels and the extent of positive, supportive adult relationships drive virtually every area of prevention/early intervention concern. • Difficulty collecting outcome data once youth no longer is receiving services. Funding does not provide means for long term follow-up • ways in which programs need to be modified, what students we are impacting the most • We will know next year. • Metropolitan Upward Bound has provided services to over 150 first generation College bound students since 2007. Findings over the last four years reflect statistically significant differences (p=.05) between beginning of the school year and end of the school year GPA. During the 4 years of service, 60% of students increased their GPA from beginning to end of school year. 89.25% UB participants have been retained over the 4 years. Because of the afterschool programming, 89% of parents agree that their child has learned new knowledge or skills; 87% agree that their child gets along better with other students; 79% report helping their child with homework at least weekly; 87% report that they are talking to their child about school at least weekly; and 86% report that the afterschool program helped them learn more ways to support their child’s education • We are learning that our programs are significantly increasing the high school graduation rate (i.e. our program graduates youth at approximately 90% compared to general school graduation rate for our partner high school of approximately 40%), but our high school graduates are not entering or staying in college at nearly as high a rate.
- 62. 62 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS WHAT ARE YOU LEARNING FROM YOUR OUTCOME DATA? (PLEASE GIVE AN EXAMPLE.) • Youth are having difficulties remaining drug free • I am learning that maybe we aren't doing a good enough job on academics as far as looking at their report cards. It also shows the days missing from school. We are always using the surveys for our own benefit. • Teachers feel more efficacious, students are spending more time on task, and disciplinary referrals are decreasing. • The majority of youth are returned to their home or to an alternative living arrangement arranged by their parents. • Data for example in 2011, 97% did not enter the juvenile justice system and remain at home. • The vast majority of youth make positive choices despite the difficulties presented in this community. • Lack of quality education and a living wage job are the main challenges for youth and adults. • The programs we are trying to measure only have collected data up to a point or have not been collecting long enough. Best practices in reduces delinquency might work in one community and not in another. One size does not fit all. • Our students on average are maintaining 3.0 or higher GPA and have a very low school absence which is positive. While 89% of our youth report that they will obtain a 4 year college degree, 44% of students are unsure or think they probably cannot afford to attend college using financial aid, scholarships, or their family’s resources. Staff are targeting the 44% as a data point to improve this year. Staff are also part of a collaborative evaluation group and in meetings where program staff compare data across agencies, they have learned from other agencies who are getting better results in some areas. Staff have incorporated these ideas into program improvement plans (for instance we are restructuring our tutoring services based on what we’ve learned in collaboration meetings) HOW ARE YOU ANALYZING THAT DATA? • Manager and mid-management analyzed the data and also use it to compare previous year's data to evaluated the success of the program and determine the direction for the next year • We review it via discussion and theorize on the why. • Our Performance Improvement Department analyzes the data, in addition to our Managing for Results process. • Benchmarking performance internally and with other providers • Use of reports at various levels (executive, program manager level) with filters for dosage, etc. • We are in our first year. • We use SPSS to conduct analysis and then provide summaries that include visuals (charts) indicating changes that were seen over time. We provide descriptive data back to program staff summarizing the data that was received. Additionally, we complete statistical analysis (t-tests, ANOVAS, ChiSquare) if we have sufficient power and return on data from programs. • We run queries in our database to identify the number of participants who are meeting the targeted outcomes and comparing those results to the outcomes for their counterparts who did not participate in our program. We analyze the data during department leadership meetings, during staff supervision meetings, and through monthly reports. • We occasionally review our data - on an annual basis, we complete an agency wide planning and evaluation document. We also do program evaluation when we complete our annual program plans. We are also COA accredited and they review our outcomes as well as our policies and procedures. • Manually • Evaluator use SPSS • I use eConerstone every day • Statistical analysis/SPSS (i.e., Correlations, Analysis of Variance • Simple mathematics • We will be with school support
- 63. 63 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW ARE YOU ANALYZING THAT DATA? • The data is being analyzed by focusing on the indicators, the success and changes or improvements that are needed in specific areas. Staff is involved and meeting often to review the data. • Various levels of staff meet periodically to review the data. • Conducting assessments every two years in order to prioritize programming. Obtain feedback from a sampling of clients to determine program effectiveness. • Not scientifically. Looking at trends. Reduction in crime, • Data reports are analyzed at least quarterly. Program staff looks at overall results to understand big picture trends, and program staff also drills down to look at individual student data like grades and attendance rates. They develop action plans tailored to individual student needs. We are also in the process of building reports to correlate data such as GPA and program attendance. This will be a new layer of data analysis for staff in the coming months.
- 64. 64 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS HOW ARE YOU USING THE DATA? • We use the data to evaluate the program. We present the data to funders and use it to apply for future funding. • It helps us determine what programs and services we want to keep, eliminate or maintain. It also helps us determine if we need to change an approach or an operative structure. • To improve our processes, operation, program design and quality at the point of implementation, and to replicate successful programs and services. • To provide continuous quality improvement • Identify program modifications, things to continue doing, things to stop doing and things to tweak • Reporting and improvement. • Improve programming, funding reporting, use for marketing, increasing the donor base, or funding proposals, to benchmark against other organizations or entities, to educate staff and schools about anticipated and final results from yearly programming, accreditation, and recruiting staff. • We are using the data to determine where to invest our resources and/or to determine which program activities to discontinue or redevelop, to determine which partners to seek out to address the shortcomings in our programming, and to determine which grants to apply for. • We share outcome data with program staff and brainstorm possibilities to enhance programming. • For program accountability and future program adjustments/development • Program development and funding opportunities • We are using the data to track our Teen-REACH participants after graduation. • Annual Outcome Reports and Performance Improvement Reports • Reporting info to the state • To notice if prevention deters substance use in future • The use of the data helps in understanding the type of intervention needed. • Primarily for our own planning. At times it is helpful to grant writing or reporting. • Adjusting program activities based on the needs of clients and whether or not outcomes are being achieved. • To find where there are gaps in services are. To learn what we are doing right and what we can improve upon. • At the end of the program year, staff analyzes reports and identifies at least two data points they want to target for improvement (or if there are positive data results staff may develop plans to maintain those scores into the next year). Staff develops an improvement plan that aligns with SMART goal criteria and resources are allocated to support their efforts. IF YOU AREN'T ANALYZING YOUR OUTCOME DATA, WHAT ARE THE BARRIERS TO ANALYZING YOUR OWN OUTCOME DATA? • Not enough time to perform analysis because it is the responsibility of one person to complete reports that has many other duties. • We are analyzing the data • Though we do it, again, time and personnel availability make it difficult to do it as we'd like. • Analysis of outcomes must be improved with more defined outcomes, metrics, and the appropriate technology and staff to analyze makes any necessary changes. • Finances to purchase a program that would make data collection easier to collect and share by multiple entities.
- 65. 65 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS WHAT IS YOUR UNDERSTANDING OF BUDGETING FOR RESULTS? • BFR is a tool for government agencies to set priorities according to goal achievement in the services provided so that funding can be given to the services that reach the best results. • That an organization sets up the annual budget to support the outcomes you want to achieve. • BFR is intended to identify spending priorities by adopting performance based budgeting. • Budgeting for Results is the state efforts to set funding priorities and put limited funding into programs and services that work • Tying money to outcomes • Meeting the bottom line while meeting program objectives. • "The National Advisory Council on State and Local Budgeting (NACSLB) has identified four essential principles of effective budgeting. The specific principles include: (1) set broad goals to guide decisions, (2) develop strategies and financial policies, (3) design a budget supportive of strategies and goals and (4) focus on the necessity of continually evaluating a government’s success at achieving the goals that it has set for itself (i.e., performance). • Budgeting For results means strategically investing financial resources to most cost-effectively produce the desired outcomes. • Tying financial resources to positive outcomes. • Funds received from the state will be tied to outcomes and impact results • Looking at funding and performance based measures • From my understanding Budgeting for Results have something to do with the budget and having available dollars to the most significant programs and activities. • Determining how most effectively to use the funds to best meet the needs of the students/families and school. • Governor Quinn signed into law Budgeting For Results (BFR), an historic spending reform act requiring the state of Illinois to institute a results-based budgeting process that will end the automatic funding of programs. By requiring the State to live within its means and focus on performance, BFR will transform the way that state officials, legislators – and the public – prioritize, think about and implement the State’s budget. Going forward, the State will fund only those programs that can demonstrate effectiveness and help the State achieve its stated outcomes and goals. • Unknown • Using funds wisely to see outcomes • A strategic alternative to incremental budgeting, which resources are allocated based on how effectively programs or service achieves established goals and objectives. • I am unfamiliar with this. • State budget resources allocated based on effective a program or services meet goals and objectives. • Have not had any formal training on BFR. Although we do our very best to maximize every dollar using volunteers, in kind contributions, it's hard to budget for results when the payments you rely on are severely delayed. It's hard to expect the maximum results under these circumstances when every program that yields the best results relies on a steady cash flow.
- 66. 66 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS WHAT ARE THE IMPLICATIONS OF BFR FOR YOUR ORGANIZATION? • BFR means we have to justify our budget requests based on the results of our services. The results are based on the data we collect. So we have to have very good data collection methods. • Sometimes leadership uses it to reduce in areas that are not a priority or preference. If you use BFR there will be less of that. • Uncertain at this point • We must be able to demonstrate successful program outcomes to secure and maintain funding to services • It is good to show taxpayers that their money is being used for important projects as I have seen a lot of waste in the past. Funded agencies need to be accountable for how they use money. All nonprofits should have an outcomes measurement process. If not, they should not be funded (or provide funds to create one). Hoping that BFR will be in line with current research/thinking around outcomes from existing sources and not created from scratch as that could lead to a separate outcomes process for organizations and create inefficiencies. For example, there is a lot of work being done around social emotional learning - these concepts should be used in BFR because most organizations that have an outcomes process will also use these concepts. • Unknown at this time. • We receive funding from entities involved in this process in Illinois and recognize that funding will be more likely tied to priorities and results from those priorities. We are at the table with those making the recommendations for priorities and results. • BFR for our organization would require a thorough analysis of each department's budget in comparison to the produced outcomes and then an agency-wide budget -outcome analysis to find out which programs and services are most cost-effective and which are least effective. We would then need to use that data to decide which programs and/or services to continue, to expand, or to discontinue. • Not sure. • More resources to build up our technology capacity and capability. • Improvement in program performances • I am working the Teen-REACH after school program and in that component we do have a budget but that is basically handled through the Management office. • A quality program is the result of using funds that are assisting students in making better decisions for their future and showing improvement in their educational decisions and their positive role in the community. Career choices are also an intricate part of their future success. • We are required to use evidence-based, empirically validated interventions. • Unknown • None • To prioritize and focus on quality service that will bring about long-term results • Will assist our agency to better empower our clients. • I'm not sure since I have not been educated on the concept of BFR.
- 67. 67 APPENDIX E –AGGREGATED DATA FROM THE YOUTH PROVIDER SURVEYS WHAT TYPE OF SUPPORTS (OR CAPACITY BUILDING) WOULD YOUR ORGANIZATION NEED TO SUCCESSFULLY MEET THE OUTCOMES IN BFR? • We find that ETO is a good tool for our collecting data to evaluate our outcome in programs. However, we are only using ETO in certain programs, we need further assistance, technology and financial, to implement it in other programs in the agency • Training for our admin team and understanding of the process • Need more information • Funding to maintain contact and track long term outcomes of youth after they are no longer receiving program services • Funding to support a data manager/outcomes specialist. Training on managing data. • Unknown at this time. • Access to institutional data (school data) and aligning systems with better data integration such as DMH, WIIA, DHS, CDBG. • Our organization would require funding for and/or an assigned evaluator who has experience in strategic financial planning and operations to help determine whether the data we are collecting is accurately measuring what we desire and, if it isn't what data we should collect, help us develop the tools to collect the data if we do not already have them, and assist the PQI department and department leaders in analyzing the data that is collected. • Not sure. • More resources to assist with building our technology capability. • Not sure at this point • Fundraisers, training centers, consultants, ministers throughout the Illinois area working together, local authorities, St. Clair County Housing Authority. Some of the staff might need to acquire some skills, find resources, and work as a team for the same vision. • An improvement in our obtaining professionals and programs to support our life skills i.e. anti-drug/alcohol education for the students. • Cooperation of schools, parents, and the school board. • Unknown • Keep tracking system updated • Staff training to understand the concept in order to provide quality service that delivers the intended results. • As previously described and additionally training, technical assistance in constructing appropriate outcomes, metrics, and sufficient technology. • Again, I can only imagine having the software and the education to implement BFR
- 68. 68 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS BUDGETING FOR RESULTS FOCUS GROUP AGENDA JANUARY 16TH AND 18TH , 2013 I. Introductions II. Purpose of the Focus Group/Setting the Context III. Developing a Shared Definition of Positive Youth Development IV. Developing a Shared Definition and Set of Outcomes V. Necessary Supports to Successfully Implement BFR Organizationally & for the State VI. Additional Thoughts or Concerns about BFR
- 69. 69 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS BUDGETING FOR RESULTS STRATEGIC PLAN
- 70. 70 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS INTRODUCTION The State of Illinois is in the beginning stages of implementing a comprehensive Budgeting for Results (BFR) process to change the way it allocates over $33 billion per year in general revenue spending. Prior to the signing of this law in July 2010, Illinois utilized a traditional, incremental budgeting approach, in which the current year’s budget served as the baseline for development of the next year’s spending proposal. Budgeting for Results is a strategic alternative to incremental budgeting, in which resources are allocated based on how effectively a program or service achieves established goals and objectives rather than historical funding levels. Implementation of BFR will be a multi-year process to identify, develop and execute the structures and processes that will lead to an outcomes-based budget. As with any new system, this will require resources and knowledge that the state may not currently have. This strategic plan reflects our understanding of the activities needed to implement BFR at this time. In this document we have identified where additional resources and/or expertise would be helpful, however the timelines assume implementation will move forward with existing resources only. This plan will be updated as our understanding of BFR, and availability of outside resources, evolves. MISSION/VISION/VALUE STATEMENTS The mission of the BFR implementation team is to implement a comprehensive Budgeting For Results process in Illinois that incorporates all of the components necessary to determine funding priorities, assess program performance, inform decision making, and deliver results for the people of Illinois. Our vision is to be a state in which agencies have the tools they need to collect, track and assess program performance and in which funding, policy and contracting decisions are based on the quality and return on investment of publicly-funded programs. We believe the people of Illinois deserve the best value for their taxpayer dollars, and quality services that deliver the intended results. ORGANIZATIONAL DESCRIPTION The State of Illinois is the fifth largest state with over 12.8 million people, a gross state product of $652 billion and an operating budget of over $33 billion general revenue funds in FY’2012. The State government has over 51,000 employees working in nearly 50 agencies reporting directly to the Governor. Faced with increasing budget pressures, such as growing pension, healthcare and debt obligations, and diminishing state revenues, Governor Quinn recognized the need to begin addressing the structural deficits in Illinois’ budget. First, the Governor signed a prospective pension reform into law that is projected to reduce Illinois’ actuarial accrued liability by $200 billion over the next several decades. Next, the Governor signed a temporary tax increase to provide additional revenue to help cover the state’s structural deficit of expenses consistently in excess of revenues. Third, the Governor signed Budgeting for Results into law, to help inform the difficult budget decisions that must be made to get our fiscal house in order.
- 71. 71 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS GOVERNANCE STRUCTURE Budgeting for Results is a fundamental change that will impact all levels of state government as well as organizations that receive funding from state agencies. The governance structure for BFR must recognize and account for this complexity. A bi-partisan Budgeting for Results Commission, comprised of state legislators, business and academic leaders and experts in state budgets and fiscal policy, was created by statute and appointed by Governor Quinn in August 2011. The Commission was created to advise the Governor and General Assembly on the implementation of a budgeting for results process in creating the Governor’s introduced budget and throughout the appropriations process. The Governor’s Office of Management and Budget is the lead agency responsible for implementing Budgeting for Results in the Executive Branch of state government. In this role, GOMB will coordinate statewide adoption of BFR across all Executive Branch agencies. GOMB has established a BFR Team to lead implementation efforts and will recruit a Chief Performance Officer to lead this project on a full-time basis. At the agency level, each state agency Director has identified a person to be the BFR point of contact responsible for implementing BFR within their respective agencies. This person will attend all BFR training sessions and meetings, work in collaboration with their agency’s senior leadership, budget staff and performance management staff to ensure appropriate BFR processes are in place, and report performance metric data to the Director of GOMB and the Chief Performance Officer on a regular basis. BFR PROCESS There are seven main steps in the budgeting for results process. Below we outline our goals and strategies under each step. It is important to note that these steps are not necessarily linear. The BFR Implementation Team will engage in them according to the timeline provided in Appendix A. STEP 1: IDENTIFY AVAILABLE RESOURCES Budgeting for Results begins by identifying the resources “available” in the coming fiscal year rather than the resources “needed.” For the process to be successful, revenue forecasts must be reliable. Reliable revenue forecasts instills confidence in stakeholders that programs will not be cut unnecessarily due to faulty revenue forecasting. State source tax revenue forecasting is an ongoing process or cycle. The following steps outline the annual revenue forecasting cycle. December 1. Department of Employment Security (DES) economists forecast employment for Illinois in the coming fiscal year. 2. Department of Revenue (DOR) adjusts wage and salary forecasts from IHS Global Insight based on DES employment forecasts. 3. DOR economists run econometric models for each tax source to determine expected revenues for the coming fiscal year. 4. DOR/DES economists present results to the Council of Economic Advisors (CEA) and Governor’s Office of Management and Budget (GOMB). The underlying economic assumptions are discussed by the CEA. Once revenue forecasts are approved they will become part of GOMB’s three year forecast.
- 72. 72 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS January 5. DOR/DES updates data and re-runs models with new tax data from January estimated payments and input on assumptions from CEA. 6. DOR/DES economists present results to the CEA and GOMB. Revenue forecasts are approved and become part of the Governor’s Budget Book. May 7. DOR/DES updates data and re-runs models with new tax data from April final payments, estimated final payments and estimated payments. 8. DOR/DES economists present results to the CEA and GOMB to decide whether or not to revise revenue forecasts before a budget is passed by the General Assembly. October 9. DOR/DES updates data and re runs models with new tax data from the first quarter of the fiscal year. 10. DOR/DES economists present results to the CEA and GOMB to decide whether or not the revenue forecasts presented in the budget are still reasonable. STEP 2: IDENTIFY PRIORITY AREAS A results-based budget is built around a set of strategic priorities. These priorities must be expressed in terms of the results or outcomes that are of value to the public. As part of the FY 2012 budget, Governor Quinn established six priority areas for Illinois, representing the major functions and responsibilities of state government. All state government spending was grouped into one or more of these major priority areas. The BFR Commission then reviewed the six Results and made several recommendations to make them more outcome-focused. The seven Result areas for Illinois are as follows: • Result 1: (Government Services): Illinois state government operates efficiently and transparently. • Result 2: (Education): Illinois has a quality education system that provides equal opportunity for growth for all Illinois students. • Result 3: (Economic Development): Illinois’ economy provides sufficient opportunities for residents to achieve economic well-being. • Result 4: (Public Safety): Illinois has adequate public safety mechanisms and infrastructure in place to protect the lives, safety and property of residents. • Result 5: (Healthcare): All Illinois residents have access to quality, affordable health care. • Result 6: (Human Services): Illinois assures that all residents, but particularly children, the elderly and the disabled, are able to experience a quality life. • Result 7: (Quality of Life): Illinois maintains quality cultural and environmental resources for Illinois residents and visitors.
- 73. 73 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS STEP 3: BUILD A PROGRAM INVENTORY Programs at the agency level are the unit of analysis for Budgeting for Results; therefore program identification is vital to successful implementation. Because traditional budgeting is based on line items and funds, rather than programs, there has never been a need to compile information at the program level. Budget for Results fundamentally changes this dynamic. Agencies were asked to self-identify programs and submit them to GOMB via the Budget Book system by April 1, 2012. GOMB will review the submitted programs and verify that they are appropriate and amenable to measurement. A program must be broad enough to capture the tasks that go into producing a result but not be too vague or encompass too much as to undermine measurement efforts. It is expected that the process of program identification will take some time and refinement as GOMB analysts and agencies develop the appropriate parameter. STEP 4: EVALUATE PROGRAMS AGAINST PRIORITY RESULTS Measuring the performance of state programs and evaluating their success in achieving results is an essential element of BFR. In order to do this, a process must be developed to objectively evaluate how a program achieves or influences an established Result. There are often two types of evaluations discussed in Budgeting for Results. The first is an evaluation of program effectiveness. This refers to evaluating a program against established metrics to determine if it is achieving results. The SMART evaluation discussed below is designed for this type of evaluation. The second is an evaluation of program impact compared to other programs, often referred to as Return on Investment. This is a much more in-depth evaluation that requires robust, state-specific, data sets. As an example, advocates for a program might cite national studies that indicate $1 invested in the program saves taxpayers $xx in future costs. However, those studies cannot speak to whether an Illinois program is of comparable quality or achieves similar returns on investment. This type of evaluation requires a sophisticated method of isolating the various programs that contribute to a given Result and determining how effective they are in achieving their goals. The Results First initiative discussed below is a promising model for this type of evaluation, but implementing such a process would be a long-term goal. STEP 5: COMPARE SCORES BETWEEN PROGRAMS A key goal of Budgeting for Results is to inform budgetary and programmatic decision making. BFR should allow the state to invest resources in programs found to be effective in achieving desired outcomes, and inform policy decisions about programs that are not as effective in producing results. STEP 6: ALLOCATE RESOURCES Once evaluations are conducted and programs scored, a process must be put in place to assign funding allocations based on available resources, established priorities, and performance toward goals. STEP 7: CREATE ACCOUNTABILITY FOR PROGRAM RESULTS The ultimate success of Budgeting for Results depends on the extent to which it is used to inform decision-making. BFR must be fully integrated into the state budgeting process, and officials must carry out their decision-making responsibilities in a way that is consistent with BFR principles. Further, because BFR is an ongoing process rather than a one-time event, it is critical that agencies be held accountable for performance year round, not just during budget development.
- 74. 74 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS BFR IMPLEMENTATION Implementation of BFR requires that GOMB create the structure and systems necessary to fully integrate the above steps into the budgeting process. The initial focus of the BFR Team has been to lay the foundation for this structure. GOMB has implemented Steps 1 – 3: developing a process to identify available resources, establishing the seven priority areas, and building a program inventory, and has begun building IT infrastructure, and developing tools to evaluate program performance. While this work must continue, GOMB must also begin building the structures for the remaining steps in the BFR process. DEVELOP AND REFINE GOALS/SUBGOALS In a BFR process, budget allocations should be based on the success of programs in contributing to the measurable achievement of established goals. Thus, measurable goals and sub-goals must be developed under each of the seven priority areas. There are certain principles that should be kept in mind when developing performance goals. • Quality over quantity. Performance goals should be relevant to the core mission of the program and to the result the program is intended to achieve. This generally argues for quality over quantity, with a focus on a few good measures. However, programs should not feel compelled to collapse complex activities to a single measure, particularly if that measure is a proxy for the true objective. • Importance to budget decisions. Performance goals should provide information that helps make budget decisions. Agencies can maintain additional performance goals to improve the management of the program, but they do not need to be included in the BFR process. • Public clarity. Performance goals should be understandable to the users and the public of what is being measured. • Feasibility. Performance goals should be feasible, but not the path of least resistance. Choose performance goals based on the relevancy of the outcomes and not for other reasons – not because you have good data on a less relevant measure, for example. If necessary, terminate less useful data collections to help fund more useful ones. • Collaboration. Agencies and their partners (e.g. grantees, contractors) need to work together in developing meaningful, measurable performance goals. For the FY ’13 budget, the BFR implementation team, in collaboration with the BFR Commission, state agency staff and the Governor’s Office, developed a preliminary list of goals and subgoals for each of the seven priority area. Each agency was asked to review this list and identify which goals and subgoals their programs helped to achieve. The goals, and associated state programs, were included in the FY’13 budget book. The BFR Team is pursuing several strategies to refine these goals for FY’14. STRATEGY MAPPING The BFR Team is working with the Government Finance Officers Association to conduct a Strategy Mapping Process to refine the state’s goals and subgoals based on the five principles mentioned above. Strategy Mapping is a process used in Budgeting for Results to identify the underlying causes or factors that drive the strategic goals of an organization (in this case, the State), and inform decisions about what investments to make to achieve those goals. This process allows decision-makers to think about which activities drive desired outcomes rather than how current activities fit within established goals.
- 75. 75 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS There are three main steps to strategy mapping 1. Identify desired outcomes/results; a. Illinois has seven established Result areas; 2. Create Strategy Mapping Teams around each Result; a. Teams should be no more than 7 – 10 people, and should include “thought leaders” as well as subject matter experts 3. Conduct a cause & effect analysis for each Outcome; a. Define the Problem b. Identify factors that drive desired Outcome c. Gather Intelligence/Research/Best Practices d. Identify which activities can drive desired result e. Determine which of those activities are/should be a function of State Government f. Prioritize those activities when establishing Goals and Subgoals. This process will result in a strategy map for each priority area that includes primary and secondary causal factors, a small number performance indicators for each goal, key strategies for pursuing each goal, and a narrative description of the cause and effect analyses that produced each map, including sources of evidence. By looking broadly at the factors that drive a desired outcome, Strategy Mapping holds promise for helping to break down programmatic silos and address concerns about state agencies being held accountable for outcomes when they don’t control all of the factors that drive a given outcome. LOGIC MODELING Logic modeling is a system of conceptualizing a program and displaying it visually, which allows for a better understanding of how a program works. Logic modeling asks basic questions about a program, such as: Who or what is the program intended to impact? What are the immediate intended results of the program activities? What are the outcomes generated from program activity? Logic modeling clarifies the necessary components that allow a program to function and simplifies development of performance measures. The goal is to complete the logic model training program by June 2012 and begin training agency staff by the end of that month. All agencies will be asked to complete a rudimentary logic model of each of their programs by October 15, 2012. GOMB will provide technical assistance to agencies in completing this task.
- 76. 76 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS DEVELOP BFR METRICS State agencies collect a vast amount of data, but current metrics tend to measure outputs rather than outcomes. Under BFR, it is the State’s intention to develop program metrics that will measure the impact of programs on achieving their stated goals. Through the process of completing logic models, agencies will have identified the various components that comprise each program, including resources, activities, and the various output measures that are currently collected. With that information in hand, consultants will engage with the agency program staff to identify new measures that will more accurately reflect the program’s progress toward meeting the state’s prioritized outcomes. This review will be in-depth and program specific, and as a result will take multiple calendar years to complete. We also recognize that this process will be more complex for those agencies that accomplish their work through providers, contractors or grantees, and will need to gather data for performance metrics from these external entities. Agencies will be directed to establish a public process for gathering input from providers and stakeholders in developing performance metrics and making refinements as needed. This process would be greatly aided by consultants with the technical expertise to assist in the development of measurable, result-oriented metrics as well as IT consultants to assist with agency implementation of the data collection and aggregation/analysis of metric data. Once new result-oriented performance measures are established, the data collected will be entered into the Budgeting for Results Performance Reporting data system, where it can be analyzed and used to inform policy decision. We understand that state grantees and providers currently track and report a variety of metrics for their state, federal government and foundation-funded programs. We are sensitive to the administrative burden these reporting requirements place on grantees. In BFR our goal is to make sure state agencies are measuring what matters. State agencies will be asked to review the data they collect, streamline reporting requirements and eliminate duplicative or otherwise unnecessary metrics that do not contribute to evaluating program outcomes. EVALUATE PROGRAMS There are hundreds of state programs, and it will not be feasible for GOMB or agencies to evaluate them all in a single year. GOMB currently prioritizing programs for and developing a timeline for completing evaluations. Once this is complete, we will need a process for compiling the data into a user-friendly format and utilizing it to evaluate program performance and make policy decision. STRATEGIC MANAGEMENT ACCOUNTABILITY REPORTING TOOL GOMB, with assistance from Patrick Mullen, Ph.D., Assistant Professor of Public Administration and Policy at the University of Illinois Springfield, has developed a Strategic Management Accountability Reporting Tool (SMART) to access performance of state programs. SMART is based on the Program Assessment Rating Tool (PART), which was developed by the federal Office of Management and Budget and the U.S. General Accountability Office (GAO) to assess the effectiveness of federal programs. SMART is a 10-part questionnaire that will be completed by GOMB analysts with agency input. The questionnaire asks weighted yes and no questions about program metrics, inputs, outputs and impacts. The resulting scores are tallied and the program rated on its effectiveness. The SMART questionnaire will be finalized by May 2012. GOMB analysts will then need to be trained in the SMART begin conducting SMART evaluations in calendar year 2013. Once complete, SMART evaluations will then be used to help determine resource allocations for the Governor’s budget proposals.
- 77. 77 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS PROGRAM SCORING TOOL Through BFR, the state will not only evaluate programs against established metrics but also against similar programs in order make informed decisions about which programs are most effective in achieving the desired outcomes. The SMART scores will be one component of this evaluation but a scoring system must also be developed to account for expected return on investment as well as policy and legal considerations. Once the programs are scored, the scores will be compiled, revealing a top-to-bottom comparison of programs that can be used to allocate resources. It will be necessary to engage consultants with the technical expertise to develop scoring criteria that can account for and appropriately weight program effectiveness, mandatory/statutory activities, policy priorities and return on investment. CREATE PERFORMANCE MANAGEMENT PROCESS Accountability for performance is a key tenet of Budgeting for Results, and represents a significant culture change not only for state agencies but also state leadership. It is important to begin fostering a performance culture as soon as possible. Beginning in July 2012, GOMB analysts, Deputy Chiefs of Staff, the Governor or his designee and the Chief Performance Officer will conduct regular Budgeting for Results meetings with agencies to review agency expenditures and performance toward key metrics and goals. These meetings will also provide a forum for refining goals/metrics and tracking agency-level implementation of budgeting for results. GOMB is in the process of developing a Performance Reporting System (PRS) to begin collecting the data needed to inform Budgeting for Results. Current performance reporting solutions lack sufficient analytic capacity and user-friendly interface to meet BFR needs. In order to move forward, GOMB determined it would need to develop and implement a new performance reporting solution. The PRS is one step toward that solution. GOMB is also developing a performance management website, modeled after Virginia Performs, to ensure metrics are transparent and accessible to the public. The website should launch by June 2012. Initially the information provided on the website will be basic, however the intent is to begin populating it with metric data from the PRS once that system is up and running. RESULTS FIRST Illinois is partnering with the Pew Center on the States in their Results First initiative. Through this initiative, Pew has helped states assess the costs and benefits of policy options, and use data to make decisions based on expected return on investment. Results First is based on four elements of policy decision-making: • Use the best information. Pew helps states calculate costs and outcomes associated with various policy options. • Design policies that work together as a total package. Pew helps policy makers quantify how investments or cuts in one program affect other costs and outcomes. • Learn from other states. Pew provides information about proven practices and tools, as well as real-world approaches that other states have used to identify and build support for policies that get the best value for the investment. • Create the climate needed to make decisions based on results. Pew helps state leaders build the political will both to adopt policies and programs that are most likely to produce the greatest success in the most cost-effective way and to reform or eliminate those that are not. Participating in Results First will give Illinois access to additional resources that could help make our BFR process more robust.
- 78. 78 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS MESSAGING AND PUBLIC ENGAGEMENT One of the many benefits of BFR is its potential to help the public understand the value they get for their taxpayer dollars. Achieving this requires a robust communications and public engagement strategy. The BFR Team has worked throughout the past few months to engage interested stakeholders and ensure they have a voice in the process. The BFR Commission held two public hearings and public comments/testimony received was included as an appendix in the BFR Commission’s report. In addition, the BFR Commission Chairman, Sen. Dan Kotowski, and Governor’s Office representatives have met with a variety of stakeholders throughout the BFR process. However, we recognize that there is much more that needs to be done, as the work of developing metrics and evaluating programs moves forward and communicating the performance and outcomes of programs once data is collected and evaluated. The foundation community has expressed interest in working with the state and the BFR Commission to develop a public engagement process for Budgeting for Results. We envision that this would include a website, public meetings, focus groups, and direct communication strategies such as public service messages. OPERATIONAL CHALLENGES AND NEEDS Budgeting for Results is a fundamental change in the way Illinois develops its budget, manages its agencies, and makes decisions about allocation of resources. It will impact all levels of state government as well as organizations that receive funding from state government. Implementation is a major undertaking for the State and there are several challenges that must be recognized and addressed as we move forward. While the end-product of a results-based budget is a user-friendly document that simplifies spending decisions, implementation and management of a BFR process is a complex undertaking that cannot be accomplished in one budget cycle. As discussed in this document, BFR will require changes in IT, procurement, budget and grant/contract management processes across state agencies and numerous state-funded entities. It is important to manage expectations as BFR rolls out, both for proponents who want to see this process utilized more quickly and those concerned about what it will mean for their programs/funding streams. The state must develop a clear message and communications plan for the public as well as a robust community engagement process. STAKEHOLDER/PROVIDER ENGAGEMENT Stakeholder “buy-in” will be vital to the success of budgeting for results. It will also be important to manage expectations as BFR rolls out, both for proponents who want to see this process utilized more quickly and those who are concerned about what BFR will ultimately mean for their programs and funding streams. Because any decision to reduce or eliminate program funding is controversial for those who have a vested interest in those programs it is important that the BFR process have enough credibility to withstand the inevitable resistance. Therefore we will need to strike the right balance between engaging stakeholders vested in specific policy areas and independent experts in developing goals, metrics and evaluation or scoring tools. While the BFR process focuses on how state resources are allocated to state agencies, much of the work of state government is done by private entities under grants and contracts. Thus, in order to evaluate the effectiveness of many state programs, an agency must be able to evaluate the services provided by private grantees or vendors.
- 79. 79 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS STAKEHOLDER/PROVIDER ENGAGEMENT For this reason BFR will need to be implemented not only across state government, but across the thousands of grantees and contractors that administer public programs or provide direct services on behalf of the state. There will need to be a mechanism for these entities to report performance metrics to the state, and for the state to evaluate their performance against other service providers. Therefore, Budgeting for Results assumes an increased use of performance-based contracting, where contracts outline specific outcome measures and payment depends on vendor performance in the achieving those outcomes. Performance based contracting will inject accountability into the process by enabling agencies to make informed decisions within their allocated budgets. By incentivizing efficiency and rewarding performance, BFR and performance based contracting could enable agencies to reduce budgets without compromising services or reducing quality. While some state agencies have made great strides in implementing performance-based contracts, many of these contracts are based on output measures rather than true outcome measures. There is a need for significant education and training, of both state agency staff and contractors, for successful implementation of performance-based contracting as it is envisioned for BFR. GOMB will need to engage consultants with the technical expertise necessary to help agencies develop and implement performance-based contracts. Resources are also needed to do sufficient outreach and education to the numerous providers and contractors that will be impacted by the shift from traditional to performance- based contracting. TECHNICAL EXPERTISE Since Illinois is new to Budgeting for Results it is reasonable to expect that state agencies do not have the internal capacity needed to fully implement a BFR process. We have highlighted in this document areas where additional resources or expertise may be needed. The Governor’s Office and GOMB is specifically seeking partners with expertise in performance measurement, program evaluation and logic modeling assist with the following tasks: • Strategy Mapping • Metric development • Technical Assistance to agencies in developing provider level metrics for performance-based contracting • Developing program evaluation and scoring tools • Staff Training; and • Performance Management, including a more focused and effective agency management process
- 80. 80 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS INFORMATION TECHNOLOGY An integrated data collection and analysis system is needed to make BFR a success. In order to conduct the necessary performance measurement and program evaluation, the BFR implementation team will need to collect vast amounts of data from across over 50 state agencies, boards, commissions and vendors. Once the data is collected it will need to be analyzed and converted into useful, information. Currently GOMB is attempting to build out a SharePoint-based data gathering solution that will allow for better collection and analysis of data. However, additional capacity is necessary to accomplish the broader goals. In addition, the BFR implementation team requires assistance acquiring software solutions for data visualization and analysis to support BFR implementation in the short-term. In the long-term, we will require assistance in establishing a more robust data gathering and analysis system capable of gathering data from multiple programs from within multiple agencies, each with their own esoteric data formats and system requirements. This is the key IT challenge associated with BFR GOMB will need to engage IT consultants to help complete the BFR Performance Reporting System, and conduct user training. GOMB expects to complete development of the BFR PRS by July 2012. Data currently on the GOMB PRS will be transferred to BFR PRS. Agency staff will be trained on the new system between July – September 2012. By November 1, 2012, all agencies that currently report metrics via the GOMB Performance Reporting System will report metrics via the BFR Performance reporting system (BFR PRS). CHANGE MANAGEMENT Change Management is a critical component for ensuring successful implementation of any major organizational change. In addition, successful BFR implementation requires a culture change in how the budget is developed, and how appropriation decisions are made. It is important for Illinois to build the political will in both the Executive and Legislative branches to fund programs that produce the greatest success in the most cost-effective way and to reform or eliminate those that do not. Because BFR will impact employees throughout state government, it will be important to address the “human side” of this change to minimize resistance. Staff must understand and feel comfortable about what is happening, why the change is needed and what it means for them. It is also important that staff see high-level support for BFR, from the Governor and his leadership team to agency Directors and “champions” at all organizational levels. Government often struggles with change management and we would benefit from assistance in this aspect of the BFR project. CONCLUSION Governor Quinn is committed to improving transparency and accountability in state government, and has embraced Budgeting for Results as a key strategy toward achieving that goal. BFR represents a significant opportunity for the State of Illinois to restore public trust by helping people better understand how taxpayer dollars are spent and the return on investment of that spending.
- 81. 81 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS APPENDIX I: BFR TIMELINE/WORKPLAN Below is a timeline for Calendar Year 2012 implementation of Budgeting for Results. BFR is a complex process involving both systems and culture change. As a result, full implementation will take several years. This strategic plan will be updated on an annual basis as we continually evaluate where we are in the implementation process and what steps and resources are needed in the upcoming year. APRIL 2012 • Agencies complete self-identification of programs • Procure vendor(s) for Strategy Mapping and Logic Model Training • Conduct Agency Director and Senior Staff training on BFR MAY 2012 • SMART questionnaire finalized • Finalize logic model training program for agency staff JUNE 2012 • Begin training agency staff on logic modeling. • Begin Strategy Mapping Process • GOMB reviews list of programs submitted by agencies and verifies they are appropriate and amenable to measurement. • GOMB completes development of the BFR Performance Reporting System. • GO and GOMB begin BFR Performance Management meetings with agency leadership using goals identified in FY’13 Budget Book. JULY 2012 • Data currently on the GOMB PRS transferred to the BFR PRS • Agency staff trained on the BFR PRS • Complete Strategy Maps OCTOBER 2012 • All agencies complete rudimentary logic models of each of their programs. • GOMB sends FY’14 Budget Instructions to Agencies • NOVEMBER 1, 2012 • All agencies currently reporting metrics via the GOMB PRS will report metrics via the BFR PRS. • GOMB compares performance metrics to inform FY’14 proposed budget. • GOMB SMART training. CALENDAR YEAR 2013 AND BEYOND • Data from logic modeling used to develop program metrics • GOMB SMART implementation. • Agencies conduct stakeholder engagement process on program metrics • Develop Balanced Scorecard or other performance management tool using updated metrics • Agencies begin reporting on new metrics • Allocate resources for FY’14 proposed budget
- 82. 82 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS SELECTED AGGREGATED SURVEY DATA FOR THE BUDGETING FOR RESULTS/DONOR FORUM PROJECT JANUARY 2013
- 83. 83 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS WHAT IS YOUR UNDERSTANDING OF BUDGETING FOR RESULTS? • BFR is a tool for government agencies to set priorities according to goal achievement in the services provided so that funding can be given to the services that reach the best results. • That an organization sets up the annual budget to support the outcomes you want to achieve. • BFR is intended to identify spending priorities by adopting performance based budgeting. • Budgeting for Results is the state efforts to set funding priorities and put limited funding into programs and services that work • Tying money to outcomes • Meeting the bottom line while meeting program objectives. • "The National Advisory Council on State and Local Budgeting (NACSLB) has identified four essential principles of effective budgeting. The specific principles include: (1) set broad goals to guide decisions, (2) develop strategies and financial policies, (3) design a budget supportive of strategies and goals and (4) focus on the necessity of continually evaluating a government’s success at achieving the goals that it has set for itself (i.e., performance). • Budgeting For results means strategically investing financial resources to most cost-effectively produce the desired outcomes. • Tying financial resources to positive outcomes. • Funds received from the state will be tied to outcomes and impact results • Looking at funding and performance based measures • From my understanding Budgeting for Results have something to do with the budget and having available dollars to the most significant programs and activities. • Determining how most effectively to use the funds to best meet the needs of the students/families and school. • Governor Quinn signed into law Budgeting For Results (BFR), an historic spending reform act requiring the state of Illinois to institute a results-based budgeting process that will end the automatic funding of programs. By requiring the State to live within its means and focus on performance, BFR will transform the way that state officials, legislators – and the public – prioritize, think about and implement the State’s budget. Going forward, the State will fund only those programs that can demonstrate effectiveness and help the State achieve its stated outcomes and goals. • Unknown • Using funds wisely to see outcomes • A strategic alternative to incremental budgeting, which resources are allocated based on how effectively programs or service achieves established goals and objectives. • I am unfamiliar with this. • State budget resources allocated based on effective a program or services meet goals and objectives. Have not had any formal training on BFR. Although we do our very best to maximize every dollar using volunteers, in kind contributions, it's hard to budget for results when the payments you rely on are severely delayed. It's hard to expect the maximum results under these circumstances when every program that yields the best results relies on a steady cash flow.
- 84. 84 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS HOW WOULD YOU DEFINE POSITIVE YOUTH DEVELOPMENT? • Positive youth development is defined by us is a holistic development of youth, not only academic +- excellence, but development in leadership skills and civic engagement • A process necessary to ensure that youth are able to make and execute decisions that lead to positive life outcomes; additionally to assist youth in developing the necessary skills to be prepared for adult life • Positive youth development is a framework that fosters internal and external protective factors to moderate risk factors and enhances optimal physical, cognitive, social and emotional development. • Giving young people every opportunity to discover and grow skills and talents in a supportive environment with positive role models and adult support • Positive youth development is efforts (activities/interventions) to support youth in achieving their dreams and becoming contributing members of their community. It emphasizes the importance of focusing on youths’ strengths and providing opportunities to help others through service. • Built upon YPQA, program quality pyramid, 40 developmental assets • NA • "PYD emphasizes the importance of focusing on youths’ strengths instead of their risk factors to ensure that all youth grow up to become contributing adults." Many researchers have looked at this and have created frameworks around it. It relates to our programming because much of our prevention programming is emphasizing strengths and building capacities in order to mitigate risk factors that exist in their communities, their homes, or even their physical predisposition. • Positive Youth Development is a process which prepares young people to meet the challenges of adolescence and adulthood through a coordinated, progressive series of activities and experiences which help them to become socially, morally, emotionally, physically, and cognitively competent. • Providing strength based framework to foster kids’ growth. • Positive Youth Development involves building on assets youth or communities posses to either prevent risky behavior or to intervene at an early stage in risky behavior among youth. • PYD is a tool best utilized to develop youth leadership, character and living skills to become productive and successful members of society • Youth making the necessary changes to improve decision making. • I was once a teenager and I cut school, talked back and broke the rules but I turned out ok. In the afterschool program we have six core components; academic assistance, life skills, mentoring, recreation, parental involvement and service learning. Every day we work on those core components other than the Parental involvement part. I would define positive youth development as making a positive difference in the youth’s life. • Youth showing positive behavior to school, other students and a positive attitude toward their role in the community. They also have a perception about their future success and how education plays an important part of the success. • Positive youth development refers to intentional efforts of other youth, adults, communities, government agencies, and schools to provide opportunities for youth to enhance their interests, skills, and abilities. PYD suggests that helping young people achieve their full potential is the best way to prevent them from engaging in risky behaviors. • Improve ability to achieve normal developmental milestones. • Youth that has access to support systems, home, school, friends, family, community, etc • Positive Youth Development in Youth Service program is used to focus on the youth's strengths in order to provide the necessary assistance needed for youth to achieve his/her objectives and goals in order become productive individuals. • We focus on the assets/skills/strengths of the child/youth, and we work to build on those. • PYD is a policy perspective that emphasizes providing services and opportunities to support all young people in developing a sense of competence, usefulness, belonging and empowerment.
- 85. 85 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS HOW WOULD YOU DEFINE POSITIVE YOUTH DEVELOPMENT? • Search Institute tells us the best way to achieve positive youth development is by following the 40 developmental assets, the building blocks of healthy development. However to define, Positive Youth Development is the facilitation, support, and nurturing of young peoples growth through dependence to interdependence, having strong support systems in place, encouraging their personal and social development and enabling them to have a voice, be active, engaged, creative and responsible for their own development, and to allow them influence and a place in their communities where their positive development can be recognized and appreciated
- 86. 86 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS HOW WOULD YOU DEFINE OUTCOMES IN YOUR PROGRAMMING? • Outcome is gauged by youth graduating from high school, entering post secondary education or involved in vocational skills training that can lead to sustainable permanent employment and self sufficiency • A change in behavior or attitude as well as accomplishing a possible life altering achievement such as graduating from high school or college • Outcomes are specific short and long-term results/changes for individuals, groups, communities, etc. stemming from programs or services. • Measurable and sustained growth in student behavior and attitude. • Outcomes are the end results of the inputs and outputs of the program/service intervention. • Youth committed to learning, positive character development traits and leadership skills, youth demonstrate life and work skills, program engagement • Outcomes are youth transitioning to independent living after completing and "working" their life plan. • Based on the logic models that we develop, there are theories that form the basis for activities within programs. These in turn are tied to outputs, which could be things such as attendance in programming activities, # of participants that attend workshops, etc. These outputs affect what we expect to be the results of the program activities otherwise known as the outcomes. Since we have programming across the continuum of prevention to treatment, each of the logic models reflect outcomes that are expected within those programs. Outcomes for youth prevention reflects things such as improvement in knowledge of substance use/abuse and the effects, improved attendance at school, improved peer relationships, ability to define a post-secondary plan, ability to create a safety plan for themselves and within their communities, decreases in suspensions and expulsions, decreases in physical fighting among peers, improvements in family functioning, improvements in time management and organization skills. • Outcomes are the durable, long-term impacts that programs have on youth. • Outcomes are measurement tools we utilize to measure our effectiveness. • Measuring area of impact brought about through the programming we provide. • Reduced occurrences of drug use and new arrests. • Our outcomes have been making sure the youth are safe at all times during out program, boosting their self-esteem and confidence. We may not make a difference in each youth’s life but we strive to make a difference in their lives. We have 70- 75 youth that attend the after school program and we feel that we are doing something right because they could be doing other things. • Students’ behavior both at school, in the community and with their families is superior to that of the students that do not attend the after school program and their grades improve. • Outcomes are related to the impact that our services have had on changing the behavior and lives of our clients. • The number of youth who are returned to their home rather than become wards of the state. • Good • Youth Services define outcomes as means to determine the level of success in achieving the overall goals and objectives of the program, the changes that are expected to occur as a result of services provide • For us, outcomes are behaviors that exhibit growth toward our goals. • Positive changes which occurs in program participants as a result of their participation in our programming.
- 87. 87 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS HOW WOULD YOU DEFINE OUTCOMES IN YOUR PROGRAMMING? • Measurable deduction of juvenile arrests and recidivism from a restorative prospective (i.e. station adjustments, peer court/juries). Measurable decreases in youth going to Juvenile Detention Center. Decreases in expulsions, and suspensions. Increase in accountability to community by offender. Increase in community based restorative initiatives and practices. • The benefit for participants during or after their involvement with a program
- 88. 88 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS PLEASE LIST THE MAIN OUTCOME AREAS RELATED TO YOUR PROGRAMMING AREA • High school diploma 2. Post secondary education acceptance letter 3. employment letter • Academic outcomes Life outcomes Prevention outcomes • Reduce childhood obesity, reduce the achievement gap, increase kindergarten readiness, increase school attendance, increase academic achievement, reduce school-based behavioral incidents, reduce gang involvement, reduce juvenile recidivism, foster safer communities, increase child safety, etc. • Reduction of problem behaviors (substance abuse and pregnancy rates, mental health crises, juvenile arrests, etc) • xxxxxxx • This varies on specific program interventions but would include: passing - onto the next grade/graduation; successfully living independently (for a year); substance abuse free (for a year); arrest free (for a year) • Youth committed to learning, positive character development traits and leadership skills, youth demonstrate life and work skills, program engagement • Outcomes are youth transitioning to independent living after completing and "working" their life plan. • Social emotional changes, academic data including behavior, grades, attendance, graduation, matriculation, etc., changes in Child and Adolescent Global Assessment of Functioning (CGAS) and Daily Living Activities (DLA). • Youth who complete high school prepared for secondary education and/or a career - youth who have the communication and problem-solving skills to conscientiously identify and address challenges in their communities -youth who have developed the behaviors and attitudes to become healthy, responsible, self-reliant productive citizens and community members • Number of youth going into DCFS care Number of youth sentenced to DJJ Number of families remaining intact § Youth engagement 2) Academic achievement • Increased abstinence, decreased legal involvement • One of our core components is Parental involvement, if we can't get them to come out, we offer free computer classes, resume building classes, adult sessions once a week from Southern Illinois Healthcare Foundation. Our staff is very active in the schools that the youth attend making sure they are in a caring school environment and if now we make a referral to another department in our agency. Our outcome area is to meet the needs of the youth that we serve and their families. • Academic, life skills, attitude toward education, more and improved and more career choices, recreation, cultural and social progress, health and nutritional meals. • Consumer behavior, school complaints, and overall improvement • Number of youth who return home 2) Number of youth who become wards of the state 3) improvement in family relationships • Lower drug use a. Fewer program participants will be involved in violent and dangerous activities b. Parents will improve their parenting skills c. Youth will remain at home and in school d. truancy level among program participants will decline • Overall, in our entire program: - academic retention and achievement - development of career aspiration/choices - development of soft jobs skills - increase decision-making and problem- solving skills - increase anger-management and conflict resolution skills - increase choices in favor of physical fitness and nutrition - increase engagement with the community
- 89. 89 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS PLEASE LIST THE MAIN OUTCOME AREAS RELATED TO YOUR PROGRAMMING AREA • Organize communities to address community needs; advocate for change(s) and programming to address needs; link and or deliver direct services to youth and families to address the need; and collect and uses data ( quarterly reports. indicators and outcomes ) to demonstrate efficacy of the program. • Reduction in juvenile delinquent behaviors.. Reduction in youth entering Juvenile Detention. Reduction in expulsions, suspensions from school. Increase in positive youth development. Increase in positive recreational, cultural, and educational activities and employment opportunities for youth. Increase in alternatives to suspensions, expulsions, and juvenile detention. • Students improve school performance Students graduate from high school Students enroll in post-secondary education Improved social skills and interpersonal behavior
- 90. 90 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS WHAT ARE THE IMPLICATIONS OF BFR FOR YOUR ORGANIZATION? • BFR means we have to justify our budget requests based on the results of our services. The results are based on the data we collect. So we have to have very good data collection methods. • Sometimes leadership uses it to reduce in areas that are not a priority or preference. If you use BFR there will be less of that. • Uncertain at this point • We must be able to demonstrate successful program outcomes to secure and maintain funding to services • It is good to show taxpayers that their money is being used for important projects as I have seen a lot of waste in the past. Funded agencies need to be accountable for how they use money. All non profits should have an outcomes measurement process. If not, they should not be funded (or provide funds to create one). Hoping that BFR will be in line with current research/thinking around outcomes from existing sources and not created from scratch as that could lead to a separate outcomes process for organizations and create inefficiencies. For example, there is a lot of work being done around social emotional learning - these concepts should be used in BFR because most organizations that have an outcomes process will also use these concepts. • Unknown at this time. • We receive funding from entities involved in this process in Illinois and recognize that funding will be more likely tied to priorities and results from those priorities. We are at the table with those making the recommendations for priorities and results. • BFR for our organization would require a thorough analysis of each department's budget in comparison to the produced outcomes and then an agency-wide budget -outcome analysis to find out which programs and services are most cost-effective and which are least effective. We would then need to use that data to decide which programs and/or services to continue, to expand, or to discontinue. • Not sure. • More resources to build up our technology capacity and capability. • Improvement in program performances • I am working the Teen-REACH after school program and in that component we do have a budget but that is basically handled through the Management office. • A quality program is the result of using funds that are assisting students in making better decisions for their future and showing improvement in their educational decisions and their positive role in the community. Career choices are also an intricate part of their future success. • We are required to use evidence-based, empirically validated interventions. • Unknown • None • To prioritize and focus on quality service that will bring about long-term results • Will assist our agency to better empower our clients. • I'm not sure since I have not been educated on the concept of BFR.
- 91. 91 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS WHAT TYPE OF SUPPORTS (OR CAPACITY BUILDING) WOULD YOUR ORGANIZATION NEED TO SUCCESSFULLY MEET THE OUTCOMES IN BFR? • We find that ETO is a good tool for our collecting data to evaluate our outcome in programs. However, we are only using ETO in certain programs, we need further assistance, technology and financial, to implement it in other programs in the agency • Training for our admin team and understanding of the process • Need more information • Funding to maintain contact and track long term outcomes of youth after they are no longer receiving program services • Funding to support a data manager/outcomes specialist. Training on managing data. • Unknown at this time. • Access to institutional data (school data) and aligning systems with better data integration such as DMH, WIIA, DHS, CDBG. • Our organization would require funding for and/or an assigned evaluator who has experience in strategic financial planning and operations to help determine whether the data we are collecting is accurately measuring what we desire and, if it isn't what data we should collect, help us develop the tools to collect the data if we do not already have them, and assist the PQI department and department leaders in analyzing the data that is collected. • Not sure. • More resources to assist with building our technology capability. • Not sure at this point • Fundraisers, training centers, consultants, ministers throughout the Illinois area working together, local authorities, St. Clair County Housing Authority. Some of the staff might need to acquire some skills, find resources, and work as a team for the same vision. • An improvement in our obtaining professionals and programs to support our life skills i.e. anti- drug/alcohol education for the students. • Cooperation of schools, parents, and the school board. • Unknown • Keep tracking system updated • Staff training to understand the concept in order to provide quality service that delivers the intended results. • As previously described and additionally training, technical assistance in constructing appropriate outcomes, metrics, and sufficient technology. • Again, I can only imagine having the software and the education to implement BFR
- 92. 92 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS WHAT IS POSITIVE YOUTH DEVELOPMENT? Taken from the Positive Youth Development Resource Manual by ACT FOR YOUTH UPSTATE CENTER OF EXCELLENCE, 2006 Click here to download Resource Manual. Youth development experts have not yet agreed on a clear definition of positive youth development, but there has been consensus about key components that need to be included in this approach. With this in mind, positive youth development can be described as a philosophy or approach promoting a set of guidelines on how a community can support its young people so that they can grow up competent and healthy and develop to their full potential. • Emphasis on positive outcomes: The approach highlights positive, healthy outcomes (in contrast to reducing negative outcomes such as teen pregnancy, substance abuse, violence). Although most parents have clear ideas what positive characteristics and behaviors they would like to see in their children, there is still a lack of clarity of what exactly positive outcomes are. Since researchers only recently have focused on positive outcomes, definitions and categories of positive outcomes are still evolving. Examples of desired youth development outcomes are competence (academic, social, vocational skills), self-confidence, connectedness (healthy relationship to community, friends, family), character (integrity, moral commitment), caring and compassion. • Youth Voice: It is essential to include youth as active participants in any youth development initiative. They have to be equal partners in the process. Youth involvement presents a great challenge to adults and charges them to rethink how they have engaged in planning and program development and implementation. • Strategies aim to involve all youth: Youth development strategies are generally aimed at all youth. The assumption is that creating supportive and enriching environments for all youth will lead to the desired positive outcomes as well as reduced negative outcomes. However, experts in the field recognize the need to blend universal approaches with approaches that are targeting youth facing extra challenges. • Long-term involvement: Youth development assumes long-term commitment. Activities and supportive relationships have to endure for a long period of time to be effective. They have to accompany young people throughout their growing up years. While short-term positive results may be seen and should be built on, both community-organizing models mentioned below state that positive community-based, youth outcomes may not be measurable for 15-20 years. Youth development strategies have to embrace and ready themselves for long-term engagement. • Community involvement: Youth development stresses the importance to engage the larger social environment that influences how young people grow up and develop. This includes family and friends, but also the community they live in. Community is more than social service and youth organizations, schools, law enforcement agencies; it involves business, faith and civic groups, and private citizens who are not attached to any organization. • Emphasis on collaboration: Youth development requires people from various agencies and community groups to work together. Collaboration can express itself in different forms e.g., agencies coming together to write a grant proposal to community groups forming a coalition to achieve one common goal by sharing resources and expertise.
- 93. 93 APPENDIX F –YOUTH PROVIDER FOCUS GROUP MATERIALS WHAT IS POSITIVE YOUTH DEVELOPMENT? Taken from http://www.findyouthinfo.gov/ Based on the literature, the Interagency Working Group on Youth Programs, a collaboration of twelve federal departments and agencies that support youth, has created a definition of positive youth development: Positive youth development is an intentional, pro-social approach that engages youth within their communities, schools, organizations, peer groups, and families in a manner that is productive and constructive; recognizes, utilizes, and enhances youths' strengths; and promotes positive outcomes for young people by providing opportunities, fostering positive relationships, and furnishing the support needed to build on their leadership strengths. Positive youth development has its origins in the field of prevention. In the past, prevention efforts typically focused on single problems before they surfaced in youth, such as teen pregnancy, substance abuse and juvenile delinquency. Over time, practitioners, policymakers, funders and researchers determined that promoting positive asset building and considering young people as resources were critical strategies. As a result, the youth development field began examining the role of resiliency—the protective factors in a young person's environment—and how these factors could influence one's ability to overcome adversity. Those factors included, but were not limited to: family support, caring adults, positive peer groups, strong sense of self and self-esteem, and engagement in school and community activities. Researchers and practitioners began to report that young people possessing a diverse set of protective factors can, in fact, experience more positive outcomes. These findings encouraged the development of interventions and programs that reduced risks and also strengthened protective factors. These programs and interventions are strengthened when they involve and engage youth as equal partners, ultimately providing benefits both for the program and for the involved youth. DEFINING DEVELOPMENTAL DOMAINS, OUTCOMES, INDICATORS AND MEASURES Definitions taken from “A Shared Vision for Youth” Common Outcomes and Indicators developed by the Forum for Youth Investment, the National Collaboration For Youth and Ready by 21 Click here to download a copy of FNCY Common Outcomes. Developmental Domain: Broad developmental area including multiple related outcomes Outcome: Aspect of child or youth development that programs can influence Indicator: Evidence that an outcome has or has not been achieved Measure: Specific data collection tool (e.g. survey, interview, observation protocol)
- 94. 94 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 95. 95 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 96. 96 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 97. 97 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 98. 98 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 99. 99 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 100. 100 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 101. 101 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 102. 102 APPENDIX G –FORUM FOR YOUTH INVESTMENT WITH THE NATIONAL COLLABORATION FOR YOUTH
- 103. 103 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA Focus Group Data Springfield Group 1/16/2013 DEVELOPING A SHARED DEFINITION OF POSITIVE YOUTH DEVELOPMENT Reactions to the Definitions Shared/Reactions to the survey data/Additions you would add/ Concerns/Cautions • Lot of [our] work is empowerment and not as much the language related to the materials or survey...about having youth involved in the planning of the programs • "Empowerment" loosely stated--what is empowerment really? Living wage job, going to college. Youth are saying help me navigate through my goals • Protective factors having an impact on positive. Outcomes...where do they come in at? Issue of strides--time span of change • Positive outcomes--economic/quality of life (more + with school)...economic is education and economic development DEVELOPING A SHARED DEFINITION AND SET OF OUTCOMES Reactions to the definitions of developmental domain, outcomes, indicator, measure/Reactions to the survey data in comparison to the definitions/What is missing? Developmental domains, Outcomes, indicators, measures/What guidance would you offer to the state around outcomes?/ Concerns/Cautions • We are more interested in school success/absence of delinquency (resiliency)--being more important to connect outcomes/vague stuff...Materials were mostly was about resiliency stuff...volunteering/community service measures • Too often we deal with outputs/#s--as opposed to critical areas--that is what we ought to be doing • Programs are unique across--same tool (across all sites)--reduction of arrest/# of days and involvement/grades/more they attend • Shift from subjective measures to real hard data • Change of behavior can occur-but what about sustainability of change over time--beyond the life of our program--long terms effects of programs • Band-Aid effect/hot new term impact--but what about the family context--are giving enough to get long term effects? • What about family resiliency? • What about community? What about environmental consequences? • Question of resources--to do enough at Human Services--must be realistic about expectations • CYS--goals is to impact community/youth/family/ • BFR--what should programs be doing or not doing? • Influence behavior change that is documented • TeenREACH--parent/community/youth component--not sure if that happens across all programs • Record # of youth showing up--staff having to up their professional development to measure outcomes...CYS-TeenREACH silo(ed) from each other--Huge disconnect Guided portion of the discussion to reach consensus on the developmental domains for the outcomes: Academic/Educational Success/Learning School success Graduate middle schools Longitudinal Research
- 104. 104 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA DEVELOPING A SHARED DEFINITION AND SET OF OUTCOMES Leading Civic mindedness/engagement/responsibilities/empowerment/altruism You play a role in something bigger than your Service learning project Balanced restorative justice Developing a sense of values (important)...this leads to trouble...research says about 1/3 of youth not responding Concerns that we are moving out of our area of expertise Economics/Work Not easy to tackle As we stabilize a family--they move out & they are replaced with less stable folks Connectedness Pro-social engagement All categories intertwine to make a whole you World is changing (more injustice)--but youth want us to change more and accommodate them After all of our work--do we decrease crime--do we decrease poverty? How do we improve environments? Especially given that the base amount to us s $75k versus federal programs which get $250k NECESSARY SUPPORTS TO SUCCESSFULLY IMPLEMENT BFR ORGANIZATIONALLY & FOR THE STATE Based on the material—is there enough clarity about BFR? What would help?/Reactions to survey data defining BFR?/Reactions to survey data sharing ideas about the necessary supports?/Additional supports organizations will need? From whom?/Is the state itself ready to implement BFR?/What are your concerns • We must monitor pennies/state $ is not worth paper it is written on--my board is weary/our partners are weary/state can change its mind/grants do not cover interest when we've had to take out loans to keep things going in between when monies come • The state cancelled some of our $/rolled our program into another area and then came back and asked us for a report when the program was no longer in our hands • Teen reach zeroed out/brought back/things change too much • IDHS/DCFS--youth programs --these entities should come together and see where there is duplication--we have been trying to do this on our own-----there should be 1 unified youth grant • Silo issue is huge--but not enough resources to deal with housing/placement • Not working with Commission on technology cost? • What resources are being brought to bear for this to be more effective? • How much of this is short-term as the state shifts to the health consortium as its model?
- 105. 105 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA NECESSARY SUPPORTS TO SUCCESSFULLY IMPLEMENT BFR ORGANIZATIONALLY & FOR THE STATE Focus Group Data Chicago Group 1/18/2013 Reactions to the Definitions Shared/Reactions to the survey data/Additions you would add/Concerns/Cautions • Building on assets was a common theme in the materials • What do the definitions mean? Is it getting from one to the next? Do we define the rubric? • Broad/Jargony--lots of room for interpretation--can we measure this? At what scale? What resources would be available? • Real strategic intentional thing/Academics-40 assets... o Out of school time--still nebulous. People who give $ want a #/outcomes in plan--make sense of priorities o There is enough out there/do not need to start from scratch • If state is willing to embrace Search Institute model—let's go with that • So much there —implementation is the issue • So much more being asked of us due to cuts in school/youth also want other activities than what happens in school/how do we balance this? • Theoretical (Search) give us theoretical & let us get to the concrete in our specific programs • Who does what? How do our roles overlap?-School/social workers/What in education has worked and not worked? • Holistic approach—more longitudinal • Two different state depts. working on youth—outcomes extremely different • Our internal system for TeenREACH/adolescent education/etc. goes into different databases-- makes no sense to go in so many different directions • Something more streamline/single depository • System tracking is concerning—unfriendly/ state might get a shared set of outcomes with flexibility—but a user-friendly database is the key and it works with right supports/piloted before roll out, etc. • EOT Database--there was intentionality--but stills does not work that way • eCornerstone introduced 8 years ago-glitches/ took 2-3 years to iron out glitches--only been using it for about 5 years • Early childhood programs could be helpful in the tracking area—tracking over time • March technology to outcomes—make sure we can get useful reports (this is a recommendation) • Expected to input data and not reflected in budget--connect tasks to $ • Training must be ongoing--not just at roll out • Social/emotional development should be a part of the outcomes • Balance this with academic achievement—in TeenREACH this is real high • If given Search Institute Model and then say we are going to focus on a specific # of the assets that would work
- 106. 106 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA NECESSARY SUPPORTS TO SUCCESSFULLY IMPLEMENT BFR ORGANIZATIONALLY & FOR THE STATE Reactions to the definitions of developmental domain, outcomes, indicator, measure/Reactions to the survey data in comparison to the definitions/What is missing? Developmental domains, Outcomes, indicators, measures/What guidance would you offer to the state around outcomes?/ Concerns/Cautions • If we use a common tool, but focus on a specific set of domains by program • Search Institute Asset Based Model is developmentally based and when would be fine with that if the state chose that • Academic achievement is important but should not be the #1 for our field—not "the" data point--academic engagement-measured by the students/family/school • Grades-We have a connection with schools, but hard to get data/try from parents, but not easy • If program is really focused on math/science then perhaps scores in those areas would be relevant • Many stakeholder support outcomes--not just the burden of the youth provider--most importantly the youth themselves as stakeholders--so could this happen with youth in some sustainable way? Skills/capacity to talk about which program are impactful and why?/Many programs truly involve folks internally, but what about externally? TeenREACH --State Youth Advisory Council Convening- -what about involving them? • Whatever is developed allow youth worker to be involved that reflect the continuum • Connecting early childhood to college Guided portion of the discussion to reach consensus on the developmental domains for the outcomes: Health & Wellness--Mental Health--a lot is missed here Working/Economic Development Teen program/how to work with folks/life skills/gap in knowledge--how do you gain a career? Financial Literacy--no home economics in schools anymore Leadership/Civics/Responsibility Getting involved in their community/opinions matter/service projects Developing critical thinking skills Self-control/respecting others opinion You have students dong well academically-but they are still jerks--character building "Hired for Hard Skills--Fired for soft skills" Connectedness--peer-to-peer and with adults Caregiver/parent-engagement/community engagement Family not necessarily traditional--non-traditional engagement In eCornerstone-how do you track these nuances--how do you track talking with parents for 15 mins? What about mentor/adult engagement? This is why at the state level this must be broad–To capture the complete picture. Safety & Violence We are training staff to recognize adverse childhood experiences and map out a plan to support youth. This is difficult to measure--but training is an outcome/If trained to recognize--then long down the road this will have an impact
- 107. 107 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA DEVELOPING A SHARED DEFINITION AND SET OF OUTCOMES Resiliency/Thriving Each kid brings it/shows through the areas we have outlined Showing resiliency by being successful--count on this to let them be engaged/demonstrate it Can be nurtured/supported/facilitate* The tool must be sensitive to the intersection of all of these assets/domains Visioning of for the future (ability)/kids not able to see beyond 19/20 years of age Goal setting/visioning for the future Spirituality--sense of hope We shy away from this due to separation of church and state PQA Assessment Tool--explores what staff are doing in creating opportunities for reflection/how it’s implemented? NECESSARY SUPPORTS TO SUCCESSFULLY IMPLEMENT BFR ORGANIZATIONALLY & FOR THE STATE Based on the material—is there enough clarity about BFR? What would help?/Reactions to survey data defining BFR?/Reactions to survey data sharing ideas about the necessary supports?/Additional supports organizations will need? From whom?/Is the state itself ready to implement BFR?/What are your concerns? • Theoretically no problem--but concerned that it will be used as a way to hire lowest quality--not being tied to quality/ongoing improvement • BFR is the next hot trend--want it as a tool--then game over-disciplinary action • How does this fit with differences in different orgs and their structures--line staff vs. administrative staff • Need multiple stakeholders at the table--not just administrative and veteran line staff present– we value their expertise but how is this going to affect folks down the whole line • Appreciate state admitting it does not have expertise and will partner with the providers and others on this • You find only one person at the state level to work with who you talk with about outcomes, etc. • Everybody at the state level needs to know more than we do--more than the agencies • Who is not here and why? What about quality control? What is checking data quality--perhaps establishing a peer review process • Establishing normal quality control/compliance process--reflective accountability--not quality assurance. Quality assurance would shape program quality and implementation • If there is an agreement on moving forward--what is the commitment and support to development? How do you help folks incorporate quality assurance (possibility of peer to peer site visits?) • Better training at the individual vendor level • Agreement in minimal training for YD--"Core Competencies" • Training and ongoing support-have to get buy-in at agencies with youth, staff, etc. This is just always added to someone's job--while they are still doing their core job • Staff must be resilient/trained/kept going so they can pass this on to the youth • Program evaluation--just going for an hour-long site visit-reviewing data--is not enough...Maybe if a site ranks low--you partner them with another agency to grow them. (necessary in specific areas of growth, but for just support as well)--becomes an innovative way to get bigger bang for the buck for the youth • Money must come along with this or we are disinvesting in youth--shrinking $ • BFR becomes a pretty excuse for slashing the orgs--there must be a budget to implement/must invest more
- 108. 108 APPENDIX H –YOUTH PROVIDER FOCUS GROUP DATA NECESSARY SUPPORTS TO SUCCESSFULLY IMPLEMENT BFR ORGANIZATIONALLY & FOR THE STATE • State must make in-roads to punctuate FED$--in early childhood--Fed $ are creating hubs regionally and nationally/ nothing like this happening in PYD at the national level/This ices out some of the smaller orgs who can be very innovative--must intentionally include smaller and larger organizations • BFR would be great if not connected to the budget/we are all on the same team--let's elevate the reflection/talk with one another/innovate when tied to $/$ create competition • Design gird up/support groups with smaller orgs in different areas and support collaboration • In certain IDHS divisions {prevention models} that are good--state should take a look at what worked internally and learn and use that • Community Youth/ Center-based/ School-based youth programs are all different and the locations are different --the context varies and has to be factored into the equation • Maternal Infant Early Childhood Home Visiting Program--is an example of a cohort based support model ADDITIONAL THOUGHTS OR CONCERNS ABOUT BFR • For profit businesses need to be brought to the table--they have a stake in this too • Concerned about the state factoring in that we serve the hard to serve/at risk populations • They should look at NWEA-MAP (Long ranging-elementary to high school educational tool) has benchmarks/growth over time/breaks each domain down • Creating incentives for folks to get into policy work--practitioners--need to share info/acknowledge organizations growth