This is a recent powerpoint presentation of the Interim Evaluation Findings for ECHO, a program that develops self-esteem and resilience in youth through training and support of the adults that interact with them.

1. Presented by:
Ron Schack, Ph.D.
The Charter Oak Group, LLC

2. Purpose of Evaluation
 Collect formative process information for early and
continuing improvement of the model;
 Collect fidelity to model information for different
implementation sites
 Collect information on “how well” the model is being
implemented (fidelity to model, commitment, use of
different components)
 Collect information on whether anyone is better off
(behavior change of adults and youth)
3/16/2010 The Charter Oak Group, LLC 2

3. Results-Based Accountability(RBA) Model
 Result : Youth achieve optimal success in school and life.
 Population (Community) Indicators:
 % court involved,
 % graduating from HS,
 % employed
 Strategies:
 Increasing youth resiliency and self esteem through involvement with caring adults who convey
ECHO core messages
 Performance measures:
 % training participants rating the training as useful and relevant
 % students demonstrating understanding of ECHO core messages
 % students/athletes increasing resiliency,
 % students/athletes increasing self-esteem,
 % students/athletes reporting positive changes in teaching or coaching behavior,
 % students with good school attendance,
 % students with no post-program suspensions/expulsions
3/16/2010 The Charter Oak Group, LLC 3

4. Evaluation Methodologies
 Observation
 Participant Training Surveys
 Athlete/Student Pre-Post Surveys
 Outcome Measures
3/16/2010 The Charter Oak Group, LLC 4

5. Matrix of Implementations/Methods
Training Pre-Post Coaching Other Testing on Outcome
Participant Athlete/Stude Observation Observation/F ECHO Core Measures
Survey nt Survey Protocol ocus Groups Messages
Bridgeport Summer No (was Yes, using a Yes, for Yes No No
Basketball League (and done in sample 2010
related leagues in Year1) approach
other towns)
Roberto Clemente Yes Yes No Yes No Yes
Leadership Academy
Bassick High School Yes Yes No Yes Yes Yes
South Windsor Parks No (was Post only Yes Yes No No
and Recreation done in Year (looking of
1) year to year
differences
Hartford Afterschool Yes Yes No Yes No No
Programs
3/16/2010 The Charter Oak Group, LLC 5

6. Bridgeport Summer League
3.5 3.38 3.38
3.32 3.31 3.31
3.19
3
2.5
Pre-test
2 Post-test
1.5
1
At times, I think I I feel that I have a I am able to do I feel I do not have I feel useless at I wish I could have
am no good at all number of good things as well as much to be proud times more respect for
qualities most other people of myself
3/16/2010 The Charter Oak Group, LLC 6

7. Bassick High School
Bassick Student Responses To Core Message Questions, % with Full
Credit (N=137)
Strong Postive Relationships 55.5%
Actions For Goals 59.1%
Life Goals 73.0%
Why Not Bully? 47.4%
Example of Empathy? 62.0%
What is ECHO? 91.2%
0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% 100.0%
3/16/2010 The Charter Oak Group, LLC 7

8. Roberto Clemente Leadership Academy
3/16/2010 The Charter Oak Group, LLC 8

9. Qualitative Findings
 Bridgeport Summer League
 Games were orderly
 No trash talking
 Coaches didn’t yell or single out individuals
 Youth were attentive and listened to the coaches
 Coaches
 Appreciated ECHO support
 Found it difficult to take the time to have “dedicated” ECHO moments, but
embedded ECHO principles into their coaching
 Appreciated the more systematic and organized support provided in 2009 as
opposed to previous years
 Believe that the ECHO basketball league gives hope to kids, many of whom have few
positive experiences and positive role models
 Interns
 Interns were happy to have opportunity
 Proud to be looked up to by younger kids
 Given real responsibilities
3/16/2010 The Charter Oak Group, LLC 9

10. Qualitative Findings, Continued
 Teacher Comments On ECHO at Bassick:
 ECHO puts a name to something we all do or try to do, but it is easier
when you have a framework to follow.
 ECHO messages need to be repeated and sustained; sometimes you
may “have” a kid and then lose them, and you need to reiterate the
messages.
 The “moments of greatness” stickers and certificates seem to have a
positive effect…they also allow the teacher to “remind” the student of
prior great moments.
 The monthly support is great…it helps us re-focus on ECHO and
anticipate what we will do for the core messages in the upcoming
month.
 ECHO does not seem to work as well on the really “hard cases.”
 On the other hand, if you can target some of the “trouble-making”
leaders and turn them around using ECHO approaches, a lot of others
will follow
3/16/2010 The Charter Oak Group, LLC 10

11. Qualitative Findings, Continued
Comments From Clemente Teachers and Administrators About the
Impact of ECHO (March 1, 2010):
 One teacher reported that a student in her class befriended
the student who was most often teased, causing other
students to take notice of the new friendship. The teacher
reported a subsequent reduction in the teasing.
 The librarian felt that students overall were “calmer.”
 Teachers reported that the students were eager to earn the
moments of greatness awards and were stepping up their acts
of kindness.
 Students are beginning to verbalize the core values on their
own without teacher prompting
3/16/2010 The Charter Oak Group, LLC 11

12. Summary of Key Findings
 Participant training surveys scores have improved since
early implementation; participants find on-going technical
assistance helpful and important.
 There are statistically significant increases on some
important self-esteem questions.
 Flexible adaptability of ECHO makes it more likely to
implemented; intensity of implementation matters to
results but is not the only factor.
 Continuing commitment to ECHO by executive leadership
at implementation sites, even when alternatives
exist, shows that ECHO has provided real added value
3/16/2010 The Charter Oak Group, LLC 12

13. Significant Positive Differences In Self-Esteem, Pre (Bridgeport Summer League (N=314)), Post
(Bassick (N=134) and Roberto Clemente (N=121))--Significant differences in Red
attitude Clemente Mid-Year
positive
toward
I take a
myself
Bassick Post
Summer League Pre
Clemente Mid-Year
number most much to I feel respect
I wish I
at times myself
could
more
have
for
Bassick Post
Summer League Pre
Clemente Mid-Year
of good other be proud useless
Bassick Post
Summer League Pre
that I things as I feel I do
have a well as not have
Clemente Mid-Year
Bassick Post
of
Summer League Pre
I am able
Clemente Mid-Year
qualities people
to do
Bassick Post
Summer League Pre
Clemente Mid-Year
I feel
Bassick Post
Summer League Pre
Clemente Mid-Year
good at
times, I
think I
am no
Bassick Post
At
all
Summer League Pre
1 1.5 2 2.5 3 3.5 4
3/16/2010 The Charter Oak Group, LLC 13

14. Next Research Steps
 School-based outcomes; use of comparison groups
 Observations of coaching behavior (UCONN)
 Year-t0-year resiliency scores
 Self-esteem, resiliency, and outcomes by age and type
of youth
 Changes in fitness/nutrition behavior following
implementation of health/wellness component
3/16/2010 The Charter Oak Group, LLC 14