El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514

C
El Sistema Mitchell Music Program
DRAFT
Formative and Outcome Evaluation Report
July 2014
Ann Arbor Public Schools
Mitchell Elementary School
And
University of Michigan
School of Music, Theatre and Dance
School of Social Work
School of Education
For more information, please contact:
Dr. John Ellis Dr. Michael Spencer
Associate Dean for Administration Associate Dean of Educational Program
School of Music, Theatre and Dance School of Social Work
University of Michigan University of Michigan
jsellis@umich.edu spencerm@umich.edu

El Sistema Program: Formative and Outcome Evaluation July 2014
Program Evaluation Group, University of Michigan School of Social Work 2
“If children participate in regularly and extensive high quality,
ensemble based music education, they will develop a strong sense of
personal agency, and sense of community through developing musical
literacy” (Landin, A., et al 2013).
El Sistema Mitchell Theory of Change

Executive Summary
Program Description
The El Sistema initiative is a partnership between the University of Michigan School of Music,
Theatre and Dance (SMTD), School of Social Work (SSW), School of Education (SOE), and Ann
Arbor Public Schools (AAPS). SMTD students provided after-school string instrument
instruction to 22 fifth-grade Mitchell Elementary students in partnership with Mitchell’s school
orchestra/general music teacher. Graduate students and faculty from SSW and SMTD were
responsible for designing and conducting the program evaluation. The SOE and AAPS
administration provided input and feedback on the implementation and evaluation.
The goal for the pilot program was to assist Mitchell school personnel in: 1) improving student
academic performance; 2) cultivating a strong sense of school pride; and 3) increasing family
and community engagement, while also providing SMTD students with a teaching opportunity
involving community-focused social engagement.
The Mitchell program began the week of October 21, 2013, culminating with a final performance
at the U-M Stamps Auditorium on April 23, 2014. Fifth-grade students received 75-minutes of
after-school instruction four days per week, for a total of 68 days and 85 hours of instruction.
Two SMTD graduate students in music performance and the Mitchell orchestra/general music
teacher were the lead teachers in the pilot program. Eight project teachers were hired to give
pull-out lessons beginning in mid-February. Six of the project teachers were undergraduate
music education majors and two were music performance majors.
Evaluation Overview
The purpose of this evaluation was to: 1) provide a descriptive account of the emerging El
Sistema Inspired (ESI) curriculum at Mitchell Elementary; 2) document school and student
outcomes related to music literacy, individual and collective empowerment, sense of
community and school pride; 3) document the teaching experiences of SMTD students and their
development of social consciousness; and 4) document the pedagogical approach used by the El
Sistema model to help inform the social engagement curriculum for SMTD.
A mixed-method data collection process was used to evaluate the program. Quantitative and
qualitative data collected include: 1) pre- and post-test student surveys and focus groups; 2)
teacher ratings; 3) music instructor and school principal interviews; 4) parent questionnaire and
focus group; 5) digital video recordings of large ensemble, sectionals and pull-out lessons; 6)
weekly process notes; 7) school records on student attendance, academic achievement and
behavioral conduct.

Results
Data from the video recordings and teacher interviews were used to analyze the instructional
approach used in the pilot program. The program featured many of the “distinguishing
characteristics” described in the El Sistema literature. The structure of the learning
environment included large ensemble, sectional learning, and individual lessons, with the large
ensemble being the main learning tool. The large ensemble environment was characterized by a
fast-paced, high-energy instructional approach, team-taught by the three teachers with the
Mitchell school orchestra/general music teacher assuming responsibility for the overall pace
and structure of rehearsal.
Most of the repertoire learned was standard repertoire for American beginning string class
instruction. As described in the ESI literature, the repertoire included arrangements of classical
repertoire, along with examples of American folk, fiddling, and patriotic repertoire. Repertoire
learned included arrangements of the Star Spangled Banner and Niño Lindo, a Venezuelan folk
song. Other ESI characteristics observed included rigorous expectations for the students to
strive for excellence, a safe and supportive learning environment, flexible teaching, and
opportunities for peer mentoring.
Fifth-grade students’ feelings of acceptance and reinforcement by parents, teachers, and friends
improved significantly from the beginning to the end of the program (p < .05). Students also
showed a statistically significant positive improvement in the way they felt about school
(p < .05). Evaluators analyzed student NWEA test scores to measure academic gain. The
change in El Sistema student test scores from Spring 2013 to Spring 2014 exceeded the national
average in reading (6.2 points, 10.7% above national average) and math (9.9 points, 16% above
national average). Music teachers in the program cited many benefits to the fifth-grade
students including high levels of musical achievement, increased opportunities for learning, an
understanding of the benefits of discipline, perseverance, a positive attitude and hard work.
Parents noted increases in students’ self-confidence as result of the El Sistema program.
SMTD graduate student instructors reported personal and professional advantages of
participating in the program. In addition to developing skills and gaining experience in
teaching group instruction, SMTD instructors benefited personally from engaging students,
their families, and the community in a meaningful way which enriched their own lives.
Implementation Challenges and Study Limitations
Challenges encountered in program implementation included the following: 1) a student
average attendance rate of 65%; 2) finding a balance between private instruction, sectionals, and
full ensemble; 3) addressing individual student needs; and 4) the balance of collaboration
between the Mitchell school music teacher and the two SMTD graduate student instructors.
One unintended consequence of offering the after-school ESI program at Mitchell to strings only

rather than all fifth-graders was that it introduced an inequality of educational opportunity for
the fifth-grade band students. It is recommended future school-based partnerships that target a
specific grade level offer the opportunity to all students.
Parent involvement and support is a characteristic of ESI programs that the teachers reported
was difficult to implement in the pilot program. The teachers suggested the hiring of a
professional staff person to help coordinate the private lessons, performances, and
communication with parents. Teachers recommended that the program include additional
performances, perhaps in December and March. SMTD instructors also expressed an interest
in receiving more preparation and training for working in public settings with diverse groups
of students, perhaps a lecture, presentation, or course to teach cultural sensitivity and
humility.
Threats of validity and reliability exist because the single group pre-post design lacks scientific
rigor. The small sample size threatens the study’s reliability, and threats to validity are evident,
particularly maturation, given the intervention is delivered to elementary school-aged children.
However, when feasible, testing the impact, attrition, and instrumental effect was monitored
and managed through the implementation of the evaluation protocols. Despite these
limitations, this type of design provided key stakeholders with information to better
understand the target population, services, and attainment of expected outcomes for future
replication and expansion purposes.

Executive Summary................................................................................................................. 3
I. Purpose of the Report........................................................................................................ 7
II. Transforming Approaches to Teaching and Scholarship ..................................................... 9
A. Spheres of Influence, Collaborative Partnership..................................................................... 9
B. Role and Responsibilities of Key Stakeholders...................................................................... 11
III. El Sistema Inspired Program at Mitchell Elementary........................................................ 12
A. El Sistema Philosophy .......................................................................................................... 12
B. El Sistema Guiding Principles................................................................................................ 12
C. Conceptual Framework ........................................................................................................ 12
D. El Sistema Logic Model......................................................................................................... 13
E. Facilitating Factors for Implementation................................................................................ 13
IV. Purpose of Evaluation...................................................................................................... 15
A. Key Evaluation Questions..................................................................................................... 15
V. Methodology..................................................................................................................... 17
VI. Population........................................................................................................................ 24
VII. Formative Evaluation Results .......................................................................................... 26
VIII. Outcome Evaluation Results........................................................................................... 33
A. El Sistema Mitchell Student Outcomes................................................................................. 33
B. U M Graduate Student Instructor Outcomes........................................................................ 38
IX. Areas for Improvements................................................................................................... 39
X. Implications....................................................................................................................... 44
XI. Recommendations............................................................................................................ 49
Appendix A: References......................................................................................................... 50
Appendix B: Background and Review of the Literature.......................................................... 53
Appendix C: Conceptual Framework...................................................................................... 55
Appendix D: El Sistema Guiding Principles............................................................................. 57
Appendix E: Procedures For Scoring Student Surveys ............................................................ 59
Appendix F: Codes Used to Analyze the Videotapped Rehearsals.......................................... 61
Appendix G: El Sistema Inspired Program Music Curriculum Evaluation Criteria.................... 63
Appendix H: Preliminary Pre Test Results.............................................................................. 66

I. Purpose of the Report
The El Sistema initiative is a research partnership between the University of Michigan School of
Music, Theatre and Dance (SMTD), School of Social Work (SSW), School of Education (SOE),
and Ann Arbor Public Schools (AAPS). This report was prepared by the University of Michigan
School of Social Work Curtis Center Program Evaluation Group (CC-PEG) in collaboration with
the SMTD Department of Music Education.
The purpose of the report is to provide key stakeholders with a summary of the Year 1 pilot
evaluation results. These results will be used by the El Sistema Collaborative to: 1) guide
program improvements; 2) replicate and expand an El Sistema Inspired (ESI) curriculum and
pedagogy; and 3) inform the development of a social engagement curriculum for SMTD.
This report contains results from (1) the October 2013 pre-test survey; (2) the ongoing
instrument instruction video-data collection from October 2013 to April 2014; (3) the April 2014
post-test survey; (4) and focus groups and interviews conducted between April and June 2014;
and (5) pre and post-program student school records (i.e., report cards and test scores).
The evaluation of the El Sistema Mitchell program was comprised of three (3) main
components:
1. Process Evaluation
To gather program experiences of participants and program improvement ideas
from stakeholders;
To determine adherence to the program logic model and grant narrative; and
To capture stakeholders understanding of the program
2. Outcome Evaluation
To determine program outcome attainment, barriers to completion, and
facilitating factors;
To identify unexpected outcomes of program participants; and
To capture stakeholders understanding of program impact
3. Evaluation Capacity Building
To inform SMTD curriculum development; and
To establish ongoing evaluation systems for the El Sistema program to enhance
program sustainability

Evaluation Team (In order of contribution)
Anthony M. Provenzano, MPA, MSW Candidate
Evaluator, Curtis Center Program Evaluation Group
Michael Hopkins, PhD
Associate Professor, Department of Music Education
School of Music, Theatre and Dance
Sue Ann Savas, MSW
Clinical Assistant Professor
Director, Curtis Center Program Evaluation Group

II. Transforming Approaches to Teaching and Scholarship
El Sistema is a Venezuelan music program that has inspired a global movement due to its social
mission to fight poverty and improve the human condition. Inspired by El Sistema’s model for
social change, this Venezuelan model has emerged as the catalyst for the University of Michigan
to develop an “innovative, multi-disciplinary teaching and scholarship approach” (U-M, 2013).
For more background information on El Sistema and a review of the literature, refer to Appendix
B: Background and Review of the Literature.
Partially funded by the University of Michigan’s Third Century Initiative, the University of
Michigan SMTD, SSW, and SOE have partnered with AAPS to develop an innovative music
intervention program for diverse communities, and a social engagement curriculum for SMTD
graduate students. Through this interdisciplinary research initiative, faculty, graduate students,
and key stakeholders utilized a participatory research approach on the design, implementation,
and evaluation of the El Sistema Mitchell music program.
Existing literature, although limited, provides evidence of the effects of the El Sistema model on
improving social, health, and education outcomes. Furthermore, there are few studies on the El
Sistema pedagogical approach; however, this is one area the evaluation has made contributions
to the field. Through this research partnership, formative and outcome evaluations were used
to shape the El Sistema Inspired (ESI) music intervention program, enhance the SMTD social
engagement curriculum and pedagogy, and verify if the pilot program was effective in
achieving its objectives.
Transforming approaches to teaching and scholarship through multidisciplinary social
engagement opportunities were the precepts for this collaborative research partnership. By
drawing from the innovative pedagogical approach of El Sistema, and combing proven research
methods from music, social work, and education, faculty, graduate students, and community
partners have gained from the collective intelligence of the different disciplines. This shared
knowledge has provided opportunities for scholars and communities to develop advanced
philosophical approaches for responding to challenging health, social, education, and economic
concerns.
A. Spheres of Influence, Collaborative Partnership
There are two (2) spheres of influence that have been integral to guiding the implementation of
the El Sistema Mitchell model and designing the formative and outcome evaluation:
1) AAPS/Mitchell Elementary School; and
2) The University of Michigan SMTD

i. AAPS/Mitchell Elementary School
Administrators from AAPS responsible for research and evaluation and the Principal and Music
Teacher at Mitchell Elementary expressed the need for improving school performance, student
academic achievement, and relationships with families and the community. In response, key
stakeholders from the El Sistema Mitchell initiative have adapted the Venezuelan music model
to develop a theory of change and logic model (Refer to Page 14, the El Sistema Mitchell Logic
Model) to target the specific needs, goals, and objectives of Mitchell Elementary and AAPS.
Mitchell Elementary School is in its third year of an intensive partnership with the University of
Michigan SOE: the Mitchell Scarlett Teaching and Learning Collaborative (MSTLC). This
partnership was designed to assist school personnel to improve school achievement and to
create a context for teacher education. Through the El Sistema Mitchell initiative, Mitchell
Elementary intends to build upon other after-school and extended year programs offered by the
MSTLC that focus on literacy, math, and science instruction. In partnership with the University
of Michigan SMTD, SSW, and aligned with the El Sistema philosophy, Mitchell Elementary has
established the following school-specific program objectives for the El Sistema pilot program: 1)
improving student academic performance; 2) cultivating a strong sense of school pride; and 3)
increasing family and community engagement.
ii. University of Michigan School of Music, Theatre and Dance
The University of Michigan SMTD aims to develop a social engagement pedagogy and
community-focused curriculum for SMTD students through their participation in the El Sistema
Mitchell initiative. This objective is central to SMTD’s mission and development as a
professional performing arts school. Teaching students how to engage communities in which
they work is now a core skill for the 21st Century performing artist and arts educator. A social
engagement curriculum situates SMTD students in the center of the global concerns
surrounding income inequality, economic deprivation, lack of access to education, health care
and other basic needs. It allows students opportunities to witness how their artistry speaks to
and transforms communities coping with these stressors.
Through the collaborative partnership with Mitchell Elementary School, the El Sistema model
will be used as the catalyst to form a social engagement pedagogy and curriculum. By
enriching SMTD’s curriculum with social engagement teaching activities graduate students
will: 1) develop a sense of social consciousness; 2) increase opportunities for graduate students
to provide instruction across different community settings; and 3) increase employability
through traditional and non-traditional curricula.

B. Role and Responsibilities of Key Stakeholders
In this research initiative, the Associate Deans from SMTD and SSW were Co-Principal
Investigators responsible for project oversight and capacity building.
i. Music Intervention
SMTD was tasked with implementing the music intervention pilot program at Mitchell
Elementary School using the El Sistema model. SMTD graduate students with previous
involvement in El Sistema provided music instruction in partnership with Mitchell’s school-
based music teacher. In addition to teaching children to play music as part of the El Sistema
program, two (2) lead SMTD graduate students were also participants of the evaluation, helping
to inform the development of SMTD’s social engagement pedagogy. Eight (8) SMTD graduate
student project teachers volunteered to provide pull-out lessons beginning in mid-February.
The School Principal provided administrative and program support. The school-based music
teacher served as the program director to: i) lead rehearsals and performances; ii) organize
program logistics; and iii) provide music education instructional support to SMTD graduate
students.
ii. Evaluation
Graduate students from the SSW CC-PEG, under the direction of the Associate Dean and CC-
PEG Director, were responsible for collaborating with SMTD faculty to conduct the evaluation.
The SMTD faculty member is a Professor of music education and Lead Evaluator of the El
Sistema music curriculum. CC-PEG was responsible for designing and conducting the overall
evaluation, in collaboration with the El Sistema collaborative and SMTD Lead Evaluator. The
SOE was also a partner in the El Sistema initiative, providing input and feedback on the
implementation and evaluation. Key leadership from AAPS responsible for research and
evaluation provided direction and support in developing the pilot program and evaluation plan
at Mitchell Elementary School.

III. El Sistema Inspired Program at Mitchell Elementary
A. El Sistema Philosophy
The philosophical underpinnings of the El Sistema intervention focus on personal development
as a means to build community. “Music nourishes and is nourished by that day-to-day pace,
awakening its aesthetical sense, encouraging without artificial postures the emergence of harmony and
beauty in places such as the city, the country and even within the very individual,” (Fundacion, 2013).
The El Sistema model supports children and youth through the philosophy that music can
provide “self-concept, self-esteem, self-confidence, discipline, patience, and commitment.” This helps
young people to recognize how their individual actions impact collective goals, while
cultivating a supportive environment of families, schools, and communities. At the core of El
Sistema’s community engagement approach is the concept that music offers a “spiritual”
richness that awakens the human condition through a heightened state of mind, greater
awareness of ethical principles, and improved intellectual and emotional skills conducive to
overcoming poverty (Fundacion, 2013).
B. El Sistema Guiding Principles
Through the literature El Sistema is characterized by 10 overarching principles listed below.
Refer to Appendix D: El Sistema Guiding Principles for a detailed account of these principles.
1. Mission of social change
2. Access and excellence
3. The nucleo environment.
4. Intensity
5. The use of ensemble.
6. The CATS teacher model: Citizen/Artist/Teacher/Scholar
7. The multi-year continuum
8. Family and community inclusion
9. Connections and network
10. Ambition and Achievement
C. Conceptual Framework
El Sistema Mitchell has developed a conceptual framework incorporating the guiding
philosophical principles of the El Sistema literature, a review of social and music learning
theories, and the specific community needs of Ann Arbor, Michigan. Precepts include music as
an agent of change, excellence, community, and supporting personal growth. The theory of
change and logical model (Refer to Page 14) provide an overview of how the El Sistema Mitchell
intervention achieved its intended outcomes (Refer to Appendix C: Conceptual Framework for the
goals and objectives of the program).

The theory of change states,
“If children participate in regularly and extensive high-quality, ensemble-based music
education, they will develop a strong sense of personal agency, and sense of community
through developing musical literacy” (Landin, A., et al 2013).
D. El Sistema Logic Model
Refer to Page 14 for detailed program logic model.
E. Facilitating Factors for Implementation
To maximize program recruitment and retention, students identified to participate in the
El Sistema program were already receiving instrumental music instruction as a standard
component of Mitchell Elementary School’s fifth grade music curriculum.
In an effort to ensure student and parent participation in El Sistema Mitchell program,
students were provided transportation to all activities including daily music instruction
and musical performances. Parents were offered transportation to the Year End
Performance at U-M Stamps Auditorium.
The SMTD music instructors have participated in El Sistema global movement as
students, and later as instructors. This previous experience is an essential training
component that directly impacts the success of El Sistema intervention at Mitchell
Elementary School and development of SMTD’s pedagogy.
Key stakeholders for the El Sistema Mitchell Initiative are using a community-based
participatory approach to design the program intervention and evaluation plan.

El Sistema Mitchell Logic Model

IV. Purpose of Evaluation
The purpose of the evaluation was to: 1) provide a descriptive account of the emerging El
Sistema Inspired curriculum at Mitchell Elementary; 2) document school and student outcomes
related to music literacy, individual and collective empowerment, sense of community and
school pride; 3) document the teaching experiences of SMTD students and their development of
social consciousness; and 4) document the pedagogical approach used by the El Sistema model
to help inform the social engagement curriculum for SMTD.
Two (2) types of evaluations were used to help shape the El Sistema music intervention at
Mitchell Elementary and to measure the goals and objectives. A formative evaluation was
conducted to document the emerging curriculum from year one of the El Sistema Mitchell pilot
project. Key stakeholders intend to use the results of this evaluation to manualize the El
Sistema program for future replication and expansion purposes. An outcome evaluation was
conducted to provide evidence that the El Sistema Mitchell program is meeting its intended
outcomes.
A. Key Evaluation Questions
Formative Evaluation
1. What is the curricular content of the El Sistema model?
2. What is occurring during music lessons to create the music culture of El Sistema?
3. What are the similarities and differences in the curricular content from other forms of group
class string instruction typically found in the United States?
4. What are the skills the SMTD Instructors and Mitchell Elementary Music Teacher possess?
5. What are the concepts and key elements of the El Sistema pedagogical approach?
6. How can the El Sistema model best inform a social engagement curriculum?
Outcome Evaluation
1. What impact does participation in El Sistema Mitchell have on fundamentals of string
instrument technique (posture, instrument position, bow hold, bow stroke, tone quality,
intonation), music performance skills (performing rehearsed music, sight-reading music,
playing from memory, playing by ear, improvising), and selected aspects of music literacy?
2. What impact does El Sistema Mitchell have on students’ musical self-esteem, academic
achievement, and conduct?
3. Have students increasingly acknowledged their sense of control in evoking change in their
own lives?

4. Have students become more aware of their community?
5. Has their sense of belonging or contribution to the community increased?
6. Do students who participate in El Sistema Mitchell have higher levels of enrollment in the
middle school instrument program?
7. Has Mitchell Elementary School developed a stronger sense of pride as a result of
implementing the El Sistema program?
8. Has the School developed stronger connections with students, families, and the community?
9. Have the SMTD graduate students developed a stronger sense of social consciousness
through their work at Mitchell Elementary using the El Sistema model?

V. Methodology
A. El Sistema Inspired Program – Music Curriculum Evaluation Criteria
The evaluation criteria used to assess the music curriculum (Figure 1) of the El Sistema Mitchell
pilot program is based on a literature review of: i) Venezuelan El Sistema (ES) and El Sistema-
inspired (ESI) programs in the U.S.; ii) National Standards of Arts Education (NASAE); and iii)
1st year string instrument standards and AAPS fifth-grader outcomes.
Figure 1. El Sistema Mitchell Music Curriculum Evaluation Criteria
Music Curriculum Evaluation Criteria
ES and ESI Programs NASAE AAPS & 1st Year String
fast-paced, high energy rehearsal
environment
singing, alone and with others,
a varied repertoire of music
instrument skills—bowing
teachers have rigorous
expectations for the students to
strive for excellence
performing on instruments,
alone and with others, a varied
repertoire of music
instrument skills— Instrument
Position and Left Hand Skills
the learning environment is safe
and supportive, characterized by
caring and kindness
improvising melodies,
variations and accompaniments
music literacy—defined as “an
association of sound-to-symbol,
in a given musical context.
the teaching is characterized by
flexibility, purposeful planning,
encouragement of exploration and
curiosity, and a variety of teaching
techniques
composing and arranging
music within specific
guidelines
aural skills – implementing
patterns by ear
the structure of the learning
environment includes large
ensemble, sectional learning, and
individual lessons, with the large
ensemble being the main learning
tool
reading and notating music rehearsal skills – playing together
with other students
parent involvement and support listening to, analyzing, and
describing music
attitude – fostering a positive
attitude towards music
opportunities for peer mentoring
and student leadership
evaluating music and music
performances
regular student assessment understanding relationships
between music, the other arts,
and disciplines outside the arts
frequent performance
opportunities
understanding music in
relation to history and culture
repertoire includes arrangements
of classical repertoire, along with
examples of American music
(fiddling repertoire, jazz, etc.)
repertoire includes simple
arrangements of big pieces (i.e.,
masterworks) with big sound

B. Grant Criteria: Curricular Design and Goals
The curricular design and goals as determined by the grant proposal were also used to evaluate
the music intervention. Listed below are the activities listed in the grant.
i. SMTD undergraduate and graduate students will have opportunities to:
a. work in a socially, economically, and linguistically diverse context;
b. enact pedagogies with direct feedback on their work; and
c. see the impact of arts engagement on the overall achievement and well-being of
students in the academic and social life of the school.
ii. Instruction will combine full ensemble rehearsals with sectionals, partner lessons and
individual lessons.
iii. Instruction will primarily be carried out by the lead GSI, the assistant teacher, as well as by
SMTD students employed to teach supplementary private lessons (hereafter referred to as
“project teachers”) on a weekly basis.
iv. Outreach in the Performing Arts (MUSPERF 101/401) and Teaching String Instruments
(MUSED 201/202) will serve as training venues for the SMTD EI Sistema project teachers.
v. The lead teachers will offer regular training and assessment meetings for the project
teachers.
vi. The Mitchell Elementary School music teacher will assist with planning, teaching and
facilitation of the interaction between the EI Sistema program and the regular instrumental
program at Mitchell Elementary School.
vii. A research team of Music Education and Social Work graduate students will assess musical,
academic and social results.
viii. Beginning in October 2013, graduate students will provide monthly reports on music
assessments. Program leaders convene monthly to review graduate students reports and
videotapes.
ix. The EI Sistema initiative will complement other extended day and extended year programs
offered collaboratively with the MSTLC that focus on literacy, math, and science instruction.
C. Study Design
A single group pre-test/post-test multi-method research design was used to evaluate the
effectiveness of the El Sistema Mitchell pilot project in achieving its desired program goals,
objectives, and outcomes, as determined by the proposed conceptual framework and logic
model (Refer to Page 14, the El Sistema Mitchell Logic Model).
Single Group Pre-test-Post-test Design
O1 X O2
X = Intervention O = Observation

The El Sistema Collaborative Partnership selected a pre-experimental design to deliver a cost
effective descriptive account of the Venezuelan model. Short time constraints leading up to the
project kick-off, and limited financial resources to support the implementation and evaluation,
were cited by the Collaborative as reasons for selecting this type of design. Although the pre-
experimental design is feasible provided the restraints placed on the community, this study
design presents limitations, and positive outcomes could be a result of factors other than the
program intervention.
D. Study Recruitment and Retention
A purposeful sample was used to maximize program recruitment and retention and to help
control project costs. Twenty-two students at Mitchell Elementary School who currently receive
string instrumental instruction as a standard component of the fifth-grade music curriculum in
AAPS were targeted for this study. These students were selected to participate in the program
because the El Sistema trained SMTD graduate instructors were experienced violinist/violist
and cellist. This had excluded fifth-grade horn instrument students from participating in the
afterschool program.
The parents of string students were mailed letters providing information regarding the El
Sistema afterschool program and inviting them attend a program orientation. All
communication to the parents regarding the afterschool program were provided in both English
and Spanish. During the orientation family members were provided with: 1) the El Sistema
program permission slip for student enrollment; 2) the parental consent form for their child’s
participation in the program evaluation; and 3) a one-item questionnaire regarding the
expectations for their child in the after school program. In an effort to increase program
participation and ensure retention, the school principal contacted families via the telephone to
inquire about reasons for not participating, and to discover potential causes for students
dropping-out.
Student participation in the study was voluntary. A student’s experience in the El Sistema
Mitchell program was not positively or negatively impacted as a result of their participation in
the evaluation. Students had the right to refuse to answer any of questions, and at any point
could choose not to participate in the study. No one is able to identify students from any of
the data, analysis, findings, or reports. All 22 students participated in the program evaluation,
including the pre-test and post-test surveys and focus group.
Evaluation consent forms were signed by the parental guardians of all 22 students who
participated in the El Sistema Mitchell program evaluation. Parental consent forms were
translated for Spanish speaking families and bilingual project staff were available during the
program orientation to answer any questions parents had regarding student participation in the
program and the exploratory study. Signed parental consents forms and sensitive survey data

was kept in a locked office and located on the University of Michigan’s secured password
protected computer network.
E. Data Collection and Measurements
The multi-method data collection processes for the program evaluation was reviewed and
approved as non-regulated by the University of Michigan Institutional Review Board. Data
were gathered to assess the achievement of the program goals (Refer to Appendix C: Conceptual
Framework for detailed list of the program goals and objectives) through analysis of videotaped
rehearsals, pre- and post-program interviews with the three (3) music instructors and the school
principal, focus groups with the parents and students, researcher observations, email
correspondence with the lead teachers, project teachers, and the Mitchell Elementary school
music teacher.
The quantitative and qualitative procedures for this study design consist of conducting pre-test
(O1) and post-test (O2) student surveys and focus group, teacher ratings, music instructor and
school principal interviews, parent questionnaire and focus group. Digital video recordings of
a sample of music instruction lessons were conducted weekly to assess instrumental technique
and other aspects of music literacy. Weekly process notes were also documented to account for
changes to program implementation plans. The evaluation also examined student academic
achievement and behavioral conduct through a review of official school records that capture
student attendance, grades, and test scores to reveal any changes in academic performance.
Student pre-test surveys were completed by the El Sistema program participants on October 29
and 30, 2013 during the afterschool music program, with 18 El Sistema program participants
responding. Survey administration was split between two (2) days to account for program
agendas, respondent fatigue, and accurate survey completion. Make-up sessions were offered
the following week for students who were not in attendance. An additional attempt to collect
student survey data was made upon new student enrollment into the program. All 22 program
participants completed the pre-test student survey, for a response rate of 100%.
Student post-test surveys were completed by El Sistema program participants on April 29 and
May 1, 2014 during the general music education string instrument instruction. The surveys
were administered proceeding the El Sistema Year-End Concert and program banquet/awards
ceremony. The surveys were split between two (2) days due to time constraints. All 22
program participants completed the post-test student survey, for a response rate of 100%.
Pre-test teacher surveys were administered to and completed by both fifth-grade general
education teachers for each student participating in the El Sistema program on October 30, 2013.
The teachers completed 21 of the 22 surveys for a response of 95%. The post-test teacher rating

was administered on May 27, 2014. The teachers completed 22 surveys for a response rate of
100%.
Pre-tests and post-tests for students and teachers utilized the following standardized measures
(Figure 2):
Figure 2. Detailed Account of Student Survey and Teacher Rating Measures
KEY INFORMANT NAME OF MEASURE AUTHOR(S) DATE
DEMONSTRATED EFFECTIVE IN
MEASURING
Students
Attitudes Towards
School
Anderson, S.
A.
1999 Student’s perception of school
Students
Satisfaction With
Life
Gadermann, A.
M., et al
2010 Individual well-being
Students Conflict Behavior
Kammratrh, L.
& Dweck, C. S.
2006 Relationship conflict behavior
Students
Parent Involvement
In Students’ Music
Education
Zdinski, S. 1993 Music environment at home
Students
Self-Esteem of
Music Ability
Schmitt, M. 1979
Self-confidence; Skills and abilities
in music; and Feelings of acceptance
and reinforcement by parents,
teachers, and friends
5th Grade
Teachers
School Achievement
Motivation
Chiu, L. H. 1997
Teacher rating of student
achievement motivation
demonstrated in the classroom
The qualitative data collection included semi-structured interviews, focus groups, and
questionnaires with project stakeholders at time O1 and O2. The qualitative data collection
protocols were designed by the evaluation team with input from the El Sistema Collaborative to
ensure different perspectives in capturing information about the implementation, operations,
and outcomes.
Semi-structured interview and focus group questions for music instructors, the school principal,
parents, and program participants were developed to describe the emerging music curriculum,
address program improvements, and student outcomes in relation to music literacy, sense of
community, and sense of personal agency; perceived sense of school pride; and the music
instructors’ perceived sense of social consciousness.

A parental guardian questionnaire was distributed in both English and Spanish during the El
Sistema program orientation/kick-off event on October 16, 2013. A total of 17 of 22
questionnaires were collected for a response rate of 77%. Parents/legal guardians also
participated in a post-program focus group hosted during the program ending banquet and
awards ceremony on April 28, 2014. Four (4) separate tables were organized to control the size
of the focus groups. One of the parent focus groups was conducted in Spanish to capture the
voice of Spanish speaking families. A total of 18 parent/legal guardians participated in the
focus groups.
Post-program interviews were conducted with the Mitchell Elementary school principal and
school-based music teacher, and U-M lead graduate student music instructors in April and May
2014.
Post-program student focus groups were conducted on May 27, 2014 during the general music
education string instrument instruction. Three (3) separate groups were hosted to control for
size. All 22 students participated in the focus group.
The statements provided by the key informants in this report are direct quotes and have not
been altered.
F. Analysis Procedure
Statistical Package for Social Sciences (SPSS) version 21.0 was used to organize, manage, and
analyze data to evaluate the effectiveness of the intervention outcomes. For complete account
of the methods used to calculate and score the student surveys and teacher rating refer to
Appendix E: Procedures for Scoring Student Surveys.
Descriptive statistics were performed to report the differences in the means of the Northwest
Evaluation Association (NWEA) RIT reading and math test scores from Spring 2013 (4th grade)
to Spring 2014 (5th grade). El Sistema student data was compared to national averages using
NWEA normative data to document changes in academic performance. Student report cards
were obtained and analyzed at the end of the school year that contained grades for fall, winter,
and spring as another measure of school achievement.
Semi-structured interviews and focus groups were audio recorded and transcribed by the
evaluation team. CC-PEG evaluators analyzed transcripts and open-ended responses on
questionnaires for themes. The qualitative data collected were organized and coded into
categories using Microsoft Excel.
A video coding scheme based on Duke (1999) was developed in order to analyze the content of
the videotaped rehearsals. A list of the categories used to code the video can be found in
Appendix F. Videotaped data were coded using Scribe 4, a data analysis program developed at
the University of Texas at Austin's Center for Music Learning (cml.music.utexas.edu). In order

to determine the reliability of the coding procedure, 10% of the video used for analysis was
coded independently by the SMTD Associate Professor and a graduate student in music
education. Reliability was computed by comparing the total time in minutes coded for each
observational category (agreements/(agreements + disagreements)). Average inter-rater
reliability was .91.
For the purposes of this evaluation report, 325 minutes of video was coded and analyzed.
Video was selected from November 6 and 13, December 4 and 18, January 30, February 11 and
25, and March 18, yielding a total of 16 sample lessons. The teacher interviews conducted to
further understand the emerging music curriculum and pedagogy at Mitchell were transcribed
and analyzed based on the evaluation criteria listed in Appendix G: El Sistema Inspired Program
Music Curriculum Evaluation Criteria

VI. Population
Two (2) populations were directly affected by the El Sistema Mitchell program: 1) fifth graders
at Mitchell Elementary School; and 2) SMTD graduate students who received experience
working in public schools and teaching diverse populations.
A. Mitchell Elementary School
In the 2013-14 school year Mitchell Elementary had approximately 280 students enrolled.
Located in the southeast area of Ann Arbor, Michigan, more than 30% of the students speak a
language other than English at home. Seventy-two percent (72%) of students qualify for free or
reduced lunch (WISD, 2013). Figure 3 below provides an account of the total school population
by race and ethnicity using data from 2010, the most recent year available. In 2010, the total
census was 260 students (AAPS, 2010).
Figure 3. Mitchell Elementary School Race and Ethnicity (n=260)
B. Participant Description
i. El Sistema Students
Twenty-two (22) fifth-grade students receiving string instrument instruction as standard
component of the Mitchell Elementary School music curriculum were identified to participate in
this pilot program. Figure 4 below provides the percentages of boys and girls in the program
by gender.
0.4%
2.3%
7.3%
18.5%
23.1%
23.9%
24.6%
Other
Middle East
Asian
Multiethnic
Caucasian
Latino/Hispanic
African American
(n = 60)
(n = 64)
(n = 1)
(n = 62)
(n = 48)
(n = 19)
(n = 6)

27.3%
72.7%
Boys
Girls
Figure 4. El Sistema Students By Gender (n = 22)
There were six (6) boys and sixteen (16) girls in the program.
Figure 5 provides the percentages of program participants by race and ethnicity. There were no
students of Middle East decent or who identified as Other that participated in the program.
Figure 5. El Sistema Students By Race and Ethnicity (n = 22)
ii. U M Graduate Student Instructors
Two (2) SMTD graduate students in music performance and the Mitchell orchestra/general
music teacher were the lead teachers in the pilot program. Eight (8) project teachers were hired
to give pull-out lessons beginning in mid-February. Six (6) of the project teachers were
undergraduate music education majors and two (2) were music performance majors.
13.6%
18.2%
18.2%
18.2%
27.3%
Latino/Hispanic
Multiethnic
Asian
African American
Caucasion (n = 6)
(n = 4)
(n = 4)
(n = 4)
(n = 4)

VII. Formative Evaluation Results
A. El Sistema Inspired Music Curriculum at Mitchell Elementary
Question 1: What is the curricular content of the El Sistema model?
a. Repertoire
The repertoire learned and performed by the students included the following:
Pizzicato D and A strings on rhythmic patterns learned in general music curriculum
D and G scales bowed with straight 1/16 note rhythm
Hot Cross Buns and Twinkle, Twinkle Little Star played as follows:
o Pizzicato
o Arco on open D and A strings with rhythm of song
o Arco adding left hand finger patterns
Twinkle, Twinkle Little Star with the rhythm used in Suzuki Book 1, Variation 1—four 1/16
notes, two 1/8 notes.
Lightly Row
Bile ‘Em Cabbage Down on D string
Star Spangled Banner 3-part arrangement in G major, played pizzicato
Nino Lindo – a Venezuelan song the U-M graduate student instructors arranged for final
performance
Other repertoire learned was taken from Essential Elements 2000 Book 1 (Allen, Gillespie &
Hayes, 2001). The approach used to learning this repertoire was similar to Hot Cross Buns
and Twinkle, as described above.
o Can Can
o Ode to Joy
o Picky Pizzicato
o This Old Man
o Kookaburra
Question 2: What is occurring during music lessons to create the music culture of El
Sistema?
a. Instructional approach/learning environment
The learning environment was characterized by a fast-paced, high-energy instructional
approach. The percentage of time allocated to teacher interventions, student responses, and
other activities is presented in Figure 6 below.

Figure 6. Allocation of Instructional Time in Mitchell ESI Program
Teacher Interventions Student Responses Other
Verbal
Intervention
(Instruction,
Feedback)
Co-Verbal
Demonstration
Musical
Demonstration
Non-Verbal
Instruction
(Gestural,
Pedagogical
Touch)
Verbaliz-
ation
Performance
Trials
Tuning
26% 7% 7% 2% 2% 38% 14%
The remaining 2% of time not accounted for in Table 1 was spent transitioning between lessons,
snack and activities. The analysis revealed a very strong balance between time allocated to
teacher interventions (42%) and student responses (40%). It should also be noted that there was
verbal and non-verbal instruction that was occurring outside of the frame of the videocamera in
many of the lessons. The videocamera was positioned at the front of the room and set at an
angle where several of the violinists were often not in frame. Figure 7: Skills Address in ESI
Lessons (Refer to Page 28) contains data regarding addressing skills/outcomes.

Figure 7. Skills Addressed in ESI Lessons
Skill/Outcome Percentage of lessons where skill was
addressed by teacher during instruction
Bow Hold 44%
Bow Stroke (Weight, Angle, Speed,
Placement, Tone Quality
70%
Body Posture 39%
Instrument Position 44%
Left Hand Position 57%
Finger patterns, fingering 91%
Intonation 48%
Music Reading 35%
Aural Skills 100%
Rehearsal Skills (playing together as an
ensemble, watching, etc)
83%
Attitude 35%
b. Program organization
The Mitchell program met Monday through Thursday beginning the week of October 21, 2013,
culminating with a final performance at the U-M Stamps Auditorium on April 23, 2014. When
the regular school day ended at 3:42 PM, students would gather in the cafeteria for a snack and
then receive approximately 75-minutes of instruction, ending at 5:15 PM. As described in the
ESI literature, the structure of the learning environment included large ensemble, sectional
learning, and individual lessons, with the large ensemble being the main learning tool.
At the beginning of the program on October 23, the fifth-grade students were split into two
sections. U-M graduate students with expertise in playing cello and violin/viola whom had
previous experience in El Sistema and El Sistema Inspired programs provided string instrument
instruction. On November 6, approximately 2/3 of the instructional time was spent in
sectionals, with the students coming together in the large ensemble for the final 25 minutes of
rehearsal. The U-M graduate student instructors taught this rehearsal segment. On November

13, the class was kept together as a large ensemble for the entire rehearsal, again being team
taught by the graduate student instructors.
The week of November 20, a different instructional pattern emerged. The class began by
watching a video of Danzon no. 2 by Marquez being performed by a Venezuelan El Sistema
orchestra. This activity led into a tuning procedure that featured the following characteristics.
The school-based music teacher improvised a piano accompaniment. The students bowed a
string specified by the teachers (A, D, G, C, or E), while teachers walked around the room and
tuned the students’ instruments one string at a time using the fine tuners. The students
maintained control of their instrument, rather than the teachers taking the instruments from the
students to tune. The school-based music teacher interrupted the tuning procedure to lead the
students in singing a song, then returned to tuning other strings. The tuning procedure
continued for approximately 8 minutes. The school-based music teacher then led the rest of the
class from the piano, with U-M graduate student instructors assisting individual students, and
demonstrating on their instruments. The school-based music teacher would alternate with U-M
graduate student instructors in leading call-response non-verbal instructional segments. When
graduate student instructors were leading these segments, the school-based music teacher
provided a background piano accompaniment.
This was the general team teaching approach to the learning environment that emerged and
developed from November – April. Some alterations were made to the sequencing and
activities within this large ensemble structure as the program developed. For example, from
November through February the students would have their snack first, then come to rehearsal.
Beginning in February, the teachers decided to begin the lesson with a note-reading activity
without instruments (10-15 minutes), then take a snack break (5-10 minutes), tune instruments
(7-8 minutes), warm-up with a focus on tone and intonation development while playing scales
and patterns by ear (20-30 minutes), then spend the final 20-30 minutes of class learning
repertoire. The week of Jan 21 featured breaking out into sectionals to learn the Star Spangled
Banner, but the learning environment was largely characterized by the three teachers team
teaching the large ensemble, with the school-based music teacher as the lead teacher that
assumed responsibility for the overall pace and structure of rehearsal. There were two weeks of
the program that U-M was on break and AAPS was in session, where school-based music
teacher led the class alone.
Project teachers were hired to give pull-out lessons beginning in mid-February, while U-M
instructors and school music teacher focused on the large ensemble. The project teachers were
U-M students hired by the U-M graduate student instructors. A total of eight project teachers
were hired. Six of the teachers were undergraduate music education majors. One teacher was
an undergraduate music performance major. One was a masters student in cello performance.
The project teachers visited Mitchell once per week and provided 25-minute individual lessons
and mini-lessons with two students. In the pull-out lessons the project teachers reinforced the

concepts being taught in the group class and also introduced supplementary solo repertoire to
enhance the instruction students received.
The U-M graduate student instructors would provide the project teachers with instructions
about the materials, techniques, and concepts to be covered in the pull-out lessons when the
project teachers arrived at Mitchell. The U-M graduate student instructors would stop by the
pull-out lessons to assess progress, provide support, and answer questions. They were also
available afterwards to answer any questions the project teachers would have. The U-M
graduate student instructor and school-based music teacher reported in the focus group
interview that the pull-out lessons were very helpful for improving aspects of the fifth-grade
students’ technique, especially tone quality, and were very timely in that they began at a time
when the fifth-grade students were beginning to separate in achievement level.
c. Expectation for the students
In the focus group interview and the post-interviews the teachers spoke of communicating high
expectations for musicianship, behavior, and attendance to the students. Teachers often spoke
to students of achieving their “personal best.” The teachers also described their efforts at
maintaining consistency of expectation and routine.
The U-M graduate student instructor contrasted the expectations of the students in the Mitchell
program with those of ES. In the focus group interview he said, “I think when you are talking
about El Sistema in Venezuela you will think of people demanding their students to practice at home so
they learn the parts really well…I see that here more as a very high expectation in attitude. We are really
expecting them to show their best in the attitude to us learning and to us producing the music in the
classroom. So I think it a little bit different on the focus of the expectation, but I think the standard is
very high.”
Question 3: What are the similarities and differences in the curricular content from
other forms of group class string instruction typically found in the
United States?
a. Distinguishing Characteristics/Comparison to ES and other ESI Programs
The repertoire learned in this ESI class is standard repertoire for American beginning string
class instruction. As described in the ESI literature, the repertoire included arrangements of
classical repertoire, along with examples of American folk, fiddling, and patriotic repertoire.
While ensemble-based group instruction is a ubiquitous characteristic of American public
school music education, this approach has been criticized in recent years for giving precedence
to a group’s performance result over addressing individual student needs (Williams, 2011). In
the post-interview, however, a U-M graduate student instructor offers an alternate view—one

more in philosophical alignment with Vygotsky’s ideas on the Zone of Proximal Development
(1994),
“take the group and make the group learning and the sound of the group like a
participatory sound, of a participatory nature. The sound of the group will help kids that
don’t have the ability to play the whole piece. For example, to just participate. Be there
and try to play what they can play and then, they take the sound of their group as the
strong thing that keeps them playing and keeps them stimulated while facing individual
challenges.”
By focusing on the collective participatory sound of the large ensemble, the teacher is in fact
addressing the needs of the individual students within the ensemble. Group instruction
provides students with low levels of achievement or self-esteem an opportunity to feel
successful even if they not be able to play the music perfectly.
Question 4: What are the skills the SMTD Instructors and Mitchell Elementary
Music Teacher possess?
a. Development of musicianship
The preparation that the Mitchell students received from the school-based music teacher prior
to the program, and during the school day throughout the program, was a highly influential
factor on this ESI program. In the post-interview, one U-M graduate student instructor
remarked, “I’ve never been with a group that had such strong rhythm and solfege training prior to
picking up their instruments.”
In the focus group interview, the teachers spoke of the benefits they saw from the program. The
school-based music teacher said, “The biggest thing for me is that the kids are set up better than ever.
That is outstanding. In terms of repertoire, we’re not really further along, but the kids have a deeper
understanding of reading that we didn’t have in the past. When they leave us, they’re going to be better
musicians than ever. Their set up is better, they’re listening better, they’re creating better sounds and
they’re understanding deeper.” A U-M graduate student instructor added, “They’re reading really
strong, some of them, some of them are exceptionally strong, and all of them are functional readers at this
point.”
The school-based music teacher expressed a belief that the students’ reading ability will help
keep them interested in continuing to play in middle school and will make them leaders in the
middle school orchestra program. In the post-interview, a U-M graduate student instructor said,
“they really learned a lot of music over the course from October through April, and because of the
fact that there is a combination of the [school-based music teacher and U-M graduate student
instructors], I think they really, for the most part, do have a pretty good technical set-up…and
they have strong reading skills. I’m very happy with the kind of reading processes they’re
developing.”

Question 5: What are the concepts and key elements of the El Sistema pedagogical
approach?
a. Daily access to high level musicians
In the post-interview when I asked school-based music teacher what the program’s greatest
successes were, he said, “to hear the U-M graduate student instructors on their instruments…to see
musicians jamming on their instruments; not just teachers teaching, and to hear what a violin really
sounds like, live and played well, or a cello and all of its range. That was huge for the students, to see
what they were trying to aspire to.”
b. Peer mentoring
In the focus group the teachers described several ways that they try to encourage peer
mentoring within the class: 1) taking turns being the leader or “principal” player in their
section; 2) using inquiry to encourage musical decision making among the students; and 3)
providing opportunities for students with higher levels of achievement to help struggling peers
learn the assigned repertoire.
c. Frequent performance opportunities
The students gave three performances during the program. November 21, February 10, and
April 23. In the post-interviews the teachers expressed an interest in giving more frequent
performances. A U-M graduate student instructor noted that in ES there is typically at least one
performance per month. This is because in Venezuela performances are considered important
for the overall development of the children.

VIII. Outcome Evaluation Results
A. El Sistema Mitchell Student Outcomes
i. Instrument Assignment
Students self-selected to participate in the El Sistema Mitchell program and were assigned their
string instrument. From a total of 22 participants, nine (9) students were assigned the violin,
seven (7) students the viola, and six (6) students the cello.
ii. Student Attendance
El Sistema Mitchell program was held on sixty-eight (68) days between October 21, 2013 and
April 23, 2014, which accounted for scheduled days off and five (5) unexpected inclement
weather / snow days. Attendance was tracked on twenty-two (22) participants. The program
had an average attendance rate of 65%. Daily attendance ranged from seven (7) to twenty-one
(21) with an average of fourteen (14) students present at each session.
There was an increase in the retention rate of string instrument students as a result of the music
intervention according to the post-program interview with the school-based music teacher, he
noted, “There was less attrition because of the El Sistema program.”
Question 1: What impact does participation in El Sistema Mitchell have on
fundamentals of string instrument technique (posture, instrument
position, bow hold, bow stroke, tone quality, intonation), music
performance skills (performing rehearsed music, sight-reading music,
playing from memory, playing by ear, improvising), and selected
aspects of music literacy?
Assessment data of individual student performance and technique outcomes were not collected
during the program by the instructors. This may have been due to resource and time
limitations, or concerns about the impact of individual assessment on students' comfort level
and musical self-esteem. We recommend that future programs develop formal assessment
techniques to provide students, parents, and teachers with information regarding student
growth. Such data would be very useful for program evaluation, as the students' performance
outcomes could be compared to those of students at other area schools.

Question 2: What impact does El Sistema have on students’ musical self-esteem,
academic achievement, and conduct?
a. Musical self esteem
Results from a paired sample t-test (p<.05) of pre-test (mean = 55.68) and post-test (mean =
59.41) scores from Subscale 1: Support and Recognition from Others of the Self-Esteem of
Musical Ability (SEMA) Scale (Schmitt, 1979) indicated that students showed a statistically
significant improvement in their feelings of acceptance and reinforcement by parents,
teachers, and friends. The Self-Perception of Music Ability subscale changed in a positive
direction, but was not statistically significant. The Personal Interest and Desire sub-scale
decreased from pre to post-test, however was not significant.
Furthermore, in the post-program interviews with the music instructors, one teacher stated that,
“the program helped the fifth-grade students learn the benefits of discipline, perseverance, a positive
attitude and hard work.” Another instructor said, “Some students were more shy in the beginning, less
inclined to express their feelings or personality, but by the end of program they were more at ease.”
Another instructor stated, “Some students displayed more confidence in their music playing.”
Parents reported in the post-program focus group that students seemed to have improved their
confidence. “I feel like [El Sistema] was a great confidence builder.” Another parent stated, “She saw
she was successful for herself.” A father noted, “[He has] more self-esteem.”
b. Academic achievement
Results of the differences in the means of the Northwest Evaluation Association (NWEA)
reading and math test scores from Spring 2013 to Spring 2014 revealed El Sistema students
exceeded the national average. NWEA normative data shows the national average growth in
RIT test scores from end of 4th grade to end of 5th grade as 5.6 points in reading and 8.5 points
in math. The growth from Spring 2013 (end of 4th grade) to Spring 2014 (end of 5th grade) for
program participants was higher than the national average in reading (6.2 points, 10.7% above
national average) and in math (9.9 points, 16% above national average). However, it was
difficult to determine if the El Sistema music intervention had a direct impact on academic
achievement due to the limitations of the outcome design.
Parents shared their reservations about the effects of the program on academic achievement
during the focus group, “[My child’s] reading went up two levels, not sure if it was the [El Sistema]
program.”
c. Behavioral conduct
Results from a paired sample t-test (p<.05) of pre-test (mean = 68.32) and post-test (mean =
70.95) scores from the Student Attitudes Towards School Scale (Anderson, 1999) indicated that
students showed a statistically significant positive improvement in the way they felt about
school.

In a post-program interview, the school principal noted El Sistema students’ behavior had
appeared to improve, “I do believe the kids are more engaged in positive behaviors.” The principal
also indicted that, “There were no school suspensions [of those students who participated in El
Sistema].” Parents noted changes in their students’ behavior, during the focus group one parent
said, “He takes things more seriously. He’s more responsible.”
Question 3: Have students increasingly acknowledged their sense of control in
evoking change in their own lives?
It was challenging to identify if the students increased their sense of control in evoking change
in their own lives due to the limitations of the measurement tools used and outcome design.
Results from the paired sample t-test of pre-test and post-test scores from the Self-Perception of
Music Ability (SEMA) Scale (Schmitt, 1979) revealed no significant change in students’
perception of their interest and desire in playing music. However, the results from the SEMA
Personal Interest and Desire sub-scale were positive, indicating that students: i) enjoyed
practicing music; ii) expected a lot of themselves; and iii) had a strong desire to play music in
the future. Although limitations created barriers for answering this evaluation question, the
students’ realization of their role in determining their level of interest in continuing playing was
used as a proxy to determine the students’ realization of their sense of control in evoking
change in their own lives.
In parent focus groups, families underscored their students’ renewed sense of control in
evoking change in their own lives. Parents noted their student’s ability to adhere to prior
commitments, act responsible, and make decisions about their future, “She would work to finish
her homework, she was more organized.” Another parent shared, “I liked the way she followed through
[with El Sistema]. It is important to me for her to take on something challenging… Taking on the
responsibility shows maturity.” ”Playing the cello has made him take it a step further. He wants to play
a stand-up bass,” was noted by one student’s father.
Question 4: Have students become more aware of their community?
In the post-program interviews, the music teachers all spoke of the benefit they saw from the
year end performance at U-M Stamps Auditorium. The school-based music instructor stated
that the experience of playing in Stamps Auditorium was highly empowering for the students
and helped them to envision themselves as part of a larger community.
“The greatest success was getting the kids to the Michigan campus, getting them to North
Campus, getting them to see that, getting them to see the auditorium… the cello players were
already down on the stage getting ready with [one instructor], and [the other two instructors]
brought the violins and violas and we came in from the top, and when they looked down they
were just…they were really taken back. They could not believe they were playing there.”

The students echoed the similar sense of community they felt as being part of the final
performance at Stamps Auditorium. During the post-program focus group, one student stated
that, “It was nice to see my teacher and student teacher come to one of my performances. It was also nice
to hear my teachers talk about my performance at school.”
Question 5: Has their sense of belonging or contribution to the community
increased?
In a focus group with student’s families, one parent stated, “the impact [El Sistema] had on her
made her feel like she belonged. This was the first year she felt like she belonged to the group. She had fun
with it and enjoyed it. It made her a lot better.”
In the post-program interview, a U-M graduate student instructor attributed the performance of
El Sistema as vehicle to keep students connected said, “there are some students that kind of drifted
away from the program and came back…I was very happy that they came back. That they were in their
final performance and I know their families were some of the ones that were most happy in the wake of the
performance.”
Question 6: Do students who participate in El Sistema Mitchell have higher levels
of enrollment in the middle school instrument program?
The evaluation of the El Sistema pilot program measured program outcomes at the end of fifth-
grade (June 2014). The El Sistema student level of enrollment at the middle school should be
conducted in a subsequent evaluation after the program participants have entered 6th grade.
For example, the Mitchell School music teacher tracks the elementary students into the
instrument and orchestra program at Scarlett Middle. The instrument instruction retention rate
at the elementary level is a proxy to determine attrition rates at the middle school level
currently used by Mitchell Elementary School.
Question 7: Has Mitchell Elementary School developed a stronger sense of pride as
a result of implementing the El Sistema program?
During a post-program interview, the school principal was asked about the ways El Sistema has
contributed to the sense of school community and positive reputation in the larger Ann Arbor
community. He noted the following:
“We have become a music school in a sense where kids are interested in playing and developing
their music ability. It is really an important part of the changes that have happened here.”
“Having kids engaged at the performances and attend the initial meeting created a sense of
community.”
“For the Mitchell Community the performance at the end added to the sense of community at
Mitchell.”

“Historically, people in the larger community do not speak highly of Mitchell Elementary.
However, because of El Sistema there was a certain buzz in Ann Arbor about what is going on
here. The program is something special. This program is a positive thing for Mitchell, bringing
positive press.”
In post-program interviews, the U-M graduate student instructors and school-based music
teacher conveyed a similar sense of community within the school and across Ann Arbor. “The
media buzz was pretty cool, on NPR and two articles in the newspaper.”
The school principal and music teacher had to monitor the publicity and address issues of
equity among the school community. In a post-program interview, the school based music
teacher stated, “If we made too big a deal of El Sistema we could offend some people.” He went on to
say, “We had to harness the publicity at the end of the program to protect against the divide between the
two groups [strings and band students.]”
One unintended consequence of offering the after-school ESI program at Mitchell to strings only
was that it introduced an inequality of educational opportunity for the fifth grade students. The
fifth-grade band director and strings teacher reported that the band students expressed feelings
of envy toward the ESI program. Eventually, the Mitchell band director began inviting her
fifth-grade band students to walk to Scarlett Middle School two days a week to participate in an
afterschool practice club, in an attempt to ameliorate the disparity of instructional time between
the fifth-grade orchestra and band students.
Question 8: Has the School developed stronger connections with students, families,
and the community?
A sense of support and community developed around the program. An older brother of one of
the fifth-grade students in the program began attending and assisting the teachers with snack
and tuning instruments. The teachers also stated this helped the fifth-grade student to have her
brother attend because of some family issues. The presence of eight (8) U-M graduate students
at Mitchell to provide lessons each week also provided a sense of support to the students. The
school-based music teacher was pleased with the project lesson teachers. In the focus group, he
reported that the attendance of the U-M students had been very consistent.
During a post-program interview, the school principal stated that, “The [El Sistema program] has
helped me get to know parents in a more meaningful way.” He went on to say, “Every parent I see of
an El Sistema student will say that they love it and they will tell me.”

B. U M Graduate Student Instructor Outcomes
The two (2) U-M lead graduate student instructors participated in a post-program interview to
offer their perspective of program’s effectiveness in achieving a stronger sense of social
consciousness through their provision of music instruction in the community.
Question 9: Have the SMTD graduate students developed a stronger sense of social
consciousness through their work at Mitchell Elementary using the El
Sistema model?
The instructors discussed the opportunity for U-M students to teach in communities as
advantage and strength of program. One of the graduate students decided to attend U-M
because of the El Sistema initiative: “One of the reasons I chose U-M was because of the opportunity
to participate in the El Sistema program.” Throughout the interview, the graduate students
expressed an appreciation for the opportunity to work in a public school to develop their
expertise in teaching group instruction. “It was great for me to develop these skills and gain more
experience teaching group instruction.”
When the U-M graduate student instructors were asked about their initial expectations for the
program during the post-program interview, it was evident they both expected to connect with
the students through their music instruction. One instructor stated, “I hoped to work closely with
the kids not just learning to play music, but how to play an instrument well and for them to gain
confidence on their instruments.” The other instructor added to this notion by stating that,
“[Initially,] I was anxious to meet kids from a different culture. I wanted to be able to handle situations
with the children.”
Upon further reflection of their initial expectations and what the graduate student instructors
gained from the program, engaging students, their families, and the community in a
meaningful way while contemporaneously improving their teaching acumen emerged as the
recurring themes:
“The teacher-student relationship is a special thing in of itself, music teaching is a unique subset
of that. Ideally, helping kids to gain fluency in a new language, in a new manner of expression,
and connect with a new corpus of art in the world – is exciting in of itself. It is great when you
get to know these children and their families, it enriches your life too.”

IX. Areas for Improvements
A. Attendance
The El Sistema Mitchell program had an average attendance rate of 65%. At the beginning of
the program, there was a strict policy regarding attendance. A number of problems began to
emerge with the strict policy, and it needed to be dropped. The school-based music teacher
explained in the post-program interview, “We knew that if we were strict about the policy about
attendance, we’d lose everybody. In fact, when I look back, I think we were maybe going to end up with 2
or 3 students if we enforced the policy about attendance that we started off with. There were only 2 or 3
kids that gave us almost full attendance.” The school-based music teacher expressed considerable
frustration in the focus group with how schedule interruptions were impacting the program,
“They added programs throughout the year. Math was one that got added. It was only
supposed to be first semester for 8 weeks. They found extra money from the governor,
and all of a sudden they had another one, and the day that the kids met changed on us.
Tuesdays and Thursdays were always our big days, and it worked out well because I
would see the kids during the day for string class, with a gentle reminder, ‘you’re coming
tonight, yes? Yes. We’ll see you tonight. Good.’ Second semester that switched so our
big days ended up being Monday and Wednesday, now I don’t have the reminder, the
schedule changed, attendance was affected…All of a sudden, it was hard to define what
school activities were a must that you could miss versus an activity that you wanted to
join. They also added Science Olympiad. There was another Science Makers that was
not in place at the beginning of the year that started; so a lot of activities.”
He also noted the many disruptions in January and February,
“there were so many snow days…and then Martin Luther King Day, and then,
President’s Day…we just had so many interruptions. They went to camp. The kids said,
“I can’t be here today, I have to get ready for camp.” And then when they came back,
they said “I can’t be here because I’m really tired from camp”….so there are a lot of
dynamics that hit me probably more than them [U-M graduate student instructors], with
the teachers here in the building…. I am thinking this could have been planned
differently, we could have better communication, who’s in charge? That was making it
tough. And then Michigan’s off for a week. I was frustrated because I felt like we were
losing momentum, but it’s not because we’re not doing a good job. We have a lot of
things fighting against us.”
Regular student attendance was the most significant challenge perceived by the other
instructors. In the post-program interview, a U-M graduate student instructor said, “one of the
stresses for us was regular attendance especially as certain other programs targeting the fifth-grade
population were being added. It’s great for the school that the kids have all these opportunities, but in

terms of the kind of music community that we’re trying to build, that’s always been the strongest when
the people are coming most consistently.”
Erratic attendance by some students made the teaching more challenging and impacted the
progress and morale of the entire group. In the focus group, one of the U-M graduate
instructors stated, “there are kids that keep coming every day and they keep working hard and then we
those other have the kids that don’t come, so it’s hard to put the pacing together between those two
groups.” An instructor also noted in the focus group that this is a problem in Venezuelan ES
programs; “it’s not that large scale here as there, the difference in levels is tinier, but I think that is a real
issue.” One instructor gave a specific example of a student who attends every day and has
made great progress being frustrated by the slow progress of the group, “she is very musical and
has had a superb year…but she is one that, if we’re drilling a single thing that the whole group is not
getting, she’ll just kind of check out, you know, feet askew, slumping viola, bow claw…I’m encouraging
her more and more to demand more of herself at all of those points that she’s making sure not that just her
fingers are going down in the right place, at the right time, the bow is making good tone.” A U-M
graduate student instructor reiterated this challenge in the post-interview.
Teachers need to “find a way where every kid at every different speed of learning feels
stimulated. We had faster kids and we had very much slower kids and we had kids way in
the middle, and of course, we went for the middle, so that the slower could catch up with
some individual attention and the faster ones wouldn’t get so bored….But I think with
the faster kids we didn’t achieve that. I think some of the faster kids would benefit if we
find a way for them to find challenges in the music. Require them to concentrate and to
really think and feel that they have to, push themselves too, to achieve that relationship of
achievement and effort.”
In the post-program interview, the school-based music teacher summarized the problem of
attendance disparity on achievement, “there’s a trickle down - those who were showing up are
growing. Those who are not need extra help…Now we’re starting to have disparity between abilities and
people looking at each other and trying to keep that cool. That was a little bit of a challenge, because they
started to grow apart.”
B. Repertoire selected for program
The teachers described that learning the three-part Star Spangled Banner from December –
February was challenging for the students. In the focus group interview, the school-based music
teacher said, “that was really a hurdle, you know, with The Star Spangled Banner, because it’s a big
push and then we had to really regroup, I call it “filling in”, and you know, I felt like we were losing a
little bit of energy in some of the students because they thought that was a big deal…but I’m hoping it
will show up in exit interviews that they feel like they can read music and that they feel good about that
and they feel strong.”

C. Expectations for students
Teachers reported it was challenging to have high expectations for the students and at the same
time, make them understand that they are not expected to play perfectly.
In the post-program interview, a U-M graduate student instructor stated, “I think if we could
manage to establish a mechanism of which the kids that are more involved, coming every day, could fulfill
a little bit more of the expectations in terms of the material they’re covering, the challenges they’re faced
with, I think that would be great. I think it is a very delicate balance, and we have that issue in El
Sistema in Venezuela too.”
In the post-program interview one respondent stated, “I think another challenge, and this was one
that was less of a challenge because we were working with the school-based music teacher, the fact that it’s
an after school program, I think in general, the kids’ focus was phenomenal, considering that this was the
end of, probably, a nine-hour day for them. By the time they were going home, but I know that it took a
lot of effort from school music teacher too, to make sure that the focus was there and I think that we
managed that challenge I think very well as a whole, but it was, I think, one of the more challenging
aspects of what we were doing.”
In the focus group, the school-based music teacher summarized some of the challenges,
“instruments are going home but they’re not necessarily being practiced…numbers have fallen
some…it’s been difficult because I don’t think it’s exactly what we were hoping it would be.”
D. Delivery of Instruction
In the post-program interviews, the teachers reported challenges of working together as a team
to deliver instruction. A U-M graduate student instructor said,
“one of the challenges was finding a productive and structured way to work together as a
group. I think that’s a challenge we still face in a way. I think that we have worked out
productive things with the group teaching, but I don’t know if we have gotten to a
moment where we can say that we are getting the best of everyone there.”
In the post-program interview, one of the U-M graduate student instructors suggested that it
was not always easy to find the balance between addressing the fast-paced, high-energy
rehearsals led by school music teacher (and cited in the literature as a “distinguishing
characteristic” of ESI programs) and the need for careful attention to details of proper technique
in beginning string instruction,

“there’s always a balance to strike between keeping momentum and excitement on the
kids part, and making sure that they’re, that whatever they’re doing, they’re doing
correctly, and I know my impulse as having come mostly from private violin teaching is
to really make sure that every time the kid is about to play, to go through the checklist to
make sure that they have their bow set, their violin set, and everything set up the way it
should be set, so they are always reinforcing the correct thing….The school music
teacher’s style seems much more about keeping the excitement , ‘we’re playing now! Go,
go, go!’. It’s kind of moving forward. I appreciate that the kids need that, but at the same
time, it took a while, a period of dissonance, where especially in the beginning, when the
kids didn’t have as much practice, getting themselves into playing position correctly,
where I felt like, ‘this is exciting, but they’re not getting the practice making sure that
they’re doing it correctly.’ So I think that was one challenge, and I think it ultimately
worked out pretty fine. “
E. Parental Support
In the focus group, the teachers described challenges with parent involvement. The only
activities for parents were the parent meeting at the beginning of the program and the concerts.
In describing the challenge of finding time to communicate with parents, the school music
teacher said, “We find that everybody is so busy. These guys are busy, I’m busy, I’m feeling swamped,
and you get tired. I mean, we’ve been here every day. We haven’t missed. We’ve kept the cart rolling,
but definitely, that’s one piece I know that we haven’t been able to really accomplish.” All the teachers
agreed it would be very useful to have administrative support to help communicate with
parents.
In the focus group, the school music teacher said, “I think if we can do it again, I want to do more
community type things to get the parents in. I think we really have to move on that. I keep saying
spaghetti dinners, because it would be easy to throw a pile of spaghetti together and some sauce and have
the kids playing for their parents.”
The teachers described an interest in developing strategies for how they will relate to parents.
There was also discussion of the idea of bringing the fifth-grade students to attend a University
Symphony Orchestra concert at Hill Auditorium.
Challenges related to parental support for the program were revealed by the teachers in the
interviews. In the focus group, a U-M graduate student instructor said,

“I think some of the people that aren’t showing up are some of the kids that were
struggling a little bit more. They don’t have someone telling them, ‘look, this is going to
be great for you, you need to work with this.’ They have people going, ‘well if it’s not
easy for you, maybe you should try something else, or maybe you shouldn’t go’ and I
think that’s been a challenge just because again, you can’t count on the parents to say
‘this is a great opportunity for you, we expect you to be there, we want you to be trying
your hardest.’ And without that, it’s kind of difficult because some of these kids that feel
like trying their hardest isn’t getting them as good a result as some of the other kids, even
if we are trying to communicate that we’re happy with what they’re doing.”
In the post-program interview, the school music teacher said, “the parents don’t see this as an
intellectual exercise. They don’t see the value in it, so moving forward, that’s something we have to
address. To educate parents in a gentle way….What this really looks like. What commitment looks like.
Bring in a guest speaker, not just hearing it from us.”

X. Implications
The after-school ESI pilot program provided the participating students with many musical and
social benefits. The fifth-grade students received high-quality musical instruction, delivered by
a team with diverse areas of expertise. The caliber of music instruction offers students from
ethically, racially, and economically diverse backgrounds with the opportunity to develop their
craft, while creating a level playing field for their competition in future music engagements.
A. Social Impact
Student participation in structured afterschool activities, such as El Sistema and other MSTLC
programs, supports the developmental needs of children and youth, particularly among
marginalized groups, the underprivileged and underserved. According to the research, youth
participation in extracurricular activities is linked to indicators of positive development
including school connectedness, student achievement, and educational attainment (Eccles,
Barber, Stone & Hunt, 2003). Students in the afterschool El Sistema program directly benefited
from: 1) improved attitudes towards school and their learning environment; and 2) increases in
their perceived feelings of acceptance and reinforcements by parents, school faculty, and
friends.
These findings support the literature on El Sistema and the impact of organized and
constructive after school programming. The data revealed El Sistema students’ improved
perceptions of meaningful social bonds with their music instructors, general education teachers,
student teachers, friends, the school principal, and family members. And, although there were
no significant pre and post-program changes in the students’ perception of their: i) music
ability; ii) personal interest or desire to play; iii) achievement motivation; and/or iv) overall
well-being, the findings did discover the students’ felt positive about their own music literacy,
talent, and learning ability. They were satisfied with their lives and conveyed a strong interest
to play music in the future.
Furthermore, the added extracurricular instruction of El Sistema helped to address the issue of
equal access to high-quality out-of-school and afterschool programing that exists across the
United States (Eccles et al, 2003). Due to a variety of social, political, and economic factors, there
is a limited availability of quality programming in poorer communities that have the highest
risk of adverse developmental outcomes (Eccles et al, 2003). Mitchell Elementary School is no
exception when compared to other schools in the district and across the State. The El Sistema
program not only enriches the School’s offering of afterschool programming, but the program
targets the mediating mechanisms (i.e., positive peer and adult relationships) of school
connectedness that support developmental health (Eccles et al, 2003).

El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514

El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Viewers also liked

Viewers also liked (20)

Similar to El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514

Similar to El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514 (20)

More from Anthony Michael Provenzano

More from Anthony Michael Provenzano (6)

El Sistema Formative and Outcome Evaluation Final Report DRAFT 071514