Provenzano_Formative and Outcome Evaluation_El Sistema Music Program PART 1&2 - FINAL

El Sistema Ann Arbor Music Program 1

Running head: EL SISTEMA ANN ARBOR MUSIC PROGRAM EVALUATION
Formative and Outcome Evaluation of
El Sistema Ann Arbor Music Program
Anthony M. Provenzano
University of Michigan
Evaluation Proposal Part I & II
[Logic Model Submitted Separately]


STATEMENT OF NEED
El Sistema is a renowned Venezuelan music intervention program that has inspired a
global movement due to its social impact on fighting poverty and improving the human
condition. The El Sistema model provides high quality music instruction to children from a
broad range of socio-economic backgrounds in over twenty-five countries around the world
(Fundacion, 2012). Internationally adopted this socially innovative paradigm has reached Asian,
European, and Anglo countries including the United States and Canada (Tunstall, 2012). Now
thriving in Los Angeles, Boston, Philadelphia, Toronto, and other major cities and music centers,
this famed program fills a special need in the corridor between Ann Arbor and Detroit. The
reasons for starting a pilot program in southeast Michigan are threefold: 1) serving socially,
racially, and linguistically diverse children in the University of Michigan (U-M) community;
2) planning the scale up of the El Sistema music intervention to include schools and other venues
in Detroit; and 3) developing a Social Engagement curriculum of greater depth for University of
Michigan School of Music, Theatre, and Dance (SMTD).
Two types of evaluations will be used to help shape the El Sistema music intervention in
Ann Arbor and to verify if the pilot program is effective in achieving its objectives. A formative
evaluation will be conducted to document the emerging curriculum from year one of the El
Sistema Ann Arbor pilot project. The results of this evaluation will be used to manualize the El
Sistema program for future replication and expansion purposes. An outcome evaluation will be
conducted to offer evidence that the El Sistema program is meeting its intended outcomes.
There are two spheres of influence that have been integral to guiding the implementation
of the El Sistema model, and designing the formative and outcome evaluation:
1) Ann Arbor Public Schools/Mitchell Elementary School; and
2) The University of Michigan School of Music, Theatre, and Dance.


Spheres of Influence / Aims of the Evaluation
Ann Arbor Public Schools/Mitchell Elementary School
Mitchell Elementary is a school of approximately 280 students in the southeast area of
Ann Arbor. The school population is diverse with about 30% African American, 40%
Hispanic/Latino, 5% Asian students, 22% Caucasian and more than 30% of the students
speaking a language other English at home. Approximately 80% of students qualify for free or
reduced lunch. Administrators from Ann Arbor Public Schools responsible for research and
evaluation, and the Principal and Music Teacher at Mitchell Elementary, have expressed the
need for improving school performance, student academic achievement, and relationships with
families and the community. In response, key stakeholders from the El Sistema initiative have
adapted the Venezuelan music model to develop a theory of change and logic model (Refer to
Appendix 1: Logic Model) to target the specific needs, goals, and objectives of Mitchell
Elementary and Ann Arbor Public Schools.
Mitchell Elementary School is in its third year of an intensive partnership with the
University Of Michigan School Of Education, the Mitchell Scarlett Teaching and Learning
Collaborative (MSTLC). This partnership is designed to assist school personnel to improve
school achievement and to create a context for teacher education. Through the El Sistema
initiative, Mitchell Elementary intends to build upon other after-school and extended year
programs offered by the MSTLC that focus on literacy, math, and science instruction. In
partnership with the University of Michigan SMTD, School of Social Work (SSW), and aligned
with the El Sistema philosophy, Mitchell Elementary has established the following school-
specific program objectives for the El Sistema pilot program: 1) improving student academic
performance; 2) cultivating a strong sense of school pride; and 3) increasing family and


community engagement. The purpose of the evaluation is to document school and student
outcomes related to music literacy, individual and collective empowerment, and sense of
community and school pride; and to collect formative research data during the course of the
outcome evaluation to shape the El Sistema program.
University of Michigan School of Music, Theatre, and Dance
The University of Michigan School of Music, Theatre, and Dance aims to develop a
social engagement pedagogy and community-focused curriculum for SMTD students through
their participation in the El Sistema initiative. This objective is central to SMTD’s mission and
development as a professional performing arts school. Teaching students how to engage
communities in which they work is now a core skill for the 21st
Century performing artist and
arts educator. A social engagement curriculum situates SMTD students in the center of the global
concerns surrounding income inequality, economic deprivation, lack of access to education,
health care and other basic needs. It allows students opportunities to witness how their artistry
speaks to and transforms communities coping with these stressors. Through the collaborative
partnership with Mitchell Elementary School the El Sistema model will be used as the catalyst to
form a social engagement pedagogy and curriculum. By enriching SMTD’s curriculum with
social engagement teaching activities graduate students will: 1) develop a sense of social
consciousness; 2) increase opportunities for graduate students to provide instruction across
different community settings; and 3) increase employability through traditional and non-
traditional curricula. The purpose of the evaluation is to document the teaching experiences of
SMTD students, their development of a sense of social consciousness, and their perspective on
the potential for increased employment opportunities. The evaluation also aims to collect


formative research data to document the pedagogical approach used by the El Sistema model to
help form a social engagement curriculum for SMTD.
Key Stakeholders
El Sistema Ann Arbor is partially funded by The University of Michigan Third Century
Initiative, a collaboration between the School of Music, Theatre, and Dance and the School of
Social Work. The goal of the Third Century Initiative is to support interdisciplinary research
aimed at discovering innovative pedagogies that provide U-M students unique learning
experiences in the community. In this engagement, SMTD is responsible for implementing a
pilot program at Mitchell Elementary School using the El Sistema model. SMTD graduate
students with previous involvement in El Sistema provide music instruction in partnership with
Mitchell Elementary School’s Music Teacher. In addition to teaching children to play music as
part of the El Sistema program, the SMTD graduate students are also key informants in the
evaluation, helping to guide the development of SMTD’s social engagement pedagogy. Fifth
grade string instrument students from Mitchell Elementary School are eligible to enroll in the El
Sistema after-school program and will also be assessed as part of outcome evaluation.
Graduate students from the School of Social Work Curtis Center Program Evaluation
Group (CC-PEG) are tasked with collectively designing and conducting the evaluation of El
Sistema Ann Arbor under the direction of the Associate Dean and CC-PEG Director. In the third
year of the Mitchell Scarlett Teaching and Learning Collaborative, the School of Education is
also a partner in the El Sistema initiative, providing input and feedback on the implementation
and evaluation. Key leadership from Ann Arbor Public Schools who are responsible for
elementary education and research and evaluation provide direction and support in developing
the pilot program and evaluation plan.


EVALUATION QUESTIONS
Outcome Evaluation
1. What impact does participation in El Sistema Ann Arbor have on fundamentals of string instrument
technique (posture, instrument position, bow hold, bow stroke, tone quality, intonation), music
performance skills (performing rehearsed music, sight-reading music, playing from memory, playing
by ear, improvising), and selected aspects of music literacy?
2. What impact does El Sistema Ann Arbor have on students’ musical self-esteem, academic
achievement, and conduct?
3. Have students increasingly acknowledged their sense of control in evoking change in their own lives?
4. Have students become more aware of their community?
5. Has their sense of belonging or contribution to the community increased?
6. Do students who participate in El Sistema Ann Arbor have higher levels of enrollment in the middle
school instrument program?
7. Has Mitchell Elementary School developed a stronger sense of pride as a result of implementing the
El Sistema program?
8. Has the School developed stronger connections with students, families, and the community?
9. Have the SMTD graduate students developed a stronger sense of social consciousness through their
work at Mitchell Elementary using the El Sistema model?
Formative Evaluation
1. What is the curricular content of the El Sistema model?
2. What is occurring during music lessons to create the music culture of El Sistema?
3. What are the similarities and differences in the curricular content from other forms of group class
string instruction typically found in the United States?
4. What are the skills the SMTD Instructors and Mitchell Elementary Music Teacher possess?
5. What are the concepts and key elements of the El Sistema pedagogical approach?
6. How can the El Sistema model best inform a social engagement curriculum?


INTERVENTION: EL SISTEMA ANN ARBOR PILOT PROJECT
The philosophical underpinnings of the El Sistema intervention focus on personal
development as a means to build community. “Music nourishes and is nourished by that day-to-
day pace, awakening its aesthetical sense, encouraging without artificial postures the emergence
of harmony and beauty in places such as the city, the country and even within the very
individual (Fundacion, 2013).” The El Sistema model supports children and youth through the
philosophy that music can provide “self-concept, self-esteem, self-confidence, discipline,
patience, and commitment.” This helps young people to recognize how their individual actions
impact collective goals, while cultivating a supportive environment of families, schools, and
communities. At the core of El Sistema’s community engagement approach is the concept that
music provides a “spiritual” richness that awakens the human condition through a heightened
state of mind, greater awareness of ethical principles, and improved intellectual and emotional
skills conducive to overcoming poverty (Fundacion, 2013).
El Sistema Ann Arbor has developed a theory of change to incorporate the guiding
philosophical principles of the El Sistema literature, a review of social and music learning
theories, and the specific community needs of Ann Arbor. Precepts include music as an agent of
change, excellence, community, and supporting personal growth. The theory of change states:
“If children participate in regularly and extensive in high-quality,
ensemble-based music education, they will develop a strong sense
of personal agency, and sense of community through developing
musical literacy (Landin, A,, et al 2013.)”
El Sistema Ann Arbor Theory of Change


Target Populations
There are two populations that will directly benefit from the El Sistema intervention.
Students from Mitchell Elementary School will not only receive high-quality music education,
but they will also benefit from individual growth, improved academic performance, and a
heightened sense of connection to their school and community. SMTD graduate students will
benefit from their experience teaching diverse communities, obtaining experience working with
schools, and recognizing the intrinsic value of working in non-traditional settings.
Conceptual Framework
The theory of change and logic model provides an overview of how the El Sistema Ann
Arbor intervention will achieve its intended outcomes (Refer to Appendix 1: Logic Model). A
key underlining assumption of the program intervention is that ensemble-based music instruction
will improve students’ music literacy, heighten a sense of personal agency, and further develop
their sense of community. Resources for the project intervention include students, school,
musical instruments, music teacher, SMTD graduate student instructors, transportation, and
afternoon snacks.
Goal 1: Offer intensive after-school group music instruction over the course of 26 weeks for
fifth grade string instrument students at Mitchell Elementary School.
Objective 1.1: Increase the students’ ability to read/comprehend music, establish collective
music making skills, and develop an interest in music.
Objective 1.2: Increase students’ awareness of individual and collective responsibilities, self-
care, and respect for others.


Objective 1.3: Increase students’ ability to build new relationships, experience community
through ensemble-making music making, and develop a sense of school pride
in relation to the music program.
Other assumptions posited by the logic model include the expectations of Mitchell
Elementary School and the University of Michigan School of Music, Theatre, and Dance. The
designed intervention aims to cultivate a strong sense of school pride and will be assessed by
measuring Mitchell Elementary School’s relationships with students, families, and the
community, the School’s connection to the El Sistema movement, and the potential for
generating additional resources for the School as a result of the intervention. The resources
required to support this objective include the University of Michigan’s performing arts theatre,
music performers from the community, principal, parents, music instructors, performance
audiences, and transportation.
Goal 2: Host guest musical performances at Mitchell Elementary, provide transportation
for students to attend musical performances in community, and offer transportation for
parents to attend their child’s music concerts.
Objective 2.1: Increase school pride among student, teachers, families, and administrators
through activities of the El Sistema program.
Objective 2.2: Increase the school’s connection with students, families, and the community
through the activities of the El Sistema program.
Objective 2.3: Establish a school connection to a national movement through El Sistema.
Objective 2.4: Capture the media buzz surrounding the activities of the El Sistema initiative.


The University of Michigan School of Music, Theatre, and Dance aims to develop a
social engagement curriculum by better understanding the El Sistema pedagogical approach, and
through the support of SMTD students with increased opportunities for teaching in non-
traditional community settings. Resources from the project that contribute to achieving this
objective include U-M Third Century Initiative funding, U-M School of Social Work, School
Education, Ann Arbor Public Schools, Mitchell Elementary School, SMDT graduate instructors,
and the community of Ann Arbor.
Goal 3: Use the El Sistema pedagogical model to offer SMTD students untraditional
community teaching opportunities for the purpose of developing a social engagement
curriculum.
Objective 3.1: Increase student teaching opportunities to include diverse populations.
Objective 3.2: Increase for SMTD students’ sense of social consciousness.
Objective 3.3: Manualize the El Sistema Ann Arbor program and develop social engagement
training modules to enhance SMTD’s curriculum.
Intervention Strategies
v To maximize program recruitment and retention, students identified to participate in the
El Sistema program were already receiving instrumental music instruction as a standard
component of Mitchell Elementary School’s fifth grade music curriculum.
v In an effort to ensure student and parent participation in El Sistema Ann Arbor program,
students will be provided transportation to all activities including daily music instruction,
civic engagements, and musical performances. Parents will be provided transportation to
all five of their student’s musical performances.


v The SMTD music instructors have participated in El Sistema global movement as
students, and later as instructors. The previous experience of the SMTD graduate
students is an essential training component that directly impacts the success of El Sistema
intervention at Mitchell Elementary School and development of SMTD’s pedagogy.
v Key stakeholders for the El Sistema Ann Arbor Initiative are using a community-based
participatory approach to design the program intervention and evaluation plan.

REVIEW OF THE LITERATURE
El Sistema is a global movement that began thirty-eight years ago by Jose Antonio
Abreu, an economist and musician, discouraged by the fact that Venezuelans could not
participate in classical music. History tells us Abreu announced his plans to start a youth
orchestra in the burgeoning cosmopolitan capital city of Caracas, Venezuela. The initial
rehearsal took place in an abandoned parking structure with only eleven student musicians
showing up (Tunstall, 2012). As the orchestra grew, so did Abreu’s understanding of what was
being created – not only a music center, but a catalyst for changing social and emotional lives of
children and their families (Tunstall, 2012, p. 71).
“To me, an orchestra is first and foremost a way to encourage better
human development within children. That is why I always said, and
I say today, that [El Sistema] is not an artistic program but a human
development program through music. It is very important to be clear
about this. Because everything that happened then, and everything that
happened since then, has been a direct consequence of this concept.”
Jose Antonio Abreu


Today, there are over 300 youth orchestras or “nucleos” throughout the Venezuela with
the majority funding provided by their federal government. Approximately 370,000 children and
youth currently participate in El Sistema, in country that has population of approximately 28
million people. Of the children and youth in the El Sistema program, there have been estimates
that between 70 and 90 percent live in poverty (Tunstall, 2012). Mitchell Elementary School has
a similar rate of poverty, approximately 80 percent, which is measured by the proportion of
students receiving free or reduced lunch. Modeling the Venezuelan program, El Sistema Ann
Arbor will deliver free music instruction, provide instruments, transportation, uniforms,
nutritious snacks, and offer access to social services.
Fundacion Musical Simon Bolivar is the Venezuelan government foundation that
administers the El Sistema program, and in 2010 had an operating budget of $120 million.
Reports from the Intra-American Development Bank (IDB), a private funder of Fundacion
Musical Simon Bolivar provides evidence that El Sistema participants have demonstrated better
academic achievement and fewer behavioral problems than children and youth who are not
involved with El Sistema (IDB, 2013). Furthermore, Venezuela’s national high school drop-out
rate is over 26 percent, although only 6.9 percent for students who have participated in the El
Sistema program (Tunstall, 2012).
In the United States, El Sistema Colorado has experienced similar success with half of
teachers at Garden Place Elementary reporting positive changes in the academic performance of
students who participated in El Sistema. These results did not show a reduction in drop-out
rates. However, it did highlight strong performance indicators that support the school’s goals in
achieving long-term retention and success in school. Data from El Sistema Colorado also
demonstrated higher levels of reading proficiency (approximately 9% greater) for students in the


El Sistema program over non-El Sistema students. Results also depicted improvements to self-
discipline (with an average increase of 8.9%), and self-control (averaging an increase of 12.5%)
among students in the El Sistema program (El Sistema Colorado, 2013). El Sistema Colorado
offers evidence of the impact the El Sistema model has on student academic achievement and
behavior in the United States. Other research studies have provided evidence that students who
engage in learning to play a music instrument had significantly higher self-efficacy scores than
those who didn’t (Ritchie, L., & Williamon, A., 2011). In one particular study, researchers
provided evidence of strong positive correlation between self-efficacy and child well-being
(Ritchie, L., & Williamon, A., 2011).
In an attempt to replicate similar findings, El Sistema Ann Arbor will evaluate students’
sense of personal agency (self-efficacy, personal responsibility, self-care, and respect of others),
sense of community, sense of well-being, and perceived self-esteem in relation to their musical
ability. The evaluation also will examine student academic achievement and behavioral conduct
through a review of official school records that capture student attendance, grades, and behavior
to show correlations between increased sense of personal agency and academic achievement.
The replication of El Sistema in United States looks significantly different than
Venezuela’s centralized government funded system. In the United States El Sistema programs
are organically grown out of music centers, philanthropic organizations, non-profits, and
universities, and challenged with the attempt to sustain funding and scale up efforts.
Furthermore, in the United States public and private funders and researchers are more concerned
with creating quantifiable measures for the purpose of evaluating the effectiveness of the of El
Sistema intervention (Tunstall, 2012).


Although there is incomplete data on the effectiveness of the El Sistema model and
related outcomes, El Sistema Ann Arbor has incorporated a variety of psychological measures
supported by social and music learning theories. Furthermore, there is limited literature on the
El Sistema pedagogical approach. However, the El Sistema philosophy is adequately
documented. In fact, El Sistema Ann Arbor has ascribed to these ten fundamental principles,
Refer to Appendix 2: El Sistema Ann Arbor Fundamental Principles for the details of each
respective component:
1. Mission of social change.
2. Access and excellence.
3. The nucleo environment.
4. Intensity.
5. The use of ensemble.
6. The CATS teacher model: Citizen/Artist/Teacher/Scholar.
7. The multi-year continuum.
8. Family and community inclusion.
9. Connections and network.
10. Ambition and Achievement.
Eric Booth, El Sistema USA, 2010
CONCLUSION
A literature review of social and music learning theories was conducted to reveal
strategies to complement the limited data on the El Sistema pedagogical approach and evaluation
research. The racial, ethnic, linguistic, and socio-economic diversity, added to limited resources
of the Mitchell Elementary community, affirms the selection of the target population and
highlights the need for a socially innovative intervention to improve overall school performance
and the development of SMTD’s social engagement pedagogy.


RESEARCH DESIGN
A single group pretest-posttest multi-method research design will be used to evaluate the
effectiveness of the El Sistema Ann Arbor pilot project in achieving its desired program goals,
objectives, and outcomes, as determined by the proposed conceptual framework and logic model
(Refer to Appendix 1: El Sistema Ann Arbor Logic Model).
Single Group Pretest-Posttest Design
O1 X O2
X = Intervention O = Observation
The El Sistema Collaborative Partnership selected a pre-experimental design to deliver a cost
effective descriptive account of the Venezuelan model. Short time constraints leading up to the
project kick-off, and limited financial resources to support the implementation and evaluation,
were cited by the Collaborative as reasons for selecting this type of design. Although the pre-
experimental design is feasible provided the restraints placed on the community, this study
design presents limitations, and positive outcomes could be a result of factors other than the
program intervention.
Threats of validity and reliability exist because this design lacks scientific rigor. The
small sample size threatens the study’s reliability, and threats to validity are evident, particularly
maturation, given the intervention is delivered to elementary school-aged children. However,
when feasible, testing the impact, attrition, and instrumental effect will be monitored and
managed through the implementation of the evaluation protocols. Despite these limitations, this
type of design will provide key stakeholders with information to better understand the target
population, services, and attainment of expected outcomes for future replication and expansion
purposes.


Study Recruitment and Retention
A purposeful sample will be used to maximize program recruitment and retention, and to
help control project costs. Twenty-two students at Mitchell Elementary School who currently
receive string instrumental instruction as a standard component of the 5th
grade music curriculum
in Ann Arbor Public Schools will be targeted for this study. These students have been selected
to participate in the program because the El Sistema trained SMTD graduate instructors are an
experienced violinist/violist and cellist. Naturally this has excluded fifth grade horn instrument
students from participating in the after school program.
The parents of string students will be mailed letters providing information regarding the
El Sistema after school program and inviting them attend a program orientation. All
communication to the parents regarding the after school program will be provided in both
English and Spanish. During the orientation family members will be provided with: 1) the El
Sistema program permission slip for student enrollment; 2) the parental consent form for their
child’s participation in the program evaluation; and 3) a one item questionnaire regarding the
expectations for their child in the after school program. In an effort to increase program
participation and ensure retention, project staff will contact families via the telephone to inquire
about reasons for not participating, and to discover potential causes for students dropping-out.
Parental consent forms will be translated for Spanish speaking families, and bilingual
project staff will be available during the program orientation to answer any questions parents
have regarding student participation in the program and the exploratory study. Student
participation in the study is voluntary. A student’s experience in the El Sistema program will not
be positively or negatively impacted as a result of their participation in the evaluation. Students
have the right to refuse to answer any of questions, and at any point can choose not to participate


in the study. No one will be able to identify students from any of the data, analysis, findings, or
reports. Signed parental consents forms and sensitive survey data will be kept in a locked office
and located on the University of Michigan’s secured password protected computer network.
Data Collection and Measurements
The multi-method data collection processes for the program evaluation will be reviewed and
approved by the University of Michigan Institutional Review Board. The quantitative
procedures for this study design consist of conducting pretest (O1) and posttest (O2) student
surveys and teacher ratings. The psychosocial surveys and teacher ratings have been researched
and collectively identified to systematically align the selected measures with the objectives of the
El Sistema music intervention. Quantitative data collected from five student surveys and one
teacher rating survey will be triangulated with qualitative data collected from parents and music
instructors to add depth to the analysis. Students and 5th
grade teachers will be evaluated using
the following proven and reliable measures:
KEY
INFORMANT
NAME OF
MEASURE
AUTHOR(S) DATE
DEMONSTRATED EFFECTIVE
IN MEASURING
Students
Attitudes Towards
School
Anderson, S.
A.
1999 Student’s perception of school
Students
Satisfaction With
Life
Gadermann, A.
M., et al
2010 Individual well-being
Students Conflict Behavior
Kammratrh, L.
& Dweck, C. S.
2006 Relationship Conflict Behavior
Students
Parent Involvement
In Students’ Music
Education
Zdinski, S. 1993 Music environment at home
Students
Self-Esteem of
Music Ability
Schmitt, M. &
Zdinski, S.
1979
Self-confidence; Skills and abilities
in music; and Feelings of acceptance
and reinforcement by parents,
teachers, and friends
5th Grade
Teachers
School Achievement
Motivation
Chiu, L. H. 1997
Teacher rating of student
achievement motivation
demonstrated in the classroom


The qualitative procedures include administering semi-structured interviews, focus
groups, and questionnaires to study participants at time O1 and O2. Interview and focus group
questions for music instructors, school faculty, and parents will address student outcomes in
relation to music literacy, sense of community, and sense of personal agency; perceived sense of
school pride; and the music instructors’ perceived sense of social consciousness. Findings from
the outcome evaluation will demonstrate the effectiveness of the program.
Digital video recordings of a sample of music instruction lessons will conducted to assess
instrumental technique and other aspects of music literacy. The evaluation also will examine
student academic achievement and behavioral conduct through a review of official school
records that capture student attendance, grades, and behavior to show correlations between
increased sense of personal agency and academic achievement.
Analysis Plan
Upon completion of collecting pretest and posttest data, SPSS software Version 21.0 will
be used to organize, manage, and analyze data to evaluate the effectiveness of the intervention
outcomes. A paired analysis of individual student surveys will be conducted. A data dictionary
will be developed to help organize and facilitate the data entry processes. Descriptive statistics
will be performed to report frequencies. Inferential statistics will be computed to explore
possible associations between measures. Chi-square tests of independence will be conducted to
identify significance at p<.05. Associations among student characteristics, school grades, dosage
(attendance), and outcomes will be analyzed. The table below provides the quantitative measure,
level of measurement, and statistical technique corresponding to the specific program objectives.
For a list of the proposal objectives, refer to pages 8 – 10.


QUANTIATIVE ANALYSIS PLAN
Objective(s) Measure
Level of
Measurement
Statistical Technique
1.1 Student Grades Ordinal
Chi-Squared Test and
Percentages
1.1
1.2
Students’ self-esteem of music ability Ordinal
Percentages
1.2
Students Behavior – Number of
suspensions
Ratio Mean/Standard Deviation
1.2
1.3
Students’ conflict behavior Ordinal
Percentages
1.2
1.3
Teacher ratings of student school
achievement motivation
Ordinal
Percentages
1.3 Students’ attitudes towards school Ordinal
Percentages
1.3 Students’ satisfaction with life Ordinal
Percentages
1.3
Students’ perception of their parent
involvement in their music education
Ordinal
Percentages
N/A
Students’ age in years
Interval
Mean/Standard Deviation
Students’ gender Nominal
Frequencies and
Percentages
Students’ Race/Ethnicity Nominal
Frequencies and
Percentages
Number of students attending each
music lesson
Ratio Mean/Standard Deviation
Semi-structured interviews with graduate student music instructors, elementary music
school teacher, and school principal will be audio recorded and transcribed by the evaluation
team. The qualitative data collection will also include collecting parent questionnaires and
conducting a posttest focus group with family members. The focus group will be audio-recorded
and transcribed. Graduate students from the School of Social Work will analyze transcripts and
open-ended responses on questionnaires looking for concepts and themes. The qualitative data
collected will be organized and coded into categories using NVivo software. The table below
provides the qualitative measure corresponding to the specific program objectives in this
proposal. For a list of the proposal objectives, refer to pages 8 – 10.


QUALITATIVE ANALYSIS PLAN
Objective(s) Measure
1.1 Students’ instrumental technique and the development of other aspects of music
literacy
1.2
Students’ executive functioning skills (memory, planning, and managing time and
attention)
1.2
1.3
Students’ sense of belonging
1.3
Students’ personal sense of control
2.1
2.2
Students’ and families’ connection to school
2.1
2.2
School faculty’s connection with families and community
2.2
Additional resources provided to the school
2.3
2.4
Community’s connection to a National movement
3.1
Experiences gained from graduate student instructors
3.2
Graduate students’ recognition of value added teaching in diverse community settings
3.3
Skills the graduate school instructors and music teacher possess.
3.3
Curricular content of an El Sistema style program
3.3
Similarities and differences in El Sistema model from other curricula
CONCLUSION
Inspired by El Sistema’s pedagogical approach, and its social impact on fighting poverty
and improving the human condition, this famed Venezuelan model has emerged as the catalyst
for the University of Michigan to develop an “innovative, multi-disciplinary teaching and
scholarship approach” (U-M, 2013). Partially funded by the University of Michigan’s Third
Century Initiative, the School of Music, Theatre and Dance (SMTD), School of Social Work, and
School of Education have partnered with Ann Arbor Public Schools to develop an innovative music
intervention program for diverse communities, and a social engagement curriculum of greater depth


for SMTD graduate students. Through this interdisciplinary research initiative, faculty, graduate
students, and key stakeholders will use a Community-Based Participatory Research approach to
collaborate on the design, implementation, and evaluation of the El Sistema Ann Arbor music
program.
Existing literature, although limited, provides evidence of the effects of the El Sistema model
on improving social, health, and education outcomes. Furthermore, there are few studies on the El
Sistema pedagogical approach, however, this is one area the proposed evaluation can make a
contribution to the field. Through this research partnership, formative and outcome evaluations
will be used to help shape the El Sistema music intervention program, enhance the SMTD social
engagement curriculum and pedagogy, and verify if the pilot program is effective in achieving its
objectives.
Transforming approaches to teaching and scholarship through multidisciplinary social
engagement opportunities are the precepts for this collaborative research partnership. By
drawing from the innovative pedagogical approach of El Sistema, and combing proven research
methods from music, social work, and education, faculty, graduate students, and community
partners will gain from the collective intelligence of the different disciplines. This shared
knowledge will provide opportunities for scholars and communities to develop advanced
philosophical approaches for responding to challenging health, social, education, and economic
concerns.


REFERENCES
Anderson, S. A. (1999). Attitudes toward school scale. Center for Applied Research, University
of Connecticut, School of Family Studies.
Booth, E. (2010). Fundamental of el sistema. El Sistema USA. Last accessed, November 4, 2013.
http://ericbooth.net/the-fundamentals-of-el-sistema/
Chiu, L. H. (1997) Development and validation of the school achievement motivation rating
scale. Educational and Psychological Measurement, 57, 292-305.
El Sistema Colorado. El sistema colorado accomplishments. Last accessed November 4, 2013.
http://www.elsistemacolorado.org/our-program/accomplishments/
Fundacion Musical Simon Bolivar, Homepage. Last accessed, November 1, 2013
http://www.fesnojiv.gob.ve/es/el-sistema.html
Gadermann, A. M., Schonert-Reichl, K. A., and Zumbo, B. D. (2010). Investigating validity
evidence of the satisfaction with life scale adapted for children. Social Indicators
Research, 96, (2), 229-247.
Inter-American Development Bank. Last accessed, November 2, 2013.
http://www.iadb.org/en/inter-american-development-bank,2837.html
Landin, A,, Roldan, C., Sandoval, E., and Zanussi, S. (2013). Say yes to assess: An exploration
in el sistema-inspired assessment practice. Sistema Fellows Program at New England
Conservatory.
Kammrath, L., & Dweck, C.S. (2006). Voicing conflict: Preferred conflict strategies among
incremental and entity theorists. Personality and Social Psychology Bulletin, 32, 1497–
1508.


Ritchie, L., and Williamon, A. (2011). Primary school children’s self-efficacy for music
learning. Journal of Research in Music Education, 59 (2), 146-161
Schmitt, M. (1979). Development and validation of a measure of self-esteem of music ability.
(Doctoral Dissertation, University of Illinois at Urbana-Champaign). Dissertation
Abstracts International, 40, 5357A. (University Microfilms No. 80-09164.
Tunstall, Tricia. (2012). Changing lives: gustavo dudamel, el sistema, and the transformative
power of music, New York, Norton.
University of Michigan. Third Century Initiative. Last accessed, December 16, 2013.
http://thirdcentury.umich.edu/about/
Zdzinski, S. (1994). Parental involvement, gender, and learning outcomes among
instrumentalists. Contributions to Music Education, 21, 73-89.


ABSTRACT
El Sistema is a renowned Venezuelan music intervention program that has inspired a
global movement due to its social impact on fighting poverty and improving the human
condition. The El Sistema model provides high-quality music instruction to children from a
broad range of socio-economic backgrounds in over twenty-five countries around the world
(Fundacion, 2012). Internationally adopted, this socially innovative paradigm fills a special need
in the corridor between Ann Arbor and Detroit, Michigan. This reasons for starting this pilot
program in southeast Michigan are threefold: 1) serving socially, racially, and linguistically
diverse children in the University of Michigan (U-M) community; 2) planning the scale up of the
El Sistema music intervention to include schools and other venues in Detroit; and 3) developing
a Social Engagement curriculum of greater depth for University of Michigan School of Music,
Theatre, and Dance (SMTD). A single group pretest-posttest multi-method research design will
be used to evaluate the effectiveness of the El Sistema Ann Arbor pilot project. The El Sistema
program will target 5th
grade string instrument students at Mitchell Elementary School. If the
intervention is successful, it is expected that by having children participate in regularly and
extensively high-quality, ensemble music education students will develop music literacy, a
strong sense of personal agency, improved academic performance, and increased sense of
community. In addition, U-M graduate student music instructors will develop a sense of social
consciousness in delivering music instruction in untraditional community settings using the El
Sistema pedagogical model. Transforming approaches to teaching and scholarship through
multidisciplinary social engagement opportunities are the precepts for this collaborative research
partnership. By drawing from the innovative pedagogical approach of El Sistema, and combing
proven research methods from music, social work, and education, faculty, graduate students, and
community partners will gain from the collective intelligence of the different disciplines. This
shared knowledge will provide opportunities for scholars and communities to develop advanced
philosophical approaches for responding to challenging health, social, education, and economic
concerns.


Appendix 1: El Sistema Ann Arbor Logic Model
[Submitted As A Separate Document]


Appendix 2: El Sistema Ann Arbor Fundamental Principles
1. Mission of social change. El Sistema is a social change/youth development program that uses music
to enable every child to feel like an asset within her or his community, inside and outside the
“nucleo.” Students feel an ownership of the music making process, taking responsibility for both
individual and group improvement.
2. Access and excellence. El Sistema includes as many children as it can, bringing young people into its
community whenever possible, as young as possible, for as long as possible, whatever their
background or abilities. As El Sistema strives single mindedly toward musical excellence for all
students, it also provides intensive training at “Academies” for the most committed and gifted,
preparing them for the highest-level national orchestras and cultivating them as leaders in their own
communities. In this way and others, the ideals of access and excellence are maintained in a
productive balance that maximizes both the fullest success for all and highest accomplishment for
some.
3. The nucleo environment. The nucleo is a physical location, within the students’ neighborhood
where students live, that embodies the values and goals of El Sistema. It is a haven of safety, fun, joy,
and friendship, with an ethos of positivity and aspiration, where all students are encouraged to explore
their potential. The nucleo‘s doors are always open, and community members convene in its
hallways.
4. Intensity. Students spend a large amount of time at the nucleo, many hours per day, and almost all
days of the week, often building up to four hours per day, six days per week. Rehearsals are fast
paced and rigorous, demanding a durable commitment, personal responsibility, and a strong work
ethic. Through frequent performances, students have many opportunities to excel and to share their
accomplishments with their peers, family and community.
5. The use of ensemble. The learning in El Sistema is based in ensemble experience in which group
achievement is balanced with individualized attention. The orchestra acts as a model society in which
an atmosphere of competition between individuals is replaced by shared struggle. [Dr Abreu: “The
orchestra is the only group that comes together with the sole purpose of agreement.”] Smaller
ensembles and choruses adopt the same ethos.
6. The CATS teacher model: Citizen/Artist/Teacher/Scholar. Those who work at the nucleo take on
many jobs and multiple roles in relationship to the students. By acting as citizens, artists, teachers and
scholars, these adults encourage their students to develop holistically: as active musicians, helpful
educators, inquisitive learners and responsible civic contributors.
7. The multi-year continuum. El Sistema provides a “conveyor belt” of services, supporting its
students from early childhood into adulthood. Despite variation in resources and practices, all nucleos
work toward a full program. The “Academies” and other national teams have formed lists of
sequential repertoire, orchestral levels, and pedagogical practices that create a through line for every
child’s learning. Although each nucleo is encouraged to develop programs that suit its community,
shared practices and unified vision allow El Sistema to provide its students with a continuous musical
experience. The learning process develops the ear as the fundamental tool before the visual.


8. Family and community inclusion. Family participation is an essential aspiration of El Sistema.
Siblings often go to the same nucleo, parents attend classes with the youngest students, and families
form the bulk of the audience at orchestra concerts. Many sites have parent musical ensembles, and
all actively work to involve the community at large through outreach concerts.
9. Connections and network. Although nucleos run independently and customize their programs, they
are strongly connected to the national leadership organization, which provides financial resources but
more importantly gives the network a unified vision. Additionally, each nucleo is indispensably tied
to the many other nucleos that form the El Sistema network. These interdependent relationships are
manifested through events such as “seminarios,” which are intensive, project-based musical retreats
where orchestras share repertoire, streamline technique, and build personal and institutional
relationships. By uniting students and teachers from disparate parts of the country, the nucleo network
embodies the El Sistema ideals of sharing and learning.
10. Ambition and Achievement. El Sistema work is more than merely good for young people. It
aspires to transform youth lives, and widely succeeds in setting a healthier, fuller trajectory for stress
young lives. What changes young lives is being part of an aligned committed community that reaches
unreasonably high together, regularly, taking risks to achieve excellence that matters to them and to
others. This cycle of ambitious yearning and achievement, repeated consistently for the sake of
beauty and contribution to the community, changes, over time, thousands of small and larger choices
a young person makes in creating his/her life.
Eric Booth, El Sistema USA, 2010


Appendix 3: El Sistema Parent Consent English Spanish Dual


Appendix 4: El Sistema Student Surveys

Provenzano_Formative and Outcome Evaluation_El Sistema Music Program PART 1&2 - FINAL

Recommended

Recommended

More Related Content

Similar to Provenzano_Formative and Outcome Evaluation_El Sistema Music Program PART 1&2 - FINAL

Similar to Provenzano_Formative and Outcome Evaluation_El Sistema Music Program PART 1&2 - FINAL (20)

Provenzano_Formative and Outcome Evaluation_El Sistema Music Program PART 1&2 - FINAL