Your SlideShare is downloading. ×
Using empowerment evaluation to strengthen talent search progamming march 2011
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.

Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Using empowerment evaluation to strengthen talent search progamming march 2011


Published on

Presentation give to annual COE Policy Seminar-Washington DC March 2011

Presentation give to annual COE Policy Seminar-Washington DC March 2011

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Annual Seminar with Department of EducationUsing EmpowermentEvaluation and Data toStrengthen Your Talent SearchProgramMaggie Cahalan and Stephaine MillerPell Institute, COEMarch 9 2011
  • 2. Presentation Topics1. Overview of Empowerment Evaluation Concepts and Steps with example from COE’s new I-3 project testing new models for Talent Search (Maggie)2. Using Data for Evaluation Program Improvement and Student Success (Stepahanie)3. Breakout Session (Speed Dating Process to move through ideas about steps in empowerment evaluation of your program)—(Everyone--35 minutes)4. Group empowerment evaluation of our session (what you liked and did not like and how we could improve (Everyone--10 minutes)
  • 3. Personal Journey in Thinking About Evaluation Contractor Project Director National Evaluation of Student Support Services National Evaluation of Talent Search TRIO performance reporting support contracts Department of Education National Evaluation of Upward Bound study final technical monitor Came to see need for taking a more participatory approach involved and began Designing Next Generation of GEAR UP
  • 4. Next Generation of Gear Up EvaluationsGrew out of challenges (problems withmeaningful comparison group, validityissues, length of time to obtain results, black boxup or down studies not always useful for programimprovement, lack of involvement of stakeholdersmeant work not viewed as very useful to them)New Approach (18 month design work—sessionswith grantees, expert papers, systemic review ofpast evaluation methods and challenges, issuedcall for entries open to all grantees to designevaluations of what they viewed as promisingpractices, funded 44 grantees proposals andprovided technical assistance, and funded 12 toimplement the project led evaluations of rigorousdesigns)
  • 5. DefinitionEmpowerment evaluation is theuse of evaluationconcepts, techniques andfindings to foster improvementand self determination. Itemploys both qualitative andquantitative methodologies.Empowerment evaluation: Knowledgeand Tools for Self-Assessment andAccountabilityFetterman, Kaftaraian, and Wandersman1996
  • 6. Empowerment Evaluation Aims to increase the probability of program success by:1 Providing program stakeholders with tools for planning and implementation, and self evaluation of their program, and2 Mainstreaming evaluation as part of the planning and management of the program organization (Wandersman, et. Al, 2005, p. 25)
  • 7. Empowerment EvaluationWidely used Joint Committee for Education Program Standards have included concepts Amercian Evaluation Association and AERA workshops Public Health Field International Development work Education school systems
  • 8. ContrastsTraditional Evaluation Empowerment Evaluation External Internal Expert Coach or Critical friend Self-determination & Dependency capacity building Independent Collaboration judgment Makes use of Data Developed when data Revolution— internet, web, real time not available to all— interactive sharing of elite with resources knowledge—all to collect and skill to publish—face analyze book, blog, twitter
  • 9. Internal or External EvaluationThe dilemma of whether touse external or internalevaluation is as false as thatbetween qualitative andquantitative methods. Thesolution is always to use thebest of both, not just one orthe other (Michael Scriven)
  • 10. Learn by DoingOrganizations and their staff evaluate theirown strategiesMay hire an evaluator to work with themRather than presenting a reportcard, empowerment evaluators coach staffthrough an evaluation of their own strategiesby providing them with the knowledge, skillsand resources they need to conductevaluation
  • 11. 10 Key Principals (Wandersman et. Al 2005)1. Community Ownership—primary responsibility with organization and not outside evaluator2. Inclusion—involves representation of key stakeholders3. Democratic participation—highly collaborative— opportunity to voices questions—every stakeholders voice is heard4. Community knowledge-promotes growth of knowledge in community—stakeholders are considered to be in the best position to understand the issues and generate solutions to problems5. Evidence Based strategies-promotes use of strategies with high quality evidence—research evidence of effectiveness—evidence strategies contextualized to fit community
  • 12. 10 Key Principals (cont)6. Accountability-provides data that can be used to determine whether a strategy has achieved its goals—negative results are not punished;rather are use to inform change in a strategy or the selection of a new strategy for the purpose of producing better outcomes7. Improvement—Helps organizations improve strategies so that they are more likely to achieve stated goals—process and outcome evaluation (Rossi 1999)8. Organizational learning—fosters a culture of learning—view positive and negative feeedback as valuable information and believe that all strategies can be improved9. Social justice—increase capacity to reduce disparities that affect marginalized by persecution, discrimination, prejudice and intolerance10. Capacity building—builds capacity of organizations to conduct their own evaluations, understand results and use them to continuously improve organization
  • 13. 10 Steps (Chinman, Imm & Wandersman, 2004)1. Conduct needs and 6. Finalizing plan resource assessment 7. Evaluate process2. Develop goals and outcomes sought 8. Evaluate outcomes3. Selecting evidence 9. Implementing based strategies continuous quality improvement4. Assessing strategies fit 10.Sustaining efforts5. Building capacity
  • 14. Fetterman (2001)Define missionTaking stock-Planning for future
  • 15. Example---COE Investing in Innovation (I-3) projectED wishes to build capacity at all levels to use data and build ininternal and external evaluations as way to improveprogramming on an on-going basisOctober COE won I-3—Investment in Innovation--Using Data toInform College Access—Power KnowledgeBuild on and help validate GE/COE college access work in 2cities (Erie and Louisville) Enhanced whole school college access services—college coaches in school Intensive service Learning Communities COE Building large data warehouse Power Knowledge Tool for use by allContribute to innovative adaptive models for establishedFederal College Access programs such as Talent Search, GEARUP, Upward Bound
  • 16. COE (I-3) project Testing and Validating ---Model---Increased access to, skill, and use of data by all stakeholders (students, parents, school, college coaches, community) will lead to increased high school and college participation and success among students in high schoolsInternal Empowerment Evaluation andExternal evaluators both using the DATA TOOL
  • 17. DICAP--Builds and Enhances on GE-COE Model Multiple Power Knowledge Learning Communities Using data among all stakeholders—key District and school staff College access service providers and planners--College Coaches Students Parents Community Internal and External Evaluators
  • 18. Goals of UN Education forSustainable Development What if every person benefited from an education promoting development that is environmentally sound, socially equitable, culturally sensitive and economically just? What if learning was about knowledge and also about doing, being, interacting with others and changing the world? What if formal learning was enjoyable, hands-on and relevant to life outside school while addressing the problems of our world? What if every person benefited from genuine learning opportunities throughout life, in the workplace, and within the community? What if education systems prepared learners to enter the workforce as well as handle a crisis, be resilient, become responsible citizens, adapt to change, recognize and solve local problems with global roots, meet other cultures with respect, and create a peaceful and sustainable society?
  • 19. Partnership—Collaborative- Empowerment ModelAll stakeholders full partners—those mostimpacted –strongest rolesStakeholders Help set goals Identify assets, challenges, describe current baseline situation and data Take responsibility for identifying and implementing steps (actions) to meet goals Take responsibility for analysis, reflection on results and become learning community Transparency among all stakeholders
  • 20. Partnership—Collaborative- Empowerment ModelAll stakeholders full partners—those mostimpacted –strongest rolesStakeholders Help set goals Identify assets, challenges, describe current baseline situation and data Take responsibility for identifying and implementing steps (actions) to meet goals Take responsibility for analysis, reflection on results and become learning community Transparency among all stakeholders
  • 21. Smaller Intensive Learning CommunityPower knowledge Learning CommunityLeadership GroupSummer BridgeOn-going school year work Project to participate in building and using the data tool Project to communicate to whole school—lead to goals Project to evaluate and assess efforts and to improve the program –internal empowerment evaluation
  • 22. Theory of Participatory Action/Change Learning Reflection and Community Prepares various Participatory Communicate/di reportings/feedba goal setting sseminate ck for findings stakeholders Identify Inputs Impact (assets, baselin assessment and Power Data Tool es, challenges recommendatio and barriers) ns for future Identify indicators Identify and monitor fidelity of Assessment of Steps/actions intermediate implementation/ch indicators/outcomes needed to allenges/counterfa meet goals ctuals
  • 23. Logic of EvaluationGoal or purposeBaseline –where before interventionProgram or interventionMeasurement (including criteria)ProcessOutcomes and impacts
  • 24. Speed Dating Stations Idea Generating 1. Goal and Strategy setting 1. Setting Goals/Mission—what do want to change or improve—how much? 2. Evidence based strategy for improvement—what concrete steps do you need to take that you think might work? Now? Next year? Next five years? 2. Taking Stock 1. Baseline—where are you now—take stock of assets and barriers? 2. How can you assess or measure if what you are trying is working-----choose an evaluation design and indicators 3. Provide for gathering credible evidence---Data 3. Plan for Future—Learning Organization 1. Use data to analyze and reflect on what you find 2. Improve organizational capacity and the particular strategy evaluated
  • 25. Resources and contact informationContact---margaret.cahalan@pellinstitute.orgEmpowerment evaluation resources on_improvement-a.pdf FettermanPresentation.pdf