Jonathan A. PluckerUniversity of ConnecticutIEFE 2013 – February 18, 2013
} Historical Background} Strategic Versus Tactical Evaluation◦ A Model of Intervention Success} The Role of Evaluation Technical Assistance
} For many decades, the prevailingeducation intervention model was to:◦ Design promising interventions◦ Implement them as best as possible◦ Collect anecdotal evidence of success◦ Revise the interventions or, more likely,design new interventions
} It’s hard to tell, because so few interventionswere evaluated, and many were not evaluatedadequately.} This made it hard to build on past success, as“success” was defined by personal opinionand not rigorous, 3rd-party evidence.
} In recent years, many countries haveacknowledged the importance of programevaluation.} Implementations of evaluation have been lesswidespread.◦ Many of those efforts have struggled.} But why?
} U.S. Department of Education (ED) began toemphasize evaluation roughly 10 years ago:◦ All recipients of ED funds required to evaluate.◦ Creation of several new programs to train people todo high quality evaluations.} After 5 years, not much to show for it:◦ Required evaluations were either poor quality orinconclusive◦ No evidence of longer-term effects} Their conclusion: They were being tactical butnot strategic.
} Strategic Level:◦ Longer-term, systems-level thinking◦ “How can I use evaluation to reach my goals for thisprogram?”} Tactical Level:◦ Shorter-term, task-level thinking◦ “How can I best answer this specific evaluationquestion?”} Common framework in many fields (business,technology, military), but not education
} BOTH strategic and tactical evaluation arenecessary for a high quality, high impactevaluation program.} In most countries, we are seeing the qualityof tactical evaluation increase, often sharply.◦ U.S., China, The Netherlands, several Middle Easterncountries} Strategic evaluation is generally notincreasing.} Charter school example: Poor tactics, butpoor strategy, too?
Managing &Communicat-ing ChangeSustainingChange ThroughEffectiveInstitutions &PoliciesCreating Capacity forChangeCreating a Model for ChangeFour Macro Strategies for Achieving Outcomesin the National Education PlanHow has leadership preparationchanged to foster changemanagement?How have institutions andpolicies changed?How much capacity wascreated? Was it of highquality?Was the modelcreated? Does it lookpromising?(not really evaluation)Tactical Evaluation Examples
Managing &Communicat-ing ChangeSustainingChange ThroughEffectiveInstitutions &PoliciesCreating Capacity forChangeCreating a Model for ChangeStrategic EvaluationExamplesWas model designed in away that increasedcapacity was achievable?Have educator-policymaker feedbackloops been created?Are they working?Do some policies fostercertain outcomes butunintentionally discourageother outcomes?
Managing &Communicat-ing ChangeSustainingChange ThroughEffectiveInstitutions &PoliciesCreating Capacity forChangeCreating a Model for ChangePut it all together for a nicemix of …… Strategic… andTacticalevaluationstrategies
} An often overlooked aspect of systemicevaluation is technical assistance.} Essentially, to what extent are systems inplace to promote high quality, successfulevaluations?
} They were doing good tactical work …} … but had forgotten to put the necessarysystems in place to support evaluation.} My team designed and implemented theEvaluation Technical Assistance Continuumthat now serves as the guide for most EDevaluation efforts.
} Tuesday workshop on problem-basedlearning, 10:00-11:30, Mecca Hall} Wednesday workshop on creativity,10:30-12:00, Dammam & Arar Halls} I’ll also be at the University of ConnecticutExhibit Hall booth Tuesday and Wednesdayafternoons.
University ofConnecticutNeag Schoolof EducationJonathan A. Plucker, Ph.D.Endowed Professor ofEducationjonathan.firstname.lastname@example.org/jonathan.plucker