Using Developmental Evaluation to Support Prototyping: A Workshop

Uploaded on

There is increasing recognition that the problems we and our communities face are complex, dynamic and not easily solved. Our solutions must be as equally complex and dynamic and social innovation …

There is increasing recognition that the problems we and our communities face are complex, dynamic and not easily solved. Our solutions must be as equally complex and dynamic and social innovation (SI) has emerged as a promising method that can guide the development of these complex solutions. A key practice in social innovation is prototyping and the ability to learn quickly in order to make rapid, yet informed, decisions about the ongoing trajectory of our work.

This workshop will introduce participants to the principles and functions of prototyping in a SI initiative and the Developmental Evaluation approach that guide learning, decision making and iteration.

Leading Boldly Network:
Calgary’s Network for Collaborative Social Innovation

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Can be thought of as a research & design process Problem Space = “wicked problem”, complexity, dynamic, lots of moving/interrelated parts
  • Fast Prototyping (service) Slow Prototyping (process)
  • Prototyping requires a particular attitude and approach to project planning, one that front-loads effort and commits to more research and development before a service option is selected and significant investment is made ~ Prototyping top tips:
  • Reflection in Action calls the learner to particularly focus on the assumption behind a chosen strategy/action
  • Learning = the process of turning speculative predictions into reliable predictions. The best indicator that you are learning is that your predictions improve. Learning is hard in experiments where delays between actions and outcomes are long b/c it makes it harder to identify what needs to change
  • Highly customized process, but apply basic principles


  • 1. Using Developmental Evaluation toSupport Prototyping:A WorkshopLeading Boldly Network:Calgary’s Network for CollaborativeSocial Innovation
  • 2. Workshop Agenda• Getting Grounded:– The Collaborative Social Innovation Process– Your own experience with or questions aboutprototyping• What is Prototyping & How Is It Unique?• What is Developmental Evaluation (DE)?– Using DE to Support Prototyping
  • 3. Collaborative Social Innovation is…… the process of developing social innovationsthat are both social in their ends andcollaborative in their means. Specifically,collaborative social innovations include newideas (services, models and resource flows)that simultaneously address complex socialissues and create new social relationships orcollaborations. In other words they areinnovations that are both good for society andenhance society’s capacity to act.Adapted from the Young Foundation
  • 4. The LBN Process of Collaborative Social Innovation*Secure resources1. Co-Initiate – Set the right conditions and culture2. Co-Sensing – Frame the issue or design question Secure resources3. Co-Presence– Generate ideas4. Prepare for Prototyping – Planning5. Co-create / Prototype – Experiment to test ideas If prototype results prove to be promising, secure resources6. Co-Evolve - Implement7. Scale Out – Share innovations with a broader set of stakeholders8. Scale Up – Systems Change* This is a working model
  • 5. What is Prototyping?Prototype literally means first impression (proto= first, typos = impression)• A series of small-scale experiments, insimulated or real environments, that test a‘first impression’ or ‘hunch’ regarding how topositively influence change within a defined“problem space”
  • 6. What is Prototyping?• A core activity within thecollaborative social innovationprocess• Action– and learning–oriented• Undertaken with the purposeof designing, testing andrapidly iterating new andunverified processes, servicesor programs in order todiscover how they trulyinfluence the complexproblem you are trying toaddress• A practice for situations thatare non-routine andunpredictable, and thereforedifferent than what is requiredfor ongoing operations• Best done collaboratively withdiverse stakeholders
  • 7. What is Prototyping & How is itUnique?4 Prototyping Principles:1. It is used at the beginning, or developmentalphase, of a new process, service or program,where the way to the desired outcome isunknown2. It anticipatesfailure. Indeed a key objective ofprototyping is to create the conditions to beable to ‘fail safe’, namely endure failure that issurvivable (hence small scale) and provideslearning to inform the next iteration.
  • 8. What is Prototyping & How is itUnique?4 Prototyping Principles3. It embraces rapid iteration informed byfeedback from the system (stakeholders, end-users, etc.) it is trying to change4. It includes reflection-in-action, theintentional recording and documenting ofwhat is being learned as projects areimplemented
  • 9. What is Prototyping & How is itUnique?The focus is on learning:• Using evidence to move from “wildguesses” to “reliable forecasts”–The focus is not on results, but onthe overall trajectory of theinitiative
  • 10. What is Prototyping & How is itUnique?Moving from Wild Guesses to Reliable ForecastsWildguessesInformedestimatesReliableforecastsPredictionLearningTime
  • 11. How is Prototyping a Unique Practice?This practice enables us to:• “Think with our hands” as a way to investigate themerit of our proposed solution;• collect user feedback; and• observe how our actions influence (for better orworse) change within the domain we are seeking totransform.“The key to dealing with complexity is to focus onhaving good conversations about assumptions.”
  • 12. Why Is Prototyping an ImportantPractice?• In a time of tight budgets, prototyping can beused to test an unverified process, service orprogram before considerable resources areinvested into a large-scale initiative.– It helps to expose flaws and limitations in anapproach before significant investment has beenmade• Can help leaders/Boards manage the risksinvolved with trying something different or new• Can help us develop better products andservices
  • 13. How to Prototype1. Know when a prototypeproject is the right courseof action– Developmental phase /uncertain outcomes2. Develop a plan– Write down what you planto do, what you expect tohappen and why(Hypothesis of Record)– “In an innovative initiativethe plan is a hypothesis”3. Develop a clear learningagenda (DevelopmentalEvaluation)– Compare results to explicitset of assumptions4. Execute the Plan– Analyze differencesbetween what you thoughtwould happen and whatactually happened5. Based on what youlearned, revise the plan
  • 14. How to Prototype
  • 15. Small Group ConversationWhat stood out for you?What is becoming clearer?What questions are you carrying?
  • 16. Developmental evaluation isa small but demanding nichein the world of evaluation.
  • 17. Five NichesOngoing DevelopmentModel ReplicationCrisis SituationsPre-Formative (Innovation)Cross-Scale Complexity
  • 18. DE: 7 Things You Need to Know1.Essence2.Purpose3.Evaluator4.Methods5.Steps6.Reports7.Accountability
  • 19. 1. The Essence of DE• Developmental evaluation isn’t some particular methods ofrecipe-like steps to follow. It doesn’t offer a template ofstandard questions. It’s a mindset of inquiry into how tobring data to bear on what’s unfolding so as to guide anddevelop the unfolding. What that means and the timing ofthe inquiry will depend on the situation, context, peopleinvolved, and the fundamental principle of doing whatmakes sense for program development (Patton 2010: pp.75-6).
  • 20. Developmentis about creativethinking.Evaluationis about criticalthinking.DevelopmentalEvaluationholds theseIn balance.The Essence of DE
  • 21. 2. Purpose• Developmentalevaluation focuses ondevelopmentalquestions: What’s beingdeveloped? How iswhat’s being developedand what’s emerging tobe judged? Givenwhat’s been developedso far and what hasemerged, what’s next?
  • 22. TRADITIONAL EVALUATIONS DEVELOPMENTAL EVALUATIONSRender definitive judgments of success orfailure.Provide feedback, generate learnings, supportdirection or affirm changes in direction.Measure success against pre-determined goals. Develop new measures and monitoringmechanisms as goals emerge and evolve.Position the evaluator outside to assureindependence and objectivity.Position evaluation as an internal, team functionintegrated into action and ongoing interpretiveprocesses.Design the evaluation based on linear cause-effect logic models.Design the evaluation to capture systemdynamics, interdependencies, and emergentinterconnections.Aim to produce generalizable findings acrosstime and space.Aim to produce context-specific understandingsthat inform ongoing innovation.Accountability focused on and directed toeternal authorities and funders.Accountability centered on the innovators’ deepsense of fundamental values and commitments.Accountability to control and locate blame forfailures.Learning to respond to lack of control and stay intouch with what’s unfolding and therebyrespond to strategically.Evaluator controls the evaluation anddetermines the design based on the evaluator’sperspective on what is important.Evaluator collaborates in the change effort todesign a process that matches philosophicallyand organizationally.Evaluation engenders fear of failure. Evaluation supports hunger for learning.
  • 23. 3.1 The Evaluator• The developmental evaluator inquires intodevelopments, tracks developments, facilitatesinterpretation of developments and theirsignificance, and engages with innovators, changeagents, program staff, participants in the process,and funders around making judgments about what isbeing developed, how it is being developed, theconsequences and impacts of what has beendeveloped, and the next stages of development(Patton 2010, p. 227).
  • 24. 3.2 Evaluator RolesSurfacing IssuesFraming ConceptsTesting Quick IterationsTracking Developments
  • 25. 3.2.1 Surfacing IssuesRaise critical questions,tensions, debates thatemerge as part of thedevelopmental process.
  • 26. 3.2.2 Framing ConceptsFlush out and flesh outproblem definitions,lenses, frameworks and‘working models’underlying theconception and designof the intervention.
  • 27. 3.2.3 Testing Quick IterationsSurface –and interpret,draw conclusions andjudge -- real time datathrough techniquessuch as rapidreconnaissance,secondary evidence,modeling, outcomemapping, simulations,etc.
  • 28. 3.2.4 Tracking DevelopmentsDocumenting concretedevelopments – e.g.learning, newmodels, designfeatures, forks in theroad – in the evolutionof a group’s work.
  • 29. 3.3 The Evaluator-User RelationshipTraditional Evaluation• Positioned outside theaction to assureindependence andobjectivity.Developmental Evaluation• Positioned as team memberintegrated into the actionand ongoing process ofgathering and interpretingdata, framing issues,surfacing and testing modeldevelopments. May besituated externally orinternally.
  • 30. 3.4 CompetenciesTraditional EvaluatorCompetencies Reflective Practice. Technical Practice Situational Practice. Management Practice. Interpersonal Practice.Extra Competenciesfor DE Comfort with adaptiveprocesses High tolerance for ambiguity Can react/adapt quickly Effective communication withhyperactive, short-attention-span, action-orientedinnovators Larger tool kit of evaluativemethods
  • 31. 4. Methods• [I]t’s worth emphasizing that no definitive list of developmentalevaluation inquiry approaches can or should be constructed.Developmental evaluation creatively adapts whateverapproaches and methods fit the complexities of the situationand are responsive, appropriate, and credible to socialinnovators in opening up new understandings and guidingfurther development.• In being creative, the developmental evaluator is also practicaland pragmatic, doing the best job possible within availableresources and other constraints. Constraints always exist and dowhat constraints do -- constrain. Our ability to think ofalternatives is limited. Resources are always limited. Time is ofthe essence. We do what we can. Part of what we can do isadapt other inquiry traditions to the purposes of developmentalevaluation.
  • 32. Methods continuedKey Features• Contingency• Real Time• FlexibleAdaptive/Evolutionary• Evaluation methodsneed to be continuallyadapted to reflect thechanges in theemerging innovationand the learnings andquestions of theinnovators.
  • 33. 5. Steps
  • 34. 6. Reports• Dynamic complexities don’t slow down or wait for evaluatorsto write their reports, get them carefully edited, and thenapproved by higher authorities. Any method can be used butwill have to be adapted to the necessities of speed, real-timereporting and just-in-time, in-the-moment decision-making.• That is a major reason the developmental evaluator is part ofthe innovation team, to be present in real time as issues ariseand decisions have to be made […] Contrary to the usualpractice in evaluation of fixed designs that are implementedas planned, developmental evaluation designs can change asthe innovation unfolds and changes (Patton 2010, p. 335-6).
  • 35. Emerging Practices on ReportsOngoing Real Time Feedback• A variety of formats thatcapture key evaluationquestions, results andlearnings, and newdevelopments (e.g. memos,PPT, visual diagrams, etc.) ina way that is fits the patternof information flow anddecision-makers of theevaluation users?Periodic “DevelopmentalAccounts”• A record of the majordevelopments that haveemerged in adevelopmental process sofar (interim accounts)and/or at the end ofsignificant developmentalperiod (e.g. a shift intoformative evaluation).
  • 36. The Tracking the EvolutionOf the Intervention
  • 37. 7. AccountabilityTo Whom• People responsiblefor developingand/or adapting anintervention.For What• Developing a betterunderstanding of theissue being addressed,leverage points forchange, and developingan intervention.• This may – or may not –develop into a workable,stable or ultimatelysuccessful intervention.For How• A rigorous, data-based, user-friendlyand responsive,process of realitytesting that balancescreative and criticalthinking.• Guided by standardevaluation principlesand ethics.
  • 38. Tamarack Interview withMichael Quinn Patton onDE 4th Ed.Michael Quinn PattonThe DE Primer.Jamie Gamble. to Maybe.Frances WestleyBrenda ZimmermanMichael Quinn PattonResourcesDE 201: Practitioners GuideDozois, Langlois,Blanchet-CohenJW McConnell SiteDevelopmental EvaluationMichael Quinn Patton
  • 39. Extras for Prototyping
  • 40. types of prototyping•exploratory prototyping – testing the demand, feasibility& viability of an idea, quickly and inexpensively.•developmental prototyping – testing thedemand, feasibility & viability of an idea more detail, to seeif its works in practice.•slow prototyping – a concurrent focus on developing anidea or practice AND testing the capacity of the group tocarry out the idea.•fast prototyping – an exclusive focus on testing an ideaor practice.
  • 41. fast prototypingslow prototypingexploratoryprototypingdevelopmentalprototypingtesting a collaborative fundingmodel (simulation)exploring a new mission anddesign for employment program(scenarios)testing the efficacy of differentcampaign messages before roll out(random controlled trials)testing the buy-in, viabilityfeasibility of a fundingstrategy (role playing)developing & adapting practicesfor inter-agency collaboration(after action review)
  • 42. Questions• What questions – if any – emerge for you about DE?• What is alive for you about DE might contribute toyour work?
  • 43. Key Resources• Nesta (UK)• The Other Side of Innovation: Solving theExecution Challenge• The Open Book of Social Innovation• The Austin Centre for Design• IDEO• Human Centred Design