Naturalistic Evaluation An approach to assessment that evolved from the work of researchers at the Indiana Center for Evaluation Wolf Tymitz Guba Lincoln Has been used in numerous applications including schools, social programs, museums, and health care
Two Streams Naturalistic Evaluation has evolved from two parallel streams ‘Responsive evaluation’ from the evaluation field Naturalistic methodologies from the area of qualitative ‘inquiry’ used by anthropologists and sociologists practicing ‘ethnography’ Combining the two makes Naturalistic Evaluation an intensive endeavor
Purpose Generally used to discover what is wrong (generally) and how to fix it (in a manner that will last), or how something is working (generally) and why (so you know how to keep doing it, or for possible use by others) This is more rigorous than finding out what is wrong (according to the audience) and deciding what to do (according to the audience)
Major Role of Evaluation Guba and Lincoln believe it is to respond to audience requirements for information in ways that take account of the different value perspectives of its members Naturalistic Evaluation puts the evaluator in the role of learner – the informants teach However, criteria for scientific rigor still apply!
General Phases Familiarization phase Gaining entry Understanding the Circumstance and the players Three C’s Collection of Data Classification and Analysis of Data Confirmation of Propositions Synthesis Presentation of findings and debriefing
Naturalistic Evaluation Pros When done correctly, it can provide a very complete picture of a program’s context, stakeholders, operations, effects (qualitative and quantitative), and systems of relationship A participant-oriented ‘gold standard’ Cons When done correctly, can Take a prohibitively long time Cost a lot Be very labor intensive When done incorrectly, can give a false sense of knowledge
Responsive EvaluationO Stake (175) is one of the major developers of the naturalistic evaluation approaches.O Stake was concerned that conventional approaches were not sufficiently receptive to the needs of the evaluation client.O Stake believed that evaluators should use whatever data-gathering schemes seem appropriate relying heavily on human observers and judges.
Robert StakeO First evaluation theorist to introduce Participant Oriented Evaluation into the field of education.O Introduced the theory of Responsive Evaluation which focuses on “re-directing data gathering and interpretative efforts around emerging issues of importance to program practitioners in the evaluation setting” (Abma, 2005)O Created the Countenance Framework in 1967. The model refers to the two faces of evaluation: description & judgment.O Description includes the evaluator’s observation and list of benchmarks for the activities being evaluated. Judgment is the evaluator’s overall rating of merit.