DESIGNING AND CONDUCTING CASE STUDIES.ppt

1. DESIGNING AND CONDUCTING CASE STUDIES FOR DEVELOPMENT EVALUATIONS A Preconference Workshop for the IDEAS Global Assembly in Johannesburg March 17, 2009 Linda Morra Imas Lmorra@worldbank.org lindag1000@aol.com © IPDET 2009

2. 2 Workshop Objectives • Understand defining features of case studies • Know when to use case studies and when each type of case study is appropriate • Contrast case study methods • Know how to increase case study reliability and validity, and determine sample size • Apply some case study methods © IPDET 2009

3. 3 BOOKS TO READ 1.Robert E. Stake (1995). The Art of Case Study Research, SAGE Pub., Thousand Oaks: CA. 2. Robert K. Yin (2005). Case Study Research: Design & Methods, Third Ed. Applied Social Research Methods Series, Volume 5. SAGE Pub., Thousand Oaks: CA. 3. Robert E. Stake (2006). Multiple Case Study Analysis, Guilford Press, New York: New York. www.amazon.com © IPDET 2009

4. 4 Experience with Case Studies • Who has conducted case studies? • How did you know it was a case study? © IPDET 2009

5. 5 Definition A case study is a method of learning about a complex instance, based on a comprehensive understanding of that instance obtained through extensive description and analysis of that instance taken as a whole and in its context. © IPDET 2009

6. 6 Discussion • A case study is a method of learning about a complex instance….. – Simple instance vs. complex instance • Based on a comprehensive understanding – Fly through qualify? • Obtained through extensive description and and analysis – Qualitative, not quantitative? • Taken as a whole and in its context – ??????? © IPDET 2009

7. 7 Case Studies versus Other Approaches • Histories: no access or control– rely on documents and artifacts (the past). • Experiments: done when behavior can be manipulated– a social experiment. • Surveys: large # of cases; thin descriptions; empirical generalizations © IPDET 2009

8. 8 DEFINING THE CASE STUDY Case studies emphasize thick descriptions – Not just descriptions of people, places, and events – But also the interpretations of those people most knowledgeable – Evaluator applies principle of relativity in identifying the best explanations © IPDET 2009 © IPDET 2009

9. 9 CASE STUDY PURPOSE “The real business of case study is particularization, not generalization. We take a particular case and come to know it well…There is emphasis on uniqueness…we emphasize placing an observer in the field to observe the workings of the case, one who records objectively what is happening but simultaneously examines its meaning and redirects observation to refine or substantiate those meanings.” Robert E. Stake. 1995, pg. 8-9 © IPDET 2009

10. 10 THE APPROACH • Holistic view of complex instances • Observation • Progressive focusing • Searching for patterns • Developing assertions © IPDET 2009

11. 11 When to Use A Case Study • Preferred strategy for “how” or “why” questions. • Can use for “what” questions, but …. – “Who” or “where” questions or “how many” or “how much” or “extent” questions favor survey strategies or analysis of archival records. © IPDET 2009

12. 12 Would You Use Case Study ? • If you want to know “how frequently” project sites are obtaining monthly target outputs, would you likely use a case study? • If you want to know more about cases that illustrate a certain pattern (e.g. sites not meeting targets), is case study appropriate? © IPDET 2009

13. 13 Would You Use Case Study ? • If you want to know what a best or worst case looks like and what makes it different, would case study be appropriate? • Would you use case study to answer “How many clients did the program serve?” Or “What kinds of benefits were received most often?” • If you want to know what a typical case looks like, would you use case study? © IPDET 2009

14. 14 Use Case Study • If you want to know “why” or “how” the program worked (or did not). • Key: exploratory or explanatory purpose; not a frequency or extent purpose. © IPDET 2009

15. 15 CASE STUDIES Myth or Fact © IPDET 2009

16. 16 #1 Myth or Fact? Case studies take a long time. © IPDET 2009

17. 17 # 2 Myth or Fact? Case studies only use qualitative methods. © IPDET 2009

18. 18 # 3 Myth or Fact? Case studies provide little basis for generalization. © IPDET 2009

19. 19 # 4 Myth or Fact? You can use case studies to draw causal inferences. © IPDET 2009

20. 20 THREE CASE STUDY CATEGORIES © IPDET 2009

21. 21 Three Case Study Categories • Descriptive: describe an intervention and the context in which it occurred. • Explanatory: explain causal links in interventions; link program implementation with program effects. • Combined : Brings together findings from several case studies to answer an evaluation question. Overlap in real life! © IPDET 2009

22. 22 3 Types of Descriptive Case Studies 1) Illustrative – Add realism and in-depth examples to other information about a program, project, or policy. – Describe what is happening and why, to show what a situation is like. – Often used to help interpret survey data. – Generally selected as typical or representative of important variations. – Number generally kept small. © IPDET 2009

23. 23 Types of Descriptive Case Studies 2) Exploratory To explore those situations where little is known about the intervention or its potential outcomes. © IPDET 2009

24. 24 Types of Descriptive Case Studies 3) Critical Instance To examine a single instance of unique interest or serve as a critical test of an assertion about a program, project, problem, or strategy. © IPDET 2009

25. 25 2 Types of Explanatory Case Studies 1) Program Implementation: The case study investigates operations, often at several sites, and often normatively. 2) Program Effects: The case study examines causality and usually involves multisite and multimethod assessments. © IPDET 2009

26. 26 1 Type of Combined Case Study 1. Cumulative: Brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause and effect. THINK SYNTHESIS! © IPDET 2009 © IPDET 2009

27. 27 EXERCISE 1. Is case study appropriate? 2. If yes, what type? © IPDET 2009

28. 28 DESIGNING THE CASE STUDY EVALUATION © IPDET 2009

29. 29 Doing Case Studies Design Prepare Define Collect Analyze Conclude © IPDET 2009

30. 30 Develop design Select cases Write data collection protocol Conduct 1st Case study Conduct 2nd case study Conduct remaining case studies Write case report Write case report Write case reports Draw cross-case conclusions Modify theory Develop policy implications Write cross- case report Case Study Method: Adapted from Yin, 2003. Pg. 50. Develop theory Identify Evaluation issue

31. 31 DEFINING THE EVALUATION ISSUE • Why is this particular program being evaluated at this particular time? • Who is the main client and what are their major concerns and questions? • How will they use the evaluation results? • Who are the major stakeholders and what are their concerns and questions? © IPDET 2009

32. 32 DEVELOP THEORY OF CHANGE • Conduct a literature review • Talk with experts • Engage stakeholders in the process of articulating the program theory • Develop conceptual model of how program/project/intervention is supposed to work = TOC • Highlight assumptions and relationships between the key elements © IPDET 2009

33. 33 Develop Design • Design: the overall strategy for systematically gathering and analyzing data to address the main evaluation purpose or issue. • Begins with questions identified through stakeholder analysis and TOC/logic model. • Is case study indicated? – General approach? – Depends also on the types of questions posed. © IPDET 2009

34. 34 Questions and Case Study Design • How does the new village well typically affect the lives of the women in the village? • What types of outcomes both intended and non-intended are associated with the Rural Roads Expansion Program? Illustrative Case Study Exploratory Case Study © IPDET 2009

35. 35 Questions and Case Study Design • How critically important is the support group to the success of the women’s Microenterprise Program? • How are women using the food and nutritional supplements provided by the provincial health clinics? Critical Instance Case Study Program Implementation Case Study © IPDET 2009

36. 36 Questions and Case Study Design • Did educational services provided by the agriculture extension center result in planting of the new grain seed and increased yields? • Does the body of evaluation literature indicate that the building of roads increases trade and benefits villages economically? Program Effects Case Study Cumulative Case Study © IPDET 2009

37. 37 Using Multiple Strategies • More than one strategy can be used in a study. • Which comes first, the survey or the case study? [Sam D. Sieber. “The Integration of Fieldwork and Survey Methods”. The American Journal of Sociology, Vol. 78, No. 6, (May, 1973), pp. 1335-1359. http://www.jstor.org/stable/2776390] © IPDET 2009

38. 38 DEFINING THE SAMPLE © IPDET 2009

39. 39 MULTIPLE CASES NEEDED WHEN – Bracketing. What is happening at the extremes? What explains differences? – Best cases/Worst cases. Why is (isn’t) it working? – Cluster. How do different types of instances compare with each other? – Representative. How do different types of instances chosen to represent important variations compare and why? © IPDET 2009

40. 40 Multiple Case Studies • Same study contains more than a single case. – For example, Study of school innovations. Independent innovations occur at different sites. – Every case should have a specific purpose. • Often needed for explanatory case studies. Replication - same results predicted for each case or contrasting results for predictable reasons. © IPDET 2009

41. 41 SINGLE CASE APPROPRIATE – Exploratory. In this particular circumstance, what is happening and why? Could be an extreme or unique case. – Typical. In a typical site, what is happening and why? – Critical instance. Does this single instance support the theory? © IPDET 2009

42. 42 Embedded Case Studies • Within a single case (single public program), the analysis includes outcomes from individual projects within the program. • Embedded units selected through sampling or cluster techniques. (Project characteristics) • More frequent versus holistic approach. © IPDET 2009

43. 43 Other Sampling Strategies • Probability Sample: What is happening as a whole and why? – Not appropriate for case study selection, but can use sampling for embedded units. • Convenience Sample: In this site, selected for data collection convenience, what is happening and why? – Not a good sampling strategy © IPDET 2009

44. 44 Defining the Sampling Unit • Family/kinship group • Small group – E.g. What is a community? • Geographic area – E.g. How define boundaries of a neighborhood? • Time period © IPDET 2009

45. 45 GENERALIZING FROM THE CASE STUDY SAMPLE © IPDET 2009

46. 46 GENERALIZING FROM THE SINGLE CASE STUDY? • Naturalistic generalization . Not scientific, but intuitive, based on personal or vicarious direct experience (Stake) • Transferability and fittingness. The “fit” or similarity between two cases determines the transferability of conclusions from one to the other (Lincoln & Guba) © IPDET 2009

47. 47 GENERALIZING FROM MULTIPLE CASE STUDIES Replication • A finding emerging repeatedly across study of numerous sites yields a good working hypothesis • The more different the sites, the stronger the working hypothesis © IPDET 2009

48. 48 Analytic Generalization Robert Yin: Ruling out rival hypotheses • If two or more cases are shown to support the same theory, replication may be claimed. • The empirical results are stronger if two or more cases support the same theory but do not support an equally plausible, rival theory. © IPDET 2009

49. 49 Illustration • Proposition: Increase in number of days of school attendance per year for girls in rural areas will occur when both extra income is provided to their families and the girls’ personal safety is assured. • Select 3-4 sites in which both interventions are present. • Select 3-4 sites where only extra income is provided to the family, with the prediction being little increase in girls’ school attendance. • Might select 3-4 sites where only physical safety is assured, with the prediction being little increase in girls’ school attendance. © IPDET 2009

50. 50 DATA COLLECTION METHODS © IPDET 2009

51. 51 Data Collection Methods • See handout on data collection options • Three key approaches – Observation – Interview – Document analysis © IPDET 2009

52. 52 Data Collection Methods OBSERVATION – Participant – Non-Participant – Obtrusive – Unobtrusive Group Review of Observation DCI © IPDET 2009

53. 53 Form & Exercise Data Collection Methods: Paired Observation Parts I & II © IPDET 2009

54. 54 Data Collection Methods • INTERVIEW – Individual – Small Group – Large Group © IPDET 2009

55. 55 Data Collection Methods A GOOD INTERVIEW IS A GOOD CONVERSATION! AND, WHAT MAKES FOR A GOOD CONVERSATION? © IPDET 2009

56. 56 Data Collection Methods • DOCUMENT ANALYSIS – Contemporary records – Historical records – Personal information – Public information © IPDET 2009

57. 57 TRIANGULATION: Checking for Consistency Increases Credibility • Methods triangulation – Most common – Using different data collection methods • Data triangulation – Using different data sources • Investigator triangulation – Using several evaluators © IPDET 2009

58. 58 Convergence of Multiple Sources of Evidence Archival Records Documents Interviews FACT Observations Physical Artifacts Focus Groups FACT Source: Adapted from Yin, 2003. Pg. 100

61. 61 Increasing Reliability • Reliability: if the study was repeated  the same results • Increase: use case study protocols • TRAIN & TRAIN! • Help develop case study data bases © IPDET 2009

62. 62 Selecting Data Collectors • Field Investigators and Skills Needed – Asking questions – Listening – Flexibility – Understanding of issues being studied – Objectivity © IPDET 2009

63. 63 Training READDISCUSS PRACTICE! – why the study is being done – theoretical background – basic concepts – terminology – study issues – evidence being sought – case study methods and tactics – data collection and analysis procedures – reporting requirements © IPDET 2009

64. 65 Case Study Protocol Contains the procedures and general rules to be followed. Increases the reliability of the evaluation. Anticipates problems that might arise and identifies solutions or strategies. © IPDET 2009

65. 66 Case Study Protocol “Must Haves” 2 Page Overview 1. Study background 2. Study purpose and objectives 3. Study scope and methodology 4. Case study issues, relevant readings 1 Page Letter of Introduction 1. Sponsoring org., official, title and contact detail 2. Study title 3. Data collector name, org., and assignment © IPDET 2009

67. 71 Data Analysis • Coding schemes • Cross validation • Disconfirming evidence • Themes • Patterns Group Review of Observation Analysis © IPDET 2009

68. 72 DATA ANALYSIS • Holistic fallacy • Using both qualitative and quantitative evidence • Computer software • Within and between sites © IPDET 2009

70. 74 Content Analysis • Ethnograph 5.03 www.qualisresearch.com – Searches, marks with code words, runs analysis • atlas.ti 5.2 www.atlasti.com – Searches recorded multimedia plus print • Nvivo 8 (formerly NUD*IST) www.qsrinternational.com/products_nvivo.aspx – Also multimedia, PDF files, and most languages • MAXqda 2007 (formerly winMAX) www.maxqda.com © IPDET 2009

71. 75 Content Analysis Continued • Lewins, A. & Silver, C. (2007) Using Software in Qualitative Research: A Step- By-Step Guide, SAGE Pub., London • Comparison of software © IPDET 2009

73. 77 STRATEGIES FOR PRESENTING FINDINGS • Natural history • Critical incident • Thematic • Within and between © IPDET 2009

76. 80 SUMMARY The Good Case Study… • Design follows operational specification of the issue (e.g., a logic model) • Discovery and flexibility are accompanied by re-design • Rival explanations are addressed • Evidence (data) and interpretation are clearly distinguished • Evidence is from multiple sources © IPDET 2009

77. 81 GET REAL LINDA! OK, SO NOW I KNOW ALL ABOUT DOING GOOD CASE STUDIES. BUT REALITY IS THAT I HAVE ABOUT ONE WEEK MAX TO SPEND AT EACH OF 4 SITES. WHAT CAN I DO? © IPDET 2009

79. 83 How Did We Do? 1. Understand defining features of case studies 2. Know when to use case studies and when each type of case study is appropriate 3. Contrast case study methods 4. Know how to increase case study reliability and validity, and determine sample size 5. Apply some case study methods © IPDET 2009

DESIGNING AND CONDUCTING CASE STUDIES.ppt

You are reading a preview.

Check these out next

DESIGNING AND CONDUCTING CASE STUDIES.ppt

More Related Content

Similar to DESIGNING AND CONDUCTING CASE STUDIES.ppt (20)

Recently uploaded (20)

DESIGNING AND CONDUCTING CASE STUDIES.ppt