Qualitative Assessment “Qualitative inquiry cultivates the most useful of all human capacities—the capacity to learn from others.” Halcom’s Evaluation Laws. Cited in Patton (1990). Qualitative Evaluation and Research Methods. pp. 7. Moving Forward with Assessment Difficult Dialogues Initiative Assessment Conference Ford Foundation, NY, NY October 16-17, 2008 (H. Hernández-Gravelle)In addition to works cited, the author is indebted to the work of M. Patton; A. Driscoll; B.A. Miller; D.T. Spaulding; L.F.Gardiner, C. Anderson, and B.L. Cambridge; T.A. Angelo and K.P. Cross; and C. McTighe Musil, M. Garcia, Y.T. Moses, andD.G. Smith.
Qualitative Assessment Two Definitions “An inquiry process of understanding a social or human problem, based on building a complex, holistic picture, formed with words, reporting detailed views of informants, and conducted in a natural setting.” Creswell (1994). Research design: Qualitative & quantitative approaches. Cited in Detlor (2004). Towards Knowledge Portals. “Explores and tries to understand peoples beliefs, experiences, attitudes, behaviors and interactions. It generates non-numerical data…” http://new.wales.gov.uk/about/aboutresearch/social/glossa ry/?lang=en.
Qualitative Assessment Tidbits 1899: W.E. Dubois, undertook first social survey in the United States. 1900’s: Early phase of qualitative research. Rooted in Social Investigation. 1950’s: Government provided grants in support of educational research. Preferred modality is qualitative research. Late 1960’s: Qualitative research came into being. Development of socially responsive programs. Late 1900’s: Accountability movement in education results in increasing use of qualitative research and assessment.
Why DDI & Qualitative Assessment?Well suited in areas that are complex, multidimensionalsensitive and nuanced.Illustrative DDI Project Titles-Engaging Controversy: Religion, freedom and the topics of identityImagining the Future: Dissent, dialogue and the freedom to inquirePracticing Pluralism: Interactive theater, campus climate and academic freedomThe open nature of qualitative approach allows thesubjects to respond according to their own framework.“In assessing the impact of a faculty development workshop in adoption of newpedagogy, the teachers’ experience, reasons for participating in the workshop or thestudents that she teaches may impact her view of the value of the workshop and hersense of its applicability.”Quote from grantee.Helps to understand a program / situation as a whole.“..the holistic approach assumes that the whole is understood as complex system that isgreater than the sum of its parts.”Patton (1990). Qualitative Evaluation and Research Methods.
Why DDI & Qualitative Assessment? ContinuedTo provide a richer, more descriptive picture of theinformation being gathered or responses to thequestions being asked. For example, assessing anddescribing in a rich manner the quality of increasedcapacity for exchange between Arab and Israelistudents.On college campuses addressing pluralism issues,qualitative assessment can in itself serve as aprocess of inclusion and appreciation for multiplevalues and views.
SOME CHARACTERISTICS OF THE QUALITATIVE PROCESS Process extends through data reporting, data analysis, and utilization of findings. Researcher’s lens can impact data gathering and interpretation. Expertise in both assessment and subject area are significant. It is complex and requires multiple iterations to better uncover information (experiences, perspectives, views). Large volumes of data can accumulate quickly. Allocation of time and resources for data collection and analysis is imperative. Value: You get out what you put into it!
Qualitative Approaches Used by DDI FOCUS GROUPS: In qualitative assessment you go to the source (naturalistic group) INTERVIEWS: direct source PARTICIPANT OBSERVATIONS: To fully describe the process, you want to observe it PORTFOLIO ASSESSMENT: inductive FILM DOCUMENTATION: profiles and pictures
Assessment at its Core is about Student Learning Outcomes Question: How do we get from here to there? • Pluralism is ongoing work • Assessment is an ongoing process Tools to assess student learning outcomes: • Portfolio • Rubrics • In classroom exercise (presentations, debate) • Journal
ENHANCING DDI INDICATORS OFSUCCESS IN QUALITATIVE ASSESSMENT Ensuring indicators of success are sound: Important particularly for subjects and areas that are considered outside of the core curriculum or where the topics are controversial: “Unearth narratives about the complex history of race” Quote from grantee. Members check-in: This process can also serve to inform the constituencies and to begin process of change. Triangulation: Using multiple sources to confirm findings. Enlisting assessment advice and support from experienced and knowledgeable colleagues: Serves as a form of triangulation and check-in. Can address gaps in assessment capacity. Detailed reporting: Necessary to illustrate how the assessment team arrived at findings and to project the richness of the data.
DISTINGUISH ANECDOTAL EVIDENCE FROM QUALITATIVE DATAUnsolicited responses to programs in the form of criticism orpraise is considered anecdotal evidence. Such evidence isimportant and worthy of consideration, but it does notconstitute formal assessment. Qualitative approaches are notthe same as random responses and are designed to collectdata that can be verified and validated.The Ford Foundation is interested in vigorous assessmentthat: allows you to shape programs as they are implemented demonstrates what has been achieved enables you to examine what might be done differently asyou continue to develop programs
ANALYSIS MIRRORS PROCESS INDUCTIVE ON GOING MULTILAYERED COMPLEX ENRICHED THROUGH MEMBERS CHECK-IN
Qualitative Reporting Reporting formats are shaped by assessment purpose and by the meaning that is being conveyed. Typical qualitative reporting includes narrative accompanied by matrix or charts that demonstrate key issues or causal relationships. Ford Foundation assessment reporting seeks to communicate outcomes, indicators of program success and lessons learned through DDI programs. Use of case studies and rich description is of particular interest.