Community voices and values: Reflecting community perspectives in definitions of program success Rebecca M. Teasdale presented on how to better incorporate community voices and values into definitions of program success in evaluation. She analyzed evaluation reports and found that while effectiveness was the most common criteria of success, experience, quality, relevance and equity were also used. Criteria were typically drawn from the project, evaluation or literature, but incorporating community and partner perspectives could provide a more nuanced understanding of success. The presentation concluded by emphasizing the importance of consulting communities to identify what is important to them and exploring how projects can be relevant, equitable and have consequences for the community.

1. Outside In:
Learning from Community Voices
in Visitor Studies
Veronica Garcia Luis, Exploratorium
Cecilia Garibay, Garibay Group
Leticia Pérez Castellanos, ENCRyM – INAH / UAM-I
Rebecca Teasdale, Garibay Group / University of Illinois

2. Community voices and values:
Reflecting community perspectives
in definitions of program success
Rebecca M. Teasdale
Garibay Group
University of Illinois at Urbana-Champaign

3. Defining success
Evaluation is a practice of making judgments about the merit,
worth, or significance of a program, exhibition, project, etc.
Scriven, 1991, 2013
This assessment is based on (implicit) criteria that distinguish
a “good” or “successful” project from one that is “bad” or
“unsuccessful”.
Davidson, 2005;
Greene, Boyce & Ahn, 2011
Julnes, 2012; Schwandt, 2015;
Scriven, 1991, 2012

4. Defining success
Definitions of success (evaluative criteria) are embedded in
the entire evaluation process
Evaluation questions
Variables or constructs of interest
Conclusions

5. Research questions
1. From what domains and sources do ISE evaluators draw
evaluative criteria?
2. In what ways do ISE evaluators reflect community values
and perspectives in the criteria they use?

6. Methods
Derived framework of domains and sources from literature
Analyzed sample of reports from InformalScience.org
Evaluation questions
Variables and constructs of interest
Conclusions
Conducted survey of report authors

7. Sample (n=25)
Evaluation type**
Summative (2017): 18
Formative (2016-17): 7
**Front-end evaluation, market research
and other visitor studies were excluded
from sample
Project type*
Exhibition 9
Program 9
Media project 8
Curriculum 1
Performance 1
*Reports for formal education and
professional development projects
were excluded; 2 reports
addressed multiple project types

8. Criteria domains: Categories of “success”

9. Criteria domains: Categories of “success”
Effectiveness
25

10. Criteria domains: Categories of “success”
Effectiveness
25
Experience
16

11. Quality
14
Criteria domains: Categories of “success”
Effectiveness
25
Experience
16

12. Quality
14
Criteria domains: Categories of “success”
Effectiveness
25
Experience
16
13
Relevance

13. Quality
14
Criteria domains: Categories of “success”
Effectiveness
25
Experience
16
13
Relevance Equity
5

14. Quality
14
Criteria domains: Categories of “success”
Effectiveness
25
Experience
16
13
Relevance Equity
5
Consequence
144

15. Criteria sources: Who or what defined “success”
5

16. Criteria sources: Who or what defined “success”
24
Project-
related
5

17. Criteria sources: Who or what defined “success”
24
Project-
related
5
12
Evaluation-
related

18. Criteria sources: Who or what defined “success”
24
Project-
related
5
12
Evaluation-
related
11
Substantive
literature

19. Criteria sources: Who or what defined “success”
24
Project-
related
5
12
Evaluation-
related
Partners
14
8
11
Substantive
literature

20. Criteria sources: Who or what defined “success”
24
Project-
related
5
Visitors or
community
147
12
Evaluation-
related
Partners
14
8
11
Substantive
literature

21. Implications
Consider assessing projects based on the extent to which:
Projects are relevant to the community
Opportunities, experiences and benefits are equitable
Activities and results are consequential

22. Implications
Consult the community when defining success by:
Talking with partners about priorities, interests, and
concerns
Identifying what is important to visitors and the broader
community

23. Implications
Explore combinations and intersections
Use of multiple criteria domains can support a more
nuanced exploration of “success”
Criteria drawn from multiple sources can highlight
similarities and differences in perspectives and values

24. References
Davidson, E. J. (2005). Criteria. In S. Mathison (Ed.), Encyclopedia of Evaluation
(pp. 91-92). Thousand Oaks, CA: SAGE Publications.
Greene, J. C., Boyce, A. S., & Ahn, J. (2011). A values-engaged, educative approach
for evaluating education programs: A guidebook for practice. Urbana, IL:
University of Illinois at Urbana-Champaign.
Julnes, G. (2012). Managing valuation. New Directions for Evaluation, 133, 3-15.
Schwandt, T. A. (2015). Evaluation foundations revisited: Cultivating a life of the
mind for practice. Palo Alto, CA: Stanford University Press.
Scriven, M. (1991). Evaluation thesaurus (Fourth ed.). Newbury Park CA: Sage.
Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for
Program Evaluation(58), 1-107.

25. Thank you!
Rebecca M. Teasdale
rebecca@garibaygroup.com
rteasd2@illinois.edu

26. La Casa del Museo (Mexico City, 1972-1980)
Some ways to define and measure community impact of
museum outreach in the long term.
Leticia Pérez Castellanos
PhD Candidate. ENCRyM – INAH / UAM-I

27. To challenge and modify the static image of traditional museums
To develop new audiences and balance its visitor demographics
“An experiment in bringing the museum to the people”
1972-1980

29. Some context

31. 1972 1973 1974 1975
First stage: Observatorio

32. 1976 1977 1978 1979-80
Second stage: Santo Domingo

33. The research: mixed methods
Archival analysis
Anthropology
Field work
Qualitative
interviews
Archaeological
lenses
Staff
&
Community
members

34. Analysis themes
Observatorio
First stage
1973 – 1975
Santo Domingo
Second & third stage
1976-1980
Aims The MNA established them.
A strategy of audience
development.
To collaborate with the
community and facilitte their
own ideas
Visitor studies
approach
Visitor survey, needs
assestment, monitoring
Collaboration
Exhibitions Local history, national history,
health
Community interests:
revolution, neigborhood
history, local traditions.
Activities Arts programing: cinema,
dance, music, hadncrafts
workshops
Trips to archaeological sites
and museums, production
workshops.
Staff Multidisciplinary, controlling Multidisciplinary, partnership
Visitors Random, youth and children Solid community, women,
youth and children
Some findings

35. Theoretical approach
Professional production
Consumption
Production
Cultural participation
Complete
cultural
experience
Outcomes = Impacts
What about resonances?
Long-term (+40 years)
Personal bios & Episodic
INCLUSSIVE
CHANGE

38. “I have recollections of the provisional museum, the history museum. It lasted a short time,
around two or three months. At the beginning there were some things like a market. It had
divisions with wood. In the floor they put some valijas (pottery) people used to drink, and some
maize and things form agriculture. It was a provisional museum.”
“It was a novelty. I felt happy to visit this place and enter, it was something new and attracted me.”
Mr. Marco Antonio García, 55 years. Around 10 when visited the museums. One time visitor.

39. “Of course I remember the anthropologist and the girls. I saw him a few times later on. Some
people come here (Current community center) and talk about those times, the visit to the musems
and when the pieles rojas came”
Mr. Fernando Díaz Enciso, 68 years. Around 24 when he worked in partnership with La Casa del
Museo staff. Director of the Community Center Escuelita Emiliano Zapata.

40. Then Now

41. VISITOR STUDIES ASSOCIATION CONERENCE
CHICAGO • JULY 21, 2018
Integrating Community
Voice in Summative
Evaluation
CECILIA GARIBAY

42. Ciencia Publica
• Active engagement with the
Parklet exhibits
• Deepened understanding of
STEM content
• Cultural relevance

43. Photo by Jim Wilson/New York Times Photo via InEnArt
Context

44. Photo by ClockworkGrue via Flickr SF Print Collective. Photo by Los Angeles Times
Context

45. Centering
Community Voice

46. • xxx
• xxx
• xxx
Race, ethnicity, language, gender,
age, disability, religion, sexual
orientation, gender identity,
education. social class, geographic
location, living situation

47. Select Insights

49. Outside In: Learning from
Community Voices in
Visitor Studies
Veronica Garcia-Luis
Exploratorium
07.21.18

50. Venturing Outdoors
• Rich history of R&D
exhibit development for
almost 50 yrs
• Artist-in-Residence
began in 70s
– Wave Organ
• Fort Mason 2001
– NSF funded

51. Studio for Public Spaces (SPS)
• 2014
• Investigates new ways to apply
inquiry-based learning
theories in public spaces
• Creates exhibits and
environments that encourage
play, exploration, creativity,
and social connection
• Partner with civic agencies,
CBOs, libraries, social service
orgs, parks, and schools
Shawn Lani, Founding Director, SPS

52. Community Engagement Model
• Levels of Engagement
– Contribution
– Collaboration
– Co-creation
• Adapted from the Public
Participation in Scientific
Research project at the Cornell
Lab of Ornithology by the
Oakland Museum of California
(Lashaw & Orantes 2017)

53. SPS Project Reflections on…
• Role of Community
• Semi-permanence of
space over time
• Public space use
beyond original goals
• Content accessibility in
public spaces

54. Ciencia
Pública 2.0
• Community Partners
• Buena Vista Horace
Mann School
• Food Bank
• PTA/ELAC
• Art & Gardening
Class
• Latinx community
• Collaboration/Co-creation
• 3 year public installation
• Water usage,
consumption, and source

55. Sound
Commons
• Community Partners
• SF Office of Planning
• Hunter’s Point
Family
• Downtown Streets
• UN Plaza audiences
• Collaboration
• 2 year public installation
• 5 sound making and
sound related
experiences

56. Buchanan
Mall
• Community Partners
• Green Streets
• Citizen Film
• Trust for Public Land
• SF Rec and Park
• Youth Leadership team
• 5 blocks of
Neighbors
• Co-creation
• 3 year public installation
• Space for neighborhood
gathering; audio stories
and gardens

57. Questions we continue exploring…
• How can we learn more about community’s
impact on public spaces beyond project
timeline?
• How do we continue to engage and learn from
community participants as they evolve the use
of the space over time?

58. Thank You!
vgarcia-luis@exploratorium.edu
Studio for Public Spaces

59. Topic A: Outcomes
How do we choose criteria to reflect community values and perspectives
in addition to desired outcomes?
How do we balance the need for assessing desired outcomes with the
desire to understand value as defined by the community?
Topic B: Impact
How can we learn more about the impact of these public spaces beyond
the project timeline?
How do we trace long-term impact? How can we identify qualitative
traces or resonances of our museum work?

60. Thank you!
Veronica Garcia Luis vgarcia-luis@exploratorium.edu
Cecilia Garibay cgaribay@garibaygroup.com
@garibaygroup
Leticia Pérez Castellanos leticiaperezcastellanos@gmail.com
@LetyPerezCast
Rebecca Teasdale rebecca@garibaygroup.com
rteasd2@Illinois.edu