A Neighborhood Survey in the Nation’s Capital:  Balancing Rigor, Resources, and Respect
Upcoming SlideShare
Loading in...5
×
 

A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect

on

  • 191 views

Joint presentation by the DC Promise Neighborhood Initiative (DCPNI) and Urban Institute staff at the Eastern Evaluation Research Society's Annual Conference in 2014. Presentation focuses on DCPNI's ...

Joint presentation by the DC Promise Neighborhood Initiative (DCPNI) and Urban Institute staff at the Eastern Evaluation Research Society's Annual Conference in 2014. Presentation focuses on DCPNI's neighborhood survey - a community wide data collection project. The slides offer tips and suggestions on how to make the process as smooth as possible without compromising data collection rigor.

Statistics

Views

Total Views
191
Views on SlideShare
177
Embed Views
14

Actions

Likes
1
Downloads
5
Comments
0

3 Embeds 14

https://twitter.com 7
http://www.slideee.com 6
https://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Importance of validated items, items suggested by the PNI TA team
  • The neighborhood survey instrument includes the following domains: HousingNeighborhood amenitiesAccess to Food Neighborhood Supports Neighborhood Conditions and SafetyHousehold CompositionTravel, Education, Employment and Public AssistanceChild Health, Education, and Well-being
  • For analysesUse population weights to reflect full neighborhoodBe conscious of sample sizes
  • Differences:Respondents—family types are different, some are more/less literate, some have kids and others are singletonsInterviewers—Although everyone undergoes training,…
  • 444 surveys with 757 eligible households, which is equivalent to 59% response rate.

A Neighborhood Survey in the Nation’s Capital:  Balancing Rigor, Resources, and Respect A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect Presentation Transcript

  • A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect www.dcpni.org @dcpni 1 www.urban.org @urbaninstitute Eastern Evaluation Research Society 37th Annual Conference April 28, 2014
  • Today’s Panelists 2 Isaac Castillo Director of Data and Evaluation DC Promise Neighborhood Initiative @isaac_outcomes Samantha Greenberg Data and Evaluation Specialist DC Promise Neighborhood Initiative @eval_revolution Maia Woluchem Research Assistant Metropolitan Housing and Communities Policy Center Urban Institute Megan Gallagher Senior Research Associate Metropolitan Housing and Communities Policy Center Urban Institute April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • A Neighborhood Survey in the Nation’s Capital • The DC Promise Neighborhood Initiative’s Neighborhood Survey • It Takes a Neighborhood to Design a Survey • Getting Community Residents Excited About a Neighborhood Survey. • Data Collection in the 21st Century 3April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • The DC Promise Neighborhood Initiative’s Neighborhood Survey: When Community, Research, Evaluation, and Rigor All Come Together April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 4 Isaac Castillo Director of Data and Evaluation DC Promise Neighborhood Initiative Twitter: @Isaac_outcomes
  • As part of the White House Neighborhood Revitalization Initiative, Promise Neighborhoods align federal funding streams that invest in transforming neighborhoods of concentrated poverty into neighborhoods of opportunity. What is the Promise Neighborhood Approach? The INSPIRATION…. Harlem Children’s Zone (HCZ) inspired the U.S. Department of Education’s Promise Neighborhoods program, which launched in 2010. 5April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • What is DCPNI? • DCPNI is one of 12 communities awarded funding from the U.S. Department of Education. • DCPNI’s focus is on the Kenilworth-Parkside neighborhood of Washington, DC. • Founded as an independent nonprofit in 2012 shortly after receiving federal funding. 6April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • Kenilworth-Parkside by the Numbers • 5,725 people – 1,800 children • 98% African American, 55% are female • 50% of adults live in poverty • 49% of K-P residents experience food insecurity • 70% of K-P residents have a high school education or higher (compared to 88% in DC) • 85% of households with children are headed by single females 7April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • DCPNI’s Vision To end intergenerational poverty in the Kenilworth-Parkside community. 8April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • THE FIVE PROMISES 9April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • What does DCPNI do? 10 Fund Community-based organizations and schools to provide effective and research based programming Coordinate All services provided in the Promise Neighborhood to maximize effectiveness and avoid duplication Connect Community residents and students to available services provided by partners Train Everyone in the Promise Neighborhood to utilize information and advocate for themselves and their neighbors April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • DCPNI’s Work at a Glance 11 Fund Coordinate Provide Train Schools Partners Parents Children =+ April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • How Does DCPNI Measure Progress on Each Promise? • DCPNI has 15 federal indicators that we are required to track by the Department of Education. • Developing additional indicators on issues that are unique to Kenilworth-Parkside. – Teen birth rate – Mother’s educational attainment – Food insecurity • These indicators help us determine if we are progressing towards the Promises. 12April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • Sample of DCPNI Indicators • Parents read to their 0-5 year old children. • Increased feelings of safety among community residents. • Children ages 0-5 that have a medical home. 13April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute These data points need to be collected directly from community residents.
  • The Need for a Neighborhood Survey • Collect information directly from community residents. • Limited other publically (or privately) available data sets for the Kenilworth-Parkside population. • Desire to measure community level change over time. • What little data that was available was old, limited to parts of the community, or collected using low rigor methods. 14April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • The Most Significant Challenges • Very short timeframe – needed to complete entire process (from survey design to final analysis of data) in 9 months. • Low levels of literacy in the community (and low comfort level with technology). • High levels of suspicion of outsiders (and even suspicion of those from another part of the community). • High levels of distrust of government and government-like entities. 15April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • It Takes a Neighborhood to Design a Survey: The Methodology Behind the Kenilworth- Parkside Neighborhood Survey April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 16 Megan Gallagher Senior Research Associate Metropolitan Housing and Communities Policy Center Urban Institute
  • Methodology Overview April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 17 • Instrument Design • IRB Review • Sampling • Sampling frame • Strata, or key groupings • Random selection • Response Rates • Population Weights
  • Instrument Design April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 18 • Create a baseline assessment of the neighborhood from which to assess change • Many considerations, including • Department of Education Government Performance and Results Act (GPRA) indicators • DCPNI programmatic priorities • DCHA Choice Neighborhoods data needs • Urban Institute suggestions • Resident interests and concerns • Instrument length, nature of questions
  • Instrument Design April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 19 • Topics include: • Housing • Neighborhood amenities • Access to Food • Neighborhood Supports • Neighborhood Conditions and Safety • Household Composition • Travel, Education, Employment and Public Assistance • Child Health, Education, and Well-being
  • IRB Review April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 20 • Institutional Review Board (IRB) makes sure that research minimizes risks to human subjects • Reviewed and approved the plan for: • Requiring confidentiality pledges from staff • Obtaining consent from research subject • Roles for community resident and City Year corps member • Providing incentives for completed surveys • Keeping data confidential
  • Sampling April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 21 • Used a list of addresses for 2 census tracts that make up Kenilworth Parkside • Separated them into 3 sub-neighborhoods, or strata • Group 1: Kenilworth Courts • Group 2: KPRMC, Mayfair, Paradise, Lotus Square and Victory Square • Group 3: Eastland Gardens and Paradise • Randomly select addresses within each sub- neighborhood (SAS Proc Surveyselect) • Oversampled Group 1
  • Response Rate April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 22 • Eligible households • 872 addresses randomly selected • 115 unoccupied • 872 – 115 = 757 eligible households • Completed surveys • 444 completes • Response rate • 444 / 757 = 59% response rate
  • Population Weights April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 23 Sub- neighborhood Population % of Population Completed Surveys % of Surveys Weight Group 1: Kenilworth Courts 283 12% 133 30% =12%/30% =.3857 Group 2: KPRMC, Mayfair, Paradise, Lotus Square and Victory Square 1620 68% 217 49% =68%/49% =1.4012 Group 3: Eastland Gardens and Paradise 476 20% 94 21% =20%/21% =.9731 Total 2379 100% 444 100% 1.0000
  • Doing the Impossible: Getting Community Residents Excited About a Neighborhood Survey April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 24 Samantha Greenberg Data and Evaluation Specialist DC Promise Neighborhood Initiative Twitter: @eval_revolution
  • April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 25 Be inclusive Gain community support Build relationships Be culturally sensitive Add unique value Goals for Kenilworth-Parkside Survey:
  • April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 26 Survey Administration Teams: Kenilworth- Parkside Community Resident • Navigated the neighborhood • Greeted person at door • Introduced survey • Documented households visited • Handled incentives • Consent process • Asked survey questions • Documented answers on tablet • Handled technology Step 1: Build an Inclusive Process
  • 27 Community residents: • Familiar with neighborhood • Have relationships with survey takers • Knowledgeable about community history and assets Step 1: Build an Inclusive Process April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 28 City Year corps team members : • Versed in technology • Able to offer confidentiality (not from K-P) • Experienced serving D.C. communities Step 1: Build an Inclusive Process April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 29 Community residents: • Recruitment and screening • Background checks • Mandatory training session City Year Corps Team: • Partnership with City Year • Recruitment at full team event • Mandatory training session Hiring Process April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 30 NEIGHBORHOOD SURVEY TRAINING 30 • Structure of each work day • Safety precautions • How to introduce survey • How to document responses from potential survey respondents Training Process April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 31 Kenilworth- Parkside Community Resident Advantages of Two-person Teams • Increased likelihood of “open doors” • Decreased # of survey days needed • Easy navigation of community • Division of labor • Two people available to answer questions • Balance between confidentiality and trust • Increased safety April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 32 Challenges of Two-person Teams • Massive recruiting and hiring efforts • DCPNI staff capacity • Difficult to ensure fidelity • Downtime during surveys • Answering questions about survey when approached on street Kenilworth- Parkside Community Resident April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 33 Step 2: Prioritize cultural sensitivity • Survey questions tested with community • Community feedback encouraged throughout • Survey administrators participated in focus group • Concerns from community members noted for next survey April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 34 Step 3: Add Unique Value • I heart data booklet provides data snapshot • Data now exists on topics for which there are no other sources • Ongoing conversations with the K-P community about data April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
  • 35April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute Step 4: Build Relationships
  • 36April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute • Impacts of Kenilworth-Parkside neighborhood survey: Inclusive data collection Gained community support Built relationships Culturally sensitive survey Unique data collected Results: Community Support for Survey:
  • Data Collection in the 21st Century: The Use of Tablet Computers for a Neighborhood Survey April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 37 Maia Woluchem Research Assistant Metropolitan Housing and Communities Policy Center Urban Institute mwoluchem@urban.org
  • Challenges for Our Survey • Several teams approaching hundreds of doorsteps over several points in time • Each respondent was different from the last • Each interviewer is different from the last • Presenting several challenges – Monetary costs – Confidentiality – Standardization – Flexibility – Ease April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 38
  • Paper vs Plastic April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 39 Feature Paper Surveys Tablet Surveys Money Printing supplies, associated materials, post survey dataentry $200 per tablet,KeySurvey subscription Confidentiality Maintaining security of documentsin field, post survey Tablet’s secure connection and privacy capabilities Standardization Interview cues Flexibility May involve skipping pages Skip logic, conditionality Ease
  • Program • Used KeySurvey as our survey platform – Allowed ability to build conditional sections and complicated skip patterns – Allowed personalization based on household traits • Cost-reducing in terms of time, money and flexibility April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 40
  • Back-End v. Front-End • Level one: 159 questions and hundreds of answer choices • Level two: Variable names April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 41
  • Back-End v. Front-End • Level Three: Skip patterns – Sometimes a simple skip: April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 42
  • Back-End v. Front-End • Sometimes it’s something else entirely: April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 43
  • Back-End v. Front-End • Everything was automated, making things easy for everyone— • Setting: April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 44
  • Back-End v. Front-End (Consent) April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 45
  • Back-End v. Front-End (Personalization) April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 46
  • Back-End v. Front-End • Once programmed, survey looked like this— April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 47
  • Back-End v. Front-End April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 48
  • Administration: Who conducted the survey? • AmeriCorps members and college volunteers – Read aloud the questions to the householder – Householder responds verbally • Why? – Literacy rates among the community – Unfamiliarity with the technology – Standardization of questions and answers April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 49
  • Training • Back-end was sophisticated but front-end was very user friendly. • Training session has some ground rules: – Trust the programming – Don’t be pushy • And some golden rules: – Read the exact wording of the question as presented – Obtain consent – Work with your partner April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 50
  • Partnerships in Action • Morning Briefing (DCPNI Offices) – Community Resident: • Receive ten addresses, gift cards (4), survey receipts, door hangers, and a walkie talkie – Interviewer: • Receive same ten addresses and a tablet computer • Walk – Resident: Fill out tracking form to document address condition April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 51
  • Partnerships in Action • Doorstep – Community Resident: • Explain survey and present opportunity for giftcard • Answer questions householder may have • Document what happens – Interviewer: • Obtain consent • Administer survey • Administration (Interviewer & Community Resident) – At a comfortable place – Resident Role April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 52
  • Partnerships in Action • Post-Survey – Community Resident: • Offer respondent choice of incentives • Everyone signs receipt • What if no one is home? – Resident: • Leave door hanger • Document address so that another team could make attempt April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 53
  • Tablets are not Without Challenges • What isn’t done by hand must be done by computer – Careful time-intensive programming necessary • Technical difficulties – Potential for tablet failure (need for backup tablets) • Post-launch errors are data errors – Examples of missing answers or misplaced variables – Typing answers in leads to some data quality errors April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 54
  • But the Benefits? • Benefits greater than challenges (in our case) • Data/Tablet Security – End of day wipe and upload to UI SFTP – Confidential drives at UI – Cerberus for tablet security • Substantial results – Over four weekends in October and November, got 444 responses, a response rate of 59% overall April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 55
  • Audience Q&A April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute 56 Isaac Castillo Director of Data and Evaluation DC Promise Neighborhood Initiative @isaac_outcomes Samantha Greenberg Data and Evaluation Specialist DC Promise Neighborhood Initiative @eval_revolution Maia Woluchem Research Assistant Metropolitan Housing and Communities Policy Center Urban Institute Megan Gallagher Research Associate Metropolitan Housing and Communities Policy Center Urban Institute