Piong_GSH Final Consulting Report

An Evaluation Framework for GSH Programs
A report prepared for:
Good Shepherd Housing Inc.
By:
Alex Dutton, Anthea Piong, Chao Zhang, and Gus Zimmerman
McCourt School of Public Policy
Georgetown University
April 27th
2015

Acknowledgements
First, we would like to thank Ms. Gail Williams for making this report possible.
Next, we would like to thank our client, Good Shepherd Housing Inc., for being so
accommodating to us throughout this project. Special individuals we would like to thank are Mr.
David Levine, Ms. Patricia Lopez, Mr. Ryan Nibblins, Mr. Chuck Rifae, and Ms. Karen Jupiter.
They were wonderful to work with and we hope that this report helps them further their mission
of providing affordable housing to those who need it.
And last but most definitely not least, we want to thank our professor, Dr. Micah Jensen, for his
tireless support and endless advice these past nine months. Without his input and constant
encouragement, this report would not be.

Table of Contents
Executive Summary .......................................................................................................................2
Introduction....................................................................................................................................7
Project Methodology......................................................................................................................9
Literature Review.........................................................................................................................10
Nonprofits and Evaluation........................................................................................................10
Housing Program Evaluation Methods.....................................................................................10
Emergency Assistance Program Evaluation Methods...............................................................11
Defining and Measuring Self-Sufficiency ....................................................................................13
I. ABC Measure...................................................................................................................14
II. ES Measure.......................................................................................................................16
Current Data Collection Procedures .............................................................................................19
I. ABC..................................................................................................................................19
II. ES.....................................................................................................................................22
Data Collection Recommendations ..............................................................................................25
Encoding data...........................................................................................................................25
I. ABC..................................................................................................................................26
II. ES.....................................................................................................................................27
Data Analysis Recommendations .................................................................................................31
I. Evaluation for ABC – Descriptive Statistics.....................................................................31
II. Evaluation for ABC – Regression Analysis......................................................................37
III. Evaluation for ES – Descriptive Statistics ....................................................................44
Possible Alternatives ....................................................................................................................47
I. ABC..................................................................................................................................47
II. ES.....................................................................................................................................49
Conclusion ...................................................................................................................................52
References....................................................................................................................................54
Appendix......................................................................................................................................56

1
List of Tables
Table 1 – Summary of Data Collection Recommendations and Benefits .......................................4
Table 2 – Summary of Data Analysis Recommendations and Benefits ..........................................6
Table 3 – Overview of ES follow-up questions............................................................................28
Table 4 – Benefits and costs of text message surveys...................................................................29
Table 5 – Example calculation of Client A’s change in net income compared to entry................35
Table 6 – Example calculation of Client A’s incremental change in net income over time ..........35
Table 7 – Example calculation of all ABC clients’ change in net income over time ....................36
Table 8 – Coding rules for ABC control variables .......................................................................40
Table 9 – Example calculation of ES evaluation ..........................................................................45
Table 10 – Benefits and Costs of Conducting Phone Surveys for ES...........................................49
Table 11 – Benefits and Costs of Conducting Online Surveys for ES ..........................................50
Table 12 – Benefits and Costs of Sending Postal Mail Surveys for ES ........................................50
List of Figures
Figure 1– Debt-Income ratio for ABC clients .............................................................................32
Figure 2 – Proportion of ABC clients below and above the debt-income ratio threshold .............33
Figure 3 – Debt-Income ratio for ABC clients by periods in program.........................................34
Figure 4 – Recommended OLS Model for Analyzing ABC Program Impact on Debt-Income
Ratio.............................................................................................................................................41
Figure 5 – Recommended OLS Model for Analyzing ABC Program Impact on Net Income ......42

2
Executive Summary
Background
Since 1974, Good Shepherd Housing Inc. (GSH) has been committed to reducing homelessness,
increasing community support, and promoting self-sufficiency among the working poor in Fairfax
County, Virginia. In these 40 years, GSH has grown from a small volunteer-led initiative into a
sizeable organization owning over 77 property units and operating with an annual budget of about
$2.5 million. In 2014, GSH showed their commitment to their mission statement by providing over
700 families with housing or services.
GSH programs – ABC and ES
GSH works to achieve their goals through four main programs: Apartments Budgeting Counseling
(ABC), Emergency Services (ES), Housing as Top Priority (HTP), and the Housing Locator
Program (HLP). GSH asked our team to focus on the larger two of the four programs; namely,
ABC and ES.
The ABC program is GSH’s largest program. Clients are typically the working poor who, usually
because of a low credit score, are unable to buy or rent housing in the open market. GSH acts as a
landlord and rents out housing to these clients in order to help them build good credit and eventually
find their own apartment or house by the end of the 2-year program. Our team sought to make
recommendations on how GSH can understand more about how they impact their clients’ self-
sufficiency throughout the program’s duration.
GSH’s ES program provides grants to clients to prevent evictions, avoid utility disconnections, pay
for first month’s rent, or pay for a new apartment’s security deposit. Qualifying clients are eligible
for a grant up to $350 within a 12-month period, and the grant has to go toward the above-
mentioned housing emergencies. Although these clients have minimal follow-up interaction with
GSH, GSH wants to determine how ES grants affect their clients’ self-sufficiency. Our report
makes recommendations for how GSH can achieve this despite the hard-to-evaluate nature of the
ES program.
In their five-year strategic plan released in 2011, GSH expressed the interest to “increase program
effectiveness to better meet clients’ needs” and outlined the need to develop performance
management capability. To help meet these goals, this evaluation project sought to answer the
following research question:
What methods should GSH use to assess and evaluate the effectiveness of the ABC and ES
programs in supporting GSH client’s self-sufficiency?

3
Project Methodology
In order to address GSH’s interest in program evaluation, our team engaged in three distinct
strategies to develop a basis for our recommendations.
1. We closely examined the relevant literature by seeing what methods similar programs use
for evaluation and by determining the current state of knowledge about housing program
effectiveness and self-sufficiency.
2. We interviewed key GSH personnel, who provided perspectives of internal staff on GSH
program operations and staff’s current practices.
3. Finally, we documented GSH’s current data collection methods and evaluation
procedures, which provided our team with a thorough look at how GSH operates its
programs.
Defining and Measuring Self-Sufficiency
Based on our review of the literature, we define self-sufficiency as a continuum of economic
security where a person is more or less self-sufficient, rather than a binary outcome where a person
is or is not self-sufficient.
Due to its focus on improving the economic situation of its clients as well as its extensive collection
of clients’ financial data, we recommend GSH use its clients’ debt-income ratio to measure ABC’s
effect on clients. The ratio captures both debt and income changes, and can serve as a marker of
client success as well as a program evaluation tool. Additionally, we recommend GSH use a client’s
monthly net income measurement as another variable of interest to measure client success. The
net income variable would be analyzed as a comparison between entry and current levels of net
income at the time of the analysis, in order to show progress over time.
As for ES, we believe its role is to prevent their clients from becoming less self-sufficient by
providing grants to clients facing evictions, utility shut-off, or an inability to move into a new
apartment due to first month’s rent or security deposit payments. Therefore, we recommend GSH
use housing security as the most important concept, measured by how many months a client has
paid their rent and utilities in full.
Data Collection Recommendations
We recommend that GSH adopt an evaluation strategy that accomplishes three main objectives:
1. Builds upon current GSH practices of data collection
2. Implements quantitative measures of analysis for both programs
3. Creates and maintains a dataset that can be used for analysis over time.
For ABC, we recommend administering a survey to all clients that captures their demographic data,
their personal history, income history, debt history, and tracks their progress in the ABC program.
This survey should be administered in intervals of 3 months, 6 months, 12 months, and at exit (or
graduation to tenancy) at 24 months.

4
As ABC clients are under the purview of GSH case management while they are in the program,
there are fewer challenges in collecting this survey data than in the ES program. Our
recommendations of what data to collect build off of ABC’s current data collection practices, but
also add our proposed dependent variables of interest, debt-income ratio and net income as
measures to track.
As for ES, we recommend the questions that GSH should be asking in their follow-up surveys and
suggest that GSH should collect follow-up data at entry, 2 months, and 6 months after the grant.
Our primary recommendation for ES data collection is a text message survey, administered at 2-
month and 6-month follow-up intervals. We specify our recommendations in 2 and 6 month follow-
up contact dates because of the diminishing effect the one-time ES grant has on a client’s self-
sufficiency over time.
A summary of the details of our data collection recommendations, as well as their benefits to GSH,
is laid out in Table 1 below.
Table 1 – Summary of Data Collection Recommendations and Benefits
Data Collection
Time Interval Method Benefits to GSH
ABC 3, 6, 12, and 24 months In-person survey Provides a standardized way to
collect and digitize data, integrates
well into current GSH data collection
process
ES 2 and 6 months after grant Text message
survey
Standardizes follow-up process and
presents simple, reliable way to
collect data
We also recommend that GSH consider requiring clients to remain contactable for at least 6 months
and ask that they respond to any follow-up questions that ES requests.
Data Analysis Recommendations
Variables of interest:
ABC 1: Debt-Income ratio
(Debt ÷ income)
ABC 2: Change in Net Income
(Current income – current debt) – (Entry income – Entry debt)
ES – Rent/security deposit assistance: On –time payment of rent
ES – Utility assistance: On-time payment of utility bill
ES – Housing Security: On-time payment of both rent and utilities
For the ABC program, we recommend two different variables of interest; a debt-income ratio, and
a change in monthly net income. For each of these variables of interest, we recommend using
descriptive statistics and regression analysis to determine program effectiveness.

5
For descriptive statistics of the debt-income ratio, we recommend that GSH pool individual client
ratios or create groups for clients at differing points of the program. After gathering that data, we
recommend GSH sort them into those with a ratio above 0.43 (the standard for gaining a qualified
mortgage) and those below. This analysis would give GSH a general picture of how many clients
could gain a mortgage and how participation in the program affects client ratios over time.
For descriptive statistics of the net income, we recommend that GSH group clients based on their
number of months in the program, compare their net income to their baseline, and then take an
average of the net incomes for each group. This analysis, while less statistically powerful, can give
GSH a general picture of how client’s income change over the program.
The key independent variable of interest for ABC’s regression analysis is months. This variable
measures the number of months each client has been participating in ABC’s program, collected at
our 3, 6, 12, and 24-month intervals. This variable is intended to capture GSH’s influence on ABC
client’s success over time. We recommended ABC’s regression models control for other influences
that might affect a client’s debt-income ratio or net income, such as age, race, education, and other
demographic controls.
For the ES program, we recommend GSH use descriptive statistics after collecting standardized
follow-up data. The statistics will come from our recommended survey design questions which
focus on stability after an ES grant, which varies depending on the type of grant each client received.
For example, if a client received a grant for utilities, the survey question would ask how many
months they paid their utilities in full at the follow-up intervals of 2 and 6 months after receiving
the grant. We recommend that GSH create an average number of months with both rent and utilities
paid in full after receiving the grant for each group in the follow-up period (i.e. average number of
months with rent and utilities paid in full at 2 months following grant, average number of months
with rent and utilities paid in full at 6 months following grant).
A summary of the details of our data analysis recommendations, and their benefits to GSH, can be
found in Table 2 on the following page.

6
Table 2 – Summary of Data Analysis Recommendations and Benefits
Data Analysis
Measures of Self-
Sufficiency
Key Independent
Variable
Methods Benefits to GSH
ABC Debt-Income Ratio;
Net Income
Months
participating in
the ABC program
Descriptive
statistics and
Regression
analysis
Determine overall
economic independence
of clients, measure
impact of program on
clients' self-sufficiency
ES Months with utilities
paid in full; Months
with rent paid in full;
Months with both
rent and utilities paid
in full
N/A Descriptive
statistics
Determine program's
success in maintaining
clients' housing security,
determine if clients are
delaying paying utilities
for rent (or vice versa)
Possible Alternatives
In the event that GSH finds it is not able to carry out our primary recommendations, we have laid
out some possible alternatives for GSH to explore.
For ABC, we recommend some additional data collection methods to increase client responsiveness
and ensure the collection of complete datasets. Some of these methods include the use of incentives,
sending out reminders, and seizing client-interaction opportunities. We also include quantitative
analysis alternatives such as having GSH staff undergo training, purchasing/downloading statistical
software, or acquiring external help with the analyses.
For ES, our alternatives focus more on the challenge of data collection as the analysis is simple and
easy to carry out. Each option in the chapter includes a table listing the relevant costs and benefits
in order to aid GSH in considering the viability of these alternatives.
1. Phone survey to client, landlord, or utility company
2. Online survey to client
3. Mail-in survey to client

7
Introduction
Good Shepherd Housing (GSH) is a nonprofit organization headquartered in Alexandria, Virginia,
that aims to reduce homelessness in Fairfax County. Over the past 40 years, GSH has gradually
grown from a small volunteer-led initiative to an organization that now provides housing to over
77 households a year. GSH has not conducted a formal evaluation of the impact of their programs
in the past, but in the summer of 2014, they asked our team of graduate students at Georgetown
University’s McCourt School of Public Policy to recommend an evaluation plan for two programs,
Apartments Budgeting Counseling (ABC) and Emergency Services (ES).
In 2011, GSH launched a five-year strategic plan comprised of three core goals to achieve their
vision of becoming the “best-in-class” provider of stable housing for households at risk of
homelessness in their service area. Our project focused on one of those three goals, “increas[ing]
program effectiveness to better meet clients’ needs” (GSH 2011). A strategy outlined by GSH in
achieving this goal included “developing performance management capability,” and three specific
sub-strategies (or tactics) were proposed. Our project pursued two of these tactics from the strategic
plan: developing quantitative measures for GSH’s program effectiveness and developing
comprehensive surveys to administer to clients.
For the last nine months, we have been studying GSH’s goals, programs, and existing evaluation
procedures, and reviewing research into the ways that similar programs have been evaluated. In
this report, we describe what we have learned from our research and present our recommendations
on how to answer the research question:
What methods should GSH use to assess and evaluate the effectiveness of the ABC and ES
programs in supporting GSH client’s self-sufficiency?
The first chapter of this report, titled Project Methodology, outlines the steps we took to answer the
research question. Specifically, we describe how we came to our key definitions and
recommendations through research, interviews, and close consultation of GSH staff members.
Next, the Literature Review chapter describes the important research we surveyed and analyzed
throughout the course of the project. This chapter is intended to give an overview of the academic
research that helped us connect ABC’s and ES’ evaluation needs to tested models in similar fields.
This chapter leads into Defining and Measuring Self-Sufficiency, which gives a brief background
of relevant literature and our recommended concepts of self-sufficiency for both programs for the
purposes of evaluation.
The following chapter, Existing Data and Procedures, outlines GSH’s current data collection
practices for both programs. After the documentation of GSH’s current data collection methods,
this report builds upon GSH’s current practices and offers recommendations for future data
collection in the Data Collection Recommendations chapter.

8
The Data Analysis Recommendations chapter then highlights how GSH should use the data that is
collected to implement and interpret evaluations for both programs. Finally, in the Possible
Alternatives chapter, we offer alternative suggestions for evaluation that GSH may wish to consider.

9
Project Methodology
In order to develop our recommendations for GSH’s two programs, Ms. Gail Williams, the Deputy
Director at GSH at the time, introduced us to the project by providing us with relevant documents
and information. Upon examining the original proposal drafted by GSH, our team began our
research into what the ABC and ES programs were and how we could best approach the task of
developing a program evaluation strategy for each program.
We surveyed research findings from similar programs relating to self-sufficiency, transitional
housing, emergency grant effectiveness, and nonprofit housing evaluation. Based upon this
research and in conjunction with GSH’s project specification, we developed a series of questions
for our first client meeting. Our meeting with Ms. Gail Williams on October 3rd
, 2014, helped us
better understand the context of each of the two programs we were analyzing and shaped our
perception of GSH’s current challenges and expectations for our project. Key expectations included:
that we would develop a data-driven approach to conducting impact analysis of each of the two
programs; that we would seek cost-effective ways to incorporate our recommendations into current
GSH practices; and that we would make recommendations that would help GSH shape its future
strategic plans based on evidence-based analysis. Challenges that were identified in our initial
meeting included GSH’s current lack of digitized data for its ABC and ES programs and the
inconsistent collection of follow-up data for both programs.
After meeting with Ms. Gail Williams and examining more documentation about each of the two
programs (e.g. example entry forms, Web R reports, and Strategic Planning documents.), we began
a series of interviews with Mr. Ryan Nibblins and Ms. Patricia Lopez, the head administrators of
the ABC and ES programs respectively. Our interviews with Mr. Nibblins and Ms. Lopez helped
us understand how each program was administered by providing us with detailed information about
intake procedures, program goals, and issues they wanted our final evaluation report to address.
Our team developed a Research Plan that was shared in our research plan presentation in the fall
semester on December 5th
, 2014, where Executive Director Mr. David Levine, Director of
Development Ms. Karen Jupiter, and Deputy Director Ms. Gail Williams attended. During this
meeting we presented our proposed methodology and preliminary research findings to GSH and
had a conversation that again helped shape what was most important to GSH and this project.
Beginning in January 2015, our team continued to assess GSH’s current practices by conducting
interviews with Mr. Chuck Rifae, GSH’s Housing Director, Mr. Ryan Nibblins, and Ms. Patricia
Lopez regarding the ABC program’s housing units, the ES program, and GSH’s current data. After
our informational interviews were finished, our team began developing our recommendations. Our
recommendations are a result of the combination of academic research conducted on similar
programs, conversations with GSH and its program administrators, careful consideration of GSH’s
needs and expectations, and application of statistical modeling techniques to develop the evaluation
approaches.

10
Literature Review
In seeking to answer our research question, we first attempted to survey the literature on non-profit
evaluation, housing program evaluation methods, self-sufficiency, and emergency assistance
program evaluation methods in order to determine the body of knowledge relevant to our project.
This step was crucial in determining our evaluation methods by giving us a firm background in
these topics relevant to GSH’s programs and goals. Our research findings on self-sufficiency are
outlined in the Defining and Measuring Self-Sufficiency chapter. This chapter outlines the results
of our research on the other topics and describes its relevance to GSH.
Nonprofits and Evaluation
Internal and external forces drive nonprofits to evaluate their programs. Internally, nonprofits are
motivated to improve their programming to serve more constituents, to provide their clients with
better and more comprehensive services, and, ultimately, eventually to get clients to a place of self-
sufficiency or stability through effective programming (Alaimo 2008). Externally, nonprofits are
driven to evaluate to impress funders and other stakeholders who require them to demonstrate their
successes in order to get more funding, prestige, and grant awards (Alaimo 2008).
During our research, we found several housing program evaluations studies that used mixed
methods research and found their approach to evaluation the most relatable to ABC’s current
operations. Mixed methods research refers to analysis that uses both quantitative (numbers/data)
and qualitative (feelings/personal survey) measures. This type of analysis, commonly found in the
social sciences, can be applied to GSH’s ABC and ES program evaluations because of its flexibility
and its ability to meet the unique needs of each specific program through design. While quantitative
analysis makes it possible to analyze program results systematically and over time, qualitative
analysis is valued for its ability to reveal the complexities and nuances that make every one of GSH
clients unique.
Housing Program Evaluation Methods
One evaluation of housing assistance programs commissioned by the U.S. Department of Housing
and Urban Development (HUD) and conducted by Lance Freeman in 2005 looked at the
relationship between housing assistance and dependency on federal assistance programs. To
evaluate HUD’s housing assistance programs, the model that Freeman used predicted the likelihood
of a participant exiting the program based on different factors that occur over time (Freeman 2005).
The study found that demographic factors of increased age, having children, and being married
reduced the chances of exiting housing assistance because of the major disruption that moving
homes or losing financial assistance causes for that population.
Six years later, another housing assistance program evaluation was commissioned by HUD to
evaluate the effectiveness of one of HUD’s programs, the Family Self-Sufficiency (FSS) program
(Silva 2011). The FSS program was designed to connect housing assistance recipients with tools
to promote financial self-sufficiency, eventually allowing participants to graduate from the program.

11
Silva’s evaluation model was used to estimate which measured factors (age, race, gender, and FSS
program size) were useful predictors of whether or not a participant was likely to graduate the
program. The study found that if participants had a high school diploma prior to entering the FSS
program, they were twice as likely to graduate the program compared to those who did not have a
high school diploma (2011). These studies led us to realize that the lack of a program “exit” goal
for ABC would limit GSH’s ability to evaluate program impact. Therefore, we determined that we
would need a clear measure of client success to generate a successful evaluation strategy.
Emergency Assistance Program Evaluation Methods
Though we found several studies on housing program evaluation, few studies have evaluated
emergency assistance programs similar to ES. Nonetheless, we did find some useful results from
studies of emergency utility assistance programs, which provided us with an idea of how to measure
the utility-assistance side of ES. According to a paper by David Hasson (2002), there exist two
parts to most utility financial assistance packages: demand reduction, and bill relief. GSH’s ES
program bears the most similarity to a bill relief model, such as the federal Low Income Home
Energy Assistance Program (LIHEAP).
A recent report examined the impact of the LIHEAP on household energy security (Murray and
Mills 2014). LIHEAP is the largest bill relief program in the US, offering one-time financial
assistance to low income and vulnerable households in paying their home heating or cooling bills.
LIHEAP funding is distributed to other governmental entities, states, or directly to utility
companies, and are often available on a first-come-first-serve basis. In Virginia, LIHEAP funding
for 2014 was $81.9 million and was available for three periods of the year: heating, cooling, and
crisis (the coldest months). Recipients received an average grant of $306 for heating in 2012.
Virginian residents are eligible for LIHEAP if they are below 130% of the poverty level (LIHEAP
2014).
The Murray and Mills (2014) report aimed to measure whether participation in the LIHEAP
program had any impact on a household’s energy insecurity. A family is more likely to be energy
insecure if they have a high energy burden - the portion of income spent on utilities - because they
will be more susceptible to energy price shocks. On average, low-income households spend 13.6%
of their income on utilities alone, almost double the national average of 7%. In their quantitative
analysis model, Murray and Mills controlled for variables such as household demographic
characteristics, residential characteristics, and regional characteristics, as these can each play a role
in affecting a household’s LIHEAP participation and energy insecurity. Murray and Mills found
that energy-insecure households significantly benefited from receipt of LIHEAP and that
reductions to LIHEAP have a strong negative impact on both low-income households and utility
firms.
However, there are important differences between utility assistance programs such as LIHEAP and
GSH’s ES program. The biggest difference is that ES assists not only with utility payments, but
also with rent and security deposit payments. ES also has subjective eligibility requirements, unlike
LIHEAP, which only uses income and energy spending as an objective metric for eligibility. For
example, ES clients will only receive a grant if they can show that they are in a one-time crisis

12
situation that has caused a temporary hardship. By screening their clients in such a way, ES is
already trying to maximize their impact by choosing clients who will most benefit from this grant.
These studies led us to conclude that LIHEAP’s method of analysis in each study was not entirely
applicable to ES because of ES’ multiple grant categories, which include grants for emergency
rental assistance, security deposit, and first month’s rent. The LIHEAP studies pointed to using
housing security as a literature-based measure of self-sufficiency, while the Murray and Mills study
contributed to our understanding of self-sufficiency within the context of energy security concerns
(LIHEAP 2014, Murray and Mills 2014).
As a result of this research, and after conversations with Ms. Gail Williams, Mr. David Levine Mr.
Ryan Nibblins, Mr. Chuck Rifae, and Ms. Patricia Lopez, we decided to frame our evaluation of
each of GSH’s two programs in terms of self-sufficiency for GSH clients. In addition to our
conversations with GSH staff, one of the goals of the ABC program stated in GSH’s ABC
Curriculum Guidelines is to increase the self-sufficiency of its clients (Good Shepherd Housing
and Family Services, Inc. 2014). Furthermore, the current screening procedures in the ES program
reveal that one of its primary purposes is preventing individuals from becoming less self-sufficient
due to a loss of an apartment or utility shut-off.
In the next chapter, we summarize the relevant literature relating to self-sufficiency in the context
of GSH’s ABC and ES programs. The Defining and Measuring Self-Sufficiency chapter describes
how to define, measure, and conceptualize the benefits and limitations of our recommended
definition and application of client self-sufficiency to each program.

13
Defining and Measuring Self-
Sufficiency
Self-Sufficiency and GSH
One of the goals of the ABC program, as stated by GSH in their ABC Curriculum Guidelines, is to
increase the self-sufficiency of its clients (Good Shepherd Housing and Family Services, Inc. 2014).
The current screening procedures in the ES program reveal that one of its primary purposes is
preventing individuals from becoming less self-sufficient due to a loss of an apartment or utility
shut-off. We confirmed that self-sufficiency is an important goal for both programs in
conversations with GSH staff. Self-sufficiency could mean a number of things, so if this is a
primary benchmark GSH should use to measure the success of its programs, it is important to
clearly define and determine how best to measure it. We reviewed research using self-sufficiency
as an evaluation measure, and in this chapter, we report our key findings and offer possible ways
to define and measure self-sufficiency.
Defining Self-Sufficiency
Our team’s review of the literature found that previous studies used a variety of measurements of
self-sufficiency.1
These measures mainly fell into three camps: those who used economic variables
such as income or savings; those who used psychological variables such as self-esteem or feelings
of control; and those who used a combination of both variables. The most relevant study to GSH
was a 1997 survey of nonprofit housing providers in the wake of a major shift of federal housing
policy towards the concept of self-sufficiency (Bratt and Keyes 1997). After their initial review of
the literature, the authors conceptualized self-sufficiency mainly as economic independence, where
an individual was self-sufficient if they did not rely on government or non-profit assistance. After
identifying 130 people to contact, they eventually spoke with staff members from 72 organizations
about their perceptions of self-sufficiency. They found that self-sufficiency was better
characterized not as a binary outcome or fixed point, but rather as a continuum. They came to this
conclusion by recognizing that, for some individuals, reaching their previous definition of
economic independence was impossible or very unlikely, as those individuals might have serious
illnesses or some form of severe drug addiction. For individuals unable to provide for themselves,
self-sufficiency might mean gaining more control over their own life or finding some other form
of independence. Others might need additional skill building or access to resources to become more
self-sufficient. The traditional binary conceptualization of self-sufficiency failed to adequately
capture the differences between individuals and their situations and thus the researchers
recommended a continuum of self-sufficiency to fix those failures.
Given the high quality of this study and its relevance to GSH as a non-profit housing provider, we
recommend that GSH adopt its definition of self-sufficiency as a continuum of economic
1
See, for example, Rosenthal’s 2007 study examining housing self-sufficiency programs.

14
independence – where a person is more or less self-sufficient depending on how much government
or non-profit assistance they receive. Given this definition, we also recommend GSH focus on
economic measures. While we found studies that also examined psychological factors that affected
self-sufficiency, we believe that economic measures alone are more appropriate for GSH’s
evaluation. This belief is due to the fact that both ABC and ES focus on improving the economic
situation of GSH clients and that economic measures are easier to quantify. GSH could eventually
incorporate psychological factors into their evaluation, but we recommend they establish a firm
evaluation method with economic measures before doing so.
Measuring Self-Sufficiency
We now turn to how we recommend GSH measure self-sufficiency for the ABC and ES programs
to determine a program’s success. Additionally, GSH can use these measures to determine
individual client’s progress. We have established these standards after a close careful consideration
of the research as well as goals and current practices of GSH’s programs. This chapter describes
the research and rationale for each program’s measures and describes how GSH should construct
them.
I. ABC Measure
Relevant Research
In developing our recommendations, we started from our previous recommendation on the
definition of self-sufficiency. We conceptualize self-sufficiency as a continuum where a person is
more or less self-sufficient based on the amount of government or non-profit assistance they receive,
rather than a binary outcome where a person is or is not self-sufficient. We believe this
conceptualization makes the most sense for GSH as it comes from a comprehensive survey that
interviewed over 70 non-profit housing providers similar to GSH (Bratt and Keyes 1998) and is
widely used by both the non-profit community (Massachusetts Community Action Program 2003)
and local governments, including Fairfax County (Fairfax County Human Services Council 2012).
Using this definition, ABC seeks to help its clients increase their level of self-sufficiency (i.e. move
further along on the self-sufficiency continuum). Thus we examined the literature to find possible
measures that could capture an increase in self-sufficiency. We also examined variables which GSH
already collects in one form or another in order to make best use of GSH’s current procedures.
We found a meta-analysis that examined nearly 20 studies which looked at housing assistance
programs’ impact on self-sufficiency (Rosenthal 2007). Fourteen included some form of income as
a measure of interest, which indicated to us that income is an important component of the self-
sufficiency literature. We also believe that since every ABC client is expected to be employed, a
change in household income is an important measure of an individual client’s level of self-
sufficiency.
While income may be good indicator of self-sufficiency, we believe that is an incomplete measure
of success. The ABC program also places significant import on helping clients reduce their debt
and increase their savings, often measuring these outcomes through increased credit scores. There

15
is support for using debt or savings as a measure of self-sufficiency, often in conjunction with other
measures; we found four studies that examined transitional housing similar to ABC that focused
on debt or savings as a measure of self-sufficiency (Washington 2002, Kleit 2004, Santiago and
Galster 2004, Verma, et al. 2013).
The federal government also recognizes the importance of reduced debt and increased savings in
increasing an individual’s self-sufficiency. The Family Self-Sufficiency program, the largest
program dedicated to increasing self-sufficiency for public housing recipients, requires participants
to place a portion of their income into a savings account which they then receive after graduation
from the program (Brennan 2014). While participants can lose their account if they drop out of the
program, reducing debt and increasing savings is an important measure of a client’s level of self-
sufficiency (Brennan 2014).
Constructing the ABC Measure
While we used the literature as a guide for determining the best measure of self-sufficiency for
ABC, we also wanted to tailor our recommendation specifically to the program’s needs. Given that
the most relevant concepts from the literature we found are income and debt/savings, we looked to
find concepts that capture those variables into a workable method for the ABC program. We
examined ABC’s current data collection procedures as well as its current proposed changes (Good
Shepherd Housing and Family Services, Inc. 2014) to that program’s model with the aim of
minimizing the strain of our recommendations on GSH’s current processes. At program entry, GSH
currently collects detailed income data from their clients as well as debt information through credit
checks. Though GSH initially contacts and assesses client progress on a quarterly basis, GSH does
not collect detailed debt and income information thereafter, preferring to assess client progress on
smaller achievable goals. Exit interviews do not currently collect detailed income and debt data. A
GSH document detailing possible ABC program changes indicates that they are considering
collecting forms that collect both detailed debt and income information during the quarterly review
(Good Shepherd Housing and Family Services, Inc. 2014).
We recommend GSH use two measures that capture both the effect of income and debt changes: a
simple debt-income ratio (debt ÷ income), and change in net income [(current monthly income -
current monthly debt payments) - (monthly income at entry - monthly debt at entry)]. By using both
measures, GSH can account for a client’s debt as a proportion of income as well as their level of
monthly net income. We do not recommend GSH use savings as a measure of self-sufficiency as
the net income captures some of the same effect as savings, the cost of collecting that information
would outweigh the benefit of capturing the rest of the effect, and the fact that many assets are not
easily liquidated.
Both the debt-income ratio and the net income measure can be affected by changes in client monthly
debt or income. Ideally, GSH would observe a decrease in the debt-income ratio, which would
indicate a client moving closer to having no monthly debt. Additionally, GSH would hope to
observe an increase in the change in net income measure for each client at the same time. By
capturing both reduction of debt and increase of income, these two measures would provide a
measure of GSH’s positive impact on clients’ financial self-sufficiency over time.

16
We recommend GSH define debt as the monthly amount owed to creditors for the household,
including rent, credit card debt, car loans, student loans, overdue bills sent to collections, and any
personal loans such as payday loans. We also recommend GSH define income as any monthly
income earned from work excluding any government benefits for the household, as we want to
isolate a client’s level of economic independence separate from government benefits. These
definitions provide for simplicity of calculations and ensure that the variables are equivalent across
calculations.
Benefits and Limitations
The chief benefits of using these measures of self-sufficiency are their support by previous research,
the fact that GSH already collects much of the data needed to implement them, and their simplicity
of use and interpretation. They require no special statistical training to understand on their own.
GSH can also compare the debt-income ratio to other relevant standards such as 0.43, which is the
largest ratio a borrower can have to get a qualified mortgage (Consumer Financial Protection
Bureau 2013). Thus, these measures capture both progress towards ABC’s goal of increasing self-
sufficiency, through its primary method of doing so by reducing a client’s debt burden while
increasing income and housing security.
However, the measures have some limitations. The simplicity of the measures mean that some
factors that comprise self-sufficiency may not be captured. For example, education level, marital
status, or family size may affect an individual’s level of self-sufficiency and are not captured. In
addition, the measures rely on clients to self-report, which can make data collection difficult. They
are also not cost-free to verify since they require getting third-party data such as bank account
information or credit reports. We sought to address these concerns by also recommending that GSH
account for other factors that affect self-sufficiency through the use of regression analysis. Detailed
information on our regression analysis recommendations can be found in the Data Analysis
Recommendations chapter.
II. ES Measure
Relevant Research
As with ABC, we started our research with the definition of self-sufficiency as a continuum of
economic independence. However, we discovered that while ABC tries to increase clients’ mid-to-
long term self-sufficiency, ES’ goal is to prevent their clients from becoming less self-sufficient
due to short-term crises. GSH has designed the ES program to offer grants to individuals facing a
utility shutoff, an eviction, or those who need a security deposit or first months’ rent for an
apartment. By keeping clients in housing and with utilities on, ES helps clients maintain their level
of self-sufficiency until they are past the crisis.
We found that existing literature on rental and utility assistance programs does not easily apply to
ES. Most programs similar to ES are government-run, and those programs tend to focus on more

17
technical standards of success such as number of clients served or program costs (Hasson 2002).
In a study of non-profit agencies with similar programs, we found researchers did not evaluate the
programs similar to ES as those programs were only a small portion of the agencies’ overall mission
(Edin and Lein 1998).
Given the lack of systematic reviews of programs similar to ES, we broadened our search for any
research on programs with similar goals as ES. We found a meta-analysis of five studies examining
“Housing First” programs, which provide housing to homeless individuals in an attempt to improve
their health and other related outcomes (Groton 2013). Groton found that four of the five studies
included a measure of housing retention for program clients, usually measured within an interval
of months or years. While this study sought to determine the effectiveness of programs aimed at
homeless individuals, it still shows that measuring number of months or years clients remain in
housing is an important measure for programs seeking to improve housing security for low-income
individuals. In addition to its recognition as an important concept in the housing field, housing
retention also reflects the fundamental goal of the ES program – keeping clients in their housing
with their utilities on.
Constructing the ES Measure
Again, we used the literature as a guide for determining our recommended approach, but sought to
ensure its appropriateness for ES. After identifying housing security as the most important goal for
ES, we then examined ES’ procedures to see how best to fit GSH’s current practices into our
recommendations. ES follow-up forms currently collect data on client’s housing security by asking
clients if the grant prevented their eviction or utility shut-off. While information on whether the
crisis was averted does reflect housing retention and can serve as an indicator of program success,
it does not quantify how long ES actually helped the client weather their temporary crisis, and does
not capture if the ES grant helped to maintain the client’s housing security. For example, an ES
client who received a grant for rental assistance may stop paying their electrical bill in the next
month to pay their rent. The current ES follow-up process would not accurately measure that
client’s situation with respect to housing security.
We instead recommend that GSH ask their ES clients four questions: 1) was the crisis averted, 2)
how many months they have paid their rent in full since the ES grant, 3) how many months have
they paid their utilities in full since the ES grant, and 4) how many months have they paid both
their rent and utility bills in full since the ES grant. These four questions will allow GSH to better
determine if the ES grant improved a client’s self-sufficiency by determining their ability to keep
themselves in housing and with their utilities on following the ES grant, and if the grant was simply
successful at averting the crisis. If a client has trouble paying both rent and utility bills following
the grant, it would suggest that the ES grant did not prevent a short term crisis from having a longer
term impact on a client’s level of self-sufficiency.
Benefits and Limitations
The benefits of this measure include its support from previous research, the fact that ES already
collects a similar variable (were utilities kept on or was eviction prevented), and the fact that it

18
requires only four questions to be asked at follow-up. This measure is also flexible in that it allows
for ES staff to determine client success (did a client remain in housing with their utilities on?) as
well as program success (is the average months in housing with utilities on increasing for all our
clients?). ES staff can easily incorporate these questions into their follow-up process and might
have little difficulty understanding the concepts behind the measures as ES’ follow-up process
already incorporates housing security as a concept.
The potential limitations with the measures relate to their simplicity and possible difficulties in
collecting them precisely. Much like with ABC, these simple measures leave out some factors that
may contribute to self-sufficiency. Unlike with ABC, our ES analysis cannot account for those
factors given that it relies on simpler descriptive statistics rather than regression modeling. The
second possible limitation with these measures is that they may be difficult to collect due to the
fact that ES clients often change phone numbers or addresses. In order to maximize the chances of
getting the required information for our measures, we have kept our recommendation simple to
keep required follow-up contact shorter than in ABC. Detailed information on how we address
these problems can be found in the Data Analysis Recommendations chapter.
Conclusion
In this chapter, we have outlined our recommendations for defining and measuring self-sufficiency
in the ABC and ES programs. We recommend GSH define self-sufficiency as a continuum of
economic independence, where a person is more or less self-sufficient based on the amount of
government or non-profit assistance they receive. For ABC’s goal of increasing self-sufficiency,
we recommend that a debt-income ratio and a change of net income are the best measures of self-
sufficiency. For ES’ goal of maintaining self-sufficiency, we recommend that how many months a
client pays their rent and utility bill in full as the best measure for program and client success. We
believe these recommended measures best capture the purpose of GSH’s programs and their effect
on client self-sufficiency.

19
Current Data Collection Procedures
To understand what kind of data are available for ABC and ES programs and how GSH collects
them, we reviewed GSH internal documents that explained the purposes of the programs, analyzed
the sample application forms, and interviewed the program directors of ABC and ES, Mr. Ryan
Nibblins and Ms. Patricia Lopez. We learned that most of the existing data collected on participants
of GSH’s ABC and ES programs are collected during the intake process. These data are mostly in
paper form, and the limited data that are digitized are not relevant for our evaluation purposes.2
Also, we learned that follow-up data for both ABC and ES programs are not consistently collected
due to resources constraints. This chapter describes the current data collection procedure for each
program, and documents the existing data collected at each stage.
I. ABC
Applicants for the ABC program come from different sources: referrals from other agencies, walk-
ins, and online inquiries. GSH conducts brief phone interviews, followed by 30-minute in-person
interviews to assess the eligibility of an applicant. Data collected during these assessments can
provide a baseline against which progress may be evaluated. In theory, additional client information
is gathered and updated during and at the exit of the program. We learned, however, that the actual
practice often varies from the design.
Initial Data Collection
(A) Intake Application (Paper-based and Partially Digitized)
At intake, a GSH case manager first identifies whether the applicant is a referral from another
agency. Next, applicant information is collected through a paper-based intake application form
which is filled out by the applicant together with a GSH case manager. This form documents the
basic personal information for the applicant and his or her household dependents. It also requests
that the applicant provide his or her housing and employment information for the past 5 years in
order for GSH to get an overall picture of the applicant’s financial situation. GSH also gathers
information on clients’ current monthly income and benefits.
In addition to financial information, GSH is also interested in learning about the background
information of the applicants. It collects information on their applicants’ personal financial history,
current payments, past behaviors, criminal history, health and mental health conditions, household
2
GSH manages another dataset aside from those directly used for administering ABC and ES programs. This dataset
helps GSH contribute client data to the Fairfax County Web-based Reporting and Invoicing System (Web-R Report).
These reports show data on income level, race, ethnicity, household structure, and employment status for GSH’s clients
on a monthly basis. Since the information collected for Web-R Reports represents GSH’s clients across all their
programs, participants in ABC and ES are indistinguishable in the dataset, thus rendering the data unusable in our
analysis. In addition, GSH only consistently reports the number of client households, making demographic analysis
very difficult.

20
members’ history of utilizing social services, additional need for supportive services, difficulties in
paying rent/utilities or holding a job, and access to social resources.
This information is later partially digitized into an Excel spreadsheet by the case manager.
(B) ABC/HTP Program Applicant Screening Form (Paper-based)
The screening form, which is filled out by the case manager, contains three parts: a balance sheet
documenting monthly expenses and current debt, applicant expectations, and the case manager’s
evaluation of the applicant. The purpose of this form is to help clients gauge their current financial
stability as well as establish their short-term goals.
The case manager fills out a balance sheet that documents the actual monthly expenses on a number
of items for the applicant, as well as their expected spending on each of them. The case manager
also calculates the monthly totals for both actual and expected spending, and lists out the
outstanding debts for the applicant. For the second part of the form, applicants are asked to express
their expectations for the ABC program on the following four aspects: type of assistance needed,
goals to achieve in the next two years, interests in training or education for professional
development, and safety concerns. For the third part, a case manager assesses the applicant’s
eligibility for the ABC program based on the information collected and personal impressions.
Applicants who request a one-bedroom unit should have an annual income of over $30,000. The
income thresholds for two-bedroom and three-bedroom units are $35,000 and $40,000, respectively.
(C) Individualized Action Plan (Paper-based and Partially Digitized)
Once GSH determines that the applicant is eligible for the ABC program, the case manager helps
the applicant – now an ABC client – develop an individualized action plan (IAP). The IAP is
designed to track clients’ progress in the ABC program and is used in follow-up contacts. The IAP
asks clients to list up to four personal life goals to prioritize. A GSH case manager assigns relevant
tasks to each client in order to help them achieve these goals and tracks when these tasks are
completed. The case manager also asks clients to self-evaluate their strengths, weaknesses, and
needs for further community resources.
Lastly, the case manager creates a balance sheet for each client’s monthly budget. It lists out all the
income sources and amounts, and provides a detailed breakdown of expenses. The case manager
then calculates the total monthly income and expenses, generates a net income for each client, and
sometimes creates visualizations of this information for an individual client.
(D) Fairfax County Self-Sufficiency Matrix (Paper-based and Digitized)
GSH also collects client data using the Fairfax County Self-Sufficiency Matrix (SSM) information
during the intake process. The SSM was developed by Fairfax County based on the Arizona Self-
Sufficiency Matrix (Fairfax County Human Services Council 2012). The SSM contains 19
questions, covering clients’ income, employment status, current housing type, food security, access
to health care, access to transportation, credit worthiness, adult education, family relations, legal

21
issues, life skills, mental health conditions, substance abuse, community involvement, housing
safety, access to childcare, access to children’s education, parenting skills, and citizenship status.
Each question provides five choices that indicate a client’s situation from not being self-sufficient
to being self-sufficient. A score from one to five is assigned to each choice with one representing
the worst and five representing the best scenario. GSH matches each client’s situation to one of the
five categories for each question, calculates the total score for each client, and converts into a
percentage score. This SSM is filled out by only the case manager, who manages an Excel
spreadsheet that documents clients’ SSM scores. Such information is not shared with the clients
themselves.
In our interviews we learned that GSH does not use the SSM for evaluation purposes because it is
not targeted to GSH’s client population. The SSM is required by Fairfax County and mostly used
to rank individuals for priority receipt of government programs. GSH records and keeps SSM data
for auditing reasons.
(E) Credit and Criminal History (Paper-based)
GSH also checks applicants’ credit and criminal history during the intake process using United
States Homeland Investigations Inc. (USHII) screening. The process may include: checking
consumer credit with FICO scored, running a national criminal database search, Virginia statewide
criminal search, corresponding federal criminal search, and tracing social security numbers. Credit
information and criminal history are not digitized.
GSH’s uses for this information are twofold. First, the detailed credit history helps GSH evaluate
applicants’ financial history in order to identify ways to improve their credit. Second, GSH will
scrutinize the cases of applicants with criminal history to see if they can be accepted into the
program. According to the program manager, roughly 10% of the applicants are rejected because
of their criminal records.
Follow-up Data Collection
(A) Quarterly Reviews (Paper-based)
The case manager of the ABC program conducts quarterly reviews with property managers to
follow up on clients’ progress. These reviews are conducted at the 3-month and 6-month mark for
every client after he or she joins the ABC program. In these reviews:
 Clients’ IAPs are reviewed to track their progress on self-set goals;
 Bank statements are requested to make sure that clients are making progress financially.
Clients who have made progress in improving credit scores and are able to pay rent will be
considered “self-sustained” clients. These clients will no longer be under GSH’s intense case
management as long as they keep paying timely rent, but will still be reviewed by GSH annually.
Clients who are not considered self-sustained will be reviewed every three months and will no
longer be eligible for the program if they show no effort to make progress.

22
(B) Attendance in Financial Counseling Classes (Digitized)
ABC requires that its clients take financial budgeting class with “Our Daily Bread” (ODB), a
nonprofit organization that provides short-term safety net services to people living in the Fairfax
County area. Whereas ODB offers a total number of 11 financial training modules, the ABC
program requires every client to attend only one module on budgeting (“Money Matters”, module
4) once accepted into the program. If clients are struggling financially within their first six months
of tenancy or falling into behind on rent, the case manager will recommend (but not require) these
clients complete the full training course with ODB. GSH keeps a record for clients’ completion of
the financial counseling classes, but does not conduct any follow-up assessment on this matter.
(C) Fairfax County Self-Sufficiency Matrix (Paper-based and Digitized)
GSH re-administers the SSM with clients and documents their SSM scores during the follow-up
interviews at the 6 month mark. However, this follow-up SSM score (overall percentage) is
available only for some of the clients due to a lack of resources in conducting follow-up interviews.
This inconsistent collection of data makes it difficult to measure overall changes in client self-
sufficiency with the SSM.
Exit Data
Compared to the data collected during the intake process, exit data have not been regularly collected
in GSH. When exit data has been collected, only a client’s exit date and exit reason (i.e. eviction
or leaving voluntarily) have been recorded.
II. ES
Clients for ES program either contact GSH directly or are referred to GSH by Coordinated Services
Planning (CSP), a local government organization that provides information, referral, linkage, and
advocacy to public and private human services to Fairfax County residents. As most ES clients are
one-time clients, data are consistently collected only at the time of the grant application. GSH relies
heavily on volunteers to collect follow-up data, and as a result, these data are collected sporadically
through a follow-up survey depending on the availability of the volunteers. Existing clients’ data
are in two forms: a paper-based form as well as a digitized Microsoft Access database that captures
a subset of the paper-based data.
Initial Collection of Data
(A) Client Intake Application (Paper-based)
The case manager interviews those who are eligible for ES and collects their information using the
“Client Interview and Intake Application.” This process applies to both direct clients and CSP
referrals. The intake application begins by identifying whether a client has used GSH’s services

23
before. If this is a returning client, the case manager further documents the types of GSH services
that the client participated in the past and the date of their previous experiences. Demographic
information for client’s household and contact information of the client and his or her landlord are
also collected, which could be used to conduct follow-up surveys.
To assess the financial health of the household, GSH asks ES clients to report detailed household
information on monthly income, benefits and expenses. Based on this information, GSH calculates
the total amount of monthly income, benefits and expenses of the client’s household.
In order to better understand clients’ emergency situation, GSH identifies the scope of and reasons
for the crisis and the amount of money needed to help the client. GSH categorizes clients’ need for
assistance into one of the four reasons: preventing eviction, paying first month’s rent, paying a
security deposit, or avoiding utility disconnection. GSH then documents the amount of money
clients owe and the amount requested by the clients. For CSP referrals, GSH further records the
names of other agencies that have committed to help and the amount to be paid by these agencies.
GSH also analyzes the reasons for the client’s emergency in order to make sure it is just a temporary
crisis.
(B) CSP Financial Request Referral (Digitized)
Some additional information for CSP referrals is available to GSH through a digitized referral form,
CSP Financial Request Referral, provided by CSP. This process sometimes creates difficulties in
data management because client information may not have been transferred consistently from CSP
to GSH. In addition to the information already collected directly by GSH from the intake
application, the referral form documents a detailed description of the crisis, utility vendor
information, and a breakdown of the assistance package and grantors other than GSH.
(C) Access Database (Digitized)
With the exception of three items listed below, data from the entire ES application data are
transferred into an Access database. The three exceptions are:
 Household demographic information is entered into Access as a numerical total instead of
the detailed information of household members
 Information on household monthly non-housing expenses is not transferred to the database
 Explanation of crisis is reduced to a few words (i.e. car trouble, unexpected medical
expenses)
Follow-up Data
Follow-up Report (Paper-based)
GSH conducts follow-up surveys with past clients or their landlords using the “Emergency Services
Follow-up Report” at least 60 days after a client receives an ES grant. These follow-up data are
essential for evaluating the ES program’s impact on clients. However, according to our interviews

24
with Ms. Patricia Lopez, the program director of ES, GSH has not been able to collect these data
most of the time due to the unavailability of volunteers.
The follow-up survey collects information on the type of assistance provided by GSH, date of
assistance provided, and the date of the follow up. It also asks a series of Yes/No questions on
client/landlord’s willingness and availability to respond, whether the assistance was received and
successfully prevented the crisis, and whether the household is still living in the same address.
Three open-ended questions are asked at the end of the survey. The first question asks those who
do not stay in the same address about their reasons of moving, the second question seeks comments
from household or landlord on GSH’s work, and the third question seeks to find out how secure
clients feel about staying in their current housing.
Conclusion
Our analysis for GSH’s current data and data collection procedures finds that GSH has been
collecting useful information on their clients when they enter the program, but most of these data
are not digitized and are thus not suitable for statistical analysis. Follow-up data has also not been
collected consistently, making it difficult to analyze clients’ progress. Lastly, we believe that GSH
can benefit from collecting some additional client information as it may be useful in assessing the
impact of the ABC and ES programs on GSH’s clients.
We provide recommendations on data collection in the next chapter to show what data GSH should
collect and how GSH could collect them in order to conduct program evaluations.

25
Data Collection Recommendations
In this chapter, we will present data collection recommendations based on what we have learned
about GSH’s current data, the organization’s capacity, and on our analytical recommendations,
which are described in the next chapter.
Currently, GSH does not have a unified dataset which could be used to perform quantitative
analyses. As outlined in the chapter on Current Data Collection and Procedures, both ABC and
ES do not currently have data that allows for thorough quantitative analysis. This chapter will
provide recommendations on how each program should collect data, what variables to collect, and
when to collect the data.
The data that GSH currently collects is quite comprehensive and gives a good profile of a client,
but much of it is paper-based, and so is unsuitable for statistical analysis. Furthermore, while
questions that are on the form provide useful qualitative information to GSH and case managers, it
is not necessary to have all the fields digitized if the cost to GSH is too high to do so. As a result,
our recommendations will focus only on the data fields which we believe should be digitized in
order to carry out our analytical recommendations.
Encoding data
Data will need to be carefully encoded in order to prepare it for statistical analysis. Standardizing
the way data are coded is necessary for statistical analysis as only then is it possible to make
comparisons across observations. There are three main ways in which data can be encoded; as a
continuous variable, a categorical variable, or a binary variable.
Continuous: A variable is said to be continuous if it is a rational number. Most of these variables
can be entered as a dollar amount (e.g. $5,347) or as an integer (e.g. 4).
Categorical: A categorical variable is a variable that indicates the category of an individual
observation. It assigns each individual to a particular group or “category” by taking on one of a
limited, and usually fixed, number of possible values. For example, if the variable married is coded
categorically, 0 could indicate being single, 1 could indicate being married, 2 could indicate having
been divorced, and 3 could indicate being widowed.
Binary: A binary variable is one that is coded as 0 or 1. Variables will typically be coded as a 1 if
the observation has that characteristic. Categorical variables can be split up by its categories and
be its own binary variable. Using the same example in the Categorical chapter above, a person who
is married will be coded as 1 for the married variable, and 0 for the all other variables such as single,
divorced, or widowed. Likewise, a person who is single will be coded as 1 for the single variable,
and 0 for all else.
A guide to the detailed form of the variables that we recommend is provided in Appendix A1, A2,
and B. An electronic form can also be found in the Excel file that accompanied this guide. This

26
form can be adapted to the format appropriate for database or statistical software that GSH chooses
to purchase in the future.
I. ABC
As detailed in the chapters above, GSH currently collects detailed income and debt data from their
clients at program entry, but not at the quarterly assessments or at program exit. A document
detailing possible ABC program changes expresses GSH’s desire to switch to forms which collect
detailed income and debt information during the quarterly review (Good Shepherd Housing and
Family Services, Inc. 2014). This is in accord with our recommendations as this will allow GSH to
measure a client’s change in self-sufficiency over the duration of the ABC program.
GSH is currently undergoing the process of purchasing a new database and client management
software. As this is an on-going process, we are unable to recommend a specific and tailored way
to collect data. However, we have provided GSH with a framework which should be easily
adaptable to whichever software GSH chooses, along with a sample Excel file showing what the
collected data should look like.
Recommended New Variables
After evaluating the ABC data collection procedures, we identified two important variables that are
not currently collected but which we recommend be added to the existing procedures. As these are
both our primary variables of interest for evaluating the ABC program, we recommend that GSH
calculate and record this variable at program entry and at each follow-up assessment. We
recommend that these variables be calculated when the data is being collected at both entry and
follow-up so as to prevent having to do additional calculations before analyzing the data.
Debt-income ratio: This variable is calculated by taking a client’s monthly debt payment (debt)
and divide it by monthly wages (income), creating a new variable, debt_income.
Net monthly income: This variable is calculated by taking a client’s monthly wages (income), and
then subtracting total monthly debt (debt) from it, creating a new variable, net_income.
Collection methods
During the intake application process, we suggest that this new data be collected by case managers
in their assessments with the client. Also, case managers should ensure clients agree to remain
contactable for follow-up data collection for at least 24 months after entering the program. As we
have chosen to focus on economic indicators of self-sufficiency which are easier to quantify, the
procedures we recommend do not collect lengthy open-ended answers, but short and simple ones
which are more suitable for analysis.
For the ABC program, we recommend that GSH use two variations of the same form in order to
collect data on entry and on follow-up. The forms are separate in order to reduce redundancies and
speed up data collection, as some variables only need to be answered once (such as demographic

27
information), whereas other variables might change over time and will need to be monitored (such
as debt and income). Guidelines to these forms are provided in Appendix A1, A2, and B. A digital
version of these forms can also be found in the Excel file accompanying this document.
GSH already has a procedure in place for collecting such data upon entry and follow-up. However,
if incorporating our additional measures creates an unacceptable burden upon GSH staff, we have
proposed some ideas GSH may want to consider in order to minimize stress placed on resources.
These ideas can be found in the Possible Alternatives chapter.
Follow-up intervals
The collection of follow-up data is essential to evaluating GSH’s impact on a client’s self-
sufficiency over time. We recommend collecting data at entry into the program, then again at 3
months, 6 months, 12 months, and at 24 months. Since ABC clients have less frequent check-ins
after their first 6 months, our follow-up recommendations follow the current semi-regular intervals
so as to minimize the impact on case and property managers.
As per our recommendations, two different forms should be used over the course of the client’s
participation in the ABC program. The form used at entry (Appendix A1) is longer and more
comprehensive as it must capture client demographic and personal history information. The form
for subsequent follow-up (Appendix A2) data collections is shorter and only requires the client to
provide information on the variables that are likely to change over time.
II. ES
The short-term nature of the ES program means both that detailed information on clients may be
unavailable and that long-term evaluation is not warranted. Nevertheless, some meaningful analysis
of ES’ impact may still be conducted by examining descriptive statistics. As mentioned in the
Defining and Measuring Self-Sufficiency chapter, we recommend only minor modifications to
GSH’s existing data collection procedures: ES data collection will simply consist of asking clients
four questions: whether their crisis was averted, how many months they have paid their rent in full
since receiving the ES grant, how many months they have paid their utility bills in full since
receiving the ES grant, as well as how many months they have paid both their utility bills and rent
in full since receiving the grant. After a discussion with Patricia, we learned that it was more
important to GSH for clients to pay their bills in full rather than on time, as any late fees have a
lesser negative impact on self-sufficiency than keeping a balance on the bill.

28
Table 3 – Overview of ES follow-up questions
Question 1 Question 2 Question 3 Question 4
Survey
question:
Was the crisis
averted?
How many
months have you
paid your rent in
full since the ES
grant?
How many
months have you
paid your utility
bill in full since
the ES grant?
How many
months have you
paid both your
rent and your
utility bill in full
since the ES
grant?
For Question 1, clients should respond with a 0 for “no” or a 1 for “yes”. For the last three questions,
clients should respond with a number integer between 0 (minimum) and the month of follow-up
since receiving an ES grant (maximum). The ES Follow-Up Form found in Appendix B can be
used as a guide for how the variable should be collected. A digital version of this form can also be
found in the Excel file that accompanies this report.
For example, imagine that a client who has been in their current housing for the past two years
recently received an ES grant for rent assistance. At the 2-month follow-up, the client is asked the
questions. The client should answer 0 for “no” or 1 or “yes” to Question 1. The client should answer
with a whole number ranging from 0 to 2 for the other three questions, despite having been at the
same address for the past two years. Collecting the data in this way will ensure that a client’s prior
utility or rent history does not obscure the short-term effect receiving an ES grant will have on a
client’s self-sufficiency.
Collection methods
ES clients move quickly through the program, and GSH staff do not necessarily have regular
interactions with them, making it difficult to collect data consistently. We also learned that ES
clients are more likely than ABC clients to move around, and they are harder to contact and follow
up with. Therefore, we recommend that data collection methods make it easy for clients to respond
to requests for follow-up information quickly and succinctly.
To encourage higher completion rates, ES might want to consider adding a condition requiring
clients to remain contactable for at least 6 months after receiving the grant. Clients should also be
asked to agree to respond to any follow-up questions that ES administers. While it may be difficult
to enforce this, stating this at the time of the grant will serve to inform the client of this expectation
and any attempts by ES staff to collect follow-up data should not come as a surprise.
We foresee that collecting this follow-up data from clients may place a considerable financial or a
security burden on them. For example, there may be a cost associated with receiving data collection
surveys, or that responding might inadvertently reveal sensitive information. Therefore, we ask that
GSH obtain consent from clients in order to preserve client confidentiality. If clients do not consent

29
to the collection of follow-up data, we do not recommend attempting to contact them for any
information after the grant has been administered.
We recommend that GSH utilize text message survey technology in order to gather data on ES
clients as many ES clients own cell phones. Ms. Patricia Lopez from GSH’s ES mentioned that
many clients lose their cell phones or change their phone numbers, but she also mentioned that this
occurs with only a minority of clients. Thus, we think that it would be possible and beneficial for
GSH to collect follow-up data through conducting text message surveys. Clients would receive one
scheduled automated text message per question at the follow-up intervals, and clients should
respond as instructed in the text.
GSH might want to consider conducting a pilot project for data collection to test how receptive
clients would be in responding, and to see if using text message surveys are a more effective way
of collecting follow-up data compared to the existing methods of phone follow-ups or administering
in-person surveys. Doing this will help GSH establish a routine for this data collection process. At
the outset, GSH may want to manually send the text messages and record the data. If this is
successful, GSH could consider scaling up by using computer programs to automate the process.
There are a few web-based software packages that GSH could purchase to carry out this function.
If GSH decides to go with this option, they will have to choose which packages would be best
suited to their needs and their budget. Some available packages include Qualtrics SMS Surveys,
Poll Everywhere, and SMS Poll, to name a few.3
Each has different advantages and disadvantages,
and GSH would need to spend some time and resources choosing a program to purchase. However,
we believe this would be a good investment as automating a system such as this will save GSH
staff time that might be better spent in other ways. Table 4 below lists some costs and benefits of
administering text message surveys to ES clients:
Table 4 – Benefits and costs of text message surveys
Benefits: Costs:
Most clients have cell phones Need to purchase software
Easy for clients to respond to quickly Clients may not have reliable cell phones
or phone numbers
No need for manual data entry Text messages are easy to ignore
Cost-efficient No capacity to ask longer, open-ended
questions
3
More information about the above-mentioned packages can be found at these following links respectively:
http://www.qualtrics.com/university/researchsuite/distributing/more-distribution-methods/sms-surveys/,
http://www.polleverywhere.com/, and http://www.smspoll.net/.

30
Follow-up Intervals
We suggest GSH collect data from their ES clients at entry, and again at 2 months and 6 months
after the grant. Data collected at entry will include all that ES currently collects, such as biographic
and crisis details – we recommend no changes here.
The 2-month interval was chosen because this is consistent with ES’ current 60-day minimum
follow-up after the grant. This is also long enough after the grant has been given, but short enough
that it is likely GSH still has contact with the client. The 6-month interval was chosen because the
grant would still have been awarded recently enough for ES’ impact to reasonably be felt on client
self-sufficiency. As time passes, it would become more difficult to attribute a client’s increase or
decrease in self-sufficiency to the ES grant, rather than an unrelated life event such as an increase
in income or voluntarily moving houses.
In the next chapter, we present our recommendations for analyzing the data that our recommended
data collection procedures will generate. GSH can begin to conduct this evaluation of the impact
of the ABC and ES programs after a year for descriptive statistics or 30 pieces of data for regression
analysis.

31
Data Analysis Recommendations
In this chapter, we will present our recommendations for analyzing the data that will be collected
under our recommended procedures for both the ABC and ES programs as well as suggest an
evaluation timeline.
To evaluate the ABC program’s effectiveness using the data collection methods we described in
the previous chapter, we recommend that GSH analyze the collected data using descriptive statistics
and regression analysis. We suggest evaluating two different dependent variables, debt-income
ratio and change in net income, to capture ABC’s impact on clients’ financial self-sufficiency. For
the ES program, we suggest GSH evaluate the program effectiveness by only calculating
descriptive statistics due to the more short-term nature of the program.
Methodology for Descriptive Statistics & Regression Analysis
The first evaluation method that we recommend for evaluating GSH programs is to use descriptive
statistics. This approach allows GSH to get a quick overview of clients’ financial self-sufficiency
and to track their changes in their self-sufficiency over time. It is important to note that any such
changes may or may not be a result of program participation as clients are affected by many factors
outside of GSH’s control. Nonetheless, this analysis would help to identify clients who may be in
need of additional financial education or other assistance. This evaluation method applies to both
the ABC and ES programs.
The other evaluation method that we recommend for the program is to use regression analysis.
Regression analysis will allow GSH to examine how specific factors affect client self-sufficiency
and by how much. Compared to descriptive statistics, regression analysis will have more statistical
power in explaining the ABC program’s impact on the change in client’s self-sufficiency, though
it may not prove with certainty that GSH programs cause any positive or negative changes to self-
sufficiency. Regression analysis also controls for other variables in the model, which would allow
GSH to see how each variable influences self-sufficiency in its own way. This method can be used
to estimate ABC’s effect on different types of clients, and help GSH better allocate its resources to
target those who might benefit the most from this program. However, regression analysis is not
applicable to the evaluation of the ES program due to the difficulty of collecting follow-up data
and the unique design of the program.
I. Evaluation for ABC – Descriptive Statistics
Measure 1: Debt-Income Ratio
We recommend GSH generate descriptive statistics to gain an overview of ABC clients’ debt-
income ratios over time. These descriptive statistics are not difficult to generate and could act as
useful “nuggets” of information when presented to funders.

32
We recommend that a debt-income ratio of 0.43 be used as a threshold for comparing the
descriptive statistics. This threshold is cited in the Consumer Financial Protection Bureau’s (CFPB)
new mortgage lending rules as the highest ratio a borrower can have to get a Qualified Mortgage
(CFPB 2013). Additionally, having a ratio of 0.43 or lower will signify to lenders that the client
has the ability to pay back home loans. As GSH aims to help their clients become financially
eligible to qualify for fair market rentals and/or home ownership, we believe that this is a good
target for evaluating clients progress though GSH can use others based on their preferences.
There are two ways in which GSH can generate descriptive statistics using the threshold; comparing
all clients at present and comparing clients across periods of time in the program.
1. Compare how many clients are below the threshold to clients who are above
For this analysis, GSH should use only the most recent data collected for each client. That is to say,
if GSH has collected data for Client A at entry into the program, at three months after, and at six
months after, GSH should only use the data collected at the 6-month mark for this summary.
GSH should then calculate the number of clients who have a debt-income ratio below the threshold
(in our example, 0.43), and the number of clients above the threshold. An example of how this
would look like (using simulated numbers) is shown in Figure 1 below. A proportion might also be
calculated, as shown in the pie chart in Figure 2.
Figure 1– Debt-Income ratio for ABC clients

33
Figure 2 – Proportion of ABC clients below and above the debt-income ratio threshold
The more ABC clients are below the threshold (or the higher the proportion of clients below the
threshold), the more self-sufficient the overall population of ABC clients is.
This statistic is not meant to have predictive value as it only shows a snapshot of all of ABC’s
clients at present. In the example given above, it is clear that more than half of ABC clients are
above the threshold, and therefore not eligible for a mortgage loan at fair market value based on
CFPB’s threshold.
This is useful to GSH as it may suggest where GSH should allocate its resources. For example, if
GSH has many clients who are below the threshold – thereby qualifying for a mortgage loan, then
GSH may want to focus on helping ABC clients transition to fair market housing or toward taking
out a mortgage loan. However, if many clients are above the threshold, then GSH may want to
examine the data further to observe a finer breakdown as shown in the next descriptive method,
which compares between clients over time.
2. Compare between clients below and above the threshold over time
This descriptive statistic will offer GSH a finer assessment of the overall self-sufficiency of clients
in its ABC program and their debt-income ratio over time.
Unlike the first statistic, GSH must manage the dataset differently before generating any summaries.
First, GSH should use all historical data that they have recorded for all ABC clients. Next, GSH
should sort the data by the variable month (which records the months the client has been in the
program at the time the data was collected), and split client data into these sub-categories:

34
 0 – 3 months
 4 – 6 months
 7 – 12 months
 13 – 24 months
These are the intervals at which we have recommended GSH follow up and collect data on their
client. Special attention should be paid to the uneven spacing of these intervals, as it mimics the
different frequencies in which ABC case managers interact with their clients. Once GSH has
‘binned’ the dataset into these categories, they should count the number of clients within each bin
who fall below or above the debt-income threshold of 0.43. The outcome, visually, might look
something like Figure 3 below:
Figure 3 – Debt-Income ratio for ABC clients by periods in program
This example shows an outcome that GSH would like to see. Since a debt-income ratio that is lower
is better, increases in the proportion of clients who have a ratio lower than 0.43 are a sign of
increased self-sufficiency among ABC clients. If GSH observed such a trend, then it may suggest
that ABC clients seem to improve their debt-income ratio the longer they stay in the program.
As we have noted, such analysis cannot prove with certainty that the changes shown in these
examples were caused by GSH’s ABC program. Nonetheless, these two calculations are quick and
straightforward to perform, and we believe GSH could use them to provide useful snapshots of the
overall self-sufficiency of their clients to their donors and other stakeholders.

35
Measure 2: Net Income
In this chapter, we will recommend a way to observe and track ABC’s impact on a clients’ net
monthly income over time. This is similar to the recommendation to track debt-income ratios over
‘bins’ of time, but using the average change in net monthly income instead. These ‘bins’ of time
should be split into these categories:
 0 – 3 months
 4 – 6 months
 7 – 12 months
 13 – 24 months
To calculate the average change in net income in each ‘bin,’ GSH should first find the change in
net income compared to entry. An example of Client A in Table 5 below shows more clearly the
calculation that needs to be made. This information about net income can be used as a partial
measure of self-sufficiency, and can tell GSH how much more or less economically independent a
client is since entry into the ABC program.
Table 5 – Example calculation of Client A’s change in net income compared to entry
Entry 3 months 6 months 12 months
Net Monthly Income $1,000 $1,500 $1,750 $1,200
Change in net income compared
to entry
$0 $500 $750 $200
Next, GSH could calculate the incremental change in net income between periods of time. GSH
can calculate this statistic by taking the difference between each month’s change in net income
compared to entry. For example, the incremental change in net income for Client A from program
entry to three months in the program is $500. Consequently, his incremental change in net income
from three months in the program to six months is $250. This example calculation of this change
can be seen in Table 6 below. This calculation is a useful step to determine in which period in the
ABC program a client experiences the most change in net income. This step is also important in
order to carry out the final calculation.
Table 6 – Example calculation of Client A’s incremental change in net income over time
0-3 months 4-6 months 7-12 months
Incremental change in net
income over time
$500 $250 -$550
Finally, the calculations in Tables 5 and 6 must to be repeated for all clients in the dataset. After
which, GSH should take the average of change in net income within each time bin. An example
calculation is shown in Table 7 below:

36
Table 7 – Example calculation of all ABC clients’ change in net income over time
Change in net monthly
income
0-3 months 4-6 months 7-12 months 13-24 months
Client A $500 $250 -$550
Client B $300 -$350 $700 $200
Client C $200 $200
Client D $300
Average change $325 $33 $75 $200
For the above calculation in Table 7, the change in net income for all clients should be calculated
only if there is available data. Since clients enter the ABC program at different points in time, some
clients might have fewer data points than others. Nonetheless, if the averages are calculated
properly (i.e. divided by the right number of observations) and within each time period, the
summary will be able to provide GSH with useful information. For example, the calculations for
the 0-3 month period and the 7-12 month period will look like this:
 0-3 months: ($500 + $300 + $200 + 300)/4 = $325
 7-12 months: (-$550 + $700)/2 = $75
For the interpretation, GSH should take into account the number of data observations averaged in
each period. The more data points there are the more accurate the summary. For example, the
average change for the 0-3 month period is a more trustworthy estimate as it has four observations,
whereas the average change for the 13-24 month period only has one observation and would thus
be less accurately representative of the overall program.
Observing the incremental change of net income between time periods in the ABC program can
give GSH a sense of its general impact on clients. In the example above, it seems that clients
improve the most in the first time period of 0-3 months but not by much in the next time period of
4-6 months. If this trend is still observed with more data, GSH could focus on helping their clients
improve more in the second period. However, as with all descriptive statistics, we caution GSH to
not make any causal predictions with this information as many other factors could have affected a
client’s net income over time. GSH may also want to calculate the confidence intervals for these
averages as it can give you an idea of the range in which the true mean lies. If GSH wishes to find
out more information on confidence intervals, statistical textbooks such as Wooldridge can provide
additional guidelines and their uses (Wooldridge 2012). Descriptive statistics are a way for GSH to
get a more detailed look at how their average client progresses over their time in the ABC program.

37
II. Evaluation for ABC – Regression Analysis
Dependent Variables
As discussed in the Defining and Measuring Self-Sufficiency chapter, we recommend that debt-
income ratio (debt ÷ income) and change in net income [(current monthly income - current monthly
debt payments) - (monthly income at entry - monthly debt at entry)] are the preferred measures of
self-sufficiency for the ABC program’s clients.
The following example illustrates how the two dependent variables should be generated: imagine
a client had a monthly income of $5,000 and monthly debt of $1,500 when he or she joined the
program and a monthly income of $5,000 and monthly debt of $1,000 after participating into the
program for six months. For this client, the debt-income ratio at the 6-month mark would be
($1,000/$5,000), or 0.20 compared to 0.3 at entry. For this same client, the net income at the entry
point would be ($5,000 - $1,500), or $3,500, and the net income at the 6-month mark would be
($5,000 - $1,000), or $4,000. Then change in net income would be ($4,000 - $3,500), or $500. Both
the decrease in the debt-income ratio and the increase in net income suggest that this client’s self-
sufficiency has increased.
Conversely, now imagine a client who had the same debt-income ratio of ($1,500/$5,000), or 0.3
at entry, but at the 6-month mark had a monthly income of $5,000 and monthly debt of $2,000,
resulting in a debt-income ratio of 0.4. This client’s change in net income would be ($3,000 -
$3,500), or -$500. Both the increase in the debt-income ratio and the decrease in net income suggest
that this client’s self-sufficiency has decreased.
Key Independent Variable of Interest
The key independent variable of interest is the number of months elapsed since a client joined the
ABC program, which is coded as month. This is a continuous variable that captures how many
months a client has participated in the ABC program. As GSH collects entry data and follow-up
data at 3 months, 6 months, 12 months, and 24 months with the ABC program, this variable will
only take the value of 0, 3, 6, 12, and 24 in our recommended analysis.
As we think the relationship between months since a client entered in the ABC program and his or
her self-sufficiency might be non-linear – which is saying that the ABC program might impact a
client differently as the time spent in the program changes – we also generate the square of month
and include this new variable, month2
, to adjust for this non-linearity.
Using standard ordinary least squares (OLS) regression procedures such as those described in
Wooldridge (Wooldridge 2012), a regression model can then be estimated and its results analyzed.
For example, using this method, we would be able to estimate how much a person’s age affects his
or her income level. Knowing this, it is then possible to further predict a person’s income based on
his or her age.

38
Control Variables for Regression Analysis
A naïve model that contains only the dependent variable and the independent variable mentioned
above would be likely to produce an incorrect estimated effect of ABC participation on the change
in a client’s debt-income ratio and a change in net income. If we leave out the factors that affect
the dependent variable and are related to the independent variable of interest, we may attribute their
effect to the key independent variable. The risk of falsely attributing the influence of these other
factors onto the key independent variable creates a need to add control variables to our model.
To do so, we first control for the effects on the change in dependent variables that are associated
with demographic factors, including:
 Race/Ethnicity – Several categories are created to indicate a client’s race and ethnicity.
These categories include “White”, “African American”, “Asian”, “Native American”,
“Hispanic” and “Others”.
 Age of Client – This is a continuous variable that captures the information of a client’s
age.
 Marital Status – A client’s marital status is categorized as one of the following three
categories: “single”, “married”, “divorced”, and “widowed”.
 Education Level – A client’s education level is categorized as “below high school”,
“high school graduate” and “college and above”.
 Gender – A binary variable that indicates a client’s gender (0 for male, 1 for female).
Second, we control for client’s characteristics which may both affect the change in dependent
variables as well as correlate with clients’ participation in the ABC program. These characteristics
are reflected in the clients’ personal history, including:
 Bankruptcy – A binary variable that shows whether or not a client has experienced
bankruptcy before.
 Substance Abuse – A binary variable that captures whether or not a client has a history
of substance abuse.
 Mental Illness – A binary variable that indicates whether or not a client has a history of
mental illness.
 Crime History – A binary variable that shows whether or not a client has ever committed
a crime.
 Gambling – Clients’ gambling habits are categorized as “Never”, “Occasionally” (1 to
3 times per week), and “Frequently” (more than 3 times per week).
Third, we control for three household-related factors that may contribute to the changes in the
dependent variables. These factors include:
 Size of Household – A continuous variable that shows how many members a household
has.

39
 Housing Voucher – This binary variable indicates whether or not a household has ever
received housing vouchers.
We also control for clients’ participation in additional financial classes. Since the case manager
recommends that clients who are struggling within their six months of tenancy or falling into rental
arrears take additional financial classes, we think this is a good proxy for a client’s first six months’
performance.
 Being Recommended to Take Additional Budgeting Classes – A binary variable that
shows whether a client has been recommended by a case manager to take additional
budgeting classes.
Table 8 below provides a list of recommended control variables and how they should be coded. A
digital version of this table can also be found in the accompanying Excel file to this report.

40
Table 8 – Coding rules for ABC control variables
Variable Coded as Coded name
Months since entry into ABC Number month
Months squared Month * Month month2
Race/Ethnicity:
White, African American, Asian,
Hispanic, Native American, Other
race
Age of client Number age
Single 0 if not single, 1 if single
single (omitted as
reference)
Married 0 if not married, 1 if married married
Separated/Divorced
0 if not separated or divorced, 1 if
separated or divorced
divorced
Widowed 0 if not widowed, 1 if widowed widowed
Below high school education
0 if not below, 1 if below high school
education
below_hs (omitted as
reference)
High School Education
0 if not high school education, 1 if high
school education
highsch
College degree or above
0 if not college degree or above, 1 if college
degree or above
college
Gender of client 0 if female, 1 if male male
History of bankruptcy 0 if no history, 1 if history bankruptcy
History of substance abuse of
client
0 if no history of abuse, 1 if history of
abuse
sub_abuse
History of mental illness of client
0 if no history of mental illness, 1 if history
of mental illness
mental
Criminal history of client 0 if no criminal history, 1 if criminal history crime
Never Gamble (client doesn’t
gamble any times per week )
0 if client does gamble, 1 if client does not
gamble
gamble_never (omitted as
reference)
Client gambles 1-3 times per week
0 if client does not gamble 1-3 times per
week, 1 if client gambles 1-3 times per week
gamb_occasionally
Client gambles 3 or more times
per week
0 if client does not gamble 3 or more times
per week, 1 if client does gamble 3 or more
times per week
gamb_frequently
Household size of client Number size
Client has a Housing Voucher
0 if client does not have housing voucher, 1
if client does have housing voucher
voucher
Attendance of additional financial
counseling course
recommendation by case manager
0 if not recommended for more classes, 1 if
recommended for more classes
att
0 if not race of client, 1 if race of client
white (omitted as
reference), black, asian,
Hispanic, natAm,
other_race

41
Model Specification 1: Debt-Income Ratio
For a richer analysis of the impact of the ABC program, we recommend GSH use regression
analysis. This type of analysis, which can be done with any statistical software application, uses
the data collected to estimate an equation such as the one shown in Figure 4 below. In this equation,
the factors that are believed to influence a client’s debt-income ratio (in this case) are listed on the
right side of the equal sign. Each “regression coefficient,” or 𝛽̂ , estimates the impact of an
independent variable on the dependent variable, while controlling or accounting for the effect of
the other factors. In this case, the values of 𝛽̂1 and 𝛽̂2, once estimated from the collected data, will
allow GSH to test whether clients who remain in the program are likely to see improvement in their
debt-income ratios, even after accounting for other factors that might play a role, such as education
or bankruptcy.
Figure 4 – Recommended OLS Model for Analyzing ABC Program Impact on Debt-Income Ratio
𝑌𝑑𝑒𝑏𝑡_𝑖𝑛𝑐𝑜𝑚𝑒 = 𝛽̂0 + 𝛽̂1 𝑋 𝑚𝑜𝑛𝑡ℎ + 𝛽̂2 𝑋 𝑚𝑜𝑛𝑡ℎ
2
+ 𝛽̂3 𝑋ℎ𝑖𝑠𝑝𝑎𝑛𝑖𝑐 + 𝛽̂4 𝑋 𝑏𝑙𝑎𝑐𝑘 + 𝛽̂5 𝑋 𝑎𝑠𝑖𝑎𝑛 + 𝛽̂6 𝑋 𝑛𝑎𝑡𝐴𝑚
+ 𝛽̂7 𝑋 𝑜𝑡ℎ𝑒𝑟_𝑟𝑎𝑐𝑒 + 𝛽̂8 𝑋 𝑎𝑔𝑒 + 𝛽̂9 𝑋 𝑚𝑎𝑟𝑟𝑖𝑒𝑑 + 𝛽̂10 𝑋 𝑑𝑖𝑣𝑜𝑟𝑐𝑒𝑑 + 𝛽̂11 𝑋 𝑤𝑖𝑑𝑜𝑤𝑒𝑑
+ 𝛽̂12 𝑋ℎ𝑖𝑔ℎ𝑠𝑐ℎ + 𝛽̂13 𝑋𝑐𝑜𝑙𝑙𝑒𝑔𝑒 + 𝛽̂14 𝑋 𝑚𝑎𝑙𝑒 + 𝛽̂15 𝑋 𝑏𝑎𝑛𝑘𝑟𝑢𝑝𝑡𝑐𝑦 + 𝛽̂16 𝑋𝑠𝑢𝑏_𝑎𝑏𝑢𝑠𝑒
+ 𝛽̂17 𝑋 𝑚𝑒𝑛𝑡𝑎𝑙 + 𝛽̂18 𝑋𝑐𝑟𝑖𝑚𝑒 + 𝛽̂19 𝑋 𝑔𝑎𝑚𝑏_𝑜𝑐𝑐𝑎𝑠𝑖𝑜𝑛𝑎𝑙𝑙𝑦 + 𝛽̂20 𝑋 𝑔𝑎𝑚𝑏_𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑡𝑙𝑦
+ 𝛽̂21 𝑋𝑠𝑖𝑧𝑒 + 𝛽̂22 𝑋 𝑣𝑜𝑢𝑐ℎ𝑒𝑟 + 𝛽̂23 𝑋 𝑎𝑡𝑡
Coefficient Interpretations and Statistical Tests
While the constant 𝛽0 does not need to be interpreted, the other coefficients have different ways of
interpretation depending on the type of variable with which they are associated. In this chapter, we
provide examples for interpreting the estimated coefficients, and discuss how to identify
coefficients that are statistically significant – that is, which variables are likely to have an impact
on all ABC clients, even possibly on those whose data has not been collected yet.
a) Interpretation of the key independent variable
Since we adjusted our regression model to capture the effect – which might change over time – of
staying in the ABC program, the predicted effect of a one-month increase in the months a client
has participated in the ABC program on the change in debt-income ratio is (𝛽̂1 + 2𝛽̂2 𝑋 𝑚𝑜𝑛𝑡ℎ).4
GSH can plug in a specific value for the month variable to see the predicted effect at a different
month, other than the months in which data was collected.
To illustrate this interpretation, imagine that the estimated coefficient of month, 𝛽̂1, is -0.024, and
the coefficient of month2
, 𝛽̂2, is 0.0014. In this case, the predicted effect is (0.002𝑋 𝑚𝑜𝑛𝑡ℎ − 0.024).
This means that for a client in his third month with the ABC program, an increase in program
participation by one month is associated with a (0.002 ∗ 3 − 0.024) = −0.018 change in his debt-
income ratio, holding everything else constant. Whereas in his sixth month with the program, an
4
This effect is calculated by taking derivative of the dependent variable with respect to month.

42
increase in program participation by one month is associated with a (0.002 ∗ 6 − 0.024) =
−0.012 change in the debt-income ratio, holding everything else constant. Therefore, the
conclusions that GSH would be able to infer from this case are that the ABC program helps lower
clients’ debt-income ratio, but also that the program effect on improving clients’ self-sufficiency at
the 6-month mark is less than the effect at the 3-month mark, holding everything else constant.
Ideally, GSH would want to observe a negative predicted effect at any given month. This would
suggest that participating in the ABC program would lower clients’ debt-income ratio, a sign of
clients’ improved financial management skills. As we mentioned in the Defining and Measuring
Self-Sufficiency chapter, a lower debt-income ratio indicates that a client is more self-sufficient
based on our definition.
b) Interpretation of the control variables
The interpretation for the coefficient of a continuous variable (coded as numbers) is that a one-unit
increase in this continuous variable is predicted to change the debt-income ratio by the estimated
coefficient, holding everything else constant.
The interpretation for the coefficient of a binary variable (coded as 0 or 1) is that the average
difference of the debt-income ratio between clients in the category compared to the omitted
category, holding everything else constant.
c) Statistical tests
In order to be as confident as possible about the proposed model, GSH should test the statistical
significance of the estimated coefficients in order to ensure that the relationship between any
independent variable and the dependent variable is meaningful. To test whether the key
independent variable is statistically significant, GSH should conduct a joint significance test. For
the other control groups, GSH may want to check the corresponding p-value reported with the
coefficient estimates. We recommend consulting a standard econometrics textbook such as
Wooldridge (Wooldridge 2012) for additional guidance on these procedures.
Model Specification 2: Net Income
The regression model for this dependent variable would be:
Figure 5 – Recommended OLS Model for Analyzing ABC Program Impact on Net Income
∆𝑌𝑛𝑒𝑡_𝑖𝑛𝑐𝑜𝑚𝑒 = 𝛽̂0 + 𝛽̂1 𝑋 𝑚𝑜𝑛𝑡ℎ + 𝛽̂2 𝑋 𝑚𝑜𝑛𝑡ℎ
2
+ 𝛽̂3 𝑋ℎ𝑖𝑠𝑝𝑎𝑛𝑖𝑐 + 𝛽̂4 𝑋 𝑏𝑙𝑎𝑐𝑘 + 𝛽̂5 𝑋 𝑎𝑠𝑖𝑎𝑛 + 𝛽̂6 𝑋 𝑛𝑎𝑡𝐴𝑚
+ 𝛽̂7 𝑋 𝑜𝑡ℎ𝑒𝑟_𝑟𝑎𝑐𝑒 + 𝛽̂8 𝑋 𝑎𝑔𝑒 + 𝛽̂9 𝑋 𝑚𝑎𝑟𝑟𝑖𝑒𝑑 + 𝛽̂10 𝑋 𝑑𝑖𝑣𝑜𝑟𝑐𝑒𝑑 + 𝛽̂11 𝑋 𝑤𝑖𝑑𝑜𝑤𝑒𝑑
+ 𝛽̂12 𝑋ℎ𝑖𝑔ℎ𝑠𝑐ℎ + 𝛽̂13 𝑋𝑐𝑜𝑙𝑙𝑒𝑔𝑒 + 𝛽̂14 𝑋 𝑚𝑎𝑙𝑒 + 𝛽̂15 𝑋 𝑏𝑎𝑛𝑘𝑟𝑢𝑝𝑡𝑐𝑦 + 𝛽̂16 𝑋𝑠𝑢𝑏_𝑎𝑏𝑢𝑠𝑒
+ 𝛽̂17 𝑋 𝑚𝑒𝑛𝑡𝑎𝑙 + 𝛽̂18 𝑋𝑐𝑟𝑖𝑚𝑒 + 𝛽̂19 𝑋 𝑔𝑎𝑚𝑏_𝑜𝑐𝑐𝑎𝑠𝑖𝑜𝑛𝑎𝑙𝑙𝑦 + 𝛽̂20 𝑋 𝑔𝑎𝑚𝑏_𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑡𝑙𝑦
+ 𝛽̂21 𝑋𝑠𝑖𝑧𝑒 + 𝛽̂22 𝑋 𝑣𝑜𝑢𝑐ℎ𝑒𝑟 + 𝛽̂23 𝑋 𝑎𝑡𝑡
We denote the change in net income over time as ∆𝑌𝑛𝑒𝑡_𝑖𝑛𝑐𝑜𝑚𝑒. This dependent variable allows
GSH to track clients’ progress in improving their total monthly net income.

43
Coefficient Interpretations and Statistical Tests
Similar to the regression model above, GSH should interpret the coefficients of different variables
differently.
a) Interpretation of the key independent variable
As mentioned in the previous regression model, the predicted effect on the change of net income
from a one-month increase in the time a client has spent in the ABC program is (𝛽̂1 + 2𝛽̂2 𝑋 𝑚𝑜𝑛𝑡ℎ)
dollars. As before, the effect of the key independent variable “month” changes depending on the
value of “month”. In order to figure out the predicted effect at each month, GSH will need to replace
𝑋 𝑚𝑜𝑛𝑡ℎ with the actual month value.
For example, imagine that the coefficient of “month”, 𝛽̂1, is 500, and the coefficient of “month2
”,
𝛽̂2, is -25. In this case, the predicted effect is (−50𝑋 𝑚𝑜𝑛𝑡ℎ + 500). For a client in his third month
with the ABC program, spending one more month in the program is associated with a
(−50 ∗ 3 + 500), or $350 dollar increase in the change in net income, holding everything else
constant. Whereas in his sixth month with the program, spending one more month is associated
with a (−50 ∗ 6 + 500), or $200 dollar increase in the change of net income, holding everything
else constant. If this pattern were observed for many clients, the conclusions that GSH would be
able to make in this case is that the ABC program helps increase clients net income over time, but
the program’s impact on clients’ net income at 6-month mark is less than the impact at the 3-month
mark, holding everything else constant.
Ideally, GSH would want to observe a positive predicted effect at any given month. This would
show that participating in the ABC program would increase clients’ net income, which is an
indicator of client’s improved financial situation. As we mentioned in the Defining and Measuring
Self-Sufficiency chapter, a positive change in net income suggests that a client has become more
self-sufficient based on our definition.
b) Interpretation of the control variables
The interpretation for the coefficient of a continuous variable is that a one-unit increase in this
continuous variable is predicted to affect the change in net income by the estimated coefficient,
holding everything else constant.
The interpretation for the coefficient of a binary variable is that the average difference of the change
in net income between clients in the category compared to the omitted category, holding everything
else constant.
c) Statistical tests
As within the previous regression model, GSH should test the statistical significance of the estimate
coefficients in order to make any meaningful conclusions about the regression estimates. The
methods that we recommend GSH employ to test significance are the same as described in the
previous model. GSH can find additional guidance on these procedures from a standard
econometrics textbook such as Wooldridge (Wooldridge 2012).

44
III. Evaluation for ES – Descriptive Statistics
Methodology
We propose to use descriptive statistics as the approach to analyze the effectiveness of the ES
program. The unique nature of the ES program renders a regression analysis on impact evaluation
impractical due to the following two reasons:
 Follow-up data may be difficult to obtain because ES clients do not necessarily have to
comply with GSH’s communication condition after receiving a grant;
 Other unexpected factors, which will be difficult to capture, may positively or negatively
impact clients’ self-sufficiency after receiving an ES grant.
Calculation
We suggest GSH use the four questions outlined in the Data Collection Recommendations chapter
to gather standardized information from all ES grant recipients:
1. Was the crisis averted?
2. How many months have you paid your rent in full since the ES grant?
3. How many months have you paid your utility bill in full since the ES grant?
4. How many months have you paid both your rent and your utility bill in full since the ES
grant?
For the first question, we recommend GSH use the Yes/No survey responses to demonstrate ES
grant effectiveness at its most basic level. Using client survey data in this fashion allows GSH to
examine the effectiveness of the grant on remedying the immediate crisis at hand faced by ES
clients. For example, if 500 ES grants were awarded in 2014 and 400 respondents replied with a
confirmation that their crisis was averted, GSH could say that 400/500 or 80% of grants were
effective. GSH could potentially use this type of information to continue to quantitatively prove to
funders, stakeholders, and clients that ES grants have an immediate benefit to the community that
they serve in the form of crisis prevention.
For the next three questions, we recommend GSH use the average number of months a client has
made in-full payments as the measure of interest for each question. ES clients will respond with a
number between zero and six, which indicates the number of months they have made in-full
payments for rent, utilities, or both.
We recommend collecting standardized data across all ES grant types to better understand the
grant’s effect on a client’s self-sufficiency. As described in our Defining and Measuring Self-
Sufficiency chapter, looking at a client’s housing security is vital to the analysis of ES grant
effectiveness. Measuring whether clients paid both rent and utility payments in full could help GSH
understand if a client has maintained their level of self-sufficiency since delivery of the ES grant.
If a client is sacrificing utilities to make a rental payment or sacrifices rent to cover utilities (or vice
versa), a client is not housing secure and thus may be less self-sufficient despite receiving the ES
grant.

45
Using the same example scenario as above, a way for GSH to generate descriptive statistics of the
in-full payment survey questions would be to imagine that every client responded to the follow-up
text message survey. If ES clients do not respond, we recommend dropping them from the sample
of data for the purposes of this analysis. We suggest dropping clients who are unresponsive because
the reason for their unresponsiveness and the unknown status of in-full payments could be either
positive or negative and would be impossible to infer. Evaluating them would unfairly bias our
results.
Table 9 – Example calculation of ES evaluation
Based on the example table, the calculations we propose GSH use for descriptive statistic analysis
for ES follows the same pattern as ABC. To calculate the average months of on-time payments for
each question, GSH should multiply the number of respondents who answered in each month
category (number of months * number of clients) and divide that sum by total clients, which is 500
in this example:
Number of clients who paid their rent in full:
Average = ((3*50) + (4*50) + (6*400))/500 = 5.5 months
Number of clients who paid their utility bill in full:
Average = ((0*50) + (5*100) + (6*350))/500 = 5.2 months
Number of clients who paid both their rent and utility bill in full:
Average = ((0*50) + (3*100) + (6*350))/500 = 4.8 months
This example calculation of descriptive statistics for ES clients demonstrates that, on average, the
ES client base pays their rent in full more often than they do their utility bills. Also, clients averaged
4.8 months of in-full payments for both rent and utilities, which for GSH may be the most important
number to focus on. If clients are unable to pay for rent or utilities, they may not be housing secure,
an important indicator of a client’s self-sufficiency. We recommend using descriptive statistics for
ES clients in this fashion to allow GSH to gain an understanding of how clients progress after the
Month after ES Grant is Provided 0 1 2 3 4 5 6 Average
Number of Clients Who Paid Rent in
Full
50 50 400 5.5
Number of Clients Who Paid Utility
Bill in Full
50 100 350 5.2
Number of Clients Who Paid Both
Rent and Utility Bill in Full
50 100 350 4.8

46
grant and where future clients may need additional resources offered to them based on the
measurement of their in-full payments over time.
Evaluation Timeframe
We recommend that GSH collect follow-up data at the 2- and 6-month mark after the ES grant is
administered. We do not think it is necessary to look beyond the six month timeframe for two
reasons. First, the ES program has the most effect on clients in the immediate-term. Since the
program is designed to help clients overcome a temporary crisis and provides only a one-time
intervention, its long-term effect on clients is uncertain. Second, the difficulty of collecting follow-
up data increases as the timeframe gets longer. Allocating resources to collect follow-up data
beyond six months might not be cost-effective or meaningful to collect because of the ambiguity
of the grant’s effect on client self-sufficiency over the long run.

47
Possible Alternatives
This chapter will present alternative options that GSH might consider should resource constraints
prevent GSH from implementing the recommendations outlined above.
I. ABC
Data Collection Alternatives
The methods which GSH currently use to collect data on their clients is not different from our
recommendations. In a client’s first six months in the ABC program, data may be easily collected
as the client will have constant interactions with their case managers. However, after that period, it
may be more difficult to follow up with clients. Incomplete data or inconsistent data collection will
prevent using the regression model that we have recommended.
Here are a few suggestions for ensuring the collection of follow-up data:
 Make completing the surveys a requirement of staying in the program and/or receiving
program benefits.
o ABC can adjust its requirements according to how much importance it wishes to
place on data collection.
 Administer the follow-up survey at/during budgeting classes.
 Send out reminders to clients via post mail or email.
 Incentivize clients to self-report at the appropriate interval.
o This could be in the form of a small cash prize, a gift card, or some other giveaway.
Quantitative Analysis Alternatives
We understand that GSH’s resources and needs may change over time, and that our primary
recommendations may not always be feasible. Here, we provide some alternatives that GSH may
want to explore in conducting their quantitative analysis of the ABC program, so as to make best
use of the data they have collected with the capacity they have.
A. Offer employee training
If someone within GSH has the willingness and the time to receive training, it might be worthwhile
for them to attend classes in statistical methods at a local community college or university. There
are also textbooks on introductory econometrics that are available for purchase or for loan from the
local library. Many online resources such as edX, Coursera, and Khan Academy offer free lessons
on statistical analysis. Though this requires a time commitment and a willingness to learn, having
someone internally who can perform the quantitative analysis could be a valuable asset to GSH as
that person would have an intimate knowledge of how the ABC program works and would modify
and improve our recommended model to adapt to any changes within the organization over time.

48
Training someone would not only add to that person’s professional development, but the skills
acquired are highly transferable and could be applied to evaluating GSH’s other programs.
If ABC decides to go with this alternative, then GSH might choose to acquire some statistical
software for the employee to use. The statistical package STATA is a strong tool which provides
users with a user-friendly interface and has large and active online support forums.5
Its wide base
of users make it easy for a user to find help, whether using their comprehensive native help book
or searching the internet for other people who have encountered similar problems. STATA/IC 13
is the most affordable version of STATA that would be able to carry out all the functions
recommended in our quantitative model. An annual license for STATA/IC 13 currently costs $595.6
An alternative package to STATA would be R.7
R is open-sourced, meaning that it is a free software.
However, while it is an incredibly powerful tool for performing regression analysis, it is less user-
friendly than STATA and would be more challenging for those without a coding background.
RStudio is an accompanying software which improves R’s user-interface, but it still requires some
training and technological knowledge to navigate.8
B. Seek volunteer or part-time assistance
If no one at ABC has the training or time to carry out the analysis we have recommended, we
suggest that GSH look to hire someone – either on a full-time or a part-time basis – to take over
this task. Even if the data are collected fully and completely, complications with the model might
arise that are at present hard to anticipate and correct, but someone with adequate quantitative
analysis skills would be able to adapt and refine the model as more data are collected.
In the event that GSH is unable to hire someone to perform quantitative analyses on the data
collected, they may want to consider seeking the help of a graduate student in a nearby university
to help. University students are often looking for experience and opportunities to work with
organizations such as GSH’s, and GSH can offer valuable experiences to a student who would be
willing to practice his or her quantitative skills with a real dataset. If possible, GSH might try to
find two graduate students to come on board at the same time. As students are may be less likely
to be quantitative experts, having two students collaborate on this project can increase their
confidence in their results.
5
More information on STATA can be found here: http://www.stata.com/.
6
As of April, 2015.
7
R is free to download here: http://www.r-project.org/.
8
An open-sourced version of RStudio is free to download here: http://www.rstudio.com/.

49
II. ES
Data Collection Alternatives
Emergency Services faces many hurdles to collecting complete data from their brief interaction
with their client base. Our recommended method of using text message surveys requires some
upfront investment, but in the long run, we believe it could save ES staff time and money due to
the automated process. However, in the event that ES staff might not have the time or resources to
look into buying new software and carrying out our main recommendation, here are some
alternative ways that GSH may still collect data. GSH may also consider using our main data
collection recommendation in conjunction with any of these other alternatives to capture as much
data as possible.
A. Phone surveys (to client, landlord, or utility company)
At the follow-up intervals, GSH could call the client and administer the survey questions over the
phone. As they are simple and straightforward questions, data should not be difficult to collect. A
question asked over the phone will also be harder to ignore than a text message.
In the event that the client is unable to be contacted, GSH should then attempt to contact the
landlord or the utility company to gather this information. Landlord contact information is collected
when a client applies for an ES grant to assist with rent or a security deposit, and ES works directly
with the landlord in handling the grant distribution. GSH can use this information to get in touch
with the landlord to collect this data.
In our interviews with GSH staff, we have also learned that ES has built up a good reputation with
the local Fairfax utility companies. In the event a client who received utility assistance is unable to
be contacted, GSH should attempt to contact the utility companies directly to gather information
on the client’s recent payment history. ES also has access to the utility company database, and it is
possible for GSH staff to log in and check a client’s recent bills. If, however, a client no longer has
a utility account, it should be recorded as a missing value as it may be difficult to track down the
client. Table 10 below details the benefits and costs of carrying out this recommendation.
Table 10 – Benefits and Costs of Conducting Phone Surveys for ES
Benefits: Costs:
Contact information is already currently
being collected
Requires manual data entry
Can leverage existing relationships with
utility company
Somewhat labor intensive
Low-cost
B. Online surveys
If ES clients mostly have access to the internet, and have an email account, GSH could consider
sending their clients an online survey via email. With this option, ES staff would create a survey

50
with the suggested questions, and send the client an email at the recommended follow-up interval.
There are many affordable ways for GSH to do this, using free or low-cost services like Google
Forms, SurveyMonkey, or Qualtrics.9
This method is a cost-efficient way of surveying clients, and
the many available digital survey services make data collection automated and easy to manage.
However, clients can easily ignore these emails, so GSH may want to send follow-up emails to
encourage a response. Table 11 below outlines the benefits and costs of conducting these online
surveys.
Table 11 – Benefits and Costs of Conducting Online Surveys for ES
Benefits: Costs:
Data would be digitized Clients would need a reliable email
address and internet access
Cost-efficient Emails are easy for the client to ignore
Emails could be automated to follow up at
the right interval
Possible to ‘piggyback’ other longer
follow-up questions to the surveys
C. Mail-in responses
An effective technique that some companies use to collect survey data is to mail surveys to
households, together with a crisp dollar bill as a reward for not discarding the mail. The included
instructions should state that if the survey is completed and returned, a five-dollar bill will be mailed
to the household. Some survey designs may vary, and provide a crisp five-dollar bill outright, along
with a blank survey. Numerous studies have shown that this is an effective, albeit expensive, way
to increase survey response rates (Edwards et al 2002, Jobber et al. 2004). ES staff could explore
this option as a potential way to gather survey responses. However, this could be labor intensive
(in sending mail and digitizing responses), costly, and does not guarantee a good completion rate.
ES might also conduct further research into the costs and risks of using this method, some of which
are outlined in Table 12 below.
Table 12 – Benefits and Costs of Sending Postal Mail Surveys for ES
Benefits: Costs:
Cash incentivizes clients to respond Expensive
Possible to ‘piggyback’ other longer
follow-up questions to the surveys
Labor-intensive
9
More information about the above-mentioned services can be found at these following links respectively:
https://forms.google.com, https://www.surveymonkey.com/, and http://www.qualtrics.com/.

51
Quantitative Analysis Alternatives
The model that we have proposed for analyzing the data collected for ES is fairly straightforward
and simple, primarily using descriptive statistics to show program impact. However, if the capacity
of the ES program and staff grows in the future, ES staff may want to consider revising our data
collection recommendations to gather more comprehensive data in order to perform more
sophisticated quantitative analyses.

52
Conclusion
Based on our academic research on similar programs, interviews with GSH staff, and careful
consideration of GSH’s organizational needs and objectives, we developed a series of
recommendations to assist GSH in evaluating their ABC and ES programs. We recommend that
GSH define self-sufficiency as a continuum of economic independence where a person is more or
less self-sufficient based on the amount of government and non-profit benefits they receive. For
ABC, we recommend the use of financial measures to track program effectiveness, with debt-
income ratio and change in net income as the measures of self-sufficiency. For ES, our
recommended measure of self-sufficiency is the number of months a client pays their rent and
utility bill in full following the grant receipt. This measure aims to capture the stabilizing effect of
the grant (preventing clients from becoming less self-sufficient through the loss of housing security).
We also developed a series of data collection methods that build upon GSH’s current collection
methods, and included the new variables needed to conduct evaluation of ABC and ES. We
recommend GSH survey their ABC clients in person at the client meetings that occur three, six, 12,
and 24 months after program entry. Given that GSH has significantly fewer points of contact with
ES clients and the likely shorter-term impact of the grant, GSH should send text message surveys
to ES clients two and six months following the client’s receipt of the grant.
Our main recommendations focus on the methods GSH should use to analyze the new data collected
from ABC and ES clients. For ABC, we recommend using both descriptive statistics and regression
models to determine GSH’s impact on the self-sufficiency measures of a client’s debt-income ratio
and change in net income. Descriptive statistics of the measures will allow GSH to gain a picture
of their clients’ current levels of self-sufficiency, including how those measures change over the
clients’ participation in the program. Regression models will allow GSH to isolate its impact on
clients’ self-sufficiency through our recommended independent variable of the number of months
a client has participated in the program. Regression also allows GSH to control for other factors
that affect self-sufficiency such as level of education, family size, and having previously declared
bankruptcy.
For ES, we recommend that GSH perform descriptive statistics using the proposed measure of self-
sufficiency. GSH should calculate averages based on the three questions asked: how many months
a client has paid their rent in full, how many months a client has paid their utility bill in full, and
how many months a client has paid both rent and utilities in full. Calculating an average of the first
two questions can show GSH if clients lack housing stability by avoiding paying utilities bills to
cover rent (or vice versa). Calculating an average of the third question will allow GSH to evaluate
ES’ overall success at maintaining their clients’ level of self-sufficiency by helping maintain their
housing stability.
We are proud to have been a participant in GSH’s process to incorporate evaluation and
effectiveness measures into their programs. This report outlines what we believe to be the best
methods for GSH to move forward in evaluating the ABC and ES programs. We hope that this

53
report will serve as a useful guide for GSH when they develop evaluation methods and goals for
the rest of their activities.

54
References
Alaimo, S. P. 2008. Nonprofits and evaluation: Managing expectations from the leader’s
perspective. In J. G. Carman & K. A. Fredericks (Eds.), Nonprofits and evaluation. New
Directions for Evaluation, 119, 73–92.
Bratt, Rachel G., and Langley C. Keyes. 1998. New Perspectives on Self-Sufficiency: Strategies
of Nonprofit Housing Organizations. Medford, MA: Tufts University, Department of
Urban and Environmental Policy.
Brennan, Maya. Strengthening Economic Self-Sufficiency Programs: How Housing Authorities
Can Use Behavioral and Cognitive Science to Improve Programs. Ideals for Housing
Policy and Practice. Center for Housing Policy, June 2014.
http://www.nhc.org/EconomicSelfSufficiency_final_web.pdf.
Consumer Financial Protection Bureau. “What Is a Debt-to-Income Ratio? Why Is the 43% Debt-
to-Income Ratio Important?” December 30, 2013.
http://www.consumerfinance.gov/askcfpb/1791/what-debt-income-ratio-why-43-debt-
income-ratio-important.html.
Edin, Kathryn, and Laura Lein. “The Private Safety Net: The Role of Charitable Organizations in
the Lives of the Poor.” Housing Policy Debate 9, no. 3 (January 1, 1998): 541–73.
doi:10.1080/10511482.1998.9521307.
Edwards, Phil, Ian Roberts, Mike Clarke, Carolyn DiGuiseppi, Sarah Pratap, Reinhard Wentz,
and Irene Kwan. “Increasing Response Rates to Postal Questionnaires: Systematic
Review.” BMJ 324, no. 7347 (May 18, 2002): 1183. doi:10.1136/bmj.324.7347.1183.
Fairfax County Human Services Council. “Economic Self-Sufficiency.” Roundtable Discussion,
February 6, 2012.
http://www.fairfaxcounty.gov/hscouncil/minutes/economic_self_sufficiency.pdf.
Freeman, Lance. 2005. “Does Housing Assistance Lead to Dependency? Evidence From HUD
Administrative Data.” Cityscape 8, no. 2. 115–33.
Good Shepherd Housing and Family Services, Inc. 2011. GSH Strategic Plan: Looking to the
Future by Building on the Past. Alexandria, VA.
Good Shepherd Housing and Family Services, Inc. ABC Curriculum Guidelines. 2014.
Alexandria, VA.
Groton, Danielle. “Are Housing First Programs Effective? A Research Note.” Journal of
Sociology and Social Welfare XL, no. 1 (March 2013): 51–63.
http://www.wmich.edu/hhs/newsletters_journals/jssw_institutional/institutional_subscriber
s/40.1.Groton.pdf.
Hasson, David S. “Water Utility Outcomes for Low-Income Assistance Programs.” American
Water Works Association 94, no. 4 (April 1, 2002): 128–38.
http://www.jstor.org.proxy.library.georgetown.edu/stable/41298504.

55
Jobber, David, John Saunders, and Vince-Wayne Mitchell. “Prepaid Monetary Incentive Effects
on Mail Survey Response.” Journal of Business Research 57, no. 1 (January 2004): 21–
25. doi:10.1016/S0148-2963(02)00280-1.
Johnson, R. Burke, Anthony J. Onwuegbuzie. 2004. Mixed Methods Research: A Research
Paradigm Whose Time Has Come. Educational Researcher, Vol. 33, No. 7, pp. 14-26.
American Educational Research Association.
Kleit, Rachel Garshick. “Designing and Managing Public Housing Self-Sufficiency Programs
The Youngs Lake Commons Program.” Evaluation Review 28, no. 5 (October 1, 2004):
363–95. doi:10.1177/0193841X04265649.
LIHEAP. “LIHEAP: Fighting Poverty in Virginia,” 2014. http://liheap.org/states/va/.
Massachusetts Community Action Program. Do You Know the Way to Self-Sufficiency? A Case
Study Report, September 30, 2003. http://www.masscap.org/workforce/fnlstudies9-24-
3.pdf.
Murray, Anthony G., and Bradford F. Mills. “The Impact of Low-Income Home Energy
Assistance Program Participation on Household Energy Insecurity.” Contemporary
Economic Policy 32, no. 4 (October 2014): 811–25. doi:10.1111/coep.12050.
Rosenthal, Larry A. “A Review Of Recent Literature On Housing Assistance And Self-
Sufficiency.” Berkeley Program on Housing and Urban Policy, September 1, 2007.
http://escholarship.org/uc/item/6ps2v9d7.
Santiago, Anna M., and George C. Galster. “Moving from Public Housing to Homeownership:
Perceived Barriers to Program Participation and Success.” Journal of Urban Affairs 26,
no. 3 (August 1, 2004): 297–324. doi:10.1111/j.0735-2166.2004.00201.x.
Silva, Lalith de., Imesh Wijewardena. 2011. Evaluation of the Family Self-sufficiency Program:
Prospective Study. Planmatics, Inc. Prepared for: U.S. Department of Housing and Urban
Development; Office of Policy Development and Research.
Verma, Nandita, Betsy L. Tessler, Cynthia Miller, James A. Riccio, Zawadi Rucks, and Edith
Yang. Working Toward Self-Sufficiency: Early Findings from a Program for Housing
Voucher Recipients in New York City. SSRN Scholarly Paper. Rochester, NY: Social
Science Research Network, January 17, 2013. http://papers.ssrn.com/abstract=2202444.
Washington, Thomas Alex. “The Homeless Need More Than Just a Pillow, They Need a Pillar:
An Evaluation of a Transitional Housing Program.” Families in Society: The Journal of
Contemporary Social Services 83, no. 2 (January 1, 2002): 183–88. doi:10.1606/1044-
3894.36.
Wooldridge, Jeffrey M. Introductory Econometrics: A Modern Approach. 5th ed. Cengage
Learning, 2012.

56
Appendix
APPENDIX A1
Variable names
First Name
Last Name
Date of Birth
Sex male 0: Male 1: Female
Age age
Ethnicity
Hispanic hispanic 0: Non-hispanic 1: Hispanic
Race
(check one) White 0: Not White 1: White
Black black 0: Not Black 1: Black
Asian asian 0: Not Asian 1: Asian
Native American natAm 0: Not Native
American
1: Native
American
Other other_race 0: Not Other 1: Other
Education
(check one) Less than High School 0: Not Less than
High School
1: Less than
High School
High School highsch 0: Not High
School
1: High School
College and above college 0: Not College 1: College and
above
Marital Status
(check one) Single 0: Not Single 1: Single
Married married 0: Not Married 1: Married
Separated/Divorced divorced 0: Not Separated/
Divorced
1: Separated/
Divorced
Widowed widowed 0: Not Widowed 1: Widowed
Do you receive housing vouchers? voucher 0: No 1: Yes
Size of household? size
Do you have a history of mental illness? mental 0: No 1: Yes
Have you been bankrupt before? bankruptcy 0: No 1: Yes
Have you abused any substances in the past 6 months? sub_abuse 0: No 1: Yes
Have you ever been convicted of a crime? (felony, probation, parole, incarcerated, etc.)crime 0: No 1: Yes
Have you gambled in the past 6 months? gamb_occ 0: No 1: Yes
(If yes) How many times a week do you gamble? gamb_fre
0 times/week 0: No 1: Yes
1-2 times/week gamb_1 0: No 1: Yes
3+ times/week gamb_2 0: No 1: Yes
Monthly wages ($) income
Credit card debt ($)
Loans ($)
Student ($)
Car ($)
Other ($)
Overdue utility bills ($)
Uninsured medical bills ($)
Other outstanding debt ($)
Total monthly debt ($) debt
Total monthly debt ($) / Monthly wages ($) debt_income
Monthly wages ($) - Total monthly debt ($) net_income
How many months since entering the ABC program? months
Have you been recommended to take additional budgeting classes? att 0: No 1: Yes
ABC Client Changes Over Time
#
Calculations
#
#
#
#
#
#
#
#
ABC Entry Data Collection Form
Coding Rules
#
#
Income History
Debt History
Personal History
Administrative Information
#
#
#

57
APPENDIX A2
Variable names
Monthly wages ($) income
Credit card debt ($)
Loans ($)
Student ($)
Car ($)
Other ($)
Overdue utility bills ($)
Uninsured medical bills ($)
Other outstanding debt ($)
Total monthly debt ($) debt
Total monthly debt ($) / Monthly wages ($) debt_income
Monthly wages ($) - Total monthly debt ($) net_income
How many months since entering the ABC program? entry
Have you been recommended to take additional budgeting classes? att 0: No 1: Yes
#
Income History
#
ABC Follow-up Data Collection Form
Debt History
#
Coding Rules
ABC Progress
#
#
#
#
#
#
#
Calculations
#
#

58
APPENDIX B
First Name
Last Name
Date of ES grant
Months since ES grant
Contact information
Was the crisis averted? 0: No 1: Yes
How many months have you paid your utility bill in full since
the ES grant?
Question 3
How many months have you paid both your rent and your
utility bill in full since the ES grant?
#
Question 1
Question 2
#
Administrative Information
ES Follow Up Data Collection Form
Coding rules
#
#How many months have you paid your rent in full since the
ES grant?

Piong_GSH Final Consulting Report

More Related Content

Viewers also liked

Similar to Piong_GSH Final Consulting Report

Piong_GSH Final Consulting Report