1. Evaluation of America Works
Contextualized Literacy
Programs
Swati Desai, Ph.D.
Rockefeller Institute of Govt. SUNY
Andrew Silverstein
Graduate Center, CUNY
Ashley Putnam
America Works of New York

2. What is Contextualized Literacy?
Background:
 In welfare and employment fields the focus has changed from
only Work First or only Education to a combined approach.
 New York City’s Human Resources Administration (HRA)
included a focus on contextualized literacy as a part of its
employment contracts.
 A Contextualized Literacy eligible participant is identified as a
person without a high-school diploma or equivalent degree
who is seeking employment through the Back 2 Work program.

3. Fellowship Program
 America Works of New York as an HRA employment contractor
implemented these programs and requested the evaluation.
 In June, America works dedicated their Literacy Fellowship
Program to designing and implementing Contextualized
Literacy Programs.
 Literacy Fellows are young professionals interested in careers
in law, public policy, education and social service. The
fellowship created space for research and innovation while
adding new energy to current programs.
 The first iteration of these programs was the Customer
Service course which was recommended to eligible
participants starting in August. Later courses included Food
Service and TASC prep.

4. Research Questions
 Are Contextualized Literacy participants more likely to
get a job compared to those who don’t participate in the
program?
 Do those who participate in Contextualized Literacy
programs get jobs that pay more compared to those who
are eligible but do not participate?
 Do different demographic groups have different
employment outcomes at the conclusion of the
contextualized literacy programs?
 How does the short-term evaluation of impact help
improve the program?
To answer these questions, we used a semi-experimental
design.

5. Data
 Treatment Group (n=700): those who were eligible and offered
Contextualized Literacy Classes
CL Eligible Clients who were assigned to AW from September to December
2014
 Control Group (n=324): those who were identified as eligible for
Contextualized Literacy Programs but not offered additional services
CL Eligible Clients who were assigned to AW from January to April 2014
NOTE: Both groups did not have HS diploma or GED
 America Works provided basic demographic data, TABE scores, case
history, class attendance, and employment data.
 AW of NY received data from HRA through NYCWAY database and
collected internal data from intake process, attendance records, and
information from employers.

6. Summary of Findings
 The offering of courses itself, regardless of
participation, increased wages and engagement over
treatment period.
 Wage increases are particularly noticeable for women
identified in these groups.
 Initial 2015 data indicates that employer involvement
might lead to better outcomes. More research is
necessary to confirm hypothesis.

7. * Indicates statistical
significance

8. Treatment and Control Groups
do not look alike
An analysis of the baseline characteristics shows that the Control and
Treatment group differ in more aspects that just the implementation of the
CL program. The differences in baseline characteristics are:
 Age: (Control 35.48, Treatment 33.69, p<.05)
 Percent assigned to Brooklyn: (Control 67.28%, Treatment 75%, p<.05)
 TABE Math Score at EP: (Control 3.95, Treatment 4.21, p<.10)
 Months on PA 3-10 years prior to start (Control 12.35, Treatment 8.22, p<.10)
These differences could be due to program strategy or they could be random.
Queens and Brooklyn had differences in baseline characteristics, including higher
percentage of males, black participants, and individuals with reading difficulty in
Brooklyn.

9. Methods
 We used the variables described above as covariates in inverse-
probability- weighted regression-adjustment (IPWRA) models.
 This is a two stage analysis. In the first stage, probabilities of being
in the treatment group are calculated for each observation using a
logit model. The inverse of these probabilities are then used as
weights in a weighted regression to predict treatment level predicted
outcomes.
 Given that in both stages of analysis, the model is controlling for
differences in the observables between the Treatment and Control
groups, this estimator is considered to be double-robust.
 In addition, we separately run our model for only Brooklyn and only
Queens participants in order to see if there were distinct treatment
effects due to differences in recruitment, programming and labor
markets. Additional estimates were done for subgroups (Blacks;
Black-Women; and Women in Households 2+).

10. Weighted

11. Results
Participation:
 Treatment participants were significantly less likely to receive
FTR/FTCs (1.12 less).
 This could be in part due to HRA’s kinder gentler policy of greater
leniency during this time.
 The could also be due to increased case management and
additional help from literacy fellows.
 The offering of classes and greater leniency did not increase time
in program.
 Treatment group was 17.84 percent more likely to exit the program
with 150 days of entry.

12. Participation Case Study:
Takeesha Thomas
Takeesha attended Customer Service and later
enrolled in the HSE course during the very early
stages of the program. She struggled with math,
but was encouraged by relationship with the
fellows.
Takeesha says, “There were days I felt like I
couldn't take it any more. It isn't because I'm not
smart enough, but this test is extremely hard. I
learned to use these fellows at America Works-
they are serious. They may be human, but they
have S's on their chests."
Takeesha took the TASC exam in November of
2014 and failed by two points on the math section.
In May of 2015, after continuing to prepare
intensively with fellows, she passed the TASC
exam with a 558 on her math score.
Takeesha says: “At the end of the day, you have
to do it for yourself.”

13. Results
Positive Labor Market Effects:
 For Brooklyn participants there was no increase in rate of finding
employment, but there was a 7.31% increase in wages.
 For Queens participants there was a 13.81 % increase in finding
employment but no increase in wages.
 For All participants, there a 4.71% increase in wages and a 4.1%
increase in employment in the treatment group compared to the control
group, but the employment effect is not significant.
Differences in effect between Brooklyn and Queens, may be due to
differences in the programming between the two sites and differences in
labor markets between the two boroughs.

14. Subgroup Analysis

15. Subgroup Analysis
This analysis shows interesting results for different
groups in samples.
 Females with household size more than 1 have
the lowest FTR/FTC rate.
 Blacks in the treatment group are 18.65% more
likely to exit within 150 days compared to the
control group.
 Higher wages in the treatment group are due to
the 4.27% higher wages for women.
 Women are 6% more likely to stay employed
after 30 days of employment.
 These results suggest: children inspire mothers.
Also, higher grant level might allow them to stay
in the program and achieve their goals.

17. Participation
 Only 34% of enrolled treatment group participants
attended class, and only 6.71% earned certificates.
 The majority of individuals in the treatment group
attended customer service class (88.5%). The more
classes taken in this treatment group do not lead to more
employment.
 The improved labor market outcomes are most likely a
result of being more engaged in the overall program with
fellows, as opposed to a direct result of the course itself.
 Over time, participants may be more motivated to
continue their education and gain meaningful labor
market skills.

18. Conclusion : 2014 Analysis
 It difficult to determine causality between training and employment
outcomes for this initial contextualized literacy program.
 Decreased in FTC/FTR could be an indicator of increased engagement
due to courses or HRA policy change.
 Other research suggests that literacy and education programs may have
positive impact on employment and wages after longer period (2-5
years). Thus, it is too early to determine the impact.
 Interestingly, women seem to have higher wages, participate longer in
the program and stay employed longer as shown in the subgroup
analysis. This suggests that they are motivated and might benefit
significantly by successfully completing courses.

19. Recommendations
 AW might be able to offer flexible classes on weekend and after work
to facilitate the completion of classes and getting a certificate.
 Using a cohort model rather than open enrollment might increase
participation and completion.
 The cohort model also creates moral support from fellow students
improving their motivation to complete
 Cohort model could facilitate better relationship with instructors
 America Works should analyze the impact of current programs that
provide more sector-specific training and employer directed training.

20. Changes since 2014
With the initial analysis of the Customer Service program,
America Works has rolled out additional courses in 2015.
 Career Specific Courses: Eligible clients are now offered
additional contextualized literacy courses in Food Handler’s,
OSHA, Security, and CDL-C. These courses are designed
with employer partners around industry-recognized
certificates.
 Post-Employment Courses: With the help of Baruch EMPA
faculty, America Works created a Professional Development
course to help individuals advance in the workplace. We’ve
also added weekend hours.
 Literacy Courses: America Works now offers a pre-HSE
course (5th-9th grade TABE scores) and HSE course (9thgrade
+), as well as ESOL classes. These courses run in a six-
week cohort model.

21. *enrollment into class has requirements per employer’s stipulation
Initial 2015 Overview
Course Enrolled % Attended % Working Wages Hours
Customer Service 824 25% 11% $9.86 28
OSHA 126 48% 11% $13.10 33
Food Service 518 47% 13% $10.06 29
Security 265 53%* 33% $10.00 30
CDL 120 98%* 21% $11.77 35
GED 117 79% 9% $9.51 33
ESL 255 74% 12% $9.82 25
Average 47% 15% $ 10.21 29

22. Initial 2015 Overview
Course Enrolled % Attended % Working Wages Hours
Customer Service 824 25% 11% $9.86 28
OSHA 126 48% 11% $13.10 33
Food Service 518 47% 13% $10.06 29
Security 265 53%* 33% $10.00 30
CDL 120 98%* 21% $11.77 35
GED 117 79% 9% $9.51 33
ESL 255 74% 12% $9.82 25
Average 47% 15% $ 10.21 29
Finding: Wage gains are higher for sector-specific courses that provide
an industry-recognized certificate upon completion.

23. Initial 2015 Overview
Course Enrolled % Attended % Working Wages Hours
Customer Service 824 25% 11% $9.86 28
OSHA 126 48% 11% $13.10 33
Food Service 518 47% 13% $10.06 29
Security 265 53%* 33% $10.00 30
CDL 120 98%* 21% $11.77 35
GED 117 79% 9% $9.51 33
ESL 255 74% 12% $9.82 25
Average 47% 15% $ 10.21 29
Finding: Attendance rates are higher for literacy courses after
implementing the six-week cohort model. These clients may also be more
motivated to attend classes and less interested in employment.

24. Initial 2015 Overview
Course Enrolled % Attended % Working Wages Hours
Customer Service 824 25% 11% $9.86 28
OSHA 126 48% 11% $13.10 33
Food Service 518 47% 13% $10.06 29
Security 265 53%* 33% $10.00 30
CDL 120 98%* 21% $11.77 35
GED 117 79% 9% $9.51 33
ESL 255 74% 12% $9.82 25
Average 47% 15% $ 10.21 29
Finding: Employer-specific courses limit participants due to requirements
regarding background check, drug test, and driver’s license, but overall have
the highest employment rates.

25. Initial 2015 Overview
● Wage gains are higher for sector-specific courses that provide an
industry-recognized certificate upon completion.
● Attendance rates are higher for literacy courses after implementing
the six-week cohort model. These clients may also be more motivated to
attend classes and less interested in employment
● Employer-specific courses limit participants due to requirements
regarding background check, drug test, and driver’s license, but overall
have the highest employment rates.

26. Policy Conclusions
 Sector-Specific: Courses tied to industry-recognized
certification show the most wage gains and more
attendance from participants.
 Employer-Driven: Work with employers to create
courses that fit their hiring needs and provide
opportunities for continued growth.
 Delivery mechanism: The America Works relationship
with employers and sensitivity to private sector demands
benefitted program participants.
The 2015 results indicate it may not have been the literacy
classes themselves, as much as the direct ties to industry
credentials and employers that helped students improve
wages.

27. Next steps:
 More research is necessary to determine if employer-
designed and sector-specific courses have wage gains
over time.
 Department of Labor data on participants should be
examined 2-5 years out from program to see if long-
term gains in employment are visible.