This document summarizes a presentation given at the NAWB Forum in Washington DC on March 13, 2016. The presentation focused on how states can use quality data to do good better by developing inclusive, aligned, and market-relevant education and workforce data systems. It discusses the importance of evidence-based policymaking and describes the Workforce Data Quality Campaign's 13-point state blueprint for a high-quality data infrastructure. The presentation outlines some elements that states have achieved success in, like cross-agency councils and improving labor market information, as well as promising state practices from places like Oregon, Alaska, and Kansas. It concludes by emphasizing the importance of data sharing agreements and continued progress in developing better data systems.

1. Open Platforms & Data Smarts:
How We Can Do Good Better
NAWB Forum, Washington DC
March 13, 1016

2. Our Practice
We help public servants, philanthropists, and
world-changers find what works in solving
their communities' toughest challenges.
Our approach to research, evaluation and
hands-on-technical assistance is rooted in a
commitment to equity and focus on learning
– so we all grow smarter by working together.
(And we love data, obviously…)

3. WHY DATA? WHY NOW?

4. States Doing Good Better with
Quality Data
Jenna Leventoff, Policy Analyst
March 13, 2016

5. • Advocate for inclusive, aligned and market-relevant education and
workforce data that can help our nation’s human capital policies meet the
challenges of a changing economy.
• Promote federal and state reforms for data systems that provide useful
information for policymakers, students and workers, business
leaders and educators.
• State Blueprint with 13 key features of a high-quality data infrastructure
• Address federal legislation, funding and technical assistance
• Policy agenda developed by broad coalition of national organizations,
state leaders and technical experts across education/workforce
spectrum
WDQC Mission

6. Apollo Education
Group
Bill & Melinda
Gates Foundation
Joyce Foundation
Laura & John
Arnold Foundation
Lumina Foundation
National Partners Funders

7. • New laws (like WIOA) clearly emphasize the
importance of evidence based policy.
• Why? Evidence based policy = better policy!
• Evidence is also useful to help secure funding!
The Importance of Evidence Based Policymaking

8. • Our 13-point state blueprint outlines the elements of a high
quality data infrastructure
• They require a data system to be:
⎻ Well Governed
⎻ Sufficiently Funded
⎻ Inclusive
⎻ Used for Accessible Analysis
WDQC State Blueprint

9. State Progress
• Officials from 47 states and the District of
Columbia submitted responses.
• Blueprint survey results reveal net improvement
on almost all elements.

10. • Cross Agency Councils: Councils including members from K-12,
Labor, Higher Education and CTE
• 43 states
• Labor Market Information: Improving labor market information data
collection, analysis, and distribution
• 43 states
• Know if Graduates Get Jobs: Linking employment and earnings data
to see if graduates of workforce and education programs get jobs
• 39 states
• Cross State Data Sharing: Sharing employment data across states,
typically through participation in WRIS-2
• 36 states
Elements Where States Achieved the Most Success

11. • Success Securing Sustainable State Funding
• Oregon
• Alaska
• Kansas
• New Data Sharing Agreements
• New Jersey
• States Newly Recommending the Need for Better Data
• California
• New York
Promising State Practices

12. Contact
Jenna Leventoff
Policy Analyst
JennaL@workforcedqc.org
202-223-8355, ext. 114
WorkforceDQC.org

13. Greg Weeks
Forecasting Division
Office of Financial Management
National Association of Workforce Boards Forum
Washington, DC
March 2016
Evaluation & Your Education Data Warehouse:
What’s a Workforce Board to Think?
&
EDUCATION RESEARCH
DATA CENTER

14. Why listen to me?
• Long career in evaluation research and workforce economic analysis
• 12 years as Washington state LMI Director
• Private sector consulting experience
• Currently research economist at Washington Education and Data
Research Center (ERDC)
• Currently conducting a series of studies on the returns to education for the
ERDC.
• And… I’m entertaining!

15. Outline
1. Overview and brief discussion of non-experimental evaluation research.
2. Description of Propensity Score Matching (PSM)
3. Impacts of PSM
4. STEM Results (with and without PSM)
5. Relevance for a Workforce Board?
6. Discussion/Next steps.
But first, a word from our sponsor…

16. Washington State
Education Research & Data Center
16
• ERDC created in 2007 to:
• Act as objective broker for education and workforce data
• Assemble, link and analyze education and workforce data
• Provide research focusing on student transitions
• Make data available to the education agencies and institutions
• Located in Governor’s budget agency (Office of Financial
Management)
• Work closely with State Education Agency (OSPI)
• Working on second SLDS and WDQI grants
• Focus on research and reporting projects
• Broadening subject areas to human services, corrections and data
visualization
• Continue to operate the ERDC data warehouse

17. Rigorous evaluation studies matter for programs that enrich
human capital
• Often required for US Department of Labor grant funded
programs
• Help define evidence-based approaches that work
• Best practice
• Efficiency
• Help target audiences
• Often analytically challenging

18. The problem
With random assignment (such as clinical trials and experimental evaluation
designs):
1. The outcome of the treatment is conditionally independent from the treatment.
• Chosen at random, the treatment and control groups are statistically identical
• The only difference is one group has the treatment, the other does not.
2. In observational non-experimental studies this assumption is invalid, resulting in
“selection bias.”
• The treatment group may have better outcome measures even in the absence of a
treatment.
• Measured outcomes reflect both the differences in the groups and the differences
attributable to the treatment.

19. Selection bias
• Selection bias occurs when observable or unobservable factors influence
both the decision to participate in the treatment and the outcomes.
• For example, our Bachelor’s degree study assumes that college graduates
differ from high school graduates in ways that affect both the likelihood of
attending and completing college, and post-graduation earnings.
• Simple (unadjusted) comparisons of earnings by educational attainment
lead to selection-biased (over-stated) estimates of the earnings premium
associated with a college degree.

20. Propensity score matching (PSM)
• Propensity score matching is utilized to develop a closely matched
comparison group and correct selection bias.
• A propensity score is the estimated probability that an individual from the
treatment or comparison group will participate in the treatment.
• This single measure indexes all the variables in the characteristics vector
and provides a selection corrected comparison of the outcomes between
the two groups.
• Estimated propensity scores allow individual treatment group members to
be matched with and compared to individual comparison group members.

21. PSM- the counterfactual
• “PSM uses information from a pool of units that do not participate in the
intervention to identify what would have happened to participating units in
the absence of the intervention”
• Heinrich, C., Maffioli, A. and Vazquez, G. “A Primer for Applying Propensity
Score Matching”. Office of Strategic Planning and Development
Effectiveness. Inter-American development Bank. 2010. Retrieved from:
http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for
%20Applying%20Propensity-Score%20Matching.pdf?sequence=1

22. Requirements for PSM
• Comparison group roughly equivalent in size to treatment group.
• Applicants for the training or educational program not accepted into the program.
• SLDS educational data warehouse – may be able to provide an anonymized
comparison group from same high school classes, or by gender or age.
• Clearly defined treatment(s) – start date, end date, time for follow up in UI
wage record (often a six month lag).
• Pre-treatment descriptive data – the SLDS educational data warehouse
may be able to help with this.
• Clearly defined outcomes/effects – often
UI wage data.

23. Basic PSM Process
Once data is assembled for both treatment and comparison groups:
1. Use logistic regression using pre-treatment variables to predict the probability
(propensity score) of participating in the treatment (using both groups together)
2. Match comparison group members to treatment group members based on this
propensity score.
There are several matching approaches including with or without
replacement, nearest neighbor, weighted, …)
3. The difference in outcome measures of the treatment group and the matched
comparison group is the measure of program net impact or effect.
There is a substantial literature on PSM. I would recommend starting with:
http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for%20A
pplying%20Propensity-Score%20Matching.pdf?sequence=1

24. Some examples from our ERDC research
• Returns to a Bachelor’s degree by gender:
(http://www.erdc.wa.gov/sites/default/files/publications/20
1403_0.pdf)
• Returns to STEM degrees by gender and race
categories:
(http://www.erdc.wa.gov/sites/default/files/publications/Ea
rningsPremiums-STEMBachelorDegrees.pdf )
• Returns to an associate degree by gender:
(http://www.erdc.wa.gov/sites/default/files/publications/20
1501.pdf)

25. Female and male earnings trajectory, bachelor’s degree and
high school only, PSM, 2012 dollars, follow up years 1-7.
$0
$5,000
$10,000
$15,000
$20,000
$25,000
$30,000
$35,000
$40,000
$45,000
1 2 3 4 5 6 7
Dollars
Follow up years since HS graduation
Female Bachelor's degree Female HS only Male Bachelor's degree Male HS only

26. Female bachelor’s degree earnings differentials, with and
without PSM; current dollars
-$10,000
-$8,000
-$6,000
-$4,000
-$2,000
$0
$2,000
$4,000
$6,000
$8,000
$10,000
1 2 3 4 5 6 7
Female_PSM
Female_no_PSM

27. Male bachelor’s degree earnings differentials, with and
without PSM; current dollars
-$15,000
-$10,000
-$5,000
$0
$5,000
$10,000
1 2 3 4 5 6 7
MalePSM Male_noPSM

28. Female and male STEM earnings premium in current dollars for years
before and after graduation (year 0), no PSM adjustment
-$5,000
$0
$5,000
$10,000
$15,000
$20,000
$25,000
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
Male Female

29. Female and male STEM earnings premium, 2013 dollars, years after HS
graduation (year 0), with PSM adjustment
-$5,000
$0
$5,000
$10,000
$15,000
$20,000
1 2 3 4 5 6 7 8
Male STEM earnings premium Female STEM earnings premium

30. Proportion of STEM graduates in occupations by
gender

31. Median overall wage rates for top occupations of STEM
graduates by gender

32. Relevance for a workforce board?
• These studies are methodologically comparable to workforce evaluation
studies.
• We are assessing a program that increases human capital and looking at
the net effects (impacts) on subsequent earnings.
• This is very similar to an evaluation of a training or job search assistance
program.

33. Takeaways
• Rigorous evaluations of job skills training and job search
assistance programs are more often possible than
sometimes assumed.
• Selection bias should be taken into account whenever
possible (PSM)
• A rigorous PSM study implies rigorous data requirements
(treatment and comparison groups)
• Use your SLDS education data warehouse as a source of
data and as a partner.

34. Thank you for your time and attention.
Questions?
Greg Weeks, Ph.D.
Greg.weeks@ofm.wa.gov
(360) 902-0660

35. WHY SHARING DATA
WITH THE PUBLIC IS AWESOME
& HOW TO DO IT

37. World-Changers
2.5Q (Bytes/Day)
Data used to be
scarce.
No more.
Abundant Data
The Cloud. Software as
service. Mobile.
Enterprise grade
systems in the palm of
your hand, for you and
Sergey Brin.
Accessible Tools
Why #CivicTech?
(#OpenGov #OpenData #Gov20 #DigitalGov)
Millennials. Encore
careerists. People who
want to be more than
“customers.” They want
to be citizens.

38.  The activities of a company associated with buying and selling a product.
 It includes advertising, selling and delivering products to people.
 People who work in marketing departments try to get the attention of target.
This is Way Hard*
(*Hat-tip to @MattBailey)
Intelligence
out
Data
in

44. Who
→People who are interested in data and
technology
→Hobbyists and professionals
→Students and retirees (and in-between)
→Public and private sector champions,
sponsors, partners
→Community-based cheerleaders
What
→Community-powered nonprofit
organization building civic data
projects that help all kinds of
people do more public good
→Runs Hack University
→Located in Portland, OR
→Works statewide

45. About this word “hack…”
→There’s no deep, dark, or
otherwise dangerous web
involved.
→“It’s good, heartwarming and
nifty.”
→It’s about citizens collaborating
with government to extend
capacity, solve problems, and
improve our communities for
everyone.
→It’s about learning how to adapt
to the new world of abundant
data together.

46. Here’s a hack…

47. Here are another 12 . . .

52. Cause&(Raise)Effect?

54. 40/40/20?

57. 12Filters

58. 12Filters

62. 10 Lessons:
1. The smartest person in the room is the room. (Hat-tip David Wenberger)
2. Regular people can work miracles when they care about what they are working on.
3. Data literacy is a big deal.
4. Experimentation/prototyping is also a big deal.
5. Pictures are better than words.
6. Sharing data begets questions, but they are much better questions.
7. There’s no “done.”
8. Data is not evidence, but you need it to build evidence, which helps
you work smarter.
9. Evidence-based policy making requires a cultural shift.
10. See #1.

63. Government is better when we all work together.
Data Genius ( Kristin gave him this
title)
Greg.Weeks@ofm.wa.gov
360.902.0660
Greg Weeks, Ph.D.
Policy Analyst
JennaL@workforcedqc.org
202.223.8355, ext. 114
Jenna Leventoff
Director, Technical Assistance &
Training
Vinz_Koller@spra.com
831.277.4726
Vinz Koller
Adjunct, Senior Analyst
kwolff@thinkers-and-doers.com
503.888.1022
Kristin Wolff