Open Platforms & Data Smarts: How We Can Do Good Better
•Download as PPTX, PDF•
1 like•537 views
This document summarizes a presentation given at the NAWB Forum in Washington DC on March 13, 2016. The presentation focused on how states can use quality data to do good better by developing inclusive, aligned, and market-relevant education and workforce data systems. It discusses the importance of evidence-based policymaking and describes the Workforce Data Quality Campaign's 13-point state blueprint for a high-quality data infrastructure. The presentation outlines some elements that states have achieved success in, like cross-agency councils and improving labor market information, as well as promising state practices from places like Oregon, Alaska, and Kansas. It concludes by emphasizing the importance of data sharing agreements and continued progress in developing better data systems.
Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Viewers also liked
Viewers also liked (15)
Similar to Open Platforms & Data Smarts: How We Can Do Good Better
Similar to Open Platforms & Data Smarts: How We Can Do Good Better (20)
More from Kristin Wolff
More from Kristin Wolff (20)
Recently uploaded
Recently uploaded (20)
Open Platforms & Data Smarts: How We Can Do Good Better
- 1. Open Platforms & Data Smarts: How We Can Do Good Better NAWB Forum, Washington DC March 13, 1016
- 2. Our Practice We help public servants, philanthropists, and world-changers find what works in solving their communities' toughest challenges. Our approach to research, evaluation and hands-on-technical assistance is rooted in a commitment to equity and focus on learning – so we all grow smarter by working together. (And we love data, obviously…)
- 3. WHY DATA? WHY NOW?
- 4. States Doing Good Better with Quality Data Jenna Leventoff, Policy Analyst March 13, 2016
- 5. • Advocate for inclusive, aligned and market-relevant education and workforce data that can help our nation’s human capital policies meet the challenges of a changing economy. • Promote federal and state reforms for data systems that provide useful information for policymakers, students and workers, business leaders and educators. • State Blueprint with 13 key features of a high-quality data infrastructure • Address federal legislation, funding and technical assistance • Policy agenda developed by broad coalition of national organizations, state leaders and technical experts across education/workforce spectrum WDQC Mission
- 6. Apollo Education Group Bill & Melinda Gates Foundation Joyce Foundation Laura & John Arnold Foundation Lumina Foundation National Partners Funders
- 7. • New laws (like WIOA) clearly emphasize the importance of evidence based policy. • Why? Evidence based policy = better policy! • Evidence is also useful to help secure funding! The Importance of Evidence Based Policymaking
- 8. • Our 13-point state blueprint outlines the elements of a high quality data infrastructure • They require a data system to be: ⎻ Well Governed ⎻ Sufficiently Funded ⎻ Inclusive ⎻ Used for Accessible Analysis WDQC State Blueprint
- 9. State Progress • Officials from 47 states and the District of Columbia submitted responses. • Blueprint survey results reveal net improvement on almost all elements.
- 10. • Cross Agency Councils: Councils including members from K-12, Labor, Higher Education and CTE • 43 states • Labor Market Information: Improving labor market information data collection, analysis, and distribution • 43 states • Know if Graduates Get Jobs: Linking employment and earnings data to see if graduates of workforce and education programs get jobs • 39 states • Cross State Data Sharing: Sharing employment data across states, typically through participation in WRIS-2 • 36 states Elements Where States Achieved the Most Success
- 11. • Success Securing Sustainable State Funding • Oregon • Alaska • Kansas • New Data Sharing Agreements • New Jersey • States Newly Recommending the Need for Better Data • California • New York Promising State Practices
- 12. Contact Jenna Leventoff Policy Analyst JennaL@workforcedqc.org 202-223-8355, ext. 114 WorkforceDQC.org
- 13. Greg Weeks Forecasting Division Office of Financial Management National Association of Workforce Boards Forum Washington, DC March 2016 Evaluation & Your Education Data Warehouse: What’s a Workforce Board to Think? & EDUCATION RESEARCH DATA CENTER
- 14. Why listen to me? • Long career in evaluation research and workforce economic analysis • 12 years as Washington state LMI Director • Private sector consulting experience • Currently research economist at Washington Education and Data Research Center (ERDC) • Currently conducting a series of studies on the returns to education for the ERDC. • And… I’m entertaining!
- 15. Outline 1. Overview and brief discussion of non-experimental evaluation research. 2. Description of Propensity Score Matching (PSM) 3. Impacts of PSM 4. STEM Results (with and without PSM) 5. Relevance for a Workforce Board? 6. Discussion/Next steps. But first, a word from our sponsor…
- 16. Washington State Education Research & Data Center 16 • ERDC created in 2007 to: • Act as objective broker for education and workforce data • Assemble, link and analyze education and workforce data • Provide research focusing on student transitions • Make data available to the education agencies and institutions • Located in Governor’s budget agency (Office of Financial Management) • Work closely with State Education Agency (OSPI) • Working on second SLDS and WDQI grants • Focus on research and reporting projects • Broadening subject areas to human services, corrections and data visualization • Continue to operate the ERDC data warehouse
- 17. Rigorous evaluation studies matter for programs that enrich human capital • Often required for US Department of Labor grant funded programs • Help define evidence-based approaches that work • Best practice • Efficiency • Help target audiences • Often analytically challenging
- 18. The problem With random assignment (such as clinical trials and experimental evaluation designs): 1. The outcome of the treatment is conditionally independent from the treatment. • Chosen at random, the treatment and control groups are statistically identical • The only difference is one group has the treatment, the other does not. 2. In observational non-experimental studies this assumption is invalid, resulting in “selection bias.” • The treatment group may have better outcome measures even in the absence of a treatment. • Measured outcomes reflect both the differences in the groups and the differences attributable to the treatment.
- 19. Selection bias • Selection bias occurs when observable or unobservable factors influence both the decision to participate in the treatment and the outcomes. • For example, our Bachelor’s degree study assumes that college graduates differ from high school graduates in ways that affect both the likelihood of attending and completing college, and post-graduation earnings. • Simple (unadjusted) comparisons of earnings by educational attainment lead to selection-biased (over-stated) estimates of the earnings premium associated with a college degree.
- 20. Propensity score matching (PSM) • Propensity score matching is utilized to develop a closely matched comparison group and correct selection bias. • A propensity score is the estimated probability that an individual from the treatment or comparison group will participate in the treatment. • This single measure indexes all the variables in the characteristics vector and provides a selection corrected comparison of the outcomes between the two groups. • Estimated propensity scores allow individual treatment group members to be matched with and compared to individual comparison group members.
- 21. PSM- the counterfactual • “PSM uses information from a pool of units that do not participate in the intervention to identify what would have happened to participating units in the absence of the intervention” • Heinrich, C., Maffioli, A. and Vazquez, G. “A Primer for Applying Propensity Score Matching”. Office of Strategic Planning and Development Effectiveness. Inter-American development Bank. 2010. Retrieved from: http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for %20Applying%20Propensity-Score%20Matching.pdf?sequence=1
- 22. Requirements for PSM • Comparison group roughly equivalent in size to treatment group. • Applicants for the training or educational program not accepted into the program. • SLDS educational data warehouse – may be able to provide an anonymized comparison group from same high school classes, or by gender or age. • Clearly defined treatment(s) – start date, end date, time for follow up in UI wage record (often a six month lag). • Pre-treatment descriptive data – the SLDS educational data warehouse may be able to help with this. • Clearly defined outcomes/effects – often UI wage data.
- 23. Basic PSM Process Once data is assembled for both treatment and comparison groups: 1. Use logistic regression using pre-treatment variables to predict the probability (propensity score) of participating in the treatment (using both groups together) 2. Match comparison group members to treatment group members based on this propensity score. There are several matching approaches including with or without replacement, nearest neighbor, weighted, …) 3. The difference in outcome measures of the treatment group and the matched comparison group is the measure of program net impact or effect. There is a substantial literature on PSM. I would recommend starting with: http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for%20A pplying%20Propensity-Score%20Matching.pdf?sequence=1
- 24. Some examples from our ERDC research • Returns to a Bachelor’s degree by gender: (http://www.erdc.wa.gov/sites/default/files/publications/20 1403_0.pdf) • Returns to STEM degrees by gender and race categories: (http://www.erdc.wa.gov/sites/default/files/publications/Ea rningsPremiums-STEMBachelorDegrees.pdf ) • Returns to an associate degree by gender: (http://www.erdc.wa.gov/sites/default/files/publications/20 1501.pdf)
- 25. Female and male earnings trajectory, bachelor’s degree and high school only, PSM, 2012 dollars, follow up years 1-7. $0 $5,000 $10,000 $15,000 $20,000 $25,000 $30,000 $35,000 $40,000 $45,000 1 2 3 4 5 6 7 Dollars Follow up years since HS graduation Female Bachelor's degree Female HS only Male Bachelor's degree Male HS only
- 26. Female bachelor’s degree earnings differentials, with and without PSM; current dollars -$10,000 -$8,000 -$6,000 -$4,000 -$2,000 $0 $2,000 $4,000 $6,000 $8,000 $10,000 1 2 3 4 5 6 7 Female_PSM Female_no_PSM
- 27. Male bachelor’s degree earnings differentials, with and without PSM; current dollars -$15,000 -$10,000 -$5,000 $0 $5,000 $10,000 1 2 3 4 5 6 7 MalePSM Male_noPSM
- 28. Female and male STEM earnings premium in current dollars for years before and after graduation (year 0), no PSM adjustment -$5,000 $0 $5,000 $10,000 $15,000 $20,000 $25,000 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 Male Female
- 29. Female and male STEM earnings premium, 2013 dollars, years after HS graduation (year 0), with PSM adjustment -$5,000 $0 $5,000 $10,000 $15,000 $20,000 1 2 3 4 5 6 7 8 Male STEM earnings premium Female STEM earnings premium
- 30. Proportion of STEM graduates in occupations by gender
- 31. Median overall wage rates for top occupations of STEM graduates by gender
- 32. Relevance for a workforce board? • These studies are methodologically comparable to workforce evaluation studies. • We are assessing a program that increases human capital and looking at the net effects (impacts) on subsequent earnings. • This is very similar to an evaluation of a training or job search assistance program.
- 33. Takeaways • Rigorous evaluations of job skills training and job search assistance programs are more often possible than sometimes assumed. • Selection bias should be taken into account whenever possible (PSM) • A rigorous PSM study implies rigorous data requirements (treatment and comparison groups) • Use your SLDS education data warehouse as a source of data and as a partner.
- 34. Thank you for your time and attention. Questions? Greg Weeks, Ph.D. Greg.weeks@ofm.wa.gov (360) 902-0660
- 35. WHY SHARING DATA WITH THE PUBLIC IS AWESOME & HOW TO DO IT
- 37. World-Changers 2.5Q (Bytes/Day) Data used to be scarce. No more. Abundant Data The Cloud. Software as service. Mobile. Enterprise grade systems in the palm of your hand, for you and Sergey Brin. Accessible Tools Why #CivicTech? (#OpenGov #OpenData #Gov20 #DigitalGov) Millennials. Encore careerists. People who want to be more than “customers.” They want to be citizens.
- 38. The activities of a company associated with buying and selling a product. It includes advertising, selling and delivering products to people. People who work in marketing departments try to get the attention of target. This is Way Hard* (*Hat-tip to @MattBailey) Intelligence out Data in
- 44. Who →People who are interested in data and technology →Hobbyists and professionals →Students and retirees (and in-between) →Public and private sector champions, sponsors, partners →Community-based cheerleaders What →Community-powered nonprofit organization building civic data projects that help all kinds of people do more public good →Runs Hack University →Located in Portland, OR →Works statewide
- 45. About this word “hack…” →There’s no deep, dark, or otherwise dangerous web involved. →“It’s good, heartwarming and nifty.” →It’s about citizens collaborating with government to extend capacity, solve problems, and improve our communities for everyone. →It’s about learning how to adapt to the new world of abundant data together.
- 46. Here’s a hack…
- 47. Here are another 12 . . .
- 54. 40/40/20?
- 57. 12Filters
- 58. 12Filters
- 62. 10 Lessons: 1. The smartest person in the room is the room. (Hat-tip David Wenberger) 2. Regular people can work miracles when they care about what they are working on. 3. Data literacy is a big deal. 4. Experimentation/prototyping is also a big deal. 5. Pictures are better than words. 6. Sharing data begets questions, but they are much better questions. 7. There’s no “done.” 8. Data is not evidence, but you need it to build evidence, which helps you work smarter. 9. Evidence-based policy making requires a cultural shift. 10. See #1.
- 63. Government is better when we all work together. Data Genius ( Kristin gave him this title) Greg.Weeks@ofm.wa.gov 360.902.0660 Greg Weeks, Ph.D. Policy Analyst JennaL@workforcedqc.org 202.223.8355, ext. 114 Jenna Leventoff Director, Technical Assistance & Training Vinz_Koller@spra.com 831.277.4726 Vinz Koller Adjunct, Senior Analyst kwolff@thinkers-and-doers.com 503.888.1022 Kristin Wolff
Editor's Notes
- [Vinz] Open Platforms and Data Smarts: How We Can Do Good Better Sunday, March 13, 11:00am Moderator: Vinz Koller, Director, Technical Assistance and Training, Social Policy Research Associates (SPR), Oakland, CA Presenters: Jenna Leventoff, Policy Analyst, Workforce Data Quality Campaign/National Skills Coalition, Washington, DC Greg Weeks, Research Economist, Economic Research & Data Center, Olympia, WA Kristin Wolff, Adjunct Associate, Social Policy Research Associates (SPR), Portland, OR Workforce and education agencies are awash in data. Harnessing them to provoke change remains a challenge. Fortunately, experiments involving uncommon partners show promise. This session will make evidence-based policy real. We’ll provide an overview of major workforce data trends and tools (e.g. dashboards), and review two examples of data projects leading to better policy: Washington State’s research on the "college premium” (with particular emphasis on earning by gender and race to individuals with STEM degrees) and Hack Oregon’s Education Pathways Project. We’re learning to use data to inform a better, more equitable future for all. Bios: Jenna Leventoff is a Policy Analyst of Workforce Data Quality Campaign, leading WDQC’s efforts to track and analyze legislation and regulation related to data privacy, authoring issue papers on best practices for data use, and serving as a primary point of contact for state advisors. Before joining WDQC, Jenna was an Associate at Upturn, where she analyzed the civil rights implications of new technologies and served as Manager and Legal Counsel of the International Intellectual Property Institute, leading the organization’s efforts to utilize intellectual property for international economic development. Jenna has also held internships with the American Civil Liberties Union Washington Legislative Office and Senator Sherrod Brown. Jenna holds a J.D. and a bachelor's degree in political science and English from Case Western Reserve University. She is based in Washington, DC. jennal@workforcedqc.org Vinz Koller is the Director of Training and Technical Assistance at Social Policy Research Associates. Mr. Koller’s expertise lies in using engaging and innovative training and collaborative design methods to advance the work of workforce agencies and collaboratives as well as their funders. He has worked on data, youth programs, and sustainability throughout his career and lent his expertise to communities, state and federal agencies as well as tribal governments. vinz_koller@spra.com Greg Weeks is an economist with experience in evaluation research, data systems development and performance measurement and a senior researcher in the Economic Research and Data Center (Office of Financial Management) in Washington State. He earned his Ph.D. in labor economic from Washington State University and has taught economics at several universities, including most recently, The Evergreen State College. He has 15 years of experience in state government, focusing on employment statistics, research and performance measurement and has received numerous accolades including the Vladimir Chavrid Award from the National Association of State Workforce Agencies and the Governor’s distinguished manager’s award. Greg lives in Olympia WA with his wife Peggy and three dogs and regularly dotes on his first grandson, now a year old. greg.weeks@ofm.wa.gov Kristin Wolff has worked in the jobs (un-jobs?) space since reading Dan Pink’s Free Agent Nation over a decade ago. A longtime member of the “gig” economy herself, Kristin serves as an adjunct researcher for Social Policy Research Associates (Oakland, CA), runs thinkers-and-doers, and experiments with peer platforms of many descriptions. She has served as a technical assistance provider to the Workforce Data Quality Initiative for three years and as a member of a Hack Oregon Education Pathways team in 2015. She is also a (working) board member at Hatch Innovation, a co-working and social innovation space in Portland, OR. kwolff@thinkers-and-doers.com
- [Vinz] Introduce SPR.
- [Vinz] Why this panel (because how we manage data, what we can do with it, and who the ”we” is are all changing. Provide WDQI context. - Shift from insider to public. - Performance measurement to intelligence. - Programs to system and beyond. By connecting data we’re learning much more about what works, but it’s a very big job. Our systems are only just learning how to cope, and the people who could use our data are only just learning that it’s there – we have to find ways to make bot more accessible and more useful. - Jenna will talk about trends nationwide, focusing on education and wf data. Here org is the center of gravity nationally on emerging approaches to data across states. - Greg comes from WA state, long time commitment to longitudinal data and building the capacity of the ecosystem to use it. His team has linked education and employment data, and focusing on STEM fields, can tell us a great deal about how college pays off (or does not) across different demographic groups in WA state, revealing the need for much more nuanced policy and program prescriptions than we’ve been used to working with. - Kristin works with me (and Greg) on the WDQI project and has been a champion of efforts to make data more available to the public. She will share her experience with Hack Oregon – one of many new organizations in the civictech space finding ways to both expand the capacity of government and accelerate public engagement in policy making, while experimenting with new ways to put the public back into public data.
- [Jenna]
- WDQC’s work is made possible and strengthened by the support of foundations and national partners. We also benefit from a long list of advisors based around the Washington area and out in the states.
- Evidence Based Policy Example: INSERT Funding Story: A city agency contracted with an immigrant serving non-profit to find limited English proficient persons jobs. Initially, they offered funding for six weeks. However, that agency had data showing that in the past, it took LEP’s about 12 weeks to secure good positions. So, the government agency provided funding for 10 weeks instead of six. After contracting with the non-profit, their 0% placement rate improved to a 57% placement rate.
- HAND OUT BLUEPRINT Well governed – cross agency council with input from all stakeholders Inclusive- contains a variety of information across the education and workforce spectrum, including credentials and cross-state employment data. Used for Accessible analysis – there is no point in collecting data if no one is using the data. State data systems should enable labor market research and analysis, as well as program and instution research and analysis. Information should be made easily accessible to the public, so that students, workers, program managers, and policymakers can make informed decisions.
- HAND OUT BLUEPRINT Well governed – cross agency council with input from all stakeholders Inclusive- contains a variety of information across the education and workforce spectrum, including credentials and cross-state employment data. Used for Accessible analysis – there is no point in collecting data if no one is using the data. State data systems should enable labor market research and analysis, as well as program and instution research and analysis. Information should be made easily accessible to the public, so that students, workers, program managers, and policymakers can make informed decisions.
- HAND OUT BLUEPRINT Well governed – cross agency council with input from all stakeholders Inclusive- contains a variety of information across the education and workforce spectrum, including credentials and cross-state employment data. Used for Accessible analysis – there is no point in collecting data if no one is using the data. State data systems should enable labor market research and analysis, as well as program and instution research and analysis. Information should be made easily accessible to the public, so that students, workers, program managers, and policymakers can make informed decisions.
- HAND OUT BLUEPRINT Well governed – cross agency council with input from all stakeholders Inclusive- contains a variety of information across the education and workforce spectrum, including credentials and cross-state employment data. Used for Accessible analysis – there is no point in collecting data if no one is using the data. State data systems should enable labor market research and analysis, as well as program and instution research and analysis. Information should be made easily accessible to the public, so that students, workers, program managers, and policymakers can make informed decisions.
- College Measures? More on admin support in way of budget?
- [Kristin]
- Founded 2009, first Fellows in 2012. Worked with cities to solve problems using open data and apps. 2015 – Focus areas: Health/Human Services Economic Development Safety & Justice Communications/Engagement Open Data: Open license – anyone can use Free of charge Open format (machine readable)
- This sector is now generally called #civictech. Here are the drivers.
- All of the things I just talked about – abundant data, better systems, curious people – are creating a demand for better systems. Not just spreadhseets, databases and performance reporting processes, but tools for generating intelligence about what works, for whom, and why. That’s the ‘way hard’ part. And it’s why there’s now a whole ecosystem around #civictech.
- One of the first efforts to document the field.
- A look at clusters.
- A look at how investments cluster.
- Growth chart.
- More recent - orgs.
- Hack Oregon is part of that eco-system. Started out a lot like a statewide Code for America brigade. Now a really interesting civictech nonprofit startup and more recently, a university – it’s also a provider inside the most recent TechHire submission by worksystems which is the Portland Workforce Board. Here are the basics: Runs in cycles 2/year Focuses on data about issues with demonstrated public interest Makes data known/accessible, does not advocate for solutions/policy Developing common tools/templates/approach to similar problems in different domains
- About the word hack… Do not be afraid!
- Here’s a hack form a couple of weeks ago in DC.
- Here are 12 more that came out of the WH Opportunity Project.
- And here’s two from Hack Oregon. Raise Effect – Completed. One Cycle. What it is, how it changes.
- Can search by county.
- Some of you may have seen this. We know that citizens (not lobbyists or legislators) sent and showed this to the legislature. So…we can’t demonstrate cause and effect, but it does seem likely that showing the effect of raises in differents contexts for different types of families has some impact on the decision to adopt a tiered approach – which was the only way it would pass in Oregon.
- Here’s another one that actually uses some of the data that’s inside SDLS and WDQI systems.
- Explain OBC & 40/40/20. Original sankey.
- 40/40/100 (We’re now at 20/5/50).
- Show the change.
- Create other ways to search and compare.
- Pulled out salient data points not well expressed other ways.
- And contextualized it through story.
- Here’s the team – all volunteer. Many in full-time jobs and volunteering on other projects. Story-telling team - diversity.