1. The U.S. Census Bureau faces challenges from the rise of big data sources produced outside of traditional government surveys. These new sources are generated faster and more cheaply than surveys. 2. To remain reliable sources of demographic and economic information, the Census Bureau must integrate these new big data sources with traditional surveys. This requires linking massive datasets and developing new statistical modeling techniques. 3. The Census Bureau is exploring ways to use new big data sources like web search data, social media, and e-commerce transactions to improve surveys and provide more timely, detailed information. However, maintaining privacy and developing new technology is difficult.

1. Big Data activities
at the U.S. Census Bureau
Cavan Capps
Big Data Lead
U.S. Census Bureau
February 13, 2014
Prepared for
MIT Libraries Program on Information Science Brown Bag Talk
Feb 2014

2. Big Data Challenge at the Census Bureau
“Designed Data” vs. “Organic Data”
“The world is now producing large amounts of data.. data from Internet
searches, credit card transactions, retail scanners, and social media”.
“ There also are more and more digital administrative data (e.g., tax
records, social security records, Medicare/Medicaid records, food stamp
records, HUD records). Some of these data are not directly linked to the
populations we study; some have item missing data problems; none
offer a real replacement for our surveys, but many will be useful as
auxiliary data sources.”
1

3. Big Data Challenge at the Census Bureau
Big Data is about creating information to make Big
Decisions from novel, and often massive data sources.
2

4. Big Data creates new Statistical Agency Challenges
A recent meeting of International Statistical Agencies observed:
1. The volume of data generated outside the government statistical
systems is increasing much faster than the volume of data collected
by the statistical systems; almost all of these data are digitized in
electronic files.
2. As this occurs, the leaders expect that relative cost, timeliness, and
effectiveness of traditional survey and census approaches of the
agencies may become less attractive.
3

5. Big Data creates new Statistical Agency Challenges
A recent meeting of International Statistical Agencies observed:
3.
Blending together multiple available data sources
(administrative, commercial electronic transactions and internet webpage data, search frequency data, twitter, facebook etc. ) with
traditional surveys and censuses (using paper, telephone, face-toface interviewing) to create high quality, timely statistics that tell a
coherent story of economic, social and environmental progress must
become a major focus of central government statistical agencies.
4. This requires efficient record linkage capabilities, the building of
master universe frames that act as core infrastructure to the blending
of data sources, and the use of modern statistical modeling to
combine data sources with highest accuracy.
4

6. Big Data creates new Statistical Agency Challenges
A recent meeting of International Statistical Agencies observed:
5. The Agencies will need to develop the analytical capabilities to
distill insights from more integrated views of the world and impart
a stronger systems view across different government and private
sector information systems to provide more geographical and
industry detail.
6. There are growing demands from researchers and policy-related
organizations to analyze the micro-data collected by the agencies, to
extract more timely and detailed information from the data.
5

7. Big Data Development Challenges for Statistical Agencies
The Meeting Recommended that Statistical Agencies develop:
1.
2.
3.
4.
High-speed, “big data” software/hardware systems for record
linkage and extraction of key information from massive files.
Efficient and sophisticated imputations procedures needed to make
the combined data sources jointly useful.
More use of statistical modeling for statistical estimation, to provide
more:
1. Timely estimates
2. Small area estimates
3. New measures
New ways to give secure access to micro-data for legitimate policy
and research purposes, to increase their impact of their work.
6

8. In Summary, massive challenges for the Statistical Agencies:
1.
The Internet and Private E-Transactions are generating data faster
and more cheaply than Statistical agencies can afford to do.
2.
To be reliable sources of information on the Demographics, Economy
and Social change in the U.S., this information needs to be mashed
together with traditional surveys and adjusted for bias.
3.
The sizes of the files and the number of computations to mash up the
data will be larger.
4.
Spoiled by the Internet, users expect more timely, and detailed data
provided at lower costs.
5.
Privacy/Confidentiality must be maintained.
7

9. Big Data Projects at the Census Bureau
The Census Bureau “Big Data”
Information Life Cycle
Data Collection
- Multi-Mode Data Survey Collection model
- New Data sources (Web, E-Transactions, Admin Recs)
Data Integration & Analysis
- Record Linkage
- Small Area Estimation modeling & “Now Casting”
Data Release
- Data Review for Release
- Confidentialize data for public release
8

10. Big Data Current Process
Future Process (exploring)
• Designed Data
• Designed & Organic Data
• Proprietary Software
• Next Generation Open-Source
& Proprietary Software
• Batch Processing
• More Parallel Processing
• Long processing times
• Faster processing times
9

11. Big Data Collection: Improving Survey Logistics & Cost
Improving Survey Collection and Imputation Operations(Adaptive
Design)
1.
Multi-modal data collection to reduce operational costs of
data collection
– More effective use of existing data such as
administrative records
– Incorporating new data into decennial operations
• Paradata from Internet Data Capture
• Information from Social Media Feeds
2.
3.
Edits and Imputations
Data Review
10

12. Big Data Collection: Evaluating Web Data as Inputs
Potential Internet Data Collection
1.
2.
Examine Google & Bing search frequency trend data
3.
Examine Twitter, and other social media trend data
Examine “Web Scraping” of housing data, price data, local
tax data, crime data, corporate profits etc.
11

13. Big Data Collection : Evaluating Commercial E-Transaction
Input Data
1.
Housing:
–
–
2.
Foreclosures: Use vendor data on new residential properties in
foreclosure to aid analysis of data on new construction and sales.
Building Permits: Web scrape opportunity to access local jurisdictions and
state agencies posting public records online.
Construction:
–
–
3.
Difficulty obtaining electronic data from numerous state and local agencies
Data are needed immediately to tabulate the monthly economic indicators.
Retail Sales: Evaluating electronic payment processing to fill data gaps such
as geographical detail and revenue measures by firm size
– New data products
– Improvements to current data quality
12

14. Big Data Integration & Analysis: (Current processes)
Data Integration Expertise:
• Record linkage
– Gov’t Admin Records to other Gov’t Admin Records
– Gov’t Admin Records to Gov’t Surveys
– Commercial records to Gov’t Admin Records
• Model based integration
– Small Area Poverty & Income Estimates
– Small Area Health & Income Estimates
– Longitudinal Economic & Housing Dynamics
13

15. Big Data Integration & Analysis: Exploring “Now Casting”
Exploring “Now Casting” to improve Statistical Timeliness :
1.
Some “real time” Internet data correlates with Official Statistics:
– Google search data modeled to match BLS unemployment &
CDC Flu spread
– Univ. of Michigan Twitter unemployment
– MIT Billion Price Project match to BLS CPI
2.
Census experiments with Gov’t Pension data
14

16. Big Data Lab
1.
Setting up an experimental Cluster
2.
Testing performance of Hardware
3.
Testing value of Software
– Open Source Big Data Software:
Hadoop, Mahout, Distributed R, Hbase, Pig, Hive,
Casandra, Mongo, Flume, Neo4J, I-Graph,
Allegrograph
– Internally Developed software:
TEA, DataWeb, Matching software

17. On the Horizon, Development of Big Data Center
Research, capacity building and economic Big Data
Processing:
1.
Proposal to create a new center that will include members from academy and
Census staff to:
1. Help lead work Census Bureau on practices to make sense of Big Data.
Developing principles to apply Big Data to federal statistics.
2. Facilitate CB as unbiased provider for information collected as Big Data
3. Validate new techniques and data sources it at a low cost (field staff
allow us to do ground checks, survey questions)
4. Lead on methods to integrate Big Data and develop standards
5. The Center should provide a way to bring both faculty and graduate
students to Census to facilitate Big Data capacity building at the Census
Bureau
1.
We will explore partnerships with others doing research in this area.
Universities, and Silicon Valley