This document provides an overview of techniques for collecting, cleaning, and manipulating data for investigative reporting purposes. It discusses finding and obtaining data from various sources like government databases and agencies. It also covers integrity checks, data cleaning, and evaluating outside studies. Examples are given of reports that were produced using analyzed data from sources like Medicaid and EPA. The document emphasizes being thorough in understanding data and accounting for limitations and errors.

1. Data: Collect, Clean and Manipulate
ONA 2012
San Francisco
Jennifer LaFleur, ProPublica
jennifer.lafleur@propublica.org
@j_la28

2. Why data?
It takes you beyond the anecdote
It’s easier than counting sheets of paper

4. Why data?
Contrasts are in the data

5. Caution: This slide contains extreme nerdiness

8. Why Computer-Assisted Reporting?
Contrasts are in the data
Your most powerful figures are in the data

9. Source: California
Health Dept.
data, Medicare billing
data
Findings: Some
hospitals had
“alarming rates of a
Third World nutritional
disorder among its
Medicare patients.”

11. Why data?
Contrasts are in the data
Your most powerful figures are in the data
You can make connections you might not be
able to make otherwise

12. Data: Youth
prison workers,
criminal
convictions and
grievance data
Findings:
Employees with
criminal
backgrounds
were more likely
to be accused of
abusing inmates.

13. Data: Federal
bridge
inspections and
stimulus funding.
Findings: Some
of the nation’s
worst bridges did
not get stimulus
funds.

14. Why data?
Contrasts are in the data
Your most powerful figures are in the data
You can make connections you might not be
able to make otherwise
You can test assumptions

15. Source: NHTSA
complaint data
Findings:
“…unintended
acceleration has
been a problem
across the auto
industry.”

16. HT/Florencia
Coelho

17. Collecting the data

18. Where’s the data?
If something is inspected
Licensed
Enforced or
Purchased
…There probably is a database

19. Where’s the data?
If there is a report
Or a form
There probably is a database

21. Where’s the data?
Sometimes data is readily available
online for download

22. Source: Census
Findings: “Fueled by
the dismal economy
and high
unemployment, mor
e Americans…are
doubling up”

23. Source: Medicaid nursing home
survey data and finance
data, housing data
Findings: “…a shortage of places
for the disabled to live outside a
nursing home and regulations
that critics say make it hard to
qualify for home services mean
many who want out continue to
receive expensive nursing care.”

24. Where’s the data?
Sometimes you have to scrape it.
That usually involves programs
that automate searching tasks on
Web sites.

26. Where’s the data?
More often you need to go to an
agency to get the data
This can be tricky if an agency
doesn’t want to release it. (Stay
tuned for more on that…)

27. Source: School district
credit card purchases
Findings: District card
holders made
questionable
purchases with their
cards.

30. Sometimes, there is no data.
But it’s okay because there are
techniques for sampling and building
a database.

31. ProPublica pulled a random
sample of 500 names from a
list of individuals who had
been granted or denied
pardons (around 2,000). We
created a database from
months or researching
individuals: their crime, age,
sentence…
We found that even after
controlling for other factors,
whites were more likely to get
a pardon.

33. Source: Loan details,
foreclosure information and
bankruptcy filings
Findings: Loans leading to
foreclosure didn’t always
follow conventional wisdom

34. When you have to ask for the data
Before filing a request: Ask for it
If they require a formal request, find
out who it should go to and what you
should ask for
Letter should describe what you’re
asking for
 Note that you’re willing to
negotiate
 Ask for a cost estimate

35. Dear Records Administrator:
I’m writing to request under the Texas Public Information Act an electronic copy of the current health-
related services registry database for the state of Texas. I also am requesting electronic copies or a
database of all complaints filed against health-related service registry members since Jan. 1, 2000.
I frequently deal with large raw databases, so I would be able to accept information in several formats
including ASCII, dbf, xls, etc… and can accept the data on a variety of media (computer tape, CD-
ROM, FTP, email attachment, etc...). Please include record layouts, code sheets or any other
documentation necessary to interpret the data.
I am requesting all data fields. If there are any fields that you must withhold by law, please let me
know what those fields are, so I can amend my request.
In the interest of expediency, and to minimize the research and/or duplication burden on your staff, I
would be happy to speak with your database administrator to figure out a method that is easiest for
you.
If you have questions or need more information, please contact me by telephone or email. My
telephone number is: 214-977-8509. My email address is jlafleur@dallasnews.com.
If you will be charging processing fees, please send me an itemized estimate explaining how the
costs were calculated.

36. Getting electronic information
Know the law. Know how your state treats (or
doesn’t) the records you need.
Know what information you want.
Do your homework
Know what the appropriate cost should be.
Know who does the data entry.
Get to know Leon
When something may not clearly be public use
your sourcing

37. Just another way of saying no
Huge costs
Delay tactics
“Oh you silly little journalist”
Sending you the wrong thing
“Your request was unclear”
HIPAA
Privacy
Privatization

38. Negotiating: Some examples

39. Our database is on a
mainframe and it’s
very complicated,
Missy

41. We don’t have
the authority
to do that

43. That will cost
$25,000.

45. We have processed your request. The
labor cost for the request is as
follows.
Item # of hours
RESEARCH 20
CREATING FILES 6
CODING 24
TESTING 4
Total (54 X$72) = $3,888.00

46. From Texas Public Information Act:
111.67. Estimates and Waivers of Public Information Charges
(a) A governmental body is required to provide a requestor
with an itemized statement of estimated charges if charges for
copies of public information will exceed $40, or if a charge in
accordance with §111.65 of this title (relating to Access to
Information Where Copies Are Not Requested) will exceed
$40 for making public information available for inspection. A
governmental body that fails to provide the required
statement may not collect more than $40. The itemized
statement must be provided free of charge and must contain the
following information:

47. We only keep
the
information
for 7 days

48. Check retention schedules

49. That uses
proprietary
software.

51. We don’t keep
that on
computer

52. Okay, we do,
but it’s a lot
of files

53. That
information is
protected by
law

59. Cleaning data

60. Remember that data are not perfect

61. It doesn’t mean you can’t use it…
Do integrity checks to find the flaws
Add caveats where necessary
Do your own analysis rather than relying on an
agency’s analysis of bad data

62. Integrity checks for every data set
Read the documentation. Understand the
contents of every field.
Know how many records you should have.
Check counts and totals against reports.
Are all possibilities included? All states, all
counties, correct ranges?

63. Integrity checks for every data set
Internal data checks:
Is there more money going to sub-contractors than went to
the prime contractor?
Are there more teachers than students?
Do people have birth dates in the future or so long ago they
would be long gone?

66. If your data is in
Excel, use the filter
function to see what
the values are in
individual fields.

67. Integrity checks for every data set
Check for missing data, misplaced data or blank
fields
Use a standard naming convention for files and
tables (I wouldn’t recommend “final”)
Check for duplicates
Take margins of error into account if necessary
(important if you’re using Census data).

68. 2010 Census ACS: Median HH Income by Metro Area

69. Be creative when you look for duplicates

70. Beyond the basics
Keep a notes file
Don’t work off your original database
Know the source
Check against summary reports
Use the right tool
Check for outliers when it comes to ups and
downs

71. Truck accidents by year and agency

72. Beyond the basics
Check with experts
Are there standards? (ex: a drop by more than
10 perc pts is a red flag)
Find out what others have done
Gut check
Go physically see a record or spot check
against documents

73. Voter Fraud
Dozens of St. Louis voters are being wrongly accused
of casting ballots from fraudulent addresses in last
year's Nov. 7 election.
They are among thousands of registered voters who,
based on city property records, appear to live on
vacant lots.

75. Texas test score data official
results versus district
Duncanville district reported
4th grade writing
Official report for Duncanville
4th grade writing
Courtesy Holly Hacker, The Dallas Morning News

76. Three rounds of analysis
after bouncing off subjects
and experts
Demographically based
Voir dire
Socioeconomics

77. Checks when you’re matching data
A name is not enough. Lots of people have the same name
Get dates of birth and
other information to
make sure you have
the correct person.

78. Source: Illinois health data, police data
Findings: Dangerous systemic failed to protect elderly patients in
Illinois nursing homes that also house mentally ill younger residents,
including murderers, sex offenders, and armed robbers.

79. Even people with seemingly unique names aren’t so unique

80. Evaluating outside studies
Get the questionnaire and methodology
Beware of nonscientific methods: Web surveys,
man on the street
Know the sample size..sampling error
Account for margin of error and non-response
when drawing conclusions
Run statistical tests on the data if possible

82. Reporting data
Consider reporting rates not raw numbers
Avoid false precision: 53.14 percent said … in a poll
with a 5 percentage point margin of error
Avoid number overload. About half is usually just as
useful as 51 percent in most cases
Adjust money for inflation
When analyzing income, use median rather than
average (Bill Gates factor)

83. When the data is the problem – you might still
have a story
Erroneous government databases – can often
be a story themselves

86. Manipulating data for stories
and apps

87. Know which tool to use
• Reporting individual records
• Counting/summing
• Mapping
• Statistics

88. Source: Medicaid
outcomes data for
dialysis facilities
Findings: A CMS
online tool did not
tell the whole story
about facilities. In
some counties the
gap in
measures, such as
survival rate were
vast.

90. Source: Washington Health Department data
Findings: “MRSA has been quietly killing in hospitals for decades.” But no
one had tracked it until this story.

91. Source: Dept. of Ed data and surveys of campus crisis clinics
Findings: Many campuses had lax enforcement and reporting loop holes
mean problems go unchecked.

93. Source: EPA and state data on hazardous chemical locations
Findings: Dallas County has 900+ sites that store hazardous chemicals

94. Source: Dam
inspection data
from Texas and
federal government
Findings: Dam
records had not
been updated to
account for
population growth

95. Source: 311 calls for downed trees
Findings: After a tornado swept across New York City, 311
calls for downed trees helps trace its path

97. Source: City Budget
Findings: Some neighborhoods suffer
more than others as mayor cuts budgets

100. Disparities in water
usage
“Water use highest in
poor areas of the city”
Mapping and statistical
analysis

101. Presenting the data
Include a methodology explaining what you did and
what you don’t know.
For really complicated analyses – consider a super
nerdy white paper explaining all of your findings
If you make data downloadable – include field
descriptions and anything users should watch for

102. For more information
www.ire.org
www.propublica.org
jennifer.lafleur@propublica.org