The document discusses the importance of validating data sources and understanding the methodology used to collect and analyze data. It emphasizes that data sets are dynamic and have a history or "genealogy" that is important to understand. Proper data validation includes checking for consistent definitions, completeness of records, precision of values, and outliers. The document provides examples of how invalid data can negatively impact stories and recommendations for journalists to evaluate data quality.
1. “OK, but where did that data come from?”
Data validation in the
Digital Age
Tom Johnson Cheryl Phillips
Managing Director Data Enterprise Editor
Inst. for Analytic Journalism Seattle Times
Santa Fe, New Mexico USA Seattle, Washington USA
tom@jtjohnson.com
cphillips@seattletImes.com
1
2. Data validation in the
Digital Age
Presentation by Cheryl Phillips and Tom Johnson at
National Institute of Computer-Assisted Reporting Conference
Date/Time: Friday, Feb. 24 at 11 a.m.
Location: Frisco/Burlington Room
St. Louis, Missouri USA
This PowerPoint deck and Tipsheets posted at:
http:// s d r v . m s / w N t i M 7
2
3. The methodology / = the value of the data set and your story
1
Important point
A data base (or
report) is only as
good as the
methodology used
to create it.
3
4. 2
Data sets are living things; they have pedigree and genealogy
Important points
•Most [all?] data sets are living
things.
•And they have a pedigree, a
genealogy.
•Data sets live in a dynamic
environment.
•Understand the DB ecology
4
5. How bad data can do you wrong
Illinois and Missouri sex-offender DB
•“St. Louis Post-Dispatch - 2 May 1999: A11 – “ABOUT 700 SEX
OFFENDERS DO NOT APPEAR TO LIVE AT THE ADDRESSES
LISTED ON A ST. LOUIS REGISTRY; MANY SEX OFFENDERS NEVER
MAKE THE LIST” By Reese Dunklin; Data Analysis By David Heath and Julie
Luca
•Sun, 3 Oct 2004 - THE DALLAS MORNING NEWS - PAGE-1A
“Criminal checks deficient; State's database of convictions is
hurt by lack of reporting, putting public safety at risk, law
officials say” By Diane Jennings and Darlean Spangenberger
•See stories here
6. How bad data can do you wrong
2011 - New Mexico Sec. of State’s “questionable
voters” data set – “The Big Bundle”
•~1.1m voters
•Previous SoS didn’t clean rolls
•Matched name, address, DoB and SS#
– SSA data base; NM driver’s licenses
– 2 variables “mismatch” = Questionable?
– Asked State Police (not AG’s office) to investigate
7. Problems with Sec. of State methodology
• What’s the error rate of original DB?
• Definition of “error”? (Gonzales or Gonzalez)
• Sample(s) by county and state total?
• Error rates of comparative DBs?
• Aggregation of error problem
• 2011 Help America Vote Verification Transaction
Totals, Year-to-Date, by State
https://www.socialsecurity.gov/open/havv/havv-year-
9. There be dragons!
A most
Data base
wonderful
rich with story!!!
potential
9
10. Building genealogy for target DB
1. Pre-plan 1. Acquire latest data and
•2nd monitor related docs
•“Logbook” apps 1. Do tables conform to
1. Lit. review/ interview peers record layout?
1. Do data fit theoretical 1. Do docs specify expected
models? ranges & frequencies?
1. Do a “critical biography” of 1. Are data values missing or
the data out of range?
1. Does biography raise 1. Review major checklist
critical warnings?
1. Have others run analysis of
this data?
Source: Palmer, Griff. “Flowchart/decision tree for data base analysis.” pgs. 136-146. Ver 1.0 Proceedings, IAJ Press (Santa Fe,
NM), April 2006. http://www.lulu.com/product/paperback/ver-10-workshop-proceedings/546459
11. Building genealogy for target DB
1. Pre-plan 1. Acquire latest data and
• Changes in
•2nd monitor related docs
definitions?
•“Logbook” apps 1. Do tables conform to
• review/ interview peers
1. Lit. By administrators? record layout?
• Formal or informal?
1. Do By statute?
• data fit theoretical 1. Do docs specify expected
models? ranges & frequencies?
• Changes in collection
1.methods, data entry,
Do a “critical biography” of 1. Are data values missing or
the data out of range?
vetting, updating, file
1.type/format?raise
Does biography 1. Review major checklist
critical warnings?
• Changes in users and
1.usage
Have others run analysis of
this data?
• Data cleaning
12. Data Quality checkpoints
• Constancy of definitions and coding categories?
• All at same time and location?
• Completeness: How many records have unfilled
cells? Are the tendencies of “nulls” consistent in
all records, variable types?
• Precision: Are the numbers rounded or?
• Hope for fine-grained, not summaries or aggregates
• Can be especially important with temporal and
geographic data, i.e. What is the range(s) of the time
scales?
14. Data Quality checkpoints
• Constancy of definitions and coding categories?
• All at same time and location?
• Completeness: How many records have unfilled
cells? Are the tendencies of “nulls” consistent in
all records, variable types?
• Precision: Are the numbers rounded or?
• Hope for fine-grained, not summaries or aggregates
• Can be especially important with temporal and
geographic data, i.e. What is the range(s) of the time
scales?
15. Newsroom methods for
measuring data quality
• Test frequencies on key fields
Bicycle accidents in Seattle included a time field. But
it was almost always noon when accidents occurred.
Caveat: Don’t over-reach with your conclusions or
analysis
16. Don’t over-reach with your
analysis
– Rates are good – IF you have the data to calculate
them.
17. Outliers are important
Explore the reasons behind anomalies or unexpected
trends in the data.
From the state of WA: After
going back and forth with our
analyst on this, we decided it
would be easiest for her to
just pull the data. You would
have been able to get most of
the way there through that
fiscal.wa.gov site, but there
was some stimulus money
you wouldn’t have captured
and we included the changes
so far to the current
biennium (based on the
supplemental the legislature
approved in December).
18. Other Key Data Checks
– When you update
the data, make sure
nothing has changed.
Check definitions for
expansion or
reduction and talk to
the creator of the
data.
– Be ready to nix a
story.
19. Other Key Data Checks
– Do the math: run sums, percent change, other
calculations. Test that math against the results in
the database – do they match?
– Look for unexpected nulls
– Run a group by query and sort alphabetically by
major fields to test for misspellings or other
categorization errors.
– If your data should include every city, or every
county in the state, does it? Are you missing data?
20. Other Key Data Checks
– Check with experts and have them test your
analysis. Research the methodology used with the
kind of data you are working with.
– There is version control for Web frameworks – use
some kind of version control for your database,
even if it’s in an Excel spreadsheet. Any time you
change it, log what you did and when and why.
21. Other Key Data Checks
– Test the data against source documents.
23. Building genealogy for target DB
• Pre-plan • Acquire latest data and
2nd monitor related docs
NOW you are ready to
“Logbook” apps
• Do tables conform to record
• Lit. review/ interview peers layout?
write a story•Do docs&specifyon
• Do data fit theoretical
models?
based expected
ranges frequencies?
a data base!values missing or
• Do a “critical biography” of
the data
• Are data
out of range?
• Does biography raise critical • Review major checklist
warnings?
• Have others run analysis of Analysis
this data?
24. Summing Up
• Databases are constantly dynamic, “living” things.
Look for and measure their energy and change.
• Beware of rounding error
– Always try to get the most fine-grained data possible in its
ORIGINAL data form or application, i.e. avoid PDFs with
SUMMARY data
• Beware of changing definitions
• Beware of changing data collectors, data entry
personnel, changing norms of editing and usage.
25. “OK, but where did that data come from?”
Many Thanks
Data validation in the
This PowerPoint deck and Tipsheets posted at:
http:// s d r v . m s / w N t i M 7
Tom Johnson Cheryl Phillips
Managing Director Data Enterprise Editor
Inst. for Analytic Journalism Seattle Times
Santa Fe, New Mexico USA Seattle, Washington USA
tom@jtjohnson.com
cphillips@seattletImes.com
25
Editor's Notes
“ The devil is in the data” “ How pure/faulty/legit are the “genes” in your data? =================================================== Opener: They don’t believe us (perhaps with good reason). Get some stats on public’s trust of journalism and journalists. Way to save and perhaps improve our reputation is to make sure of the truthfulness – the validity – of what we are reporting. As we do more and more analysis of data as part of our stories, make sure we are analyzing correct and valid pure–quality data becomes crucial. (We should also be sharing out methods and data with the public, but that’s a topic for another session.)
Finding the headwaters of your data Tracing the process of DB creation Type of agency? Gov’t, NGO, non-profit, profit Who’s responsible for the DB conception? Mandated by legislation, federal or state regulations, executive order? Some administrator For what purpose? Who’s responsible for designing and defining… Variables Collection methods Quantitative or qualitative data? Degree of precision in classification, geography, dates, time-factor Self-reported? Census or sampling? Training for data collectors? Training and verification of classification assignment?
The methodology determines the value of the data set and your story I’m suspicious of -- and reluctant to use – sweeping generalities and Adjectives, but in this case…. Appropriateness of method ALWAYS determines the validity of the analysis, though the method(s) (i.e. analytic tools) may vary depending on your objectives. Methods used to create a data set ALWAYS determine the validity and functionality of the data set Ergo, before we start crunching data and data mining, we need to recognize and know…. The methods used to create the data set determine: The reliability of the data set The functionality (for multiple audiences) of the data set (e.g. who called for the creation of this data set, when and why? Who is to use it for what ends? What is its “measured” value for original users and for our readers? Knowning and understanding those “methods of creation” determines the value of your analysis and, hence, your story.
Most [all?] data sets are living things . A data base, may look to be just a static matrix of text or numbers, but there are living, breathing dynamic forces at work in and around any data set that can provide an interesting context of understanding for journalists. And they have a pedigree, a genealogy. If we don’t understand that genealogy, we can’t evaluate – or properly use – that DB Data sets live in a dynamic environment. All data sets “live” in a context, in an environment in the datasphere that is constantly changing in terms of the validity of the data, who is collecting/updating/editing the data, who is using the data for what purposes and how often? How is Data Set A (or parts of it) related to DS B and C and G. And how do the administrators/analysts of the secondary data measure the quality of the data they are getting from DS A, if they do it at all? Understand the DB ecology See how the data set relates to other sets of data, agencies and users.
Tom will had hyperlinks to these stories, though we might include them in handouts Get bibliography on SSA publications
Get bibliography on SSA publications “ The biggest problem with E-Verify is that it’s based on SSA’s inaccurate records. SSA estimates that 17.8 million (or 4.1 percent) of its records contain discrepancies related to name, date of birth, or citizenship status, with 12.7 million of those records pertaining to U.S. citizens. That means E-Verify will erroneously tell you that 1 in 26 of your legal workforce is not actually legal.” http://www.laborcounselors.com/index.php?option=com_content&view=article&id=715:social-security-mismatch-and-immigration-2011-where-do-we-go-from-here&catid=44&Itemid=300008 “ The error rate for US citizens in the SSA data base is estimated to be 11 percent, meaning that 12.7 million of the 17.8 million "bad" SSNs in 2006 are believed to belong to US citizens, according to SSA's inspector general. “http://migration.ucdavis.edu/mn/more.php?id=3315_0_2_0 2011 Help America Vote Verification Transaction Totals, Year-to-Date, by State https://www.socialsecurity.gov/open/havv/havv-year-to-date-2011.html Tom: I think the answer depends on how many records are in each db. If db1 is very large in comparison to db2, then the error rate should be close to 4.5%. And vice versa. There's probably a formula for this, but I sure don't know it. I'd do the match and then check a sample of the results to estimate the combined error rate. Steve Doig ======================= Let's say each db holds similar data and is the same size, 1000 records. Let's also assume that there are no records duplicated in the two databases, either internally or from one data set to the other. Then you have 45 bad records in one set, and 137 in the other. Combining, you have (45+137) = 182 bad records, in 2000 total records, or an error rate of 9.1%. Same process can be used to calculate error rate combining data from any number of sets, of any size as long as no records are duplicated. Error LIMITS/confidence intervals would be quite a different matter. Steve Ross Ah, but what if one DB has an error rate of 73% and the other has an error rate of 82%. How could you have an error rate >100%? Ergo, the question becomes: What is the lowest “acceptable” error rate for meaningful analysis. (Whatever “meaningful” means.)
Always a VERY complex problem for analysis bcs of “definitions,” changes over time and then statistical evaluation methods Assume you can determine, from sampling, that Data Base “A” has 8.5% records with errors. Assume DB “B” has 11.3% of records with errors (how to define “error”?). If you compare one to the other, your probability of errors will be 8.5+11.3 or 19.8%. Ah, but what if one DB has an error rate of 73% and the other has an error rate of 82%. How could you have an error rate >100%? Ergo, the question becomes: What is the lowest “acceptable” error rate for meaningful analysis. (Whatever “meaningful” means.) Help America Vote Transactions? Note that New Mexico has not sought any clarifications. Social Security Makes Help America Vote Act Data Available http://www.socialsecurity.gov/pressoffice/pr/HAVA-pr.html ( Printer friendly version ) Michael J. Astrue, Commissioner of Social Security, today announced the agency is publishing data on its Open Government website www.socialsecurity.gov/open about verifications the agency conducts for States under the Help America Vote Act (HAVA) of 2002. Under HAVA, most States are required to verify the last four digits of the Social Security number of people newly registering to vote who do not possess a valid State driver's license. “ I strongly support President Obama’s commitment to creating an open and transparent government,” Commissioner Astrue said. “As we approach another federal election year, it remains absolutely critical that Americans are able to register to vote without undue obstacles. Making this data publicly available will allow the media and the public on a timely basis to raise questions about unexpected patterns with the appropriate State officials.” The data available at www.socialsecurity.gov/open/havv represents the summary results for each State of the four-digit match performed by Social Security under HAVA. # # # http://www.socialsecurity.gov/pressoffice/pr/HAVA-pr.html
DYNAMIC DATA & DATA BASE OR SET https://www.socialsecurity.gov/open/havv/havv-year-to-date-2011.html What do these terms mean? The following list describes the types of data in the HAVV dataset. Total Transactions: The total number of verification requests made during the time period. Unprocessed Transactions: The total number of verification requests that could not be processed because the data sent to us was invalid, (e.g., missing, not formatted correctly). Total Matches: The total number of verification requests where there is at least one match in our records on the name, last four digits of the SSN and date of birth. Total Non Matches: The total number of verification requests where there is no match in our records on the name, last four digits of the SSN or date of birth. Multiple Matches Found – At least one alive and at least one deceased : The total number of verification requests where there are multiple matches on name, date of birth, and the last four digits of the SSN, and at least one of the number holders is alive and at least one of the number holders is deceased. Single Match Found – Alive: The total number of verification requests where there is only one match in our records on name, last four digits of the SSN and date of birth, and the number holder is alive. Single Match Found – Deceased: The total number of verification requests where there is only one match in our records on name, date of birth, and last four digits of the SSN, and the number holder is deceased. Multiple Matches Found – All Alive: The total number of verification requests where there are multiple matches on name, date of birth, and last four digits of the SSN, and each match indicates the number holder is alive. Multiple Match Found – All Deceased: The total number of verification requests where there are multiple matches on name, date of birth, and the last four digits of the SSN, and each match indicates the number holder is deceased.
Source: Palmer, Griff. “Flowchart/decision tree for data base analysis.” pgs. 136-146 Ver 1.0 Proceedings, IAJ Press (Santa Fe, NM), April 2006. http://www.lulu.com/product/paperback/ver-10-workshop-proceedings/546459 1. Pre-plan 1a. 2 nd monitor 2a. “logbook” applications 2. Lit. review/ interview peers 3. Do data fit theoretical models? 4. Do a “critical biography” of the data 5. Does biography raise critical warnings? 6. Have others run analysis of this data? 7. Acquire latest data and related docs 8. Do tables conform to record layout? 9. Do docs specify expected ranges & frequencies? 10. Are data values missing or out of range? 11. Review major checklist
Source: http://nsu.aphis.usda.gov/outlook/issue5/data_quality_part2.pdf Constancy of definitions and coding categories ? All at same time and location? Completeness: How many records have unfilled cells? Are the tendencies of “nulls” consistent in all records, variable types? Precision: Are the numbers rounded or? Hope for fine-grained, not summaries or aggregates Can be especially important with temporal and geographic data, i.e. What is the range(s) of the time scales? Can be a lot of difference in traffic counts, for example, if the data is hourly vs. 15-minute intervals. Or in range of ages.
Source: http://nsu.aphis.usda.gov/outlook/issue5/data_quality_part2.pdf Constancy of definitions and coding categories ? All at same time and location? Completeness: How many records have unfilled cells? Are the tendencies of “nulls” consistent in all records, variable types? Precision: Are the numbers rounded or? Hope for fine-grained, not summaries or aggregates Can be especially important with temporal and geographic data, i.e. What is the range(s) of the time scales? Can be a lot of difference in traffic counts, for example, if the data is hourly vs. 15-minute intervals. Or in range of ages.
Important to note not to jump to conclusions, or try to do more analysis than makes sense. For example, rates would have been misleading because we don’t have good bicycle counts by street or intersection, much less car-traffic counts. But we could use this anecdotally in the story: In the city's annual mid-September count, there were 3,251 cyclists commuting into downtown in 2010, up from 2,273 in 2007. So, accidents are holding steady while the number of commuters is increasing.
Important to note not to jump to conclusions, or try to do more analysis than makes sense. For example, rates would have been misleading because we don’t have good bicycle counts by street or intersection, much less car-traffic counts. But we could use this anecdotally in the story: In the city's annual mid-September count, there were 3,251 cyclists commuting into downtown in 2010, up from 2,273 in 2007. So, accidents are holding steady while the number of commuters is increasing.
Important to note not to jump to conclusions, or try to do more analysis than makes sense. For example, rates would have been misleading because we don’t have good bicycle counts by street or intersection, much less car-traffic counts. But we could use this anecdotally in the story: In the city's annual mid-September count, there were 3,251 cyclists commuting into downtown in 2010, up from 2,273 in 2007. So, accidents are holding steady while the number of commuters is increasing.
Last year, editors at The Seattle Times noticed more food trucks around. There must be a story about the safety record of these trucks, they thought. So, of course, we checked it out. What we found? Food trucks were just as clean, met inspection rules, just as much as all other types of restaurants. In part, this was because their food came from prep sites most of the time and was not cooked in a mobile unit. And, just to be sure, we checked the prep sites. They got good grades too.
“ The devil is in the data” “ How pure/faulty/legit are the “genes” in your data? =================================================== Opener: They don’t believe us (perhaps with good reason). Get some stats on public’s trust of journalism and journalists. Way to save and perhaps improve our reputation is to make sure of the truthfulness – the validity – of what we are reporting. As we do more and more analysis of data as part of our stories, make sure we are analyzing correct and valid pure–quality data becomes crucial. (We should also be sharing out methods and data with the public, but that’s a topic for another session.)