It's not the documents; it's the DATA

Tom Johnson Managing Director Inst. for Analytic Journalism Santa Fe, New Mexico USA t o m @ j t j o h n s o n . c o m It’s not the documents; it’s the DATA!

It’s not the documents, it’s the DATA!
Presentation at
“ 2011 Open Government Academy” March 26, 2011
Presented by the New Mexico Foundation for Open Government ,
New Mexico Press Association and New Mexico Broadcasters Association
This PowerPoint deck and Tipsheet posted at: http:// j o h n s o n – f o g . n o t l o n g . c o m Licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License .

Important point
1 Nothing is as important – and valuable – as a good theory!

Theory of Journalistic Process
Data In  Analysis  Info Out
Data = that which, upon Analysis, yields Information. “Data” has many forms.
Analysis = Examination of data and facts to uncover and understand cause-effect and contextual relationships and patterns , thus providing basis for problem solving and decision making .
Information = that which aids in making decisions

Important point
The document is not the data.
2

Bertillon system: Public Records DB
Early public records
Intricate data collection
Potential for error in data entry
Potential for error in filing
No machine retrieval or analysis
Even today, OCR would be impossible

By 1910…
Indexing system has improved
Typewriters instead of pen
Better haircuts
But still …
Null fields
Subject to data entry errors; lost or misfiled cards/data
Limited large-scale analysis resources

Data entry potential for error
Filing potential for error
Even today, no OCR
By 1910…
Better haircuts
But still …
Null fields
Early “hard drives,” data retrieval and data analysis of public records

Data entry potential for error
Filing potential for error
Even today, no OCR
By 1910…
Better haircuts
But still …
Null fields
A public record, but one of limited usage
A DOCUMENT , but no efficient, productive, insightful way to FIND the data
A DOCUMENT , but no efficient, productive, insightful way to EXTRACT the data
Sorta like a PDF
Early “hard drives,” data retrieval and data analysis of public records

Traditional Data In  Analysis  Info Out Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Maps
How? Who?

Digital Age Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Atoms  Bits
How? Who?
New data is ubiquitous, shareable, scaleable.
Retrieval, copying and storage costs trivial
Can be validated and explored by individuals and applications

Digital Age Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Atoms  Bits
How? Who?
}
All data today requires NEW tools for ANALYSIS and STORY-TELLING
Statutes are usually adequate; the CULTURES are the challenge.

Important point
The document is not the data . Without analysis, the data are not the story.
3

Four stories
Doig: Hurricane Andrew, Data (from documents) = Pulitizer Prize & bldg. inspectors in jail
Craig Harris: “ Arizona pension systems a soaring burden ”
Waite: water, developers, land use = disappearing wet lands
UK: Investigate Your MPs Expenses “We have 458,832 pages of documents. 27,731 of you have reviewed 223,475 of them. Only 235,357 to go” MP’s expense claims on Google spreadsheet

Journalism and GIS
Steve Doig [Miami Herald]
1992
Hurricane Andrew + damage reports + building inspection = jail terms

Doig: Hurricane Andrew

Four stories
Craig Harris: “ Arizona pension systems a soaring burden ”

Analysis with real data Search Sort DB info

Four stories
Craig Harris: “ Arizona pension systems a soaring burden”
Waite: water, developers, land use = “ Vanishing Wetlands ”

Vanishing Wetlands

Four stories
Craig Harris: “ Arizona pension systems a soaring burden”
Waite: water, developers, land use = disappearing wet lands
UK: Investigate Your MPs Expenses “We have 458,832 pages of documents. 27,731 of you have reviewed 223,475 of them. Only 235,357 to go” MP’s expense claims on Google spreadsheet
EFF Seeks Cooperating FOIA Reviewers

UK MP’s expenses Solid search tools These are PDFs, POST -search

Major questions?
As participants in a liberal democracy…
How do we get the necessary data?
And from where?
And in appropriate forms?

Files, Transparency, Ease of Analysis Easier Challenging

Files, Transparency, Ease of Analysis

Data In: Objectives/Requirements
Move data from “out there” to analytic site/tools
Looking for connections; patterns

Data In: Objectives/Requirements
Seeking fine-grained data, NOT aggregations
Seek data in original form (i.e. NO PDFs)
Get data in lowest common denominator format: - Comma-delimited files in ASCII or Text
Who collected the data? Why? How?
Who proofed/edited the data? Why? How?
If from data base, first ask for “record layout” or “code sheet” or “schema”
Definitions of variables or fields. Constant or ???

Data In: “Typical” problems with gov sites
Barriers data = barriers to analysis
NO site search capability; no site map
Failure to use open-standard HTML; using closed-standard Adobe Flash/Shockwave environment.
Page formats/layouts not consistent; too many drill-downs instead of search-driven generators
Jiggly roll-overs; too much effort spent on bling
Impossible to download or scrape data for analysis
Information available only in Adobe PDF files; notoriously unfriendly to data analysis.

Good NM sites Search! Español Feedback!

NM Legis. Bill Finder Could be better: no way to find what bills were introduced by X legislator Download bill in TWO formats

Data In: Challenges
New site in New Mexico: www.sunshineportalnm.com
“ Beta ,” but facade for taxpayers; a secondary tax bcs of minimal utility; torture for journos

Data In: Challenges in SunshinePort
Comprehensive Annual Financial Reports
Possible to machine download, but laborious to format for analysis
Investment Holdings reports are far worse
They are poor-quality static image files, not machine-readable.
Tabular data roughly formatted; makes conversion for analysis an arduous, if not impossible task.

Bottom line on SunshinePortalNM.com
“ If the State of New Mexico takes the position that through this site it is discharging all of its disclosure obligations with respect to these particular records, open government is in trouble there. ”
“ This is not even a web page, it’s a Flash application, so there’s not going to be much sunlight escaping from this portal. “

Bottom line on SunshinePortalNM.com
“ If the State of New Mexico takes the position that through this site it is discharging all of its disclosure obligations with respect to these particular records, open government is in trouble there. ”
“ This is not even a web page, it’s a Flash application, so there’s not going to be much sunlight escaping from this portal. “ “ A perfect example of creating the appearance of transparency without actually being transparent.”

Good data sites – Gov and NGO
Data.gov [A beta site] www.data.gov/
Metrics www.data.gov/metric
DataSF - http://datasf.org/ a clearinghouse of datasets available from the City & County of San Francisco
San Francisco Enterprise GIS Program - http:// gispub02.sfgov.org/data.asp
Maplight.com – an example of how citizens can use data Nonprofit, nonpartisan research organization, provides citizens and journalists the transparency tools to shine a light on the influence of money on politics.
Prize-winning gov’t agency web sites: http:// www.centerdigitalgov.com/survey/88/2010

Common aspects?
All have up-front search capabilities
All are written in “data-accessible” code
All data can be downloaded with “relative” ease
Some have various languages available
ALL are run by GOVERNMENT; no commercial sites

Challenge for Watchdogs?
Failure on the part of planners/bureaucrats to simply…
Give The People THEIR Data…
In The Most Basic, Original, Straightforward Form…
And Let Them Figure Out What Should Be Done With It!
The governor agrees

Tomorrow? Public Access to Original Data Impact Why not?

It’s not the documents, it’s the DATA! Tom Johnson Managing Director Inst. for Analytic Journalism Santa Fe, New Mexico USA t o m @ j t j o h n s o n . c o m Gracias a todos

It’s not the documents, it’s the DATA!
Presentation at
“ 2011 Open Government Academy” March 26, 2011
Presented by the New Mexico Foundation for Open Government ,
New Mexico Press Association and New Mexico Broadcasters Association
This PowerPoint deck and Tipsheet posted at: http://johnson-fog.notlong.com

FOI history
The world’s rst reedom o inormation legislation was adopted by the Swedish parliament in 1766. This publication includes the English translation o this ordinance on reedom o writing and the press. The enlightenment thinker and politicianAnders
Chydenius (1729-180), rom the Finnish city o Kokkola, played a crucial
role in creating the new law. As Proessor Juha Manninen describes in his article, the key achievements o the 1766 Act were the abolishment o political censorship and the gaining o public access to government documents. Although the innovation was suspended rom 1772-1809, the principle o publicity has since remained central in the Nordic countries.
http://www.scribd.com/doc/5885744/The-Worlds-First-Freedom-of-Information-Act-SwedenFinland-1766

Early police data base: incomplete data Source: Jay, Ricky. “Grifters, Bunco Artists & Flimflammen.” Wired, Feb. 2011, p.88. http://rickyjay.com/

NM HB 406
“… information contained in information systems databases created or maintained by or on behalf of a public body … shall be subject to disclosure to any person requesting the information in the format requested.
“ The information shall be provided in the most effective and efficient manner available to the custodian , as defined in the Inspection of Public Records Act.
B. The custodian may charge a reasonable fee for production of the information requested . The fee shall not exceed the cost of the materials and reasonable charges for the personnel required to retrieve and provide the information.
But what if it wasn’t New Mexico state employees directly at fault?

Analytic Tools
Text
ThemeRiver - http://infoviz.pnl.gov/research_themeriver.stm

“ Analytic tools” also for story-telling
Spreadsheets:
Tables, charts, infographics
Data base programs
Charts, graphs, data tables
Stats programs ( SPSS or SAS or R )
Generate graphics
Social network analytic graphics
GIS

FOIA b(3) Exemptions Original: http://www.propublica.org/article/foia-exemptions-sunshine-law

Content Analysis

Content analysis of legis party text

Positive example of gov’t data
Positive example: NM Leg Bill Locator
http://www.nmlegis.gov/lcs/_session.aspx?chamber=H&legtype=B&legno=%20406&year=11
Same data available in two formats!

NM HB 406
Senate approved 39-0 on Feb. 9 http:// www.nmlegis.gov/Sessions/11%20Regular/bills/house/HB0406.html
“ An Act RELATING TO PUBLIC RECORDS; PROVIDING FOR THE INSPECTION OF ELECTRONIC RECORDS.”

“ Data In” questions Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
#1 – Keep a logbook (Try using Notesync.com)
Qualitative and/or Quantitative?
Objective: strive to get the data in the most fine-grained and original form.
Online data is rarely complete nor totally accurate
Where is the data? In what format? I-o-P? Original digital file type(s)?

“ Data In” questions Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
#1 – Keep a logbook (Try using Notesync.com)
Who created the data? Why? How? Legal catalysts for creation? If so, what do they say?
Have definitions and collection process changed?
Who could review and edit the data? What was/is the vetting process to insure accuracy?
Who has analyzed the data? For what purpose and with what methods?

Data In

Analysis

Info Out

“ Analysis” phase Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Atoms  Bits How? Who?
What are we looking for? How can we be surprised?
Previous/parallel investigations? (Start with IRE site stories and tipsheets)
Context, i.e. past environment(s) and changes? Trends past and future?
Quantitative and Qualitative methods?
Data cleaning tools?

“ Analysis” phase Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Measurement of phenomena
Strength of relationships
Change
Estimating
Counting
Statistical
Geostatistical
Social Network Analysis
Forensic accounting
Who’s your rabbi?

Data In  Analysis  Info Out Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Source
Definition
Context
Estimating
Counting
Statistical
Geostatistical
Forensic accounting

Data In  Analysis  Info Out Data In  Analysis  Info Out
Notes
Text
Numeric
Images
Charts/Graphs
Maps
Audio
Video
Atoms  Bits How?
Source
Definition
Context
Estimating
Counting
Statistical
Geostatistical
Forensic accounting
Broadcast
Web
Audio
Video
Text
Data visualization
Maps
Dynamic databases
Archives

Theory of Journalistic Process Copyright © J. T. Johnson Data In
Interviews
Text docs
Clips
Pictures
Infographics
This is a headline DATELINE -- And the traditional text story starts here and goes on and on and on. Info Out Analysis

It's not the documents; it's the DATA

by J T "Tom" Johnson

Accessibility

Categories

Upload Details

Usage Rights

Statistics