This document discusses turning unstructured documents into structured data. It notes that while data is often thought of as existing in spreadsheets, much information exists as unstructured documents. It explores different approaches to structuring document data such as putting it into spreadsheets, extracting already structured data, helping partially structured data, or manually processing documents. Common tools mentioned for this include spreadsheet programs, Tabula, Acrobat, CometDocs, Python, R, and DocumentCloud. The overall message is that documents contain valuable data that can be structured and made available to others.

1. Turning Documents into Data
@dataeditor
#ONA17
bit.ly/docs2data

2. Documents
are
Data

3. We often think of
data as spreadsheets
but much of it is
unstructured

4. Not every set of
documents needs
to be structured to
be useful

6. Let’s add
Some
Structure

7. Adding structure
often means
putting data in a
spreadsheet

8. Sometimes you can
extract the data
because it’s already
structured

10. Sometimes the data
is sort of
structured, but
needs some help

12. Sometimes you just
have to do things
the old-fashioned
way

14. And sometimes you
have to get super
creative

16. Overarching thoughts
● “Documents and data don’t lie” is true
○ Most of the time
● Government agencies keep documents in a data format
○ Some of the time
● Asking for the data is always the easiest route
○ If that fails, scraping and conversion is next step
● Assembling your own data from documents sets you apart
● Make your data available when you’re done
○ Don’t force others to reinvent the wheel

17. Tools of the trade
● Microsoft Excel or other spreadsheet programs
● Tabula
● Adobe Acrobat
● CometDocs
○ If you’re an IRE member, you can get a free premium
account
● Python or R for scraping
● Command-line tools
○ Tesseract
○ ImageMagick
● DocumentCloud

19. Turning Documents into Data
@dataeditor
#ONA17
bit.ly/docs2data