This document summarizes a training for librarians on data scientist skills. It introduces the speaker, Tom Morris, and covers the typical data analysis lifecycle of finding, preparing, analyzing, and visualizing data. Specific topics include the importance of recording data provenance, using tools like OpenRefine for working with messy data, characterizing data fields, and scaling techniques for larger data sets. Resources for further learning are also provided.

1. Data Scientist Training for Librarians
Harvard College Observatory
March 28, 2013
Tom Morris
@tfmorris

2. Who am I?
•
Independent software engineering & product
management consultant
•
Developer on open source OpenRefine project
•
Curious data geek
•
Contact:
– Twitter: @tfmorris
– Email: tfmorris@gmail.com
2013-03-28 Tom Morris @tfmorris 2

3. Data Analysis Lifecycle
●
Find / Extract
●
Prepare
– Characterize
– Clean
– Integrate / Extend
●
Analyze
●
Visualize / Report
2013-03-28 Tom Morris @tfmorris 3

4. Provenance
●
Provenance is key! - both before and after
you get data
●
Record source (e.g. download URL) and date
●
Unix command line
– build up a repeatable transformation pipeline script
– use make to keep from having to repeat steps)
●
OpenRefine maintains an undo history (but...)
2013-03-28 Tom Morris @tfmorris 4

5. Irreversible Transforms
●
Be careful of anything which isn't reversible
●
Keep source files and plan recovery strategy
●
Common gotchas:
– Character encoding – can't replace
substitution character with its original
value
– Leading 0s on identifiers
2013-03-28 Tom Morris @tfmorris 5

6. Provenance projects
●
Stanford Panda (Provenance and Data) -
http://infolab.stanford.edu/panda/
●
Open Provenance Model -
http://openprovenance.org/
●
Both focus on bi-directional traceability
2013-03-28 Tom Morris @tfmorris 6

7. Tools vs Scale
●
Editor with macro facility: emacs, vim
●
Spreadsheet: Excel, OO Calc
●
OpenRefine
●
Unix shell commands – awk, sed, grep, cut,
sort, head, tail
●
“Real” programming – Python, Ruby, Java
●
Map-Reduce
2013-03-28 Tom Morris @tfmorris 7

8. Regular Expressions
●
Useful in so many contexts
●
A little confusing to learn, but
●
Absolutely worth the effort!
2013-03-28 Tom Morris @tfmorris 8

9. OpenRefine
●
Power tool for working with messy data
●
Free and open source
●
Desktop based (data stays private)
●
Faceted browsing interface
●
Lots of input & output formats
●
Powerful transformations
●
Useful for analysis & web scraping/APIs too
2013-03-28 Tom Morris @tfmorris 9

10. OpenRefine Data Formats
●
CSV/TSV/separator based
●
Fixed width field
●
JSON & XML
●
Excel & OpenOffice Calc
●
Google Spreadsheets & Fusion Tables
●
RDF
●
URLs & zip files too!
2013-03-28 Tom Morris @tfmorris 10

11. Data Characterization
●
Coded vs free-form fields
●
Distribution of values
– Missing values – skip, impute, ...
– Outliers – cause? Can they be rescaled?
●
Delimiters & escaping (e.g. HTML, XML)
●
Formatting problems
●
Character encoding issues?
2013-03-28 Tom Morris @tfmorris 11

12. Hands-on
●
Let's play with some data!
●
http://code.google.com/p/google-refine/
2013-03-28 Tom Morris @tfmorris 12

13. Export
●
OpenRefine exports most import formats:
Excel, CSV, TSV, OpenOffice, Google
Spreadsheets, Fusion tables, JSON, RDF
●
Template-based exporter for everything else:
custom JSON formats, etc.
2013-03-28 Tom Morris @tfmorris 13

14. Scaling Up
●
Experiment with a (representative) sample of
your data
●
Reuse regexs, filters, etc with more heavy
duty tools – awk, sed, Map-Reduce
2013-03-28 Tom Morris @tfmorris 14

15. Resources
●
Berkeley Data Science course
http://datascienc.es/schedule/
– week 2 - Data Preparation has good R examples
http://berkeleydatascience.files.wordpress.com/2012/02/2012
●
Mike Loukides "Data Hand Tools"
http://radar.oreilly.com/2011/04/data-hand-tools
●
Jeremy Howard Getting in shape for the sport
of Data Science
http://media.kaggle.com/MelbURN.html
2013-03-28 Tom Morris @tfmorris 15

16. More resources
●
MIT IAP Data Science course materials
– http://dataiap.github.com/dataiap/
●
Quora
– http://www.quora.com/Data/Where-can-I-get-large-datasets-open-to-the-public
– http://www.quora.com/What-are-some-good-methods-for-data-pre-processing-in-machine-learning
●
OKFN School of Data handbook
– http://handbook.schoolofdata.org
●
Hilary Mason
– http://www.dataists.com/2010/09/a-taxonomy-of-data-science/
2013-03-28 Tom Morris @tfmorris 16

17. Resources mentioned
●
Harvard Business Review competition at
Kaggle
– Competition ends 8/27/2012 4:00 AM UTC !
– https://www.kaggle.com/c/harvard-business-review-vision-statement-prospect
●
Stanford Data Wrangler
– http://vis.stanford.edu/wrangler/
2013-03-28 Tom Morris @tfmorris 17

18. Thanks!
•
Questions now?
•
Questions later:
– Twitter: @tfmorris
– Email: tfmorris@gmail.com
2013-03-28 Tom Morris @tfmorris 18