Sarah Guido gave a presentation on analyzing data with Python. She discussed several Python tools for preprocessing, analysis, and visualization including Pandas for data wrangling, scikit-learn for machine learning, NLTK for natural language processing, MRjob for processing large datasets in parallel, and ggplot for visualization. For each tool, she provided examples and use cases. She emphasized that the best tools depend on the type of data and analysis needs.

1. Sarah Guido
@sarah_guido
Reonomy
PYTN 2015
ANALYZING DATA WITH
PYTHON

2. Data scientist at Reonomy
University of Michigan graduate
NYC Python organizer
PyGotham organizer
ABOUT ME

3. Bird’s-eye overview: not comprehensive
explanation of these tools!
Take data from start-to-finish
Preprocessing: Pandas
Analysis: scikit-learn
Analysis: nltk
Data pipeline: MRjob
Visualization: ggplot
What next?
ABOUT THIS TALK

4. So many tools
Preprocessing, analysis, statistics, machine learning,
natural language processing, network analysis,
visualization, scalability
Community support
“Easy” language to learn
Both a scripting and production-ready
language
WHY PYTHON?

5. How to find the best tool(s)?
The 90/10 rule
Simple is better than complex
FROM POINT A TO POINT…X?

6. Available resources
Documentation, tutorials, books, videos
Ease of use (with a grain of salt)
Community support and continuous development
Widely used
WHY I CHOSE THESE TOOLS

7. The importance of data preprocessing
AKA wrangling, cleaning, manipulating, and so on
Preprocessing is also getting to know your
data
Missing values? Categorical/continuous?
Distribution?
PREPROCESSING

8. Data analysis and modeling
Similar to R and Excel
Easy-to-use data structures
DataFrame
Data wrangling tools
Merging, pivoting, etc
PANDAS

9. Keep everything in Python
Community support/resources
Use for preprocessing
File I/0, cleaning, manipulation, etc
Combinable with other modules
NumPy, SciPy, statsmodel, matplotlib
PANDAS

10. File I/O
PANDAS

11. Finding missing values
PANDAS

12. Removing missing values
PANDAS

13. Pivoting
PANDAS

14. Other things
Statistical methods
Merge/join like SQL
Time series
Has some visualization functionality
PANDAS

15. Application of algorithms that learn from
examples
Representation and generalization
Useful in everyday life
Especially useful in data analysis
MACHINE LEARNING

16. Supervised learning
Classification and regression
Unsupervised learning
Clustering and dimensionality reduction
MACHINE LEARNING

17. Machine learning module
Open-source
Built-in datasets
Good resources for learning
SCIKIT-LEARN

18. Scikit-learn: your data has to be continuous
Here’s what one observation/label looks like:
SCIKIT-LEARN

19. Transform categorical values/labels
SCIKIT-LEARN

20. Classification
SCIKIT-LEARN

21. Classification
SCIKIT-LEARN

22. Other things
Very comprehensive of machine learning
algorithms
Preprocessing tools
Methods for testing the accuracy of your model
SCIKIT-LEARN

23. Concerned with interactions between
computers and human languages
Derive meaning from text
Many NLP algorithms are based on machine
learning
NATURAL LANGUAGE PROCESSING

24. Natural Language ToolKit
Access to over 50 corpora
Corpus: body of text
NLP tools
Stemming, tokenizing, etc
Resources for learning
NLTK

25. Stopword removal
NLTK

26. Stopword removal
NLTK

27. Stemming
NLTK

28. Other things
Lemmatizing, tokenization, tagging, parse trees
Classification
Chunking
Sentence structure
NLTK

29. Data that takes too long to process on your
machine
Not “big data” but larger data
Solution: MapReduce!
Processing large datasets with a parallel,
distributed algorithm
Map step
Reduce step
PROCESSING LARGE DATA

30. Map step
Takes series of key/value pairs
Ex. Word counts: break line into words, return
word and count within line
Reduce step
Once for each unique key: iterates through values
associated with that key
Ex. Word counts: returns word and sum of all
counts
PROCESSING LARGE DATA

31. Write MapReduce jobs in Python
Test code locally without installing Hadoop
Lots of thorough documentation
A few things to know
Keep everything in one class
MRJob program in a separate file
Output to new file if doing something like word
counts
MRJOB

32. Stemmed file
Line 1: (‘miss’, 2), (‘taylor’, 1)
Line 2: (‘taylor’, 1), (‘first’, 1), (‘wed’, 1)
And so on…
MRJOB

33. Map
 Line 1: (‘miss’, 2),
(‘taylor’, 1)
 Line 2: (‘taylor’, 1),
(‘first’, 1), (‘wed’, 1)
 Line 3: (‘first’, 1),
(‘wed’, 1)
 Line 4: (‘father’, 1)
 Line 5: (‘father’, 1)
Reduce
 (‘miss’, 2)
 (‘taylor’, 2)
 (‘first’, 2)
 (‘wed’, 2)
 (‘father’, 2)
MRJOB

34. Let’s count all words in the Gutenberg file
Map step
MRJOB

35. Reduce (and run) step
MRJOB

36. Results
Mapped counts reduced
Key/val pairs
MRJOB

37. Other things
Run on Hadoop clusters
Can write highly complex jobs
Works with Elasticsearch
MRJOB

38. The “final step”
Conveying your results in a meaningful way
Literally see what’s going on
DATA VISUALIZATION

39. Remember this?
DATA VISUALIZATION

40. Bar chart of distribution
DATA VISUALIZATION

41. 2D visualization library
Based on the similar R library
Wrapper around matplotlib
Wide variety of nice-looking plots
Easy to feed in Pandas
GGPLOT

42. Bar chart of distribution
GGPLOT

43. Layers
Aesthetics
aes()
No need for value_counts()!
GGPLOT

44.  Breakdown of
class per
maintenance type
GGPLOT

45. Other things
Many different kinds of graphs
Customizable
Smoothing, facets
Time series
Themes!
GGPLOT

46. theme_xkcd()
GGPLOT

47. Phew!
Which tool to choose depends on your needs
Workflow:
Preprocess
Analyze
Visualize
WHAT NEXT?

48. Pandas
http://pandas.pydata.org/
scikit-learn
http://scikit-learn.org/
NLTK
http://www.nltk.org/
MRJob
http://mrjob.readthedocs.org/
ggplot
http://ggplot.yhathq.com/
RESOURCES

49. Twitter
@sarah_guido
LinkedIn
https://www.linkedin.com/in/sarahguido
NYC Python
http://www.meetup.com/nycpython/
CONTACT ME!

50. AND FINALLY…

51. Questions?
THE END!