District Data Labs Workshop Current Workshop: August 23, 2014 Previous Workshops: - April 5, 2014 Data products are usually software applications that derive their value from data by leveraging the data science pipeline and generate data through their operation. They aren’t apps with data, nor are they one time analyses that produce insights - they are operational and interactive. The rise of these types of applications has directly contributed to the rise of the data scientist and the idea that data scientists are professionals “who are better at statistics than any software engineer and better at software engineering than any statistician.” These applications have been largely built with Python. Python is flexible enough to develop extremely quickly on many different types of servers and has a rich tradition in web applications. Python contributes to every stage of the data science pipeline including real time ingestion and the production of APIs, and it is powerful enough to perform machine learning computations. In this class we’ll produce a data product with Python, leveraging every stage of the data science pipeline to produce a book recommender.

1. Building Data Products with Python
District Data Labs

2. Links to various resources
Introduction to Python
http://bit.ly/1gJ73Tt
Github Repository
http://bit.ly/1eLBzki

3. About the Instructor
Benjamin Bengfort
Data Science:
● MS Computer Science from North Dakota State
● PhD Candidate in CS at the University of Maryland
● Data Scientist at Cobrain Company in Bethesda, MD
● Board member of Data Community DC
● Lecturer at Georgetown University
Python Programmer:
● Python developer for 7 years
● Open source contributor
● My work on Github: https://github.com/bbengfort

4. About the Instructor
Benjamin Bengfort
I am available to collaborate and
answer questions for all of my
students.
Twitter: twitter.com/bbengfort
LinkedIn: linkedin.com/in/bbengfort
Github: github.com/bbengfort
Email: benjamin@bengfort.com

5. About the Teaching Assistant
Keshav Magge
● MS Computer Science from University of Houston
● Lead Data/Software Engineer at Cobrain Company in
Bethesda, MD
Python Programmer:
● Python developer for 7 years
● Plone/Zope for 2 years, Django for 5 years
● My work on Github: https://github.com/keshavmagge

6. About the Teaching Assistant
Keshav Magge
Reach out to me to talk about all
things python/data or just about life
Twitter: twitter.com/keshavmagge
LinkedIn: linkedin.com/pub/keshav-magge/12/a2a/324/
Github: github.com/keshavmagge
Email: keshav@keshavmagge.com

7. Building Data Products

8. Hilary Mason
A data product is a product that is
based on the combination of data
and algorithms.
”
“

10. Mike Loukides
A data application acquires its value from the
data itself, and creates more data as a result.
It’s not just an application with data; it’s a
data product. Data science enables the
creation of data products.
”
“

12. The Data Science Pipeline

13. Data Ingestion
Data Munging
and Wrangling
Computation and
Analyses
Modeling and
Application
Reporting and
Visualization

14. Data Ingestion
● There is a world of data out
there- how to get it? Web
crawlers, APIs, Sensors? Python
and other web scripting
languages are custom made for
this task.
● The real question is how can we
deal with such a giant volume
and velocity of data?
● Big Data and Data Science often
require ingestion specialists!

15. ● Warehousing the data means
storing the data in as raw a form
as possible.
● Extract, transform, and load
operations move data to
operational storage locations.
● Filtering, aggregation,
normalization and
denormalization all ensure data is
in a form it can be computed on.
● Annotated training sets must be
created for ML tasks.
Data Wrangling

16. ● Hypothesis driven computation
includes design and development
of predictive models.
● Many models have to be trained
or constrained into a
computational form like a Graph
database, and this is time
consuming.
● Other data products like indices,
relations, classifications, and
clusters may be computed.
Computation and Analyses

17. Modeling and Application
This is the part we’re most familiar with.
Supervised classification, Unsupervised
clustering - Bayes, Logistic Regression,
Decision Trees, and other models.
This is also where the money is.

18. ● Often overlooked, this part is
crucial, even if we have data
products.
● Humans recognize patterns
better than machines. Human
feedback is crucial in Active
Learning and remodeling (error
detection).
● Mashups and collaborations
generate more data- and
therefore more value!
Reporting and Visualization

19. Don’t forget feedback!
(Active Learning for Data
Products)

20. What we’re going to build today
SCIENCE BOOKCLUB!!
● A book club that chooses what to
read via a recommender system.
● Uses GoodReads data to ingest
and return feedback on books.
● Statistical model is a non-
negative matrix factorization
● Reporting using Jinja (almost a
web app)

21. Workflow
1. Setting up a Python skeleton
2. Creating and Running Tests
3. Wading in with a configuration
4. Ingestion with urllib and requests
5. Creating a command line admin with argparse
6. Wrangling with BeautifulSoup and SQLAlchemy
7. Modeling with numpy
8. Reporting with Jinja2

22. Octavo Architecture (really clear DSP)
requests.py
Ingestion
Module
Raw Data
Storage Computational
Data Storage
Wrangling
Module
BeautifulSou
p
SQLAlchemy
Recommender
Module
Numpy
Reporting
Module
Jinja2Matplotlib

23. requests.py
Octavo Architecture (really clear DSP)
requests.py
Ingestion
Module
Raw Data
Storage
Computational
Data Storage
Wrangling
Module
BeautifulSoup
SQLAlchemy
Recommender
Module
Numpy
Reporting
Module
Jinja2
Matplotlib

24. How to tackle this course ...

25. How to tackle this course ...
Lean into it- absorb as much as
possible, don’t worry about falling
behind - it will be in your head!
Then afterwards - lets all digest it
together (keep in touch)