This document provides an overview of the data science process and tools for a data science project. It discusses identifying important business questions to answer with data, extracting relevant data from sources, cleaning and sampling the data, analyzing samples to create models and check hypotheses, applying results to full data sets, visualizing findings, automating and deploying solutions, and continuously learning and improving through an iterative process. Key tools mentioned include Hadoop, R, Python, Excel, and various data wrangling, analysis, and visualization tools.

1. Data Scientist Toolbox
Andrei Savu - Axemblr.com
BigData.ro 2013

2. Me
• Founder of Axemblr.com
• Organizer of Bucharest JUG (bjug.ro)
• Passion for DevOps, Data Analysis
• Connect with me on LinkedIn

3. @ Axemblr
• Service Deployment Orchestration
• Infrastructure Automation (DevOps)
• Apache Hadoop On-Demand Appliance
• Axemblr Provisionr
https://github.com/axemblr/axemblr-provisionr

4. (Big)Data in a nutshell
• Business Intelligence / Research Evolved
• Significant change in Decision Making
• Enables new Products & Features
• Enables new Business Models

5. Data Scientist
• Has a Business / Research oriented
perspective
• Knowledge of statistics & software
engineering (AI, infrastructure)
• Ability to explore questions and formulate
hypotheses to be tested

6. Data Science Project
• Focused on particular business goals
• Based on a set of important questions
• Result > Answers that support business
decisions

7. The Algorithm
• Find *Important* • Create Pipelines
Questions
• Automate & Deploy
• Identify & Extract Data
• Learn & Repeat!
• Store & Sample
• Analyse
• Visualization

8. Start w/ “Big” Questions
... answer them with (Big)Data
How can we understand & improve the conversion rate?
How can we increase customer satisfaction?
How can we find important mentions in social media?

9. Identify Data Sources
OR add more probes / sensors as needed
Google Analytics,Web server logs, Mixpanel, Custom
application metrics, Mouse tracking, Facebook metrics etc.

10. Extract Data
... to a medium that allows you to run arbitrary queries
Local filesystem, Databases, Hadoop, HBase, HDFS, Hive, Pig

11. Extract
• Database dump tool, replicas or backups
• External web services
• Apache Sqoop (SQL-to-Hadoop)
• Implement pipelines / real-time streams
• Write custom tools as needed

12. Curate
Unfortunately Data is Messy

13. Curate - Your Way
• Use or develop tools / scripts
• On large volumes there no obvious choices
• Custom ways of filtering & aggregating large
streams (e.g. twitter, sensors)
• Reuse existing software components for
data curation / validation

14. DataWrangler
Interactive System for Data
cleaning a transformation
http://vis.stanford.edu/wrangler/

15. Open Refine
Former Google Refine
https://github.com/OpenRefine/
OpenRefine

16. Sample (time, etc.)
As needed to support interactive exploration

17. Why Sample?
• Interactive exploration to create and check
assumptions, to create algorithms
• Be careful with “Statistical Significance”
• Sample Smart: By time, By location etc.

18. Analyse Sample
This is were the fun begins

19. Analyse Sample
• Create models
• Create algorithms
• Check hypotheses
• Faster feedback loops & Immediate
Gratification

20. Excel-like

21. Python

22. RStudio

23. Gephi.org

24. Analyse All
apply your results to the entire data set

25. How to Analyse All?
• “Easy” on a single machine
• Go distributed w/ Hadoop, MPI, Storm,
Oracle Exa* etc.
• Key: Leverage existing tools
• Tools: sed, awkSQL, RHadoop, Apache
Hive, Pig, Cloudera Impala, MPI, Custom MR

26. Visualization
Communicate meaning w/ Graphics

27. http://selection.datavisualization.ch/

28. Automate & Deploy
Make it part of your internal dashboard

29. Learn & Repeat
Answer most of the time generate new questions

30. Thanks! Questions?
Andrei Savu / asavu@axemblr.com
@andreisavu