From Developer to Data Scientist - Gaines Kergosien

@GAINESK
@ITCAMPRO #ITCAMP17Community Conference for IT Professionals
From Developer to Data Scientist
Gaines Kergosien
Executive Director, Music City Code
Associate Director, UBS

@GAINESK
Many thanks to our sponsors & partners!
GOLD
SILVER
PARTNERS
PLATINUM
POWERED BY

@GAINESK
GAINES KERGOSIEN
Leader, Speaker, Problem Solver

@GAINESK
• National gap for analytical expertise at 140k+ by 2017.
–McKinsey 2011
• Shortage of 100k Data Scientists by 2020. –Gartner 2012
• 90% of clients need expertise, 40% cite lack of talent.
–Accenture 2014
• Survey finds 83% of data scientists see shortage.
–Crowdflower 2016
• “I keep saying that the sexy job in the next 10 years will be
statisticians. And I’m not kidding.”
–Google’s Chief Economist
• Data Scientist the #1 job in America for 2016 AND 2017!
–GlassDoor
The Demand

@GAINESK
The Maturity
http://www.burtchworks.com/files/2016/04/Burtch-Works-Study_DS-2016-final.pdf

@GAINESK
The Salary

@GAINESK
The Industry

@GAINESK
A data scientist is a job title for an employee or
business intelligence (BI) consultant who excels at
analyzing data, particularly large amounts of data, to
help a business gain a competitive edge.
–WhatIs.com
The Definition

@GAINESK
The Definition

@GAINESK
The Recipe

@GAINESK
• Classification – Is this A or B?
• Anomaly Detection – Is this weird?
• Regression – How much -or- how many?
• Clustering – How is this organized?
• Reinforcement Learning – What should I do next?
The Five Questions

@GAINESK
• Educate the business
• Look for problems to solve
• Research new techniques
• Collate data for analysis (ETL)*
• Implement algorithms
• Design big data-capable architecture
• Present insights
The Job

@GAINESK
• Big Data
• Fast Data
• Dark Data
• Unstructured Data
• Data Mining
• Data Visualization
• Predictive Analytics
• [Deep] Neural Network
The Buzzwords

@GAINESK
Big Data
Volume

@GAINESK
The Volume

@GAINESK
Big Data
Volume Variety
• Records
• Transactions
• Tables & Files
• Structured
• Unstructured
• Semi-structured

@GAINESK
Unstructured Text
• Books
• Blog Posts
• Comments
• Tweets
• Photos
• Video
• Audio
The Variety

@GAINESK
Big Data
Volume Variety
Velocity
• Real Time
• Near Time
• Batch
• Streams
• Records
• Transactions
• Tables & Files
• Structured
• Unstructured
• Semi-structured

@GAINESK
The Velocity
Twitter
• 6,000 tweets per second
• 500 million tweets/day
Facebook
• 300 million photos/day
NY Stock Exchange
• captures 1TB of trade information each session

@GAINESK
Big Data
Big Data
Volume Variety
Velocity
• Real Time
• Near Time
• Batch
• Streams
• Records
• Transactions
• Tables & Files
• Structured
• Unstructured
• Semi-structured

@GAINESK
Data
• Define
• Collect
• Store
• Explore
The Breakdown
Science
• Hypothesis
• Plan Approach
• Analysis
• Report Results

@GAINESK
The Skills
Subject Matter Expertise
Statistics
• Choose Procedures
• Diagnose Problems
• Develop Procedures
Hacking Expertise
• Technical Skills
• Creativity
• Values
• Goals
• Constraints
Machine
Learning
Traditional
Research
Traditional
Software
Data
Science

@GAINESK
The Skills
Subject
Matter
Expertise
Hacking
Expertise
Social
Sciences
Statistics
Machine
Learning
Traditional
Software
Data
Science
Traditional
Research
Traditional
Research
Holistic
Research
Socially
Unaware
Domain
Unaware
Holistic
Software

@GAINESK
The Overlap
Data ScienceBig Data
Big
Data
Science
Big Data
Volume Variety
Velocity

@GAINESK
The Analysis Tools

@GAINESK
The Tool Trends
Python
KNIME
RapidMiner
R
SPSS
SAS
Hadoop

@GAINESK
• SQL
• Excel
• Python
• R
• MySQL
The Top Tools

@GAINESK
• SPSS
• Matlab
• Julia
• Kafka/Storm
• R
• Python
• Java/Scala
• Stata
• SAS
The Languages
http://www.kdnuggets.com/2015/05/r-vs-python-data-science.html

@GAINESK
The Languages – SAS, Phython or R?
http://www.burtchworks.com/2016/07/13/sas-r-python-survey-2016-tool-analytics-pros-prefer/

@GAINESK
The Languages – Trends

@GAINESK
The Languages – Industries

@GAINESK
The Languages – Education

@GAINESK
The Languages – Trends

@GAINESK
The Languages – The Future

@GAINESK
• R Statistical Programming Language
• Based on the S programming language
• R Development Environment
• Statistical and Visual Analysis
• Cross-Platform
• Free Open Source
• Active User Community
• Over 9,000 Extension Packages
The R

@GAINESK
• Created in 1991 to emphasize productivity and code
readability
• Easier learning curve than R
• Free Open Source
• Active User Community
The Python

@GAINESK
• Hadoop Distributed File System (HDFS)
• MapReduce vs. YARN
• Pig
• Hive
• Hbase
• Storm
• Spark
• etc.
The Hadoop Collective

@GAINESK
Sample the Data
• Random
• Stratified
Reconcile Missing Data
• Discard
• Infer
Normalize Numeric Values
• Standard Unit of Measure
• Subtract Average (Mean = 0)
• Divide by Standard Deviation
The Wrangling
Reduce Dimensionality
• Irrelevant Input Variables
• Redundant Input Variables
Add Derivative Values
• Generalize Attributes
• Discretize Attributes to Categories
• Binarize Categorical Attributes
Design Training Data
• Select
• Combine
• Aggregate
Power and Log transformation
• Approximate Normal Distribution

@GAINESK
• basic statistics (ie. p-value)
• statistical modeling
• statistical tests
• experiment design
• distributions
• maximum likelihood estimators
• probability theory
• linear algebra
• multivariable calculus
The Math

@GAINESK
• Tableau (enterprise visualization products) - www.tableau.com
• ggvis (R visualization package) - ggvis.rstudio.com
• ggplot (plotting system) - ggplot.yhathq.com
• D3.js (declarative DOM manipulation) - d3js.org
• Vega (visualization grammar)- trifacta.github.com/vega
• Rickshaw (charting library - code.shutterstock.com/rickshaw
• modest maps (map library) - modestmaps.com
• Chart.js (plotting library) - www.chartjs.org
The Visualization Tools

@GAINESK
Concepts
• k-nearest neighbors
• random forests
• ensemble methods
• …use Python libraries!
Tools
• Weka - www.cs.waikato.ac.nz/ml/weka/
The Machine Learning

@GAINESK
• Report
• Presentation
• Demo
• Prototype
• Component
The Results

@GAINESK
• Data Analyst (A)
• Data Engineer (B)
• Academic (Ab)
• Generalist (AB)
The Skills

@GAINESK
The Expertise

@GAINESK
The Degree

@GAINESK
The Degree – Trending

@GAINESK
1. Fundamentals
2. Statistics
3. Programming
4. ML
5. Text Mining
6. Visualization
7. Big Data
8. Data Munging
9. Toolbox
The Path

@GAINESK
1. Matrices & Linear Algebra
2. Hash Functions, Binary Tree
3. Relational Algebra, DB Basics
4. Inner, Outer, Cross, Theta Join
5. Cap Theorem
6. Tabular Data
7. Data Frames & Series
8. Sharding
9. OLAP
The Fundamentals

@GAINESK
10. Multidimensional Data Model
11. ETL
12. Reporting vs BI vs Analytics
13. JSON & XML
14. NoSQL
15. Regex
16. Vendor Landscape
17. Environment Setup
The Fundamentals

@GAINESK
1. Pick a Dataset
2. Descriptive Statistics
3. Exploratory Data Analysis
4. Histograms
5. Percentiles and Outliers
6. Probability Theorem
7. Bayes Theorem
8. Random Variables
9. Cumul Dist Fn (CDF)
The Statistics

@GAINESK
The Statistics
10. Continuous Distr.
11. Skewness
12. ANOVA
13. Prob Den Fn (PDF)
14. Cenral Limit Theorem
15. Monte Carlo Method
16. Hypothesis Training
17. p-Value
…

@GAINESK
1. Python Basics
2. Working in Excel
3. R Setup / R Studio
4. R Basics
5. Expressions
6. Variables
7. Vectors
8. Matrices
9. Arrays
The Programming

@GAINESK
10. Factors
11. Lists
12. Data Frames
13. Reading CSV Data
14. Reading Raw Data
15. Subsetting Data
16. Manipulate Data Frames
17. Functions
18. Factor Analysis
19. Install Packages
The Programming

@GAINESK
• Coursera - www.coursera.org
• EdX- www.edx.org
• Udacity - www.udacity.com
• Kaggle - www.kaggle.com
• Youtube - projects.iq.harvard.edu/stat110/youtube
• Boot Camps
The Training

@GAINESK
Q & A
Slides at DotNetDude.net
Subject
Matter
Expertise
Hacking
Expertise
Social
Sciences
Statistics
Machine
Learning
Traditional
Software
Data
Science
Traditional
Research
Traditional
Research
Holistic
Research
Socially
Unaware
Domain
Unaware
Holistic
Software
Big Data
Volume Variety
Velocity
Data ScienceBig Data
Big
Data
Science

From Developer to Data Scientist - Gaines Kergosien

More Related Content

What's hot

Viewers also liked

Similar to From Developer to Data Scientist - Gaines Kergosien

More from ITCamp

Recently uploaded

From Developer to Data Scientist - Gaines Kergosien