Algorithms and Feature Engineering

Algorithms
Poul Petersen @pejpgrep
CIO, BigML, Inc @bigmlcom
UI Algorithms & Feature Engineering with Flatline

BigML, Inc 2ML Crash Course - UI/Algorithms/Feature Engineering
BigML Algorithm History
2011
Prototyping and Beta
API-first Approach
2013
Evaluations, Batch
Predictions,
Ensembles, Sunburst
2015
Association
Discovery,
Correlations,
Samples, Statistical
Tests
2014
Anomaly Detection,
Clusters, Flatline
2016
Scripts, Libraries,
Executions,
WhizzML, Logistic
Regression
2012
Core ML workflow:
source, dataset,
model, prediction

The need for Machine Learning
• Can you find any pattern in this tiny data set?
Talk Text Purchases Data Age Churn?
148 72 0 33.6 50 TRUE
85 66 0 26.6 31 FALSE
183 64 0 23.3 32 TRUE
89 66 94 28.1 21 FALSE
115 0 0 35.3 29 FALSE
166 72 175 25.8 51 TRUE
100 0 0 30 32 TRUE
118 84 230 45.8 31 TRUE
171 110 240 45.4 54 TRUE
159 64 0 27.4 40 FALSE
…. but this is a simple example

Data Types
numeric
1 2 3
1, 2.0, 3, -5.4 categoricaltrue, yes, red, mammal categoricalcategorical
A B C
DATE-TIME2013-09-25 10:02
DATE-TIME
YEAR
MONTH
DAY-OF-MONTH
YYYY-MM-DD
DAY-OF-WEEK
HOUR
MINUTE
YYYY-MM-DD
YYYY-MM-DD
M-T-W-T-F-S-D
HH:MM:SS
HH:MM:SS
2013
September
25
Wednesday
10
02
text / items
Be not afraid of greatness:
some are born great, some
achieve greatness, and
some have greatness
thrust upon 'em.
text
“great”
“afraid”
“born”
“some”
appears 2 times
appears 1 time
appears 1 time
appears 2 times

Text Analysis
some are born great, some
achieve greatness, and
some have greatness
thrust upon 'em.
great: appears 4 times
Bag of Words

Text Analysis
… great afraid born achieve … …
… 4 1 1 1 … …
… … … … … … …
some are born great, some achieve
greatness, and some have greatness
thrust upon ‘em.
Model
The token “great”

occurs more than 3 times
The token “afraid”

occurs no more than once

DATASET
Evaluation
TRAIN SET
TEST SET
PREDICTIONS
METRICS

Ensembles
Diameter Color Shape Fruit
4 red round plum
5 red round apple
5 red round apple
6 red round plum
7 red round apple
Bagging!
Random Decision Forest!
All Data: “plum”
Sample 2: “apple”
Sample 3: “apple”
Sample 1: “plum”
}“apple”
What is a round, red 6cm fruit?

Logistic Regression

Logistic Regression
????

Logistic Regression
P≈0 P≈10<P<1
•x→-∞ : P(x)→0
•x→∞ : P(x)→1

Supervised Learning
animal state … proximity action
tiger hungry … close run
elephant happy … far take picture
Classiﬁcation
animal state … proximity min_kmh
tiger hungry … close 70
hippo angry … far 10
Regression
label
animal state … proximity action1 action2
tiger hungry … close run look untasty
elephant happy … far take picture call friends
Multi-Label Classiﬁcation

Unsupervised Learning
date customer account auth class zip amount
Mon Bob 3421 pin clothes 46140 135
Tue Bob 3421 sign food 46140 401
Tue Alice 2456 pin food 12222 234
Wed Sally 6788 pin gas 26339 94
Wed Bob 3421 pin tech 21350 2459
Wed Bob 3421 pin gas 46140 83
The Sally 6788 sign food 26339 51
Clustering
Anomaly Detection
similar
unusual

K-Means
K=3

G-Means

G-Means
Let K=2
Keep 1, Split 1
New K=3

G-Means
Let K=3
Keep 1, Split 2
New K=5

G-Means
Let K=5
K=5

Isolation Forest
Grow a random decision tree until
each instance is in its own leaf
“easy” to isolate
“hard” to isolate
Depth
Now repeat the process several times and
use average Depth to compute anomaly
score: 0 (similar) -> 1 (dissimilar)

Model Competence
MODEL
ANOMALY
DETECTOR
Prediction T T
Conﬁdence
86% 84%
Anomaly
Score
0.5367 0.7124
Competent? Y N
At Training Time At Prediction Time
DATASET

Association Rules
{class = gas} amount < 100
{customer = Bob, account = 3421} zip = 46140
Rules:
Antecedent Consequent

Association Metrics
Instances
A
C
Coverage
Percentage of instances
which match antecedent “A”

Association Metrics
Instances
A
C
Support
Percentage of instances
which match antecedent
“A” and Consequent “C”

Association Metrics
Coverage
Support
Instances
A
C
Conﬁdence
Percentage of instances in
the antecedent which also
contain the consequent.

Association Metrics
C
Instances
A
C
A
Instances
C
Instances
A
Instances
A
C
0% 100%
Instances
A
C
Conﬁdence
A never

implies C
A sometimes

implies C
A always

implies C

Association Metrics
Independent
A
C
C
Observed
A
Lift
Ratio of observed support
to support if A and C were
statistically independent.
Support == Conﬁdence
p(A) * p(C) p(C)

Association Metrics
C
Observed
A
Observed
A
C
< 1 > 1
Independent
A
C
Lift = 1
Negative
Correlation
No Association
Positive
Correlation
Independent
A
C
Independent
A
C
Observed
A
C

Association Metrics
Independent
A
C
C
Observed
A
Leverage
Diﬀerence of observed
support and support if A
and C were statistically
independent.

Support - [ p(A) * p(C) ]

Association Metrics
C
Observed
A
Observed
A
C
< 0 > 0
Independent
A
C
Leverage = 0
Negative
Correlation
No Association
Positive
Correlation
Independent
A
C
Independent
A
C
Observed
A
C
-1…

Machine Learning Secret
“…the largest improvements in accuracy often came from
quick experiments, feature engineering, and model tuning
rather than applying fundamentally different algorithms.”
Facebook FBLearner 2016
Feature Engineering: applying domain knowledge of
the data to create features that make machine
learning algorithms work better or at all.

Feature Engineering
2013-09-25 10:02
DATE-TIME
Automatic Date Transformation
… year month day hour minute …
… 2013 Sep 25 10 2 …
… … … … … … …
NUM NUMCAT NUM NUM

Feature Engineering
Automatic Categorical Transformation
… alchemy_category …
… business …
… recreation …
… health …
… … …
CAT
business health recreation …
… 1 0 0 …
… 0 0 1 …
… 0 1 0 …
… … … … …
NUM NUM NUM

Feature Engineering
some are born great, some achieve
greatness, and some have greatness
thrust upon ‘em.
TEXT
Automatic Text Transformation
… great afraid born achieve …
… 4 1 1 1 …
… … … … … …
NUM NUM NUM NUM

Feature Engineering
{
“url":"cbsnews",
"title":"Breaking News Headlines
Business Entertainment World News “,
"body":" news covering all the latest
breaking national and world news
headlines, including politics, sports,
entertainment, business and more.”
}
TEXT
Better representation
title body
Breaking News… news covering…
… …
TEXT TEXT

Feature Engineering
Discretization
Total Spend
7,342.99
304.12
4.56
345.87
8,546.32
NUM
“Predict will spend
$3,521 with error
$1,232”
Spend Category
Top 33%
Middle 33%
Bottom 33%
Middle 33%
Top 33%
CAT
“Predict customer
will be Top 33% in
spending”

Feature Engineering
Combinations of Multiple Features
Kg M2
101.4 3.24
85.2 2.8
56.2 2.9
136.1 3.6
95.9 4.1
NUM NUM
BMI
31.17
30.4
19.38
37.8
23.39
NUM
Kg
M2

Feature Engineering
Flatline
• BigML’s Domain-Specific Language (DSL) for
Transforming Datasets
• Limited programming language structures
• let, cond, if, maps, list operators, */+-
• Dataset Fields are first-class citizens
• (field “diabetes pedigree”)
• Built-in transformations
• statistics, strings, timestamps, windows

Feature Engineering
(/ (- ( f "price") (avg-window "price" -4, -1)) (standard-deviation "price"))
date volume price
1 34353 314
2 44455 315
3 22333 315
4 52322 321
5 28000 320
6 31254 319
7 56544 323
8 44331 324
9 81111 287
10 65422 294
11 59999 300
12 45556 302
13 19899 301
14 21453 302
day-4 day-3 day-2 day-1 4davg
0
314 314
314 315 314.5
314 315 315 314.6
314 315 315 321 316.25
315 315 321 320 317.75
315 321 320 319 318.75
Current - (4-day avg)
std dev
Shock: Deviations from a Trend

Feature Engineering
(/ (- (f "price") (avg-window "price" -4, -1)) (standard-deviation "price"))
Current - (4-day avg)
std dev
Shock: Deviations from a Trend
Current : (ﬁeld “price”)
4-day avg: (avg-window “price” -4 -1)
std dev: (standard-deviation “price”)

Feature Engineering
Fix Missing Values in a “Meaningful” Way
Filter Zeros
Model  
insulin
Predict  
insulin
Select  
insulin
Fixed 
Dataset
Amended 
Dataset
Original 
Dataset
Clean 
Dataset

Algorithms and Feature Engineering

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Algorithms and Feature Engineering

Similar to Algorithms and Feature Engineering (20)

More from BigML, Inc

More from BigML, Inc (20)

Recently uploaded

Recently uploaded (20)

Algorithms and Feature Engineering