Machine learning derives mathematical models from raw data. In the model building process, raw data is first processed into "features," then the features are given to algorithms to train a model. The process of turning raw data into features is sometimes called feature engineering, and it is a crucial step in model building. Good features lead to successful models with a lot of predictive power; bad features lead to a lot of headache and nowhere. This talk aims to help the audience understand what is a feature space and why it is so important. We will go through some common feature space representations of English text and discuss what tasks they are suited for and why. Expect lots of pictures, whiteboard drawings and handwaving. We will exercise our power of imagination to visualize high dimensional feature spaces in our mind's eye. Presented by Alice Zheng Director of Data Science at Dato.

1. #datapopupseattle
Understanding Feature Space
in Machine Learning
Alice Zheng
Director of Data Science, Dato
RainyData Datoinc

2. #datapopupseattle
UNSTRUCTURED
Data Science POP-UP in Seattle
www.dominodatalab.com
D
Produced by Domino Data Lab
Domino’s enterprise data science platform is used
by leading analytical organizations to increase
productivity, enable collaboration, and publish
models into production faster.

3. Understanding Feature Space
in Machine Learning
Alice Zheng, Dato
October, 2015
3

4. ‹#›
My journey so far
Applied+machine+learning+
(Data+science)
Build+ML+tools
Shortage+of+experts+
and+good+tools.

5. ‹#›
Why machine learning?
Model data.
Make predictions.
Build intelligent
applications.

6. ‹#›
The machine learning pipeline
I+fell+in+love+the+instant+I+laid+
my+eyes+on+that+puppy.+His+
big+eyes+and+playful+tail,+his+
soft+furry+paws,+…
Raw+data
Features
Models
Predictions
Deploy+in+
production

7. ‹#›
Three things to know about ML
• Feature = numeric representation of raw data
• Model = mathematical “summary” of features
• Making something that works = choose the right model
and features, given data and task

8. Feature = Numeric representation of raw data

9. ‹#›
Representing natural text
It#is#a#puppy#and#it#is#
extremely#cute.
What’s+important?+
Phrases?+Specific+
words?+Ordering?+
Subject,+object,+verb?
Classify:++
puppy+or+not?
Raw+Text
{“it”:2,++
+“is”:2,++
+“a”:1,++
+“puppy”:1,++
+“and”:1,+
+“extremely”:1,+
+“cute”:1+}
Bag+of+Words

10. ‹#›
Representing natural text
It#is#a#puppy#and#it#is#
extremely#cute.
Classify:++
puppy+or+not?
Raw+Text Bag+of+Words
it 2
they 0
I 0
am 0
how 0
puppy 1
and 1
cat 0
aardvark 0
cute 1
extremely 1
… …
Sparse+vector+
representation

11. ‹#›
Representing images
Image+source:+“Recognizing+and+learning+object+categories,”++
Li+Fei[Fei,+Rob+Fergus,+Anthony+Torralba,+ICCV+2005—2009.
Raw+image:++
millions+of+RGB+triplets,+
one+for+each+pixel
Classify:++
person+or+animal?
Raw+Image Bag+of+Visual+Words

12. ‹#›
Representing images
Classify:++
person+or+animal?
Raw+Image Deep+learning+features
3.29+
[15+
[5.24+
48.3+
1.36+
47.1+
[1.92
36.5+
2.83+
95.4+
[19+
[89+
5.09+
37.8
Dense+vector+
representation

13. ‹#›
Feature space in machine learning
• Raw data ! high dimensional vectors
• Collection of data points ! point cloud in feature space
• Feature engineering = creating features of the appropriate
granularity for the task

14. Visualizing Feature Space

15. Crudely speaking, mathematicians fall into two categories:
the algebraists, who find it easiest to reduce all problems
to sets of numbers and variables, and the geometers, who
understand the world through shapes.
-- Masha Gessen, “Perfect Rigor”

16. ‹#›
Visualizing bag-of-words
puppy
cute
1
1
I+have+a+puppy+and+
it+is+extremely+cute
I#have#a#puppy#and#
it#is#extremely#cute
it 1
they 0
I 1
am 0
how 0
puppy 1
and 1
cat 0
aardvark 0
zebra 0
cute 1
extremely 1
… …

17. ‹#›
Visualizing bag-of-words
puppy
cute
1
1
1
extremely
I+have+a+puppy+and++
it+is+extremely+cute
I+have+an+extremely++
cute+cat
I+have+a+cute++
puppy

18. ‹#›
Document point cloud
word+1
word+2

19. Model = Mathematical “summary” of features

20. ‹#›
What is a summary?
• Data ! point cloud in feature space
• Model = a geometric shape that best “fits” the point cloud

21. ‹#›
Classification model
Feature+2
Feature+1
Decide+between+two+classes

22. ‹#›
Clustering model
Feature+2
Feature+1
Group+data+points+tightly

23. ‹#›
Regression model
Target
Feature
Fit+the+target+values

24. Visualizing Feature Engineering

25. ‹#›
When does bag-of-words fail?
puppy
cat
2
1
1
have
I+have+a+puppy
I+have+a+cat
I+have+a+kitten
Task:+find+a+surface+that+separates++
documents+about+dogs+vs.+cats
Problem:+the+word+“have”+adds+fluff++
instead+of+information
I+have+a+dog+
and+I+have+a+pen
1

26. ‹#›
Improving on bag-of-words
• Idea: “normalize” word counts so that popular words are
discounted
• Term frequency (tf) = Number of times a terms appears in a
document
• Inverse document frequency of word (idf) =
• N = total number of documents
• Tf-idf count = tf x idf

27. ‹#›
From BOW to tf-idf
puppy
cat
2
1
1
have
I+have+a+puppy
I+have+a+cat
I+have+a+kitten
idf(puppy)+=+log+4+
idf(cat)+=+log+4+
idf(have)+=+log+1+=+0
I+have+a+dog+
and+I+have+a+pen
1

28. ‹#›
From BOW to tf-idf
puppy
cat1
have
tfidf(puppy)+=+log+4+
tfidf(cat)+=+log+4+
tfidf(have)+=+0
I+have+a+dog+
and+I+have+a+pen,+
I+have+a+kitten
1
log+4
log+4
I+have+a+cat
I+have+a+puppy
Decision+surface
Tf[idf+flattens+
uninformative+
dimensions+in+the+
BOW+point+cloud

29. ‹#›
That’s not all, folks!
• Geometry is the key to understanding feature space and
machine learning
• Many other fun topics:
- Feature normalization
- Feature transformations
- Model regularization
• Dato is hiring! jobs@dato.com
@RainyData,+@DatoInc

30. #datapopupseattle
@datapopup
#datapopupseattle

31. #datapopupseattle
Thank You To Our Sponsors