The document summarizes a research paper on Deep Crossing, a deep learning model that automatically combines features for web-scale modeling without manually crafted combinatorial features. The key points are: 1. Deep Crossing uses a neural network to automatically learn combinatorial features from individual features, avoiding the manual feature engineering required by previous models. 2. It was shown to outperform previous models like DSSM that used late feature crossing. Deep Crossing's early feature crossing was more effective. 3. Deep Crossing was able to achieve better performance than production models using much less training data, and is easier to build and maintain than manually engineered models.

1. 1
Reading Group
Deep Crossing: Web-Scale Modeling
without Manually Crafted
Combinatorial Features
Presenter:Xiang Zhang
6/03/2017

2. 6/03/2017
1.Background
2.Abstract
3.Why choose it
4.what’s the main idea
5.How it works
6.Implementation
7.Expermentation
8.Conclusions
9.Experience
Main Content
2

3. 3
1. Background
6/03/2017
Deep Crossing: Web-Scale Modeling without
Manually Crafted Combinatorial Features
 Authors from Microsoft Research
 Keywords Neural Networks, DL, CNN, etc.
 Conference KDD 2016

4. 4
1. Background
6/03/2017
KDD (A*) Conferences on Knowledge Discovery
and Data mining
SIGKDD Special Interest Group on Knowledge
Discovery in Data
KDD 2017 Aug.13-17, Halifax, Canada
December 9, 2016 (DDL)

5. 5
2.Abstract
 Combine features
Important and useful, but manually craft is time consuming
and experience needed, accuracy unguaranteed especially in
large scale, variety and volume of features
 Contribution
Deep neural network to automatically combine features
 Tool
Computational network Tool Kit, multi-GPU platform
6/03/2017

6. 6
3. Why choose it
 Automatically combine features (reduce D)
 Web-scale (Massive data)
 Feature in different types & Dimensions
 Better performance
6/03/2017

7. 7
4.what’s the main idea
What is feature extraction?
 Individual features
An individual measurable property of a phenomenon being
observed. (Representation of the data)
 Combinatorial features
Defined in the joint space of individual features, to make the
model shorter training time, simpler, better generalization.
6/03/2017

8. 6/03/2017
 Manually:
• Time
• Experience
 Automatically
4.what’s the main idea

9. 9
5.How it works
6/03/2017
Model
Architecture

10. 6/03/2017
max(0, )O I
j j j jX W X b 
( )j jm n ( 1)jn 
( 1)jm 

 ( 1)jm 
( 1)jm  ( 1)jm 
( 1)jm 
( 1)jn 
Reduce Dj jm n
5.How it works
5.1 Embedding layers

11. 6/03/2017
(0, )maxO I
j j j jX W X b 
5.1 Embedding layers
 Rectified linear unit (ReLU)
• Elements non-negative
 Activation function: a node
defines the output of that
node given an input
• ReLU
• Logistic
• Tanh
• Sigmoid
5.How it works

12. 12
5.How it works
6/03/2017
0 1, , ,O O O O
KX X X X   L
5.2 stacking layers

13. 6/03/2017
0 1, , ,O O O O
KX X X X   L
256, set 256
embedding & stacking
j jn m 
256,
stacking without embedding
jn 
Feature Number
Inputs K
Embedding then stacking n
Stacking(non-embedding) K-n
Stacking all K
5.2 stacking layers
Stacking rules:
Threshold: 256
5.How it works

14. 6/03/2017
   0 1 0 1( , , , , )OR IR IR
F W W B BX X X 
𝑋 𝐼𝑅
5.3 Residual layers
• Inputs and outputs have the same size
• Residual Unit is first used beyond image
recognition
5.How it works

15. 15
5.How it works
6/03/2017
5.4 Scoring layers
Sigmoid function:
𝑋 𝐼𝑅

16. 16
5.How it works
6/03/2017
1
1
log ( log( ) (1 )log(1 ))
N
i i i i
i
loss y p y p
N 
    
5.5 Objective function
Objective function: loss function or its negative
N: No. of samples
: sample label
: output of Model(predict)
𝑋 𝐼𝑅
𝒚𝒊
𝒑𝒊

17. 6/03/2017
5.5 Early Crossing vs. Late Crossing
Deep Crossing DSSM
(Deep Semantic Similarity Model)
5.How it works

18. 18
6.Implementation
6/03/2017
 Software
Computational Network Toolkit (CNKT)
Same theoretical foundation with Tensorflow
 Hardware
Multi-GPU platform
24 days (1 GPS) to 20 hours (32 GPUs)

19. 6/03/2017
7.Experimentation
7.1 Dataset

20. 6/03/2017
7.Experimentation
7.2 Performance on a Pair of Text Inputs
 DSSM: late crossing
 DP: early crossing
 DC>DSSM

21. 21
7.Experimentation
6 March, 2017
7.2 Performance on a Pair of Text Inputs
 Production Model: one model can be used in sponsored
search as baseline
 DSSM < DC< Production
 DC Main advantage: deal with many individual features

22. 22
7.Experimentation
6 March, 2017
7.3 Beyond Text Input
 All features works best

23. 23
7.Experimentation
6 March, 2017
7.3 Beyond Text Input
 Only counting feature is weak

24. 24
7.Experimentation
6 March, 2017
7.3 Beyond Text Input
 Counting feature is useful

25. 25
7.Experimentation
6 March, 2017
7.3 Beyond Text Input
 Performance changes a lot as features number
changes; log loss suffers a big fluctuation with
different feature combination. So that feature
selection is meaningful.

26. 26
7.Experimentation
6 March, 2017
7.4 Comparison with Production Models
 2.2 billion samples
 DC perform better with much less dataset
 DC is easier to build and maintain

27. 27
8.Conclusions
 Deep Crossing work well in automatically
feature combinatorial in large scale
 Need less time and experience
6/03/2017

28. 28
9.Experience
6/03/2017
• Deep learning (LSTM, CNN, etc.) can extract feature
automatically, we can compare the efficient of them
with this model
• we can use the raw data instead the individual
features to train
• In different domains such mobile sensing,
recommender system

29. 296/03/2017
Thanks！

30. 306/03/2017
Questions?