Marios, former world #1 Kaggle Grandmaster and the creator of KazAnova, presented StackNet Meta-Modelling framework.

1. Introducing StackNet Meta-Modelling Framework
Marios Michaildis
Research Data Scientist at
Email: marios@h2o.ai

2. What is StackNet (methodology)?
• StackNet is…
A scalable meta modelling methodology that
utilizes Wolpert’s stacked generalization (1992) of
combining multiple models assuming
a feedforward neural network architecture of multiple
levels
Each node represents a machine learning algorithm
A version of it with several algorithms is available in
Java
Supervisors:
Prof. Philip Treleaven
Giles Pavey

3. Why bother learning more about StackNet?
• It helps to improve predictions given the same input data
• Its is educational in its own way, especially in
understanding Stacking.
• Compiles the pinnacle of machine learning into one
framework-and-library.
• Has won 2 kaggle competitions (link A and Link B)
• Has helped many people get top 10 results in kaggle.
• It has helped me become kaggle #1

4. Why meta modelling?

5. Inspiration - Stacking
Wolpert in 1992 introduced stacking – a Meta-modelling
technique.
1. Split the training set into two disjoint sets.
2. Train several base learners on the first part.
3. Make predictions with the base learners on the second
part.
4. Using the predictions from (3) as the inputs, and the
correct responses as the outputs, train a higher level
learner.

6. Still confused about Stacking?
X0 x1 x2 xn y
0.17 0.25 0.93 0.79 1
0.35 0.61 0.93 0.57 0
0.44 0.59 0.56 0.46 0
0.37 0.43 0.74 0.28 1
0.96 0.07 0.57 0.01 1
A
X0 x1 x2 xn y
0.89 0.72 0.50 0.66 0
0.58 0.71 0.92 0.27 1
0.10 0.35 0.27 0.37 0
0.47 0.68 0.30 0.98 0
0.39 0.53 0.59 0.18 1
B
X0 x1 x2 xn y
0.29 0.77 0.05 0.09 ?
0.38 0.66 0.42 0.91 ?
0.72 0.66 0.92 0.11 ?
0.70 0.37 0.91 0.17 ?
0.59 0.98 0.93 0.65 ?
C
pred0 pred1 pred2 y
0.24 0.72 0.70 0
0.95 0.25 0.22 1
0.64 0.80 0.96 0
0.89 0.58 0.52 0
0.11 0.20 0.93 1
B1
pred0 pred1 pred2 y
0.50 0.50 0.39 ?
0.62 0.59 0.46 ?
0.22 0.31 0.54 ?
0.90 0.47 0.09 ?
0.20 0.09 0.61 ?
C1
Train algorithm 0 on A and make predictions for B and C and save to B1, C1
Train algorithm 1 on A and make predictions for B and C and save to B1, C1
Train algorithm 2 on A and make predictions for B and C and save to B1, C1
Train algorithm 3 on B1 and make predictions for C1
Preds3
0.45
0.23
0.99
0.34
0.05
Consider datasets A,B,C. Target variable (y) is known for A,B

7. Inspiration – Neural Networks
• Artificial networks were first created in an attempt to mimic
the biological neural networks in the human Brain. [
Rosenblatt ,1958] was the first to create – the perceptron.
• The advances in computing power and specifically the
usages of GPUs has allowed them to be run at greater
speeds in comlex structures taking the form of today’s deep
learning [Schmidhuber, 2015] .
• Their structure is considered state-of-the-art for many tasks

8. Inspiration – Why Java
• is less verbose than C and very popular
• Can be used in any operational system
• Almost every computer/device has it by default
• Statically typed and better defined
• Java Does not have Scikit-learn!

9. Available Algorithms
1st batch of models includes
• Linear Regression
• Logistic regression
• Kernel models
• K nearest neighbours
• GBMs
• Naïve Bayes
• LibFm
• Multilayer Perceptron
• Decision trees
• Random Forests
2nd batch of models includes
• H2O
• Xgboost
• LIGHTgbm
• Sklearn
• Keras
• Fast_Rgf

10. Howit works - General
• In a neural network , every node is a simple linear model (like
linear regression) maybe with some non linear
transformation.
• Instead of a linear model , StackNet proposes any modelling
function.
• In other words:

11. • Limited data based on which multiple models must be built
on , enhances the notion of a re-usable holdout
• It uses stratified k-folding – which is a hyper parameter.
Training – Reusable Holdout

12. Training - Modes
• The training process is a straight one-pass. There is no
notion of re-optimizing in multiple epochs. Convergence
needs to reached within that 1 epoch.

13. Command Line parameters
Command Explanation
sparse True if the data to be imported are in sparse format (libsvm)
has_head True if train_file and test_file have headers else false
model Name of the output model file.
pred_file Name of the output prediction file.
train_file Name of the training file.
test_file Name of the test file.
test_target True if the test file has a target variable in the beginning
params Parameter file where each line is a model.
verbose True if we need StackNet to output its progress else false
threads Number of models to run in parallel.
metric Logloss, Rmse, accuracy or auc (for binary only)
stackdata True for restacking else false
seed Integer for randomised procedures
folds Number of folds for re-usable kfold
Sample Parameter’s File
LogisticRegression Type:Liblinear C:2.0 threads:1 usescale:True
GradientBoostingForestClassifier estimators:300 shrinkage:0.10 max_depth:6 max_features:0.5
RandomForestClassifier estimators:300 threads:5 max_depth:16 max_features:0.25
RandomForestClassifier estimators:1500 max_depth:7 max_features:0.2 min_leaf:1.0
Java –jar stacknet.jar
train task=classification
sparse=false
model=model.mod
pred_file=pred.csv
train_file=sample_train.csv
test_file=sample_test.csv
params=params.txt
verbose=true
threads=3
metric=logloss
Target variable Input data

14. Top 10 example Using StackNet for
amazon classification challenge
• Popular competition - kaggle in 2013 (my first competition).
• Only 9 columns (8 unique variables and 1 duplicate).
• high cardinality – thousands of unique values.
• 90K rows combined for train and test.
• Scope: determine an employee's access needs.
• Metric to optimize was AUC(or Area Under Curve).
• competition: https://www.kaggle.com/c/amazon-employee-access-challenge
• Tutorial: https://github.com/kaz-Anova/StackNet/blob/master/example/example_amazon/EXAMPLE.MD

15. Parameters’ File
• Many models
• Diverse models
• At least one representative of common model families. Model families
defined as:
– Linear models, Radnom Forests, Gbms, Factorizations, svms, nns
• Having good (hyper)parameters for each model

16. LogisticRegression_L2
LogisticRegression_L2_SGD
LSVC_L2
LinearRegression
LibFmClassifier
softmaxnnclassifier
GradientBoostingClassifier
LogisticRegression_L1
Random Forest
Results in Graphical format
Best AUC : 0.893
AUC : 0.901 (+0.08)
LSVC_L1
0.893
0.885
0.891
0.879
0.891
0.882
0.851
0.88
0.871

17. Finding good parameters vol 1
RandomForestClassifier estimators:100 max_depth:5
RandomForestClassifier estimators:100 max_depth:6
GradientBoostingForestClassifier estimators:100
LogisticRegression C:0.5

18. Finding good parameters vol 2
• How to know which parameters to tune?
• Fin list of current available algorithms:
https://github.com/kaz-Anova/StackNet#algorithms-contained
• Click on the name or go to https://github.com/kaz-
Anova/StackNet/blob/master/parameters/PARAMETERS.MD#[Your_estimators_name]
• For example for deep learning try: https://github.com/kaz-
Anova/StackNet/blob/master/parameters/PARAMETERS.MD#h2odeeplearningclassifier
• There you can find a statement for StackNet as
well as the most important parameters.

19. Important elements for StackNet
• Having diverse models
• Having good (hyper)parameters for each
model
• Having good features
• Avoid temporal elements
• Avoid small data – StackNet is a Big Data tool

20. Useful Links and resources
• Github repository: https://github.com/kaz-Anova/StackNet .
• Facebook page : https://www.facebook.com/StackNet/ .
• Search “StackNet examples” on google for various resources.
• General blog about StackNet .
• General information on Stacking with H2O .
• Blog on StackNet winning a kaggle challenge .

21. Any questions