Random Decision Forests at Scale
•
2 likes•772 views
This document discusses random decision forests and how they work at scale. It provides an overview of decision trees and random decision forests. Individual decision trees are prone to overfitting, but a random decision forest addresses this by growing many decision trees on randomly selected subsets of the data and features. This results in greater accuracy compared to a single decision tree by averaging the predictions of the ensemble. The document demonstrates how to build random decision forests using Spark MLlib and discusses hyperparameters like the number of trees and feature subset strategy that can be tuned.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Random Decision Forests at Scale
Similar to Random Decision Forests at Scale (20)
More from Cloudera, Inc.
More from Cloudera, Inc. (20)
Recently uploaded
Recently uploaded (20)
Random Decision Forests at Scale
- 1. 1© Cloudera, Inc. All rights reserved. Random Decision Forests - at Scale Todd M. Boetticher| Solution Consultant
- 2. 2© Cloudera, Inc. All rights reserved. Overview • Decision trees • Introduction to decision tree’s • Building a decision tree with Spark • Tuning your model with Hyerparameters • Random Decision Forests • Introduction to Random Decision Forests • Handicapping individual learners • Deploying random decision trees in Spark
- 3. 3© Cloudera, Inc. All rights reserved. Decision Tree The Decision tree is one of the most commonly used classification techniques. According to a recent survey, the decision tree is the most common technique used today. A decision tree is a collection of outcomes that eventually lead to a decision. Like the game 21 questions, decision tree’s use features to split the data into subsets that will give you the best results.
- 4. 4© Cloudera, Inc. All rights reserved. Benefits and disadvantages of Decision Trees Pros • Computationally cheap to use • Easy for humans to understand learned results • Missing values OK • Capable of dealing with irrelevant feature Cons • Prone to overfitting • Lacks the performance available through other methods • Small changes in the Data can have enormous impacts on the data
- 5. 5© Cloudera, Inc. All rights reserved. Iris Dataset 6.4, 3.2, 4.5, 1.5, Iris-versacolor
- 6. 6© Cloudera, Inc. All rights reserved. Possible Decision Trees
- 7. 7© Cloudera, Inc. All rights reserved. Interpreting Models Simple graphic explanation of how the feature space can be divided into decision boundaries.
- 8. 8© Cloudera, Inc. All rights reserved. Decision Breakdown def visualize_tree(tree, feature_names): tree -- scikit-learn DecsisionTree. feature_names -- list of feature names. """ with open("dt.dot", 'w') as f: export_graphviz(tree, out_file=f, feature_names=feature_names) command = ["dot", "-Tpng", "dt.dot", "-o", "dt.png"] try: subprocess.check_call(command) except: exit("Could not run dot, ie graphviz, to " "produce visualization")
- 9. 9© Cloudera, Inc. All rights reserved. Building a Decision Tree
- 10. 10© Cloudera, Inc. All rights reserved. Name and Address Matching • Potential uses in every industry • Marketing • Defense and law enforcement • Bank Secrecy Act and Patriot Act Compliance RN-KOMSOMOLSKY LLC, 352 DeerPath Ave SW, Leesburg, Virginia, United States KOMSOMOLSKY REFINERY, 1621 Parkcrest Cir, Reston, Virginia, United States VS
- 11. 11© Cloudera, Inc. All rights reserved. 0.78, 0.0, 0.0, 0.78, 0.78, True Positive Name and Address Matching Name Address City State Country 0.81 0.0 0.0 0.78 1.0 0.6 0.6 0.2 0.0 0.0 0.2 0.0 0.0 0.0 0.0 0.91 0.91 0.91 0.91 1.0 0.78 0.0 0.0 0.78 0.78 0.4 0.4 0.36 0.0 1.0 1.0 0.0 0.0 0.0 0.0 Hits True-Positive False-Positive False-Positive True-Positive True-Positive False-Positive False-Positive
- 12. 12© Cloudera, Inc. All rights reserved. Building our first decision tree in MLlib
- 13. 13© Cloudera, Inc. All rights reserved. Evaluating a decision Tree .82243123866534172 ~82.24% accuracy
- 14. 14© Cloudera, Inc. All rights reserved. Benchmark vs Random .2718652198532764 ~27.19% accuracy
- 15. 15© Cloudera, Inc. All rights reserved. Hyperparameters Impurity: Gini, Impurity, Variance Maximum Depth Maximum Bin Measures the expected value of the information. The calculation of impurity is generally computed with Gini impurity or entropy Measures the expected value of the information. The calculation of impurity is generally computed with Gini impurity or entropy Measures the expected value of the information. The calculation of impurity is generally computed with Gini impurity or entropy trainClassifier(trainingData, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)
- 16. 16© Cloudera, Inc. All rights reserved. Impurity: Gini, Impurity, Variance 1 − 𝑃𝑖2H= - 𝑖=1 𝑛 𝑃(𝑥𝑖)𝑙𝑜𝑔2 𝑃(𝑥𝑖) Gini Impurity Entropy Entropy is defined as the expected value of information. First, we need to define information. If you’re classifying something that can take on multiple values, the information for symbol 𝑥 𝑖 is defined as Gini impurity is a measure of how often a randomly chosen element from the set would be incorrectly labeled if it were randomly labeled according to the distribution of labels in the subset 1 𝑛 𝑖=1 𝑛 (𝑦𝑖−𝜇)2 Variance vi is label for an instance, N is the number of instances and μ is the mean given by 1N∑Ni=1xi1N∑i=1Nxi.
- 17. 17© Cloudera, Inc. All rights reserved. Maximum Depth • Maximum tree depth is a limit to stop further splitting of nodes when the specified tree depth has been reached during the building of the initial decision tree. • The absolute maximum depth would be N−1, where N is the number of training samples. You can derive this by considering that the least effective split would be peeling off one training example per node. 1. 2. 3.
- 18. 18© Cloudera, Inc. All rights reserved. Decision Trees to Random Decision Forests
- 19. 19© Cloudera, Inc. All rights reserved. Wisdom of the crowds • The wisdom of the crowd is the collective opinion of a group of individuals rather than that of a single expert. • A large group's aggregated answers to questions involving quantity estimation, general world knowledge, and spatial reasoning has generally been found to be as good as, and often better than, the answer given by any of the individuals within the group. An explanation for this phenomenon is that there is idiosyncratic noise associated with each individual judgment, and taking the average over a large number of responses will go some way toward canceling the effect of this noise.
- 20. 20© Cloudera, Inc. All rights reserved. What is Random Forest Random Forests grows many classification trees. To classify a new object from an input vector, put the input vector down each of the trees in the forest. Each tree gives a classification, and we say the tree "votes" for that class. The forest chooses the classification having the most votes (over all the trees in the forest).
- 21. 21© Cloudera, Inc. All rights reserved. How to Create a Crowd?
- 22. 22© Cloudera, Inc. All rights reserved. Random Decision Forests with Spark MLlib RandomForest.trainClassifier(trainingData, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins) Number of Trees Feature Subset Strategy Number of trees in the forest. Increasing the number of trees will decrease the variance in predictions, improving the model’s test-time accuracy. Training time increases roughly linearly in the number of trees. Number of features to use as candidates for splitting at each tree node. The number is specified as a fraction or function of the total number of features. Decreasing this number will speed up training, but can sometimes impact performance if too low. ~ 98.67 % Accuracy
- 23. 23© Cloudera, Inc. All rights reserved. Trees See Subsets of Examples
- 24. 24© Cloudera, Inc. All rights reserved. Or Subsets of Features
- 25. 25© Cloudera, Inc. All rights reserved. Thank you tboetticher@cloudera.com
Editor's Notes
- Entropy: Information Theory Concept – Claude Shannon Measures mixed-ness, unpredictability of population Lower is better Gini: 1 minus probability that random guess I (probability pi) is correct Lower is better
- Table of Contents Introduction 1 Why should I know this? 2 Concept – Example 2 Describe the Iris Data Set 2 Explain what machine learning has to do with the iris data set 2 Physical representation of the decision tree concept 2 Interpreting the Model 2 Building Decision Trees 2 Introduce data set 2 Implement a Decision tree with Data Set 2 Explain how it worked 2 Split the data 2 Calculate Entropy 2 Cache the training data 2 Show current accuracy 2 Benchmark against Random 2 Confusion Matrix 2 Tuning the model (Hyper parameter) 2 Impurity: 2 Maximum Depth: 3 Maximim Bins 3 Try multiple parameters 3 Can the data be improved? 3 Random Decision Forests 3 Wisdom of the crowd 3 Introduce concept of Decision Forests 3 How to create Diversity of opinion 3 Handicap each tree 3 Subset of features 3 Aggregating Results 3 How to implement that in Spark 3 Introduction - Good Evening, Today I would like to talk about Random Decision forests and how we can implement them at scale using the Hadoop Ecosystem. I believe that this is important discussion due to the method’s popularity, and its potential impact for most organizations. Let’s Begin Concept – Example Before we can begin talking about Random Decision forests, I believe that it is important that we begin by discussing decision tree’s. If you have ever had the opportunity to play “21 questions” it is a game where an individual picks an object, or place and the competing individual slowly asks binary questions, attempting to shrink the possibilities for the object. This often begins by the person asking if it’s a person or a thing? Generally, these questions are mutually exclusive and collectively exhausted. The inquisitive individual would then ask a follow on question in order to lower the number of possibilities again and again until they know the answer or have run out of available questions. This is a perfect example of a decision tree. The decision tree is one of the most commonly used classification techniques and according to a recent survey, the technique is among most commonly implemented technique today. The utilization of this technique ranges from agriculture to physics. In physics decision trees have been used for the detection of physical particles, or classification of particle signatures. In other sectors, Decision trees have even been used to build personal learning assistant and classification of sleep patterns. Due to decisions tree’s ability to be utilized in both classification and regression, the possibilities are almost endless. Pro’s vs Cons Pro’s - The decision tree is used in so many problems due to its many strengths and short list of disadvantages. One of the key benefits of decision trees is that they implicitly perform variable screening or feature selection naturally. Additionally, the technique is naturally computationally inexpensive and it requires very little effort from the user during data preparation. Lastly, the best feature of using trees for analytics - easy to interpret and explain to executives! This is increasingly important in fields where individuals have to explain why there black box makes specific decisions at certain periods of time. Con’s – Due to the overall flexibility of this technique, models using a decision tree can be prone to overfitting. Overfitting is where the model describes errors in the data and improperly fits a relationship that generally cannot be connected. In addition to their ability to be easily over fitted, the technique also lacks the performance available through other available techniques. Lastly Decision forests can be significantly impacted by small changes in Data over time. Describe the Iris Data Set We don’t need to understand the relationship in order to model the relationships. We can learn them imperially from the dataset Explain what machine learning has to do with the iris data set Attempt to plot the iris Data-set Graphical example of Knn Iris Physical representation of the decision tree concept Interpreting the Model Introduction to Decision Trees Give Example Building Decision Trees Introduce data set Implement a Decision tree with Data Set Explain how it worked Split the data Cache the training data Show current accuracy Benchmark against Random Confusion Matrix Tuning the model (Hyper parameter) Impurity: Measure how much decision decreases impurity using gini impurity (vs. entropy) Maximum Depth: Calculates the total depth of the tree and set the number of decisions allowed before terminating a prediction. Maximim Bins Calculates the total number of rules allowed Fix the training data Try multiple parameters Can the data be improved? Random Decision Forests Wisdom of the crowd Introduce concept of Decision Forests Ensemble Learning How to create Diversity of opinion Handicap each tree Subset of features Aggregating Results Hyperparameters in a forest How to implement that in Spark Determining results from Random Forest