Classification and regression trees (CART) are decision tree models that can be used for both classification and regression problems. CART builds trees using recursive partitioning that splits nodes based on predictor variables. This example document discusses regression trees using a California housing dataset to predict median home prices based on location and other attributes. It covers terminology like parent and child nodes, recursive binary splitting to create the tree structure, overfitting, cross-validation to evaluate model performance, and pruning trees to optimize complexity.

1. MBD1202-Tree-Based Methods
Machine Learning Algorithm
Master of Science in Big Data Analytics- 2022

2. Classification and Regression Trees
(CART)

3. Classification and Regression Trees (CART)
• Decision trees can be applied to both regression and classification
problems.
• Decision trees are powerful classifiers, which utilize a tree structure
to model the relationships among the features and the potential
outcomes.
• Decision trees are built using a heuristic called recursive partitioning
(divide and conquer).
• We first consider regression problems (with continuous response Y )
and then move on to classification.

4. Regression Trees

5. Example
• In order to motivate regression trees, we begin with a simple example.
• We will look at housing data collected in California in the 1990 census
• Aggregated data are available for 20,640 neighbourhoods in California
• Our goal is to predict the median house price in each neighbourhood
using some or all of the following predictor variables:
Latitude; Longitude; Median income; Median house age; Average
occupancy per house; Average number of rooms per house; Average
number of bedrooms per house; Neighbourhood population size

7. We will begin
by considering
only latitude and
longitude as
potential predictors.....

13. Terminology
• Parent node: is the immediate predecessor of Node j.
• Child node: is the immediate successor of a parent node.
• Root node: is the top node of the tree, all observations are together
in this node.
• Terminal node: is the node that do not have children nodes,
decisions are made looking at these nodes.
• Depth: is the maximal length of a path from the root node to a
terminal node.

20. Recursive Binary Splitting

21. Recursive Binary Splitting

22. Recursive Binary Splitting

23. Recursive Binary Splitting

24. How do we choose the X?

25. The Problem of Overfitting
• A model that is excessively complex will fit the sample data very
well, but will not be good at predicting new responses
• This is called overfitting the sample data

26. Avoiding Overfitting
• We are interested in how well our model will predict the responses of
unseen data
• To assess this, we must divide our dataset into two parts: a training
set and a test set
• We will fit the model to the training data and then assess the
prediction error on the test set
• We will then choose the model with the lowest test error
• This may be achieved by a technique called cross-validation

27. Cross-Validation
• In K-fold cross-validation, we randomly divide the set observations into
K groups or folds of roughly equal size
• The model is fitted to all data excluding the kth fold and predictions
are then made for the data in the kth fold
• This is repeated for k = 1,…,K and the test errors are then combined
across folds
• 10-fold cross-validation is the most popular. When K = n, it is called
leave-one-out cross-validation

28. Cross-Validation
Example: 4-fold cross-validation for the linear model

29. Cross-Validation
Example: 4-fold cross-validation for the linear model

30. Cross-Validation
Example: 4-fold cross-validation for the linear model

31. Cross-Validation
Example: 4-fold cross-validation for the linear model

32. Cross-Validation
Example: 4-fold cross-validation for the linear model

33. Cross-Validation
Example: 4-fold cross-validation for the linear model

34. Cross-Validation
Example: 4-fold cross-validation for the linear model

35. Read about Tree Pruning

36. Classification Trees

37. Classification Trees

38. Example

39. Example

40. Example

41. Splitting Criteria