This document discusses the objectives and processes of machine learning model assessment and selection. It explains that data should be broken up into training (50%), validation (25%), and test (25%) sets for the purposes of model training, selection, and assessment respectively. The validation set is used to select the best performing model by comparing different algorithms, tuning hyperparameters, and selecting optimal input variables. The test set then assesses the performance of the selected model.

1. Model Assessment and Selection
Machine Learning

2. Concept
 Three separate objectives: Model Training, Model
Selection, Model Assessment.
 Data break up for Training, Validation and Test.
Training (50%) Validation
(25%)
Test
(25%)
Training + validation(80%) Test
(20%)

3. Break up
 Data for training: what happens in this phase?
 Validation Phase: Data for model selection
 Selection between multiple methods
 Fine Tuning parameters in the algorithm. Affecting
model complexity and regularization.
 Choosing the correct number of input variables and
correct input variables.
 Affects model complexity.
 Going beyond blanket approaches to penalize
more inputs
 Test Phase: Data for model assessment

4. Cross-Validation:
Training + validation Test
Train Train Train Validation Train
K fold cross validation(here
k=5)