The document discusses machine learning algorithms and techniques. It provides details on the SMO algorithm for support vector machines, including how it optimizes over two alphas instead of all. It also discusses overfitting, regularization, bias-variance tradeoff, ensemble methods like bagging and boosting, decision trees, random forests, and PCA.

1. Machine Learning Interviews –
Day 3
Arpit Agarwal

2. SMO Algorithm
• Optimization problem:
• Solve over 2 alphas instead of all

3. SMO Algorithm

4. SMO Algorithm
• Input: C, kernel, kernel parameters, epsilon
• Initialize b and all ’s to 0
• Repeat until KKT satisfied (to within epsilon):
– Find an example e1 that violates KKT (prefer unbound
examples here, choose randomly among those)
– Choose a second example e2. Prefer one to maximize step
size (in practice, faster to just maximize |E1 – E2|). If that
fails to result in change, randomly choose unbound
example. If that fails, randomly choose example. If that
fails, re-choose e1.
– Update α1 and α2 in one step
– Compute new threshold b

5. Updating Two ’s: One SMO Step
• Given examples e1 and e2, set
where:
• Clip this value in the natural way: if y1 = y2 then:
• otherwise:
• Set where s = y1y2

6. - What is Overfitting? How to avoid it?
- “Cross-validation, regularization, bagging”
- What is regularization? Why do we need it?
- What is Bias-Variance tradeoff?

7. Overfitting – Curve Fitting

8. Overfitting

9. Ensemble Methods
• JP wants to do CMO assignment, but he does
not know any of the answers.
• What will JP do?

10. Ensemble Methods
Original
Training data
D
.... D1
D2 Dt-1 Dt
Step 1:
Create Multiple
Data Sets
C1 C2 Ct -1 Ct
Step 2:
Build Multiple
Classifiers
C*
Step 3:
Combine
Classifiers

11. Ensemble Methods
Types
– Bagging (Helps reducing variance of the classifier)
– Boosting (Adaboost) (Helps in improving the
accuracy of the classifier)

12. Decision Trees
• JP’s very practical problem:- “Whether to go
to prakruthi for tea or not?”
Ask Rishabh if he
wants to come?
Yes No
Does Rishabh has
money for both of us?
Don’t go for tea
Yes No
Go for tea Don’t go for tea

13. Random Forests
• Ensemble method specifically designed for
decision tree classifiers
• Random Forests grows many classification
trees (that is why the name!)
• Ensemble of unpruned decision trees
• Each base classifier classifies a “new” vector
• Forest chooses the classification having the
most votes (over all the trees in the forest)

14. Random Forests
• Introduce two sources of randomness:
“Bagging” and “Random input vectors”
– Each tree is grown using a bootstrap sample of
training data
– At each node, best split is chosen from random
sample of mtry variables instead of all variables

15. Random Forests

16. Random Forest Algorithm
• M input variables, a number m<<M is specified such that at
each node, m variables are selected at random out of the M
and the best split on these m is used to split the node.
• m is held constant during the forest growing
• Each tree is grown to the largest extent possible
• There is no pruning
• Bagging using decision trees is a special case of random
forests when m=M

17. Random Forest Algorithm
• Good accuracy without over-fitting
• Fast algorithm (can be faster than growing/pruning a single
tree); easily parallelized
• Handle high dimensional data without much problem
• Only one tuning parameter mtry , usually not sensitive to it

18. PCA