Briefly describe a sign of overfitting in Naive Bayes learning, and how it can be avoided. Solution Briefly, with the Naive Bayes (NB) algorithm the \'naive\' conditional independence assumption means that interactions between variables can be ignored. What follows is: i) it has a simpler hypothesis function (compared with other algorithms e.g. logistic regression) ii) since the interactions are not modeled, some of the information in the data is ignored. This makes it an inherently high bias model; it has a high approximation error but as a result it also does not overfit. (A model with high variance attempts to model all of the data including the noise in the data). iii) Since the interactions are not modeled, less training data is needed. This is why the NB classifier is known to perform well both with small data sets and with missing data. Hereis a small experiment I did to see effect missing data and training data size have on the NB classifier. .

1. Briefly describe a sign of overfitting in Naive Bayes learning, and how it can be avoided.
Solution
Briefly, with the Naive Bayes (NB) algorithm the 'naive' conditional independence assumption
means that interactions between variables can be ignored. What follows is:
i) it has a simpler hypothesis function (compared with other algorithms e.g. logistic regression)
ii) since the interactions are not modeled, some of the information in the data is ignored. This
makes it an inherently high bias model; it has a high approximation error but as a result it also
does not overfit. (A model with high variance attempts to model all of the data including the
noise in the data).
iii) Since the interactions are not modeled, less training data is needed. This is why the NB
classifier is known to perform well both with small data sets and with missing data. Hereis a
small experiment I did to see effect missing data and training data size have on the NB classifier.