The document outlines various regression techniques including logistic, polynomial, ridge, lasso, Bayesian linear, decision tree, and random forest regression. Each technique is briefly described, highlighting their use cases, mathematical formulations, and advantages such as dealing with overfitting and feature selection. Additionally, it emphasizes the importance of visualizing relationships in polynomial regression and the collaborative nature of random forests.

1. Regression
By D.S Kaushal

12. Linear Regression

17. Logistic Regression
 Logistic regression is one of the types of regression analysis technique, which gets
used when the dependent variable is discrete. Example: 0 or 1, true or false, etc.
This means the target variable can have only two values, and a sigmoid curve denotes
the relation between the target variable and the independent variable.
 Logit function is used in Logistic Regression to measure the relationship between the
target variable and independent variables. Below is the equation that denotes the
logistic regression.
 logit(p) = ln(p/(1-p)) = b0+b1X1+b2X2+b3X3….+bkXk
 where p is the probability of occurrence of the feature.

19. Polynomial Regression
 The technique of polynomial regression analysis is used to represent a non-
linear relationship between dependent and independent variables. It is a
variant of the multiple linear regression model, except that the best fit line is
curved rather than straight.

20. Important Points:
 While there might be a temptation to fit a higher degree polynomial to get
lower error, this can result in over-fitting. Always plot the relationships to see
the fit and focus on making sure that the curve fits the nature of the
problem. Here is an example of how plotting can help:


21. Ridge Regression
 Ridge Regression is another type of regression in machine learning and is usually
used when there is a high correlation between the parameters. This is because as
the correlation increases the least square estimates give unbiased values. But if the
collinearity is very high, there can be some bias value. Therefore, we introduce a
bias matrix in the equation of Ridge Regression. It is a powerful regression method
where the model is less susceptible to overfitting.
 Below is the equation used to denote the Ridge Regression, λ (lambda) resolves the
multicollinearity issue:
 β = (X^{T}X + λ*I)^{-1}X^{T}y

22. Lasso Regression
 Lasso Regression performs regularization along with feature selection. It
avoids the absolute size of the regression coefficient. This results in the
coefficient value getting nearer to zero, this property is different from what
in ridge regression.

23.  Therefore we use feature selection in Lasso Regression. In the case of Lasso
Regression, only the required parameters are used, and the rest is made zero.
This helps avoid the overfitting in the model. But if independent variables are
highly collinear, then Lasso regression chooses only one variable and makes
other variables reduce to zero. Below equation represents the Lasso
Regression method:
 N^{-1}Σ^{N}_{i=1}f(x_{i}, y_{I}, α, β)

24. Bayesian Linear Regression
 Bayesian Regression is used to find out the value of regression coefficients. In
Bayesian linear regression, the posterior distribution of the features is
determined instead of finding the least-squares. Bayesian Linear Regression is
a combination of Linear Regression and Ridge Regression but is more stable
than simple Linear Regression.

25. Decision Tree Regression
 The decision tree as the name suggests works on the principle of conditions.
It is efficient and has strong algorithms used for predictive analysis. It has
mainly attributed that include internal nodes, branches, and a terminal node.
 Every internal node holds a “test” on an attribute, branches hold the
conclusion of the test and every leaf node means the class label. It is used for
both classifications as well as regression which are both supervised learning
algorithms. Decisions trees are extremely delicate to the information they are
prepared on — little changes to the preparation set can bring about
fundamentally different tree structures.

26. Random Forest Regression
 Random forest, as its name suggests, comprises an enormous amount of
individual decision trees that work as a group or as they say, an ensemble.
Every individual decision tree in the random forest lets out a class prediction
and the class with the most votes is considered as the model's prediction.
 Random forest uses this by permitting every individual tree to randomly
sample from the dataset with replacement, bringing about various trees. This
is known as bagging.