Linear Regression

1. RegBoost: a gradient boosted multivariate
regression algorithm
Author:
Sudi Murindanyi
2022/HD05/5583X
Presented by - Sudi
Affiliation: Makerere University
Uganda

2. 2
Important words
❑ Linear Regression: is a linear approach for modelling the relationship
between a scalar response and one or more explanatory variables (also
known as dependent and independent variables)
❑ Ensemble Learning: Ensemble learning is the process by which multiple
models, such as classifiers or experts, are strategically generated and
combined to solve a particular computational intelligence problem.
❑ Gradient Boosting: is a machine learning technique used in regression and
classification tasks, among others. It gives a prediction model in the form of
an ensemble of weak prediction models, which are typically decision trees.
❑ RMSE: Root mean square error or root mean square deviation is one of the
most commonly used measures for evaluating the quality of predictions. It
shows how far predictions fall from measured true values using Euclidean
distance.

3. 3
Introduction
Purpose
Inspired by the basic idea of gradient boosting, the study aims to design a
novel multivariate regression ensemble algorithm (RegBoos)t by using
multivariate linear regression as a weak predictor.
Design/methodology/approach
To achieve nonlinearity after combining all linear regression predictors, the
training data is divided into two branches according to the prediction results
using the current weak predictor. The linear regression modeling is
recursively executed in two branches. In the test phase, test data is
distributed to a specific branch to continue with the next weak predictor.
The final result is the sum of all weak predictors across the entire path.
Findings
Through comparison experiments, it is found that the algorithm RegBoost
can achieve similar performance to the gradient boosted decision tree
(GBDT). The algorithm is very effective compared to linear regression.

4. 4
Background
❑ The main difference between boosting methods and conventional machine
learning algorithms is that optimisation is carried out in the function space.
For example, gradient boosting selects a function that points toward the
negative gradient iteratively to maximise a cost function over the function
space.
❑ A technique for turning weak predictors into strong predictors is gradient
boosting. According to particular applications, the algorithm designer might
choose the primary learner. For example, linear regression models are used
as weak learners for weak predictions.
❑ All base learners are nonlinear models; their natural combination using
gradient boosting is possible. For example, linear regression, a linear model,
linear regression serves as the multivariate regression ensemble algorithm's
base learner. By dividing the data, nonlinearity is achieved.

5. 5
Reason why to choose the paper

6. 6
Approach
Problem
Adding multiple linear regression predictors directly, you end up with a
linear regression model.
Solution
RegBoost, divides the training data into two branches according to the
prediction results using the current weak predictor. The linear regression
modeling is recursively executed in two branches. In the test phase, test
data is distributed to a specific branch to continue with the next weak
predictor. The final result is the sum of all weak predictors across the
entire path.
Approach
Considering that the data may contain some features that are either
redundant or irrelevant and can thus be removed without incurring much
loss of information. RegBoost uses stepwise regression to select the
most important factors when constructing each weak predictor.

7. 7
Results
Root-mean-square deviation (RMSE) evaluation metric was used to compare the
performance of multiple linear regression, LightGBM, and RegBoostresults on the
three separate datasets. RegBoost performed best in the CASP data set. LightGBM
outperforms RegBoost in the CCPP and Super Conduct data sets, but the difference is
marginal, as seen in the table.

8. 8
Conclusion
❑ RegBoost is a new regression method created by the authors.
❑ The RegBoost used gradient boosting and linear regression as weak
predictors to create an ensemble algorithm.
❑ RegBoost could produce results comparable to GBDT, according to the tests
illustrated in the table above.
❑ Although it merits more investigation, the approach is generally quite
efficient in contrast to linear regression.