ensemble methods use multiple learning algorithms to obtain better predictive performance than could be obtained from any of the constituent learning algorithms alone. An ensemble is itself a supervised learning algorithm, because it can be trained and then used to make predictions. The trained ensemble, therefore, represents a single hypothesis. This hypothesis, however, is not necessarily contained within the hypothesis space of the models from which it is built.

1. Data Mining
Topic: Ensemble Learning
Group Members
Atta Ul Mustafa (4425)
Armgan Ali (4424)
Ali raza (4427)
Atif Ali (4407)
Abdul Rehman (4403)

2. Table of Content
Ensemble Learning Definition
Ensemble Learning Methodologies
Combination Schemes for multiple learners
Popular Ensemble Models

3. Ensemble Learning:
In order to improve the accuracy of the predictive model. Ensemble
learining helps improves machine learining results by combining
several models.
Essemble-Learining Methodologies:
The ensemble learining methodologies consist of two sequential
phases
a) Training phase
b) Testing phase

4. Training Phase:
The ensemble Method generates several different predictive models
from training sample.

5. Testing Phase:
For predicting an unknown value of a test sample, the
ensemble Method aggregates output ensemble Method
aggregates output of the each predictive model.

6. Combination Schemes for multiple learners:
Combination Schemes includes:
1- Global Approach
2- Local Approach
3- Multistage Combination.
Global Approach:
Global Approach is through learner’s fusion where all learners
produce on output are combined by voting , averaging or stacking. This
represent integration function for each patterns of all classifier
contribute to the final decision.

7. Local Approach:
Local approach is based on learner selection where one or
more learners responsible for generating the output are
selected based on their closeness to the sample.
Multistage Combination:
It is uses a serial approach where the next learner is trained with
or tested on only instances , where previous learners were inaccurate.

8. Popular Ensemble Models:
Bagging
Boosting
Ad boost
Random forest

9. Bagging:
Bagging a name derived from boot strap aggregation was the
first effective method It was originally designed for
classification and is usually applied to designed tree Models.
But it can be used with any type of Model for classification or
regression. The method used multiple versions of a training
set by using the bootstrap, that is sampling with replacement
Each of these data sets is use to train the different model. The
outputs of the models are combined by averaging or voting to
create a single output.

10. Boosting:
Boosting is the most widely used ensemble method and one of the
most powerful learning ideas introduce in the ensemble learning
community Originally designed for the classification It can also be
extended to regression In the boosting, the sample are reweighted to
focus the System on Samples that are not correctly classified with the
recently learned classifiers. During each step of learning:
1. Increase weights of the samples that are not correctly learned by
the weak learner
2. Decrease weights of the sample that are correctly learned by the
weak learner

11. Ad boost:
The original boosting algorithm combined three weak learners
to generate a strong, high quality learner. Simplicity an easy
implementation are the main reasons why ad boost is most
popular. It can be combined with any classifiers including
neural network, decision tree and nearest neighbor classifier.

12. Random forest:
Each classifier in the ensemble is a decision tree classifier and
is generated using a random selection of attributes at each
node to determine the split . During classification, each tree
votes and the most popular class is returned.
Two Methods to construct Random Forest:
1.Forest-RI
2.Forest-RC

13. Forest-RI (random input selection):
Randomly select, at each node, F attributes as candidates for the split
at the node. The CART methodology is used to grow the trees to
maximum size
Forest-RC (random linear combinations):
Creates new attributes (or features) that are a linear combination of
the existing attributes (reduces the correlation between individual
classifiers)