【ML】Ensemble Method indtorduction

I'll introduce the ways to ensemble some ML models.

1. Avaraging

How to "average" the prediction results from multiple models (in the case of regression)
A version that weighted to each model respect to each performance is also used.

2. Voting

A method of taking a "majority vote" (i.e. voting) among the prediction results of multiple models (in the case of classification).
As above, it can also be used as a weighting model.

3. Blending

• step1:
  Taking the prediction results of multiple models.
• step2:
  "Blending" with (linear) models (for regression: linear regression, for classification: logistic regression), NN, etc. (Linear model is generally used)

Simply put, using machine learning methods as role like "avaraging" or "voting".

4. Stacking

• step1:
  Taking the prediction results of multiple models.
• step2:
  Making new maltiple models that using step1's prediction as input, and retaking the prediction results of models.
• step3:
  "Blending" with (linear) models (for regression: linear regression, for classification: logistic regression), NN, etc. (Linear model is generally used)

How to "stacking" (= piling up) three layers of Levels 0 to 2 (※The number of intermediate levels can be increased)

As you can see from the definition above, blending and stacking are almost the same thing, just with a different number of levels. Stacking can also be said to be blending with three or more steps.

Others

In here, I'll itroduce the way to inprove the model performance by using the way like ensemble(these are called ensemble sometime.)

ex.1 Bagging(Bootstrap Aggregating)

This method extarct some data from dataset randomly, and train some model as weak learner, and calculate final prediction by aggregation(average, majority vote, stc).

ex.2 Boosting

In Boosting, fisrt, train the model, next model learning the loss of first model. After looping this some times, predict with all model and gregation each model's prediction and make up final prediction.
Typically, this method is better than Bagging, but must be careful to overfitting.