Skip to main content

Gradient Boosted Trees

Gradient Boosting

Gradient boosting is an ensemble techniques that gives you a collection of predictors (decision trees). It is an iterative boosting technique in which a weak classifier is made at the first iteration and then the consequent classifiers learns the residuals from the previous classifiers.
Mean squared error function gradient is equal to (y- yhat)   which is nothing but the residuals from the first prediction and hence iteratively it keeps on learning the residuals and gives a good estimate after n steps using a low learning rate (or decaying).
Steps:
·         Learn a Regression predictor
·         Compute the error residual
·         Learn to predict the residual
·         Combine the predictors

Each Learner is estimating the gradient of the loss function. Take a sequence of steps to reduce error function.

What does Gradient Boost do?

·         Learn a sequence of predictors
·         Sum of predictions is more accurate
·         Predictive function is increasingly complex

Parameters:
·         Num_of_trees/ Iterations (100 – 1000)
·         Learning Rate/ Shrinkage (0.001 – 0.1)
·         Min Number of samples in Terminal Node/ Leaf Node (3 -6)
·         Depth of the tree (3 – 5)
·         Subsampling (0.4 – 0.8)
·         Gamma (0 – 1)

Advantages:
·         Usually gives better predictions than Random Forests and individual Decision trees
·         They can handle categorical features easily which is not the case with Logistic Regression
·         They do not expect the features to be linear or even features that interact linearly
·         They can handle large dimensional spaces and large number of samples

Disadvantages:
·         More prone to overfitting.
·         We have to be extremely cautious while tuning the parameters (shrinkage, Num_trees, depth) i.e. they are harder to get right
·         Need more computation time






Comments

  1. Unknown28 September 2018 at 21:28

    I think things like this are really interesting. I absolutely love to find unique places like this. It really looks super creepy though!!
    Best Machine Learning institute in velachery | python machine learning course in velachery | Machine Learning course in chennai

    ReplyDelete

Post a Comment

Popular posts from this blog

Most Important steps to win any Machine Learning Competition

Gaining experience from the blogs of all the winners of online competition, I could see what approaches they follow, their solutions etc. Some of the most important points that I try to follow when I see a new machine learning challenge are: 1. Understand : Try to understand the data thoroughly, what each feature means and how it it related to the target variable. Exploring the data using plots gives an idea about features.  2. Data cleaning : Imputation of missing values and data pre-processing.  3. Feature Engineering : The most important part is feature engineering. Good feature engineering can give you an edge over others in any competition. In fact, it's the most important and most difficult step to think of and comes only with experience and practice. 4. Splitting your data:  Creating a strategy how to split your data into training and validation set. A good validation set that matches your score on the leader board is very crucial to fine tune your code....
Post a Comment
Read more

Interesting Projects in Deep Learning

Deep Learning is the latest new area in machine learning which is proving handy to solve many interesting problems which were not possible to do with the standard ML algorithms before such as Language Translation, Text to speech, Speech Recognition etc. It has moved us one step closer to what we thought of as our objective. Self Driving cars are the latest invention of this field and is growing at a very fast rate with more number of developers opting to move to deep learning. Some of the most interesting projects in deep learning that helps you learn concepts as well as apply these concepts on real world data and complex use cases are: Tranfer Learning : To use a pretrained model to find objects in any image Style Transfer : Applying the style of one image onto the other image Digit Recognizer : Use CNN to find the handwritten digits with our own custom built CNN model MNIST data : Classify over 10 objects in the MNIST data TED Talks : Creating word embeddings in a dataset...
Post a Comment
Read more