# The Gradient Descent Algorithm and its Variants

Last Updated on October 25, 2022 by Editorial Team

**Author(s): Towards AI Editorial Team**

Originally published on Towards AI the World’s Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale AI and technology startups. Let us help you unleash your technology to the masses.

#### Gradient Descent Algorithm with Code Examples inย Python

**Author(s):** Pratikย Shukla

โEducating the mind without educating the heart is no education at all.โ โ Aristotle

#### The Gradient Descent Series ofย Blogs:

- The Gradient Descent Algorithm
- Mathematical Intuition behind the Gradient Descent Algorithm
- The Gradient Descent Algorithm & its Variants (You areย here!)

#### Table of contents:

- Introduction
- Batch Gradient Descentย (BGD)
- Stochastic Gradient Descentย (SGD)
- Mini-Batch Gradient Descentย (MBGD)
- Graph Comparison
- End Notes
- Resources
- References

#### Introduction:

Drumroll, please: Welcome to the finale of the Gradient Descent series! In this blog, we will dive deeper into the gradient descent algorithm. We will discuss all the fun flavors of the gradient descent algorithm along with their code examples in Python. We will also examine the differences between the algorithms based on the number of calculations performed in each algorithm. Weโre leaving no stone unturned today, so we request that you run the ** Google Colab** files as you read the document; doing so will give you a more precise understanding of the topic to see it in action. Letโs get intoย it!

### Batch Gradientย Descent:

The Batch Gradient Descent (BGD) algorithm considers all the training examples in each iteration. If the dataset contains a large number of training examples and a large number of features, implementing the Batch Gradient Descent (BGD) algorithm becomes computationally expensiveโโโso mind your budget! Letโs take an example to understand it in a betterย way.

Batch Gradient Descentย (BGD):

Number of training examples per iterations = 1 million = 1โฐโถ

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1โฐโถ * 1โฐยณ* 1โฐโด =ย 1โฐยนยณ

Now, letโs see how the Batch Gradient Descent (BGD) algorithm is implemented.

#### 1. Stepโโโ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepโโโ2:

Next, we import some required libraries to read, manipulate, and visualize theย data.

#### 3. Stepโโโ3:

Next, we are reading the data file, and then printing the first five rows ofย it.

#### 4. Stepโโโ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,ย )

#### 5. Stepโโโ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 6. Stepโโโ6:

Next, we are normalizing theย dataset.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 7. Stepโโโ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 8. Stepโโโ8:

Next, we perform the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,ย 1)

#### 9. Stepโโโ9:

Next, we are going to calculate the cost associated with our prediction. This step is based on the following formula.

Dimensions: cost = scalarย value

#### 10. Stepโโโ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why weโre not summing over the values of the weights is that our weight matrix is not a 1*1ย matrix.

Dimensions: db = sum(200, 1) = (1,ย 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,ย 3)

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 11. Stepโโโ11:

Next, we are going to use all the functions we just defined to run the gradient descent algorithm. We are also creating an empty list called cost_list to store the cost values of all the iterations. This list will be put to use to plot a graph in furtherย steps.

#### 12. Stepโโโ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofย 0.01.

#### 13. Stepโโโ13:

Next, we are plotting the graph of iterations vs.ย cost.

#### 14. Stepโโโ14:

Next, we are printing the final weights values after all the iterations areย done.

#### 15. Stepโโโ15:

Next, we print the final bias value after all the iterations areย done.

#### 16. Stepโโโ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). As we learned in Part 1 of the Gradient Descent series, this is because the graph with the lower learning rate takes smallerย steps.

#### 17. Stepโโโ17:

Letโs put it all together.

#### Number of Calculations:

Now, letโs count the number of calculations performed in the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 200 * 200 * 1 =ย 40000

Weights:(training examples) x (iterations) x (parameters) = 200 * 200 *3 =ย 120000

### Stochastic Gradientย Descent

In the batch gradient descent algorithm, we consider all the training examples for all the iterations of the algorithm. But, if our dataset has a large number of training examples and/or features, then it gets computationally expensive to calculate the parameter values. We know our machine learning algorithm will yield more accuracy if we provide it with more training examples. But, as the size of the dataset increases, the computations associated with it also increase. Letโs take an example to understand this in a betterย way.

Batch Gradient Descentย (BGD)

Number of training examples per iterations = 1 million = 1โฐโถ

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1โฐโถ*1โฐยณ*1โฐโด=1โฐยนยณ

Now, if we look at the above number, it does not give us excellent vibes! So we can say that using the Batch Gradient Descent algorithm does not seem efficient. So, to deal with this problem, we use the Stochastic Gradient Descent (SGD) algorithm. The word โStochasticโ means random. So, instead of performing calculation on all the training examples of a dataset, we take one random example and perform the calculations on that. Sounds interesting, doesnโt it? We just consider one training example per iteration in the Stochastic Gradient Descent (SGD) algorithm. Letโs see how effective Stochastic Gradient Descent is based on its calculations.

Stochastic Gradient Descentย (SGD):

Number of training examples per iterations = 1

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1 * 1โฐยณ*1โฐโด=1โฐโท

Comparison with Batch Gradientย Descent:

Total computations in BGD = 1โฐยนยณ

Total computations in SGD = 1โฐโท

Evaluation:SGD is ยนโฐโถ times faster than BGD in thisย example.

**Note:** Please be aware that our cost function might not necessarily go down as we just take one random training example every iteration, so donโt worry. However, the cost function will gradually decrease as we perform more and more iterations.

Now, letโs see how the Stochastic Gradient Descent (SGD) algorithm is implemented.

#### 1. Stepโโโ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepโโโ2:

Next, we are importing some required libraries to read, manipulate, and visualize theย data.

#### 3. Stepโโโ3:

Next, we are reading the data file, and then printing the first five rows ofย it.

#### 4. Stepโโโ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,ย )

#### 5. Stepโโโ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 6. Stepโโโ6:

Next, we are normalizing theย dataset.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 7. Stepโโโ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 8. Stepโโโ8:

Next, we perform the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,ย 1)

#### 9. Stepโโโ9:

Next, weโll calculate the cost associated to our prediction. The formula used for this step is as follows. Because there will only be one value of the error, we wonโt need to divide the cost function by the size of the dataset or add up all the costย values.

Dimensions: cost = scalarย value

#### 10. Stepโโโ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why we are not summing over the values of the weights is that our weight matrix is not a 1*1 matrix. Also, in this case, since we have only one training example, we wonโt need to perform the summation over all the examples. The updated formula is given asย follows.

Dimensions: db = (1,ย 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,ย 3)

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 11. Stepโโโ11:

#### 12. Stepโโโ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofย 0.01.

#### 13. Stepโโโ13:

Next, we print the final weights values after all the iterations areย done.

#### 14. Stepโโโ14:

Next, we print the final bias value after all the iterations areย done.

#### 15. Stepโโโ15:

Next, we are plotting the graph of iterations vs.ย cost.

#### 16. Stepโโโ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). Again, we know this because the graph with a lower learning rate takes smallerย steps.

#### 17. Stepโโโ17:

Putting it all together.

#### Calculations:

Now, letโs count the number of calculations performed in implementing the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 1* 200 * 1 =ย 200

Weights:(training examples) x (iterations) x (parameters) = 1* 200 *3 =ย 600

### Mini-Batch Gradient Descent Algorithm:

In the Batch Gradient Descent (BGD) algorithm, we consider all the training examples for all the iterations of the algorithm. However, in the Stochastic Gradient Descent (SGD) algorithm, we only consider one random training example. Now, in the Mini-Batch Gradient Descent (MBGD) algorithm, we consider a random subset of training examples in each iteration. Since this is not as random as SGD, we reach closer to the global minimum. However, MBGD is susceptible to getting stuck into local minima. Letโs take an example to understand this in a betterย way.

Batch Gradient Descentย (BGD):

Number of training examples per iterations = 1 million = 1โฐโถ

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1โฐโถ*1โฐยณ*1โฐโด=1โฐยนยณ

Stochastic Gradient Descentย (SGD):

Number of training examples per iterations = 1

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1*1โฐยณ*1โฐโด =ย 1โฐโท

Mini Batch Gradient Descentย (MBGD):

Number of training examples per iterations = 100 = 1โฐยฒ

โHere, we are considering 1โฐยฒ training examples out of 1โฐโถ.

Number of iterations = 1000 = 1โฐยณ

Number of parameters to be trained = 10000 = 1โฐโด

Total computations = 1โฐยฒ*1โฐยณ*1โฐโด=1โฐโน

Comparison with Batch Gradient Descentย (BGD):

Total computations in BGD = 1โฐยนยณ

Total computations in MBGD =ย 1โฐโน

Evaluation:MBGD is 1โฐโด times faster than BGD in thisย example.

Comparison with Stochastic Gradient Descentย (SGD):

Total computations in SGD = 1โฐโท

Total computations in MBGD =ย 1โฐโน

Evaluation:SGD is 1โฐยฒ times faster than MBGD in thisย example.

Comparison of BGD, SGD, andย MBGD:

Total computations in BGD= 1โฐยนยณ

Total computations in SGD= 1โฐโท

Total computations in MBGD =ย 1โฐโน

Evaluation:SGD > MBGD >ย BGD

**Note:** Please be aware that our cost function might not necessarily go down as we are taking a random sample of the training examples every iteration. However, the cost function will gradually decrease as we perform more and more iterations.

Now, letโs see how the Mini-Batch Gradient Descent (MBGD) algorithm is implemented in practice.

#### 1. Stepโโโ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepโโโ2:

Next, we are importing some required libraries to read, manipulate, and visualize theย data.

#### 3. Stepโโโ3:

Next, we are reading the data file, and then print the first five rows ofย it.

#### 4. Stepโโโ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,ย )

#### 5. Stepโโโ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 6. Stepโโโ6:

Next, we are normalizing theย dataset.

Dimensions: X = (200, 3) & Y = (200,ย 1)

#### 7. Stepโโโ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 8. Stepโโโ8:

Next, we are performing the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,ย 1)

#### 9. Stepโโโ9:

Next, we are going to calculate the cost associated with our prediction. This step is based on the following formula.

Dimensions: cost = scalarย value

#### 10. Stepโโโ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why we are not summing over the values of the weights is that our weight matrix is not a 1*1ย matrix.

Dimensions: db = sum(200, 1) = (1ย ,ย 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,ย 3)

Dimensions: bias = (1, 1) & weights = (1,ย 3)

#### 11. Stepโโโ11:

Next, we are going to use all the functions we just defined to run the gradient descent algorithm. Also, we are creating an empty list called cost_list to store the cost values of all the iterations. We will use this list to plot a graph in furtherย steps.

#### 12. Stepโโโ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofย 0.01.

#### 13. Stepโโโ13:

Next, we print the final weights values after all the iterations areย done.

#### 14. Stepโโโ14:

Next, we print the final bias value after all the iterations areย done.

#### 15. Stepโโโ15:

Next, we are plotting the graph of iterations vs.ย cost.

#### 16. Stepโโโ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). The reason behind it is that the graph with lower learning rate takes smallerย steps.

#### 17. Stepโโโ17:

Putting it all together.

#### Calculations:

Now, letโs count the number of calculations performed in implementing the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 20 * 200 * 1 =ย 4000

Weights:(training examples) x (iterations) x (parameters) = 20 * 200 *3 =ย 12000

#### Graph comparisons:

#### End Notes:

And just like that, weโre at the end of the Gradient Descent series! In this installment, we went deep into the code to look at how three of the major types of gradient descent algorithms perform next to each other, summed up by these handyย notes:

1. Batch Gradient Descent

Accuracy โ High

Time โย More

2. Stochastic Gradient Descent

Accuracy โ Low

Time โย Less

3. Mini-Batch Gradient Descent

Accuracy โ Moderate

Time โย Moderate

We hope you enjoyed this series and learned something new, no matter your starting point or machine learning background. Knowing this essential algorithm and its variants will likely prove valuable as you continue on your AI journey and understand more about both the technical and grand aspects of this incredible technology. Keep an eye out for other blogs offering, even more, machine learning lessons, and stayย curious!

#### Resources:

*Batch Gradient Descent**โโโ**Google Colab**,ย**GitHub**Stochastic Gradient Descent**โโโ**Google Colab**,ย**GitHub**Mini Batch Gradient Descentโ**โโ**Google Colab**,ย**GitHub*

#### Citation:

For attribution in academic contexts, please cite this workย as:

Shukla, et al., โThe Gradient Descent Algorithm & its Variantsโ, Towards AI, 2022

#### BibTex Citation:

@article{pratik_2022, title={The Gradient Descent Algorithm & its Variants}, url={https://towardsai.net/neural-networks-with-python}, journal={Towards AI}, publisher={Towards AI Co.}, author={Pratik, Shukla}, editor={Lauren, Keegan}, year={2022}, month={Oct} }

References:

The Gradient Descent Algorithm and its Variants was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Join thousands of data leaders on the AI newsletter. Itโs free, we donโt spam, and we never share your email address. Keep up to date with the latest work in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming aย sponsor.

Published via Towards AI