# The Gradient Descent Algorithm and its Variants

Last Updated on October 25, 2022 by Editorial Team

**Author(s): Towards AI Editorial Team**

Originally published on Towards AI the World’s Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale AI and technology startups. Let us help you unleash your technology to the masses.

#### Gradient Descent Algorithm with Code Examples inΒ Python

**Author(s):** PratikΒ Shukla

βEducating the mind without educating the heart is no education at all.β β Aristotle

#### The Gradient Descent Series ofΒ Blogs:

- The Gradient Descent Algorithm
- Mathematical Intuition behind the Gradient Descent Algorithm
- The Gradient Descent Algorithm & its Variants (You areΒ here!)

#### Table of contents:

- Introduction
- Batch Gradient DescentΒ (BGD)
- Stochastic Gradient DescentΒ (SGD)
- Mini-Batch Gradient DescentΒ (MBGD)
- Graph Comparison
- End Notes
- Resources
- References

#### Introduction:

Drumroll, please: Welcome to the finale of the Gradient Descent series! In this blog, we will dive deeper into the gradient descent algorithm. We will discuss all the fun flavors of the gradient descent algorithm along with their code examples in Python. We will also examine the differences between the algorithms based on the number of calculations performed in each algorithm. Weβre leaving no stone unturned today, so we request that you run the ** Google Colab** files as you read the document; doing so will give you a more precise understanding of the topic to see it in action. Letβs get intoΒ it!

### Batch GradientΒ Descent:

The Batch Gradient Descent (BGD) algorithm considers all the training examples in each iteration. If the dataset contains a large number of training examples and a large number of features, implementing the Batch Gradient Descent (BGD) algorithm becomes computationally expensiveβββso mind your budget! Letβs take an example to understand it in a betterΒ way.

Batch Gradient DescentΒ (BGD):

Number of training examples per iterations = 1 million = 1β°βΆ

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1β°βΆ * 1β°Β³* 1β°β΄ =Β 1β°ΒΉΒ³

Now, letβs see how the Batch Gradient Descent (BGD) algorithm is implemented.

#### 1. Stepβββ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepβββ2:

Next, we import some required libraries to read, manipulate, and visualize theΒ data.

#### 3. Stepβββ3:

Next, we are reading the data file, and then printing the first five rows ofΒ it.

#### 4. Stepβββ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,Β )

#### 5. Stepβββ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 6. Stepβββ6:

Next, we are normalizing theΒ dataset.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 7. Stepβββ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 8. Stepβββ8:

Next, we perform the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,Β 1)

#### 9. Stepβββ9:

Next, we are going to calculate the cost associated with our prediction. This step is based on the following formula.

Dimensions: cost = scalarΒ value

#### 10. Stepβββ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why weβre not summing over the values of the weights is that our weight matrix is not a 1*1Β matrix.

Dimensions: db = sum(200, 1) = (1,Β 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,Β 3)

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 11. Stepβββ11:

Next, we are going to use all the functions we just defined to run the gradient descent algorithm. We are also creating an empty list called cost_list to store the cost values of all the iterations. This list will be put to use to plot a graph in furtherΒ steps.

#### 12. Stepβββ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofΒ 0.01.

#### 13. Stepβββ13:

Next, we are plotting the graph of iterations vs.Β cost.

#### 14. Stepβββ14:

Next, we are printing the final weights values after all the iterations areΒ done.

#### 15. Stepβββ15:

Next, we print the final bias value after all the iterations areΒ done.

#### 16. Stepβββ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). As we learned in Part 1 of the Gradient Descent series, this is because the graph with the lower learning rate takes smallerΒ steps.

#### 17. Stepβββ17:

Letβs put it all together.

#### Number of Calculations:

Now, letβs count the number of calculations performed in the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 200 * 200 * 1 =Β 40000

Weights:(training examples) x (iterations) x (parameters) = 200 * 200 *3 =Β 120000

### Stochastic GradientΒ Descent

In the batch gradient descent algorithm, we consider all the training examples for all the iterations of the algorithm. But, if our dataset has a large number of training examples and/or features, then it gets computationally expensive to calculate the parameter values. We know our machine learning algorithm will yield more accuracy if we provide it with more training examples. But, as the size of the dataset increases, the computations associated with it also increase. Letβs take an example to understand this in a betterΒ way.

Batch Gradient DescentΒ (BGD)

Number of training examples per iterations = 1 million = 1β°βΆ

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1β°βΆ*1β°Β³*1β°β΄=1β°ΒΉΒ³

Now, if we look at the above number, it does not give us excellent vibes! So we can say that using the Batch Gradient Descent algorithm does not seem efficient. So, to deal with this problem, we use the Stochastic Gradient Descent (SGD) algorithm. The word βStochasticβ means random. So, instead of performing calculation on all the training examples of a dataset, we take one random example and perform the calculations on that. Sounds interesting, doesnβt it? We just consider one training example per iteration in the Stochastic Gradient Descent (SGD) algorithm. Letβs see how effective Stochastic Gradient Descent is based on its calculations.

Stochastic Gradient DescentΒ (SGD):

Number of training examples per iterations = 1

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1 * 1β°Β³*1β°β΄=1β°β·

Comparison with Batch GradientΒ Descent:

Total computations in BGD = 1β°ΒΉΒ³

Total computations in SGD = 1β°β·

Evaluation:SGD is ΒΉβ°βΆ times faster than BGD in thisΒ example.

**Note:** Please be aware that our cost function might not necessarily go down as we just take one random training example every iteration, so donβt worry. However, the cost function will gradually decrease as we perform more and more iterations.

Now, letβs see how the Stochastic Gradient Descent (SGD) algorithm is implemented.

#### 1. Stepβββ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepβββ2:

Next, we are importing some required libraries to read, manipulate, and visualize theΒ data.

#### 3. Stepβββ3:

Next, we are reading the data file, and then printing the first five rows ofΒ it.

#### 4. Stepβββ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,Β )

#### 5. Stepβββ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 6. Stepβββ6:

Next, we are normalizing theΒ dataset.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 7. Stepβββ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 8. Stepβββ8:

Next, we perform the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,Β 1)

#### 9. Stepβββ9:

Next, weβll calculate the cost associated to our prediction. The formula used for this step is as follows. Because there will only be one value of the error, we wonβt need to divide the cost function by the size of the dataset or add up all the costΒ values.

Dimensions: cost = scalarΒ value

#### 10. Stepβββ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why we are not summing over the values of the weights is that our weight matrix is not a 1*1 matrix. Also, in this case, since we have only one training example, we wonβt need to perform the summation over all the examples. The updated formula is given asΒ follows.

Dimensions: db = (1,Β 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,Β 3)

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 11. Stepβββ11:

#### 12. Stepβββ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofΒ 0.01.

#### 13. Stepβββ13:

Next, we print the final weights values after all the iterations areΒ done.

#### 14. Stepβββ14:

Next, we print the final bias value after all the iterations areΒ done.

#### 15. Stepβββ15:

Next, we are plotting the graph of iterations vs.Β cost.

#### 16. Stepβββ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). Again, we know this because the graph with a lower learning rate takes smallerΒ steps.

#### 17. Stepβββ17:

Putting it all together.

#### Calculations:

Now, letβs count the number of calculations performed in implementing the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 1* 200 * 1 =Β 200

Weights:(training examples) x (iterations) x (parameters) = 1* 200 *3 =Β 600

### Mini-Batch Gradient Descent Algorithm:

In the Batch Gradient Descent (BGD) algorithm, we consider all the training examples for all the iterations of the algorithm. However, in the Stochastic Gradient Descent (SGD) algorithm, we only consider one random training example. Now, in the Mini-Batch Gradient Descent (MBGD) algorithm, we consider a random subset of training examples in each iteration. Since this is not as random as SGD, we reach closer to the global minimum. However, MBGD is susceptible to getting stuck into local minima. Letβs take an example to understand this in a betterΒ way.

Batch Gradient DescentΒ (BGD):

Number of training examples per iterations = 1 million = 1β°βΆ

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1β°βΆ*1β°Β³*1β°β΄=1β°ΒΉΒ³

Stochastic Gradient DescentΒ (SGD):

Number of training examples per iterations = 1

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1*1β°Β³*1β°β΄ =Β 1β°β·

Mini Batch Gradient DescentΒ (MBGD):

Number of training examples per iterations = 100 = 1β°Β²

βHere, we are considering 1β°Β² training examples out of 1β°βΆ.

Number of iterations = 1000 = 1β°Β³

Number of parameters to be trained = 10000 = 1β°β΄

Total computations = 1β°Β²*1β°Β³*1β°β΄=1β°βΉ

Comparison with Batch Gradient DescentΒ (BGD):

Total computations in BGD = 1β°ΒΉΒ³

Total computations in MBGD =Β 1β°βΉ

Evaluation:MBGD is 1β°β΄ times faster than BGD in thisΒ example.

Comparison with Stochastic Gradient DescentΒ (SGD):

Total computations in SGD = 1β°β·

Total computations in MBGD =Β 1β°βΉ

Evaluation:SGD is 1β°Β² times faster than MBGD in thisΒ example.

Comparison of BGD, SGD, andΒ MBGD:

Total computations in BGD= 1β°ΒΉΒ³

Total computations in SGD= 1β°β·

Total computations in MBGD =Β 1β°βΉ

Evaluation:SGD > MBGD >Β BGD

**Note:** Please be aware that our cost function might not necessarily go down as we are taking a random sample of the training examples every iteration. However, the cost function will gradually decrease as we perform more and more iterations.

Now, letβs see how the Mini-Batch Gradient Descent (MBGD) algorithm is implemented in practice.

#### 1. Stepβββ1:

First, we are downloading the data file from the GitHub repository.

#### 2. Stepβββ2:

Next, we are importing some required libraries to read, manipulate, and visualize theΒ data.

#### 3. Stepβββ3:

Next, we are reading the data file, and then print the first five rows ofΒ it.

#### 4. Stepβββ4:

Next, we are dividing the dataset into features and target variables.

Dimensions: X = (200, 3) & Y = (200,Β )

#### 5. Stepβββ5:

To perform matrix calculations in further steps, we need to reshape the target variable.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 6. Stepβββ6:

Next, we are normalizing theΒ dataset.

Dimensions: X = (200, 3) & Y = (200,Β 1)

#### 7. Stepβββ7:

Next, we are getting the initial values for the bias and weights matrices. We will use these values in the first iteration while performing forward propagation.

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 8. Stepβββ8:

Next, we are performing the forward propagation step. This step is based on the following formula.

Dimensions: predicted_value = (1, 1)+(200, 3)*(3,1) = (1, 1)+(200, 1) = (200,Β 1)

#### 9. Stepβββ9:

Next, we are going to calculate the cost associated with our prediction. This step is based on the following formula.

Dimensions: cost = scalarΒ value

#### 10. Stepβββ10:

Next, we update the parameter values of weights and bias using the gradient descent algorithm. This step is based on the following formulas. Please note that the reason why we are not summing over the values of the weights is that our weight matrix is not a 1*1Β matrix.

Dimensions: db = sum(200, 1) = (1Β ,Β 1)

Dimensions: dw = (1, 200) * (200, 3) = (1,Β 3)

Dimensions: bias = (1, 1) & weights = (1,Β 3)

#### 11. Stepβββ11:

Next, we are going to use all the functions we just defined to run the gradient descent algorithm. Also, we are creating an empty list called cost_list to store the cost values of all the iterations. We will use this list to plot a graph in furtherΒ steps.

#### 12. Stepβββ12:

Next, we are actually calling the function to get the final results. Please note that we are running the entire code for 200 iterations. Also, here we have specified the learning rate ofΒ 0.01.

#### 13. Stepβββ13:

Next, we print the final weights values after all the iterations areΒ done.

#### 14. Stepβββ14:

Next, we print the final bias value after all the iterations areΒ done.

#### 15. Stepβββ15:

Next, we are plotting the graph of iterations vs.Β cost.

#### 16. Stepβββ16:

Next, we plot two graphs with different learning rates to see the effect of learning rate in optimization. In the following graph we can see that the graph with a higher learning rate (0.01) converges faster than the graph with a slower learning rate (0.001). The reason behind it is that the graph with lower learning rate takes smallerΒ steps.

#### 17. Stepβββ17:

Putting it all together.

#### Calculations:

Now, letβs count the number of calculations performed in implementing the batch gradient descent algorithm.

Bias:(training examples) x (iterations) x (parameters) = 20 * 200 * 1 =Β 4000

Weights:(training examples) x (iterations) x (parameters) = 20 * 200 *3 =Β 12000

#### Graph comparisons:

#### End Notes:

And just like that, weβre at the end of the Gradient Descent series! In this installment, we went deep into the code to look at how three of the major types of gradient descent algorithms perform next to each other, summed up by these handyΒ notes:

1. Batch Gradient Descent

Accuracy β High

Time βΒ More

2. Stochastic Gradient Descent

Accuracy β Low

Time βΒ Less

3. Mini-Batch Gradient Descent

Accuracy β Moderate

Time βΒ Moderate

We hope you enjoyed this series and learned something new, no matter your starting point or machine learning background. Knowing this essential algorithm and its variants will likely prove valuable as you continue on your AI journey and understand more about both the technical and grand aspects of this incredible technology. Keep an eye out for other blogs offering, even more, machine learning lessons, and stayΒ curious!

#### Resources:

*Batch Gradient Descent**βββ**Google Colab**,Β**GitHub**Stochastic Gradient Descent**βββ**Google Colab**,Β**GitHub**Mini Batch Gradient Descentβ**ββ**Google Colab**,Β**GitHub*

#### Citation:

For attribution in academic contexts, please cite this workΒ as:

Shukla, et al., βThe Gradient Descent Algorithm & its Variantsβ, Towards AI, 2022

#### BibTex Citation:

@article{pratik_2022, title={The Gradient Descent Algorithm & its Variants}, url={https://towardsai.net/neural-networks-with-python}, journal={Towards AI}, publisher={Towards AI Co.}, author={Pratik, Shukla}, editor={Lauren, Keegan}, year={2022}, month={Oct} }

References:

The Gradient Descent Algorithm and its Variants was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Join thousands of data leaders on the AI newsletter. Itβs free, we donβt spam, and we never share your email address. Keep up to date with the latest work in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming aΒ sponsor.

Published via Towards AI