Taylor Series in AI.

Author(s): Surya Maddula

Originally published on Towards AI.

P.S. Read thru this article a bit slowly, word by word; you’ll thank me later 😉

— — — —

Let’s see what the Taylor Series is and how it relates to its applications in AI & Processing.

“The study of Taylor series is largely about taking non-polynomial functions and finding polynomials that approximate them near some input” — 3Blue1Brown.

What? 😵‍💫

Okay, let’s try to rephrase that to understand better:

Imagine you have a really complicated function, like a curve on a graph, and you want to understand what it looks like near a certain point. The Taylor Series helps us do this by breaking the function into a bunch of smaller, easier pieces called polynomials.

It is a way to approximate a function using an infinite sum of simpler terms. These terms are calculated using the function’s values and its derivatives (which tell us the slope and how the function changes!).

Consider this:

If you have a function f(x), and you want to approximate it near a point, say at x = a, then this is what the Taylor Series looks like:

f(x) = f(a) + f’(a)(x-a) + f”(a)/2! (x-a)² + f”’(a)/3!(x-a)³…

Take a second to go thru that again.

Here,

• f(a) is the value of the function at x = a
• f’(a) is the slope at x = a
• f”(a) is how the slope is changing at x = a

We all know that n! stands for n factorial, which is the product of all positive integers up to n.

ex: 3! = 1 x 2 x 3 = 6

— — — —

Let’s look at a very simple example to understand this better: the exponential function of e^x.

For e^x around x = 0 is:

(try formulating it yourself first, referring to the formula above 😉

e^x = 1 + x + x²/2! + x³/3! + x⁴/4! …

Conceptual

Think of the Taylor Series as a recipe for building a copy of a function near the point a, sort of like a stencil. The more terms, or in this case, ingredients you add, the closer you will get to the original function, and the closer your approximation will be.

So, if you want to estimate e^x for small values of x, you can just use the first few terms:

e^x = 1 + x + x²/2 + x³/6…

This exercise should give you a good idea of how e^x looks like at x = 0.

Pro-Tip: Repeat this exercise a few times to better grasp the concept.

Okay, so what? How is this useful in the real world?

Well, The Taylor series allows us to approximate complex functions with simpler polynomials, which makes calculations easier and faster!

Here are a few examples —

Physics

Example: Pendulum Motion

Imagine a pendulum, like a clock. Scientists use math to understand how it swings. The exact math is tricky, but for small swings, the Taylor Series helps simplify it, making it easier to predict the pendulum’s motion.

So that you can be late for school.

Engineering

Example: Control Systems

Think about a car’s cruise control, which keeps the car at a steady speed. Engineers use the Taylor Series to simplify complex math so the system can react smoothly and keep the car at the right speed.

So that you can ignore the speed limit.

Economics

Example: Interest Rates

When banks calculate interest on savings, they sometimes use complicated formulas. The Taylor series helps simplify these calculations so they can more easily determine how much money you’ll earn!

So that the government can take the right percentage of that in taxes.

Computer Science

Example: Machine Learning

In ML, computers learn from data. The Taylor series helps simplify the math behind these learning algorithms so computers can learn faster and more effectively.

So that you become lazy and spend all day on them.

Medicine

Example: Medical Imaging

When doctors take MRI or CT scans, they receive a lot of data. The Taylor Series helps turn this data into clear images of the inside of the body, making it easier for doctors to diagnose problems!

So that you ignore their advice and walk to McDonald's (cuz you don’t run XD)

Everyday Technology

Example: GPS Systems

When you use GPS on your phone, it calculates your location using satellites. The Taylor series helps make the math simpler so your GPS can quickly and accurately tell you where you are.

So that you can lie about where you are.

Weather Forecasting

Example: Predicting Temperature

Meteorologists predict the weather using complicated math. The Taylor series helps simplify these equations, allowing them to make more accurate forecasts about temperature, rain, and wind.

So that you never open the weather app and always forget an umbrella.

— — — —

So YOU might not use the Taylor Series in the real world — ever; but it’s used every day to make your life simpler!

— — — —

Now, for the interesting bit:

How do we use the Taylor Series in AI? 🔥

You’ve already taken a look into how this is used in ML above and how it helps simplify the math behind these learning algorithms so computers can learn faster and more effectively.

Let’s dive deeper:

First, where can we even use this in AI?

Forget the term AI for a while. Just think of where we use the Taylor Series in everyday mathematical and engineering problems. We can later extrapolate that into how we use it in AI and Machine Learning.

We’ve already discussed how we use it in physics, engineering, economics, CS, medicine, GPS, and weather forecasting. I suggest you scroll back to that again; it’ll click more now and at the end of this article. 🖱️

In AI, we often deal with complex math problems. The Taylor series helps simplify these problems so our AI can learn and make better decisions.

Example:

For Training AI Models:

When we train an AI model, like a neural network, we want to improve its prediction accuracy. We do this by adjusting its parameters (like weights in a neural network) to minimize errors. (w&b)

Taylor series helps here by letting us approximate how small changes in the parameters will affect the error. This approximation helps us find the best way to adjust the parameters to improve the model’s predictions.

Training Neural Networks:

When training a neural network, we want to minimize a loss function, which is how we measure the difference between the predicted outputs and the actual targets. To achieve this, we adjust the network’s parameters (weights and biases) to reduce the loss. This is usually done by using gradient-based optimization methods.

Example

Imagine you’re on a big hill and you want to find the lowest point. To get there, you need to figure out which direction to walk.

• The Hill: Think of the hill as the “loss function,” which shows how good or bad your predictions are. The steeper parts of the hill represent higher loss (bad predictions), and the flatter parts represent lower loss (better predictions).
• Finding the Best Path: When you’re on the hill, you can’t see the whole thing, just the part right around you. To decide which way to walk, you use the slope (how steep it is) right where you are. This is like the “gradient” in ML, which tells you the direction that increases the loss the most.
• Using the Slope: If you want to get to the lowest point, you walk in the opposite direction of the slope (since you want to go downhill). You keep taking small steps in this direction to lower the loss.

Where does the Taylor Series Help

The Taylor series is like having a small map that shows you how the hill looks around you. It helps you understand the local slope better, so you can make better decisions about which way to walk.

• Simple Map: The basic Taylor series is like a simple map that shows the hill’s slope around you.
• Detailed Map: If you want a more accurate map, you might also look at how the hill curves, which is like adding more details to your Taylor series.

1. Training AI Models: Gradient Descent

Cost Function

Same analogy again: Imagine the cost function as a hill we need to climb down to find the lowest point (the best solution). As stated, the lower the value, the better it is.

Gradient

The gradient tells us the direction of the steepest slope.

Gradient Descent:

The Taylor Series helps us approximate the cost function around a point, telling us how it changes when we adjust the parameters slightly. This approximation makes it easier to determine which direction to move in to reduce the cost.

Example:

Imagine you’re trying to adjust the angle of a ramp to make a ball roll into a target. The cost function tells you how far the ball is from the target. The Taylor series helps you understand how changing the ramp’s angle (parameters) will affect the ball’s position (cost) so you can make better adjustments.

2. Making Calculations Easier

Neural networks use something called activation functions to decide whether to activate a neuron (like a switch). One common activation function is the sigmoid function.

Example

Think of the Sigmoid Function as a dimmer switch that adjusts light brightness. The Taylor series helps simplify the math behind how much the light should dim based on the input, making it easier for the neural network to process. It helps a neural network decide whether to activate a neuron. The Taylor series can approximate this function and speed up calculations.

3. Approximating Complex Functions

In Reinforcement Learning, an AI learns by trying different actions and getting rewards or penalties (trial and error). The value function estimates the expected rewards for actions.

How the Taylor Series Helps

The Taylor series approximates the value function, which can be very complex. This approximation helps the AI predict rewards more easily, allowing it to choose better actions.

Example

Imagine you’re playing a video game, and you want to predict which moves will earn you the most points. The value function helps with this prediction, and the Taylor series simplifies the calculations, making it easier to decide the best moves.

4. Handling Uncertainty: Bayesian Inference

Sometimes, we need to understand how uncertain our AI model is about its predictions. The Taylor series helps us estimate this uncertainty, making our AI more reliable.

Example: Bayesian Inference

In Bayesian inference, we update our beliefs about the AI model’s parameters based on new data. The Taylor series helps simplify these updates, making them easier to calculate.

5. Understanding Model Behavior

The Taylor Series can also be employed to understand and interpret the behavior of machine learning models. By expanding the model’s function around a point, we can gain insights into how changes in input affect the output, which is crucial for tasks like feature importance analysis and debugging models.

Specific Applications

1. Neural Networks Training: In training neural networks, the backpropagation algorithm often uses the Taylor Series for calculating the gradients of weights.
2. Regularization Techniques: Some regularization techniques in machine learning, like Tikhonov regularization, can be understood and derived using the Taylor Series expansion.
3. Non-linear Models: For non-linear models, the Taylor Series provides a way to linearize the model around a point, which is useful for analysis and optimization.
4. Algorithm Development: Advanced machine learning algorithms, like Gaussian processes and some ensemble methods, sometimes use the Taylor Series for development and refinement.

“The fundemental intuition to keep in mind is that they translate derivative information at a single point to approximation information around that point” — 3Blue1Brown

— — — —

So, with the multiple examples and instances, we’ve discussed how the concept of the Taylor Series eases our lives, from real-world applications in Engineering & Computer Science to how it simplifies working with and building AI.

I think that the Taylor series is like a magic tool that turns complicated math into simpler math because it helps AI learn faster, make better decisions, and handle complex problems more efficiently. That’s the inference and understanding I got from the research I’ve done and while drafting this article.

Now, as we’re approaching the end, I want you to reflect back: What exactly do we mean when we say “Taylor Series,” instances of using it irl, examples of Taylor series’ use, and finally, the cherry on top, how do we use Taylor series in AI.

Read through the entire article again, and compare it with the understanding you have now; you’ll notice the difference, as I did 😉

That’s it for this time; thanks for Reading and Happy Learning!

References: How I learned this concept —

Taylor series | Chapter 11, Essence of calculus (youtube.com) (3Blue1Brown)

Exploring the Role of Taylor Series in Machine Learning: From Function Approximation to Model Optimization | by Everton Gomede, PhD | 𝐀𝐈 𝐦𝐨𝐧𝐤𝐬.𝐢𝐨 | Medium

A Gentle Introduction to Taylor Series — MachineLearningMastery.com

How is Taylor series used in deep learning? (analyticsindiamag.com)

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI