Image Processing with Fourier Transform

Author(s): Albert Nguyen

Originally published on Towards AI.

Image Processing is a crucial task in Computer Vision. For example, when we train a Deep Learning model with a small amount of image data, we need to synthesize new images using Image Processing methods to improve the performance.

There are many methods developed for Image Processing. This post will explore the application of 2D Fourier Transform methods to process images.

Background – Fourier Series

The Fourier series is found by the mathematician Joseph Fourier. He stated that any periodic function could be expressed as a sum of infinite sines and cosines:

Fourier Transform is a generalization of the complex Fourier Series. In image processing, we use the discrete 2D Fourier Transform with formulas:

Image in the frequency domain

Now we have the formulas, let's see what it looks like this when applied to an image:

The image on the right side is a spectrum of Fourier Transform. It is the heat map where the color represents the sine waves' amplitude, and position is their frequency. But at this point, we may get confused, like, "Hey," what does this tell me about the image?"

What we are doing with the 2D Fourier Transform is treating the image as a function of pixel position x and y. Imagine the function f(x, y) along axis (0,y):

So when the color jumps from "black (0)" to "white (255)", we say the color changes quickly, which means that a high amplitude and frequency sine wave contribute to that jump. So the low-frequency sine waves are responsible for the image's overall color, while high-frequency waves are responsible for how color changes.

To be honest, I can't find a better term for the responsibility of the high-frequency wave. However, the example below demonstrates my idea.

So, when we remove the low-frequency terms, we get an edge detection of the cat. On the opposite side, if we only keep the low-frequency waves, we will get a blurred image:

This is so cool when we all need to delete some terms, and we can do ‘blurring’ and ‘edge detection.’

Well, this is nothing surprising. We can do all this with openCV. So, what else can Fourier Transform do?

Fourier Transform and Convolution

The beauty of the Fourier Transform is we can do convolution on images by just multiplication on its frequency domain. The convolution measures the total product in the overlapping regions of 2 functions. In Deep Learning, we often know about it as a convolution layer. While mathematically, it will look like this:

If we plug the Fourier Transform formula into this operation for f and g, we will get:

This is so nice. With Fourier transform, we can change from computing a convolution to a multiplication. In the case of 2D Discrete Fourier Transform, this helps us reduce the time complexity by a lot.

Conclusion

Fourier Transform is a powerful tool and is widely used in many applications. In this blog, we have explored some usage of the FT in image processing. Fourier Transform is also used in some other applications in Deep Learning, which I find interesting and listed below:

1. Domain Adaption for Semantig Segmentation

2. Resolution-robust Large Mask Inpainting with Fourier Convolutions

3. etc.

Besides the frequency representation, the Fourier Transform also produces the phase representation of the image. This contains the spatial information inside the image. However, the plot for the phases domain is less informative, and it also makes functions that affect the spatial information become more complex.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI