I Tried Semantic Segmentation, and Here is What I Learned

Author(s): Gaurav Sharma

Originally published on Towards AI the World’s Leading AI and Technology News and Media Company. If you are building an AI-related product or service, we invite you to consider becoming an AI sponsor. At Towards AI, we help scale AI and technology startups. Let us help you unleash your technology to the masses.

Introduction

Semantic segmentation is a pixel-wise classification task with many valuable applications, such as object detection and image retrieval. Despite its usefulness, semantic segmentation can be difficult to achieve good results on. In this blog post, I’ll share my experience of trying semantic segmentation for the first time and the lessons I learned from it.

What is semantic segmentation?

Semantic segmentation is a type of computer vision technology that allows for identifying objects within an image. This is achieved by assigning each pixel in the image a class label, which denotes what object(s) are present in that particular location.

For example, in an image containing a person and a dog, semantic segmentation would identify the pixels corresponding to the person belonging to the “person” class and those corresponding to the dog as belonging to the “dog” class.

There are many potential applications for semantic segmentation. One example is self-driving cars, which need to be able to accurately identify objects on the road in order to avoid accidents. Another example is medical imaging, where semantic segmentation can automatically identify different types of tissue (e.g., cancerous vs. non-cancerous).

I decided to try semantic segmentation as I was interested in exploring its potential applications in medical imaging. My goals were to learn how to implement semantic segmentation using deep learning and to see if it could be used to identify different types of tissue from MRI scans accurately.

Why I decided to try semantic segmentation

I decided to try semantic segmentation for a few reasons:

I wanted to understand better how this technique works and its potential applications.

I was curious to see if I could improve upon the results of traditional image segmentation methods.

I wanted to experiment with a different dataset than I had used before to see if that would make a difference in the results.

To achieve these goals, I used the CamVid dataset (http://mi.eng.cam.ac.uk/research/projects/VideoRec/CamVid/). This dataset contains images of urban scenes, which I thought would be interesting to segment. Additionally, the CamVid dataset comes with pre-computed labels for each image, which made it easier to evaluate my results.

How I implemented semantic segmentation.

Semantic segmentation is a computer vision technique that classifies each pixel in an image into a predefined category. This differs from traditional image classification, which only assigns a label to an entire image. Traditional image classification models are limited because they cannot provide fine-grained information about an image. Semantic segmentation enables a more granular understanding of an image by labeling every pixel, allowing for better decision-making and improved performance in object detection and scene understanding tasks.

I decided to try semantic segmentation for two reasons:

First, to gain a better understanding of the technique, and second, to see if it could improve the results of my object detection models. My dataset consisted of images of cars, bikes, and pedestrians taken from a mounted camera on a vehicle. I used this dataset because it was small enough to be manageable but large enough to represent real-world data.

Pre-processing the data was relatively simple; I had to resize the images and split them into training, validation, and test sets. Building the semantic segmentation model was more complicated; I had to decide which architecture to use (I chose U-Net) and what loss function would be most appropriate (I chose cross-entropy). Training the model was also challenging; I had to experiment with different optimizers and learn rates before settling on Adam with a learning rate of 0.001.

Overall, I found that semantic segmentation was not as difficult as I had expected. The hardest part was choosing the correct hyperparameters, but even that wasn’t too bad once I got the hang of it. The results of my experiment were promising; on my validation set, my model achieved an IoU score of 0.87, which is quite good considering that semantic segmentation is notoriously tricky.

Results of my semantic segmentation experiment.

Evaluation metrics

After training the model, I evaluated it on the test set using two metrics: mean intersection-over-union (mIOU) and pixel accuracy. The mIOU metric measures the average overlap between the predicted segmentation map and the ground truth segmentation map. In contrast, pixel accuracy measures the percentage of correctly classified pixels.

Qualitative results

Overall, I was pleased with the qualitative results of my semantic segmentation experiment. In many cases, the model correctly identified objects in complex scenes with multiple entities.

There were also some cases where the model struggled, such as when there was significant occlusion or when the object was small in the image.

Lessons learned

One of the main lessons I learned from this experiment is that semantic segmentation is a complicated problem, and there is still much room for improvement. Additionally, I found that even small changes to the model architecture or training procedure can significantly impact the results.

Overall, I found semantic segmentation a fascinating and challenging problem, and I hope to continue exploring it in the future.

Conclusion

Overall, I found semantic segmentation to be a valuable tool with many potential applications. It was relatively easy to implement and yielded good results. If you are running a business, then there are several companies like Cogito Tech, Anolytics.AI, and others that provide semantic segmentation services. There are still some areas for improvement, but I’m encouraged by what I’ve seen.

I’m looking forward to seeing more advances in this area.

---

I Tried Semantic Segmentation, and Here is What I Learned was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Join thousands of data leaders on the AI newsletter. It’s free, we don’t spam, and we never share your email address. Keep up to date with the latest work in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI