Facial Analysis With Masks? Learn How To Achieve 96% Accuracy

Last Updated on January 6, 2023 by Editorial Team

Author(s): Panos Moutafis, Ph.D., Rakshak Talwar, Mary Lim

Masks and face coverings are here to stay. This is a guide for training robust AI models without crossing the privacy line.

Masks and face coverings have been prevalent in many cultures and work environments for decades. But if you are reading this in the year 2021, we can read your mind — you are thinking about the pandemic! Masks became a must-have accessory in our daily lives due to Covid-19.

There is a lack of datasets with people wearing masks

Analyzing people’s faces has vast applications from retail stores to corporate campuses and experiential marketing. The question is how do we train robust AI models without having access to vast datasets of people wearing masks? If this keeps you up at night, we have excellent news!

Data Augmentation

Our team addressed the lack of datasets with people wearing masks by using data augmentation techniques. One may enhance their existing datasets (or publicly available ones) by overlaying masks on top of people’s faces. Training facial analysis models such as face detection and sex prediction become an easier task once you do this.

A researcher at Georgia Tech is managing an open-source project called “Mask The Face.” The source code is available on GitHub and can be used to convert face datasets into masked-face datasets.

Running the software package on your images is a cinch!

cd MaskTheFace
# Generic
python mask_the_face.py --path <path-to-file-or-dir> --mask_type <type-of-mask> --verbose --write_original_image

# Example
python mask_the_face.py --path 'data/office.jpg' --mask_type 'N95' --verbose --write_original_image

Source code and further details can be found on the GitHub project page.

Deviations

Using the software extensively is expected to produce a few inconsistent results. In some cases, it works extremely well and addresses variations in head pose and lighting conditions. In other cases, the algorithm will miss faces or misplace the mask in the presence of strong head pose variation and lighting aberrations. This is due to the performance of the face detector used in the project.

Even though some faces in the training set will not be masked this is generally okay because the overall dataset would comprise both masked and unmasked faces (see training section below). In addition, one may also use different detector options which are more robust for better results. All in all, the method is resilient and practical.

As you may see in the corresponding sample pictures, the referenced method produces realistic results. One may also choose from a variety of different masks to increase the diversity of face coverings in the dataset. The options include different patterns, colors, and intensity values.

Training

There are many different ways to train facial analysis models whether this pertains to detection, recognition, sex, age group, and/or sentiment. For the purposes of this guide we will focus on sex prediction assuming the images have already been cropped and aligned using a detection module which is already robust with occluded faces.

We trained our own classification task head which received feature maps from a battletested backbone. This is code for illustration purposes for what would be a small but important component of a much larger system. Nonetheless, the principles remain the same and allowed us to achieve high accuracy across a wide range of tasks.

Discussion

The simple process of data augmentation results in a test accuracy of over 96% for masked faces. The real-world performance is also extremely reliable. We ran our algorithm on a short video to illustrate how well the sex prediction model performs.

We have trained our algorithms to show us their predictions and also their confidence in the predictions they make. Because our system captures multiple impressions from the same face during inference we have configured it to use only high confidence predictions to further increase its robustness.

Considerations

Before we share our concluding remarks it is important to highlight a few key considerations.

First and foremost, there is a big difference between, say, age prediction and facial recognition. Identifying a person becomes more difficult when increasing the database size. This is not the case for other types of facial analysis. As a result, any loss of information will have a greater impact on recognition performance compared with the accuracy reduction when detecting demographic data.

The implications for real-world use cases are also quite different. Facial recognition often focuses on security applications such as controlling access where small errors can be very costly. On the other hand, headcount and demographics analytics are leveraged for understanding the target audience on a high-level. Small deviations from the ground truth typically have negligible impact on these insights.

The impact of masks and face coverings is also less pronounced for classification tasks compared with regression problems. For example, a regression model would need to be trained in order to quantify the happiness level of a person based on their facial expressions. Such models produce scores at inference time which tend to behave differently for masked versus non-masked faces.

An AI system can be only as good as the data provided to it

Last but not least, we would like to emphasize the importance of people’s right to identify with different genders. A trained AI model can be only as good as the data provided to it and the underlying biological and biometrics differences. Excluding predictions whose confidence score is low (see discussion section) and focusing on the right use cases is extremely important.

What the future holds

Developing artificial intelligence models requires careful planning and exhaustive testing. This is particularly true when it comes to applications intertwined with intimate data such as a person’s face.

We are humans and still have many things to learn. Nonetheless, our team remains extremely bullish about the future of facial analysis even in the presence of masks and face coverings.

We will continue to work diligently and incorporate safeguards which protect people’s privacy. You are invited to support our mission and join our journey!

Stay safe. Stay positive. Test negative.

1. Some images are sourced from FairFace and used under the CC-BY-4.0 license. A few of the images were modified with mask augmentation.
2. Additional images and videos are sourced from Zenus, Wikimedia Commons, Pexels, and Unsplash. They may have been modified from their original form.

Facial Analysis With Masks? Learn How To Achieve 96% Accuracy was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Published via Towards AI