Join thousands of students in our LangChain and Vector DBs in Production course, with over 50+ lessons and practical projects for FREE!.


Facial Analysis With Masks? Learn How To Achieve 96% Accuracy
Computer Vision

Facial Analysis With Masks? Learn How To Achieve 96% Accuracy

Last Updated on January 6, 2023 by Editorial Team

Author(s): Panos Moutafis, Ph.D., Rakshak Talwar, Mary Lim

Masks and face coverings are here to stay. This is a guide for training robust AI models without crossing the privacy line.

Demographics annotations (sex & age) predicted by an AI algorithm using face masks on a couple of people.
The Zenus co-founders demonstrating live facial analysis with masks. Full video available on this link.

Masks and face coverings have been prevalent in many cultures and work environments for decades. But if you are reading this in the year 2021, we can read your mind — you are thinking about the pandemic! Masks became a must-have accessory in our daily lives due to Covid-19.

There is a lack of datasets with people wearing masks

Analyzing people’s faces has vast applications from retail stores to corporate campuses and experiential marketing. The question is how do we train robust AI models without having access to vast datasets of people wearing masks? If this keeps you up at night, we have excellent news!

A picture of Mona Lisa with and without face mask to highlight the challenge of facial occlussions.
Now, is she smiling or not?

Data Augmentation

Our team addressed the lack of datasets with people wearing masks by using data augmentation techniques. One may enhance their existing datasets (or publicly available ones) by overlaying masks on top of people’s faces. Training facial analysis models such as face detection and sex prediction become an easier task once you do this.

A researcher at Georgia Tech is managing an open-source project called “Mask The Face.” The source code is available on GitHub and can be used to convert face datasets into masked-face datasets.

Running the software package on your images is a cinch!

cd MaskTheFace
# Generic
python --path <path-to-file-or-dir> --mask_type <type-of-mask> --verbose --write_original_image

# Example
python --path 'data/office.jpg' --mask_type 'N95' --verbose --write_original_image

Source code and further details can be found on the GitHub project page.


Using the software extensively is expected to produce a few inconsistent results. In some cases, it works extremely well and addresses variations in head pose and lighting conditions. In other cases, the algorithm will miss faces or misplace the mask in the presence of strong head pose variation and lighting aberrations. This is due to the performance of the face detector used in the project.

Examples of face mask augmentation being applied correctly and being ommitted by the AI algorithm.
Left image: The face on the bottom right was detected and mask augmentation was applied. Right image: The face in the upper left corner was not detected, and thus, no mask augmentation was performed.

Even though some faces in the training set will not be masked this is generally okay because the overall dataset would comprise both masked and unmasked faces (see training section below). In addition, one may also use different detector options which are more robust for better results. All in all, the method is resilient and practical.

Original and altered photo using AI software to apply face masks.
Illustration of a mask augmentation applied correctly with a variety of masks.

As you may see in the corresponding sample pictures, the referenced method produces realistic results. One may also choose from a variety of different masks to increase the diversity of face coverings in the dataset. The options include different patterns, colors, and intensity values.

Examples of different types of face masks and face coverings.
Mimic real-world scenarios with varying colors and types of masks


There are many different ways to train facial analysis models whether this pertains to detection, recognition, sex, age group, and/or sentiment. For the purposes of this guide we will focus on sex prediction assuming the images have already been cropped and aligned using a detection module which is already robust with occluded faces.

Picture depicting a male and a female person with and without mask augmenation.

We trained our own classification task head which received feature maps from a battletested backbone. This is code for illustration purposes for what would be a small but important component of a much larger system. Nonetheless, the principles remain the same and allowed us to achieve high accuracy across a wide range of tasks.


The simple process of data augmentation results in a test accuracy of over 96% for masked faces. The real-world performance is also extremely reliable. We ran our algorithm on a short video to illustrate how well the sex prediction model performs.

Picture of diverse individuals putting on a face mask. Sex annotations and face detections included.
Illustration of Zenus’ trained model working with and without face masks.

We have trained our algorithms to show us their predictions and also their confidence in the predictions they make. Because our system captures multiple impressions from the same face during inference we have configured it to use only high confidence predictions to further increase its robustness.


Before we share our concluding remarks it is important to highlight a few key considerations.

First and foremost, there is a big difference between, say, age prediction and facial recognition. Identifying a person becomes more difficult when increasing the database size. This is not the case for other types of facial analysis. As a result, any loss of information will have a greater impact on recognition performance compared with the accuracy reduction when detecting demographic data.

Infographic explaining the difference between facial recognition and facial analysis.
Personal identification vs. Anonymous statistics

The implications for real-world use cases are also quite different. Facial recognition often focuses on security applications such as controlling access where small errors can be very costly. On the other hand, headcount and demographics analytics are leveraged for understanding the target audience on a high-level. Small deviations from the ground truth typically have negligible impact on these insights.

Several images of the same person with varying facial expressions with and without face masks.
Can you tell which expressions correspond to a happier face with and without mask?

The impact of masks and face coverings is also less pronounced for classification tasks compared with regression problems. For example, a regression model would need to be trained in order to quantify the happiness level of a person based on their facial expressions. Such models produce scores at inference time which tend to behave differently for masked versus non-masked faces.

An AI system can be only as good as the data provided to it

Last but not least, we would like to emphasize the importance of people’s right to identify with different genders. A trained AI model can be only as good as the data provided to it and the underlying biological and biometrics differences. Excluding predictions whose confidence score is low (see discussion section) and focusing on the right use cases is extremely important.

What the future holds

Developing artificial intelligence models requires careful planning and exhaustive testing. This is particularly true when it comes to applications intertwined with intimate data such as a person’s face.

We are humans and still have many things to learn. Nonetheless, our team remains extremely bullish about the future of facial analysis even in the presence of masks and face coverings.

We will continue to work diligently and incorporate safeguards which protect people’s privacy. You are invited to support our mission and join our journey!

Stay safe. Stay positive. Test negative.

  1. Some images are sourced from FairFace and used under the CC-BY-4.0 license. A few of the images were modified with mask augmentation.
  2. Additional images and videos are sourced from Zenus, Wikimedia Commons, Pexels, and Unsplash. They may have been modified from their original form.

Facial Analysis With Masks? Learn How To Achieve 96% Accuracy was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Published via Towards AI

Feedback ↓