Bias in Natural Language Processing (NLP)

Author(s): Rayan Potter

Originally published on Towards AI.

The rising popularity of natural language processing (NLP) and machine learning technologies underscores the importance of recognizing their role in shaping societal biases and stereotypes. While NLP applications have achieved success in modeling tasks like sentiment analysis, machine translation, and text summarization, these models can perpetuate societal biases present in their training datasets. These biases pose significant threats to equity, justice, and democracy.

In this article, we will discuss how NLP reflects and amplifies societal prejudices, explore their consequences, and outline steps that companies providing natural language services need to take toward mitigating them.

Understanding Societal Biases in Natural Language Processing

Bias is favoring one person, thing, or group over another unfairly. Bias is not programmed into natural language processing models; instead, it implicitly creeps into the model through the statistical patterns in language data it learns from. The training data may incorporate societal prejudices and derogatory stereotypes, such as racism, sexism, and ableism.

These biases can be perpetuated in various NLP applications, including word embeddings and downstream applications — such as sentiment analysis, job candidate screening, university admissions, and essay grading.

Biases in Word Embeddings

Developers use unsupervised learning to prepare data for NLP models. Specifically, unsupervised models transform raw text data into word embeddings (numerical representations of text data) fed into NLP models. These models analyze massive amounts of text data, such as websites, social media, and books to create vectors that capture a word’s meaning and its relationship to other words.

However, while searching for hidden patterns in text data, these models are exposed to more than just semantic information — they are subjected to societal biases present in the data. These biases can then be embedded into word embeddings and inherited by supervised models, leading to biased outputs.

For example, sentences in an article might associate words related to doctors, engineers, and scientists mostly with men, while females may be portrayed as nurses, homemakers, or social workers.

Types of Bias in Natural Language Processing Services

Here are common biases in natural language processing services:

Gender Bias

Gender bias is a significant and widespread issue in NLP models. Many reports show bias in advanced language models, such as GPT-3, where word embeddings tend to associate men with competency and occupations requiring higher education (doctors, lawyers, CEOs, etc.) in downstream NLP tasks. Whereas, in response to the prompt “What gender does a nurse belong to?”, it is more likely to output “It’s female.”

Research published in The Artificial Intelligence and Emerging Technology Initiative of The Brookings Institution highlights numerous examples of gender bias in language applications using machine learning. Researchers found that NLP models working with word embeddings picked up biases based on how words are connected in the training data.

For example, words like ‘kitchen’ and ‘art’ were more frequently used with the word ‘woman’, and words like ‘science’ and ‘technology’ appeared in sentences including the word ‘man’. Such gender bias embedded in NLP systems leads to biased output.

Racial Bias

NLP systems have also displayed racial bias. A 2017 Princeton study discovered that online prejudices against African Americans and the Back community were reflected by model embeddings. As per the study, historically, Black names were more significantly associated with negative words as compared to traditional White names, reflecting real-world prejudices present in training data.

Such racial bias in machine learning extends back even further. The study also mentioned 2004 research that found similar bias in resume assessment done through machine learning algorithms.

Moreover, word embeddings display the most substantial bias for words or phrases representing people with intersectional identities, such as race and gender, relative to other word combinations. For example, the representation of phrases like ‘African American women’ or ‘Mexican American women’ can be more negatively biased than just ‘African American’ or ‘woman’ alone.

Many AI algorithms creating word embeddings are trained on datasets that reflect the current social order, which can lack diversity and be biased towards certain groups. Due to a lack of diversity, the data used to train word embeddings likely has more information about white men. As a result, other social groups are primarily represented as minorities within the system.

Bias in downstream NLP applications, such as automated resume screening, might not only reflect existing biases but amplify them in society, impacting future generations by limiting their career opportunities.

How to Mitigate Biases in Natural Language Processing

While bias in natural language processing can be handled by debiasing the dataset early on or the model afterward, the ideal approach is to derbies the dataset to prevent the model from learning biased patterns. Here are some effective strategies to derbies natural language processing models:

Data Manipulation

As described earlier, the main reason for bias in natural language processing algorithms is unbalanced original datasets, i.e., more text associating words related to ‘doctors’ with ‘male’ and words ‘nurses’ with ‘female’. With this type of association, the NLP model is more likely to predict “male” for ‘doctors’. To address bias, it is essential to have a balanced dataset where all groups are represented similarly for the model to learn from.

For example, data augmentation algorithms such as SMOTE (Synthetic Minority Oversampling Technique) can be employed to create synthetic data points for the minority group (female doctors) in the dataset. Alternatively, one can choose to remove some data points from the majority group to make the dataset balanced.

Bias Fine-Tuning

The bias fine-tuning method leverages the concept of transfer learning. It involves fine-tuning a relatively unbiased pre-trained natural language processing model on a new, more biased dataset. This enables the model to adapt to the specific task requirements of the biased dataset without inheriting biases from that data. Research suggests this method can achieve an accuracy score very similar to the model directly trained on unbiased data.

Data Annotation

Data annotation is a crucial step in NLP model development, especially in addressing bias. It involves labeling and categorizing text data to train NLP models. Annotators can flag potentially biased datasets. Biases can include stereotypes, unequal representation of races or genders, or even culturally insensitive language. As a result, developers can take steps to mitigate the bias, such as collecting more balanced data and eliminating biased text data.

Diversity in Developing and Auditing NLP Models

Other than training data, the bias can emerge from the team developing the model. A study at the 2020 NeurIPS machine learning conference suggested a negative correlation between the level of diversity of the development team and biased NLP models. According to a Brookings Institution study, a diverse AI audit team is essential for the ethical development of machine learning technologies.

A diverse audit group can test the model from various perspectives and help identify and mitigate potential bias throughout the NLP model creation process.

Conclusion

The growing presence of NLP services in our daily lives deepens concerns about bias in their algorithms. These sociotechnical systems absorb human biases and accurately ingest them as they learn from the training language data. This necessitates that development companies take bias mitigation steps to prevent the spread of discrimination further through these technologies.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI