Why Ethics in AI Matters: Tackling Bias and Building Fair Machine Learning Systems

Author(s): Yuval Mehta

Originally published on Towards AI.

After learning that a test AI hiring tool discriminated against resumes that contained the word “women’s,” Amazon quietly discontinued it in 2018. The model had successfully taught itself that male candidates were preferred after being trained on resumes submitted to the company over ten years. There was no coding problem here. It was prejudice, ingrained in decision-making, supported by the model, and inherited from the data.

This is not a singular occurrence. Critical systems that affect healthcare, employment, justice, education, and finance have now included AI. Biased models have cultural as well as technological repercussions when they make judgments at scale.

This article explores what AI bias really is, how fairness is defined in machine learning, where bias emerges in the ML pipeline, and what’s being done, or should be, to mitigate it. Understanding these issues is essential for developers, product leaders, and policymakers working in or around artificial intelligence today.

Understanding AI Bias and Fairness

Systematic mistakes that lead to unfair results, like giving one group preference over another, are referred to as bias in AI. Biased labeling techniques, unbalanced training data, or the goals established during model optimization are some of the ways it can appear. These biases are frequently inherited from past data that reflects societal disparities rather than being purposefully imposed.

Fairness, on the other hand, is a complex and context-dependent concept. Various frameworks exist for defining fairness in algorithmic systems, including:

• Statistical parity: which seeks equal outcomes across groups.
• Equal opportunity: requires similar true positive rates across groups.
• Individual fairness: which emphasizes that similar individuals should be treated similarly.

There is a difference between these definitions. The application and social setting frequently influence which definition is best. Fairness in healthcare diagnostics, for example, can call for a different approach than fairness in lending decisions.

Bias is important for pragmatic as well as ethical reasons. When used in a variety of real-world contexts, biased systems frequently perform poorly, undermine confidence, and pose legal risks.

How Bias Enters Machine Learning Pipelines

Bias can enter the machine learning lifecycle at multiple points and often before a single line of model code is written.

Data Collection

The majority of machine learning models are only as good as the training data. Models can replicate and even magnify patterns of discrimination, underrepresentation, or societal preconceptions that are reflected in data. For instance, a crime prediction algorithm that has been trained on neighborhoods with excessive police presence is likely to continue to target those same communities.

Data Labeling

When labeling, human annotators contribute their own cultural background and presumptions. This can lead to systematic labeling bias in subjective tasks, including identifying poison in text or emotions on faces.

Model Optimization

The majority of models prioritize efficiency or accuracy over justice. Models will prioritize the majority class, frequently at the expense of minority classes, in the absence of clear limits or regularization strategies.

Deployment and Feedback Loops

Biased decisions may reinforce themselves after deployment. For example, a healthcare resource allocation model may routinely under-treat a population. This reinforces the bias over time by further distorting the training data.

Tools and Frameworks for Fairness Auditing

To assist practitioners in identifying, quantifying, and reducing bias in machine learning systems, a number of open-source libraries and tools have surfaced. While these instruments offer a framework for accountability, they do not ensure fairness.

Fairlearn

Fairlearn, a Microsoft product, provides metrics and algorithms that let data scientists assess and lessen injustice. It includes visualization tools for comprehending model performance across demographic groups and interfaces readily with scikit-learn workflows.

AI Fairness 360 (AIF360)

AIF360, one of the most complete fairness toolkits, was developed by IBM Research. In addition to tutorials and real-world datasets, it offers over 70 metrics for bias detection and prevention.

What-If Tool

The What-If Tool is an interactive visual interface for ML model analysis that was created by Google’s PAIR team. Without having to create code, it enables users to investigate counterfactuals, fairness measurements, and group-wise performance.

Each of these tools encourages a different lens on fairness and offers technical teams a starting point for critical self-assessment.

Real-World Examples of Bias in AI Systems

Several high-profile incidents have shown the real-world consequences of unaddressed algorithmic bias.

Amazon’s Recruitment Tool

The resume screening algorithm used by Amazon punished resumes that included phrases related to women’s experiences, such as “captain of the women’s chess club.” Resumes sent to the company over a ten-year period, when the applicant pool was predominantly male, were used to train the model. After internal reviews revealed the tool’s discriminatory nature, Amazon finally discontinued it.

Twitter’s Image Cropping Algorithm

When users saw that Twitter’s automatic image cropping algorithm seemed to prefer white faces in previews, the social media platform came under fire in 2020. Twitter deactivated the algorithm and pledged to be transparent in future AI deployments after receiving a lot of negative feedback and conducting user-led trials.

Healthcare Risk Prediction

It was shown that a popular healthcare algorithm in the US consistently understates Black patients’ probability of getting sick. The model suggested Black patients were healthier than they actually were because it used healthcare prices as a proxy for need, and historically, Black patients received less care. Despite being seen as “high-performing” by conventional accuracy standards, this resulted in notable discrepancies in access to care.

Challenges and Best Practices

Addressing bias in AI is difficult, not least because fairness is a moving target. Technical interventions can reduce one kind of bias while exacerbating another.

Common challenges include:

• Lack of representative data: Minority groups are often underrepresented or poorly labeled.
• Trade-offs between performance and fairness: Improving fairness can sometimes reduce overall accuracy.
• Ambiguous fairness goals: Organizations often lack clarity about which definition of fairness they want to achieve.
• Regulatory uncertainty: While new laws are emerging, the compliance landscape remains fragmented.

Some best practices:

• Incorporate fairness assessments during the model development lifecycle, not after deployment.
• Perform disaggregated evaluations, assess performance separately across demographic groups.
• Document datasets and models using tools like Model Cards and Data Sheets.
• Collaborate with domain experts, ethicists, and impacted communities when designing high-stakes systems.

The Future of Ethical AI

Responsible AI is a rapidly developing field. Academic frameworks for causal fairness and explainability, specialized regulatory measures (such as the EU AI Act), and interdisciplinary teams in large enterprises are also emerging.

In the next few years, we can expect:

• Greater integration of fairness metrics in mainstream ML libraries
• Legal mandates for algorithmic audits and transparency reports
• The rise of fairness-by-design tools and automated monitoring systems

Ethics in AI will no longer be an afterthought, it will be a core competency.

Conclusion

Fairness is a dynamic and changing objective rather than a set standard since bias in AI is a complicated problem with roots in data, design, and social environment. It is crucial to identify and proactively correct these biases as AI systems have a growing impact on important facets of life, such as healthcare and employment. True ethical AI necessitates continuous cooperation between technical, ethical, and domain specialists, even though technologies like Fairlearn and AIF360 offer helpful assistance in recognizing and reducing unfairness. The task at hand involves not only identifying prejudice but also committing to ethical AI methods that promote accountability, equity, and trust at every level of machine learning development.

References

1. Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
2. Vincent, J. (2021). Twitter retires image cropping algorithm after discovering racial bias. The Verge. https://www.theverge.com/2021/5/19/22443538/twitter-photo-crop-changes-black-faces-bias-algorithm
3. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
4. Barocas, S., Hardt, M., & Narayanan, A. (2019). Fairness and Machine Learning. https://fairmlbook.org

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI