AI’s Got Some Explaining to Do

Author(s): Paul Ferguson, Ph.D.

Originally published on Towards AI.

Why We’re Demanding Answers from Our Smartest Machines

Artificial intelligence is making decisions that impact our lives in profound ways, from loan approvals to medical diagnoses. Yet, for all their sophistication, they often can’t explain their choices — this lack of transparency isn’t just frustrating — it’s increasingly problematic as AI becomes more integrated into critical areas of our lives.

Enter Explainable AI (XAI), a field dedicated to making AI’s decision-making process more transparent and understandable.

What is Explainability AI (XAI)?

Explainability AI (XAI) is concerned with understanding how AI systems make decisions, and as AI becomes more integrated into critical areas of our lives, it’s crucial that we can trust and validate its decisions [1].

XAI isn’t just about transparency; it’s about accountability and fairness [2]. Imagine an AI system denying you a loan or misdiagnosing a medical condition…wouldn’t you want to know why?

Techniques for Peering into the AI Mind

Scientists and researchers have developed several techniques to make AI more explainable:

1. Model Attention:

Helps us understand which parts of the input data the AI focuses on most. It’s particularly useful in natural language processing [3].

2. Saliency Maps:

Provide visual explanations for image-based decisions, highlighting the most important parts of an image for the AI’s prediction [4].

3. LIME (Local Interpretable Model-agnostic Explanations):

Creates a simpler, interpretable model to explain individual predictions [5].

4. Shapley Values:

Borrowed from game theory, this method assigns importance values to different features in the decision-making process [6].

The Challenge of Modern AI Models

While these XAI techniques work well for traditional ML models, modern AI systems like Large Language Models (LLMs) present new challenges.

These massive models, from OpenAI and others, can process and generate human-like text, but understanding their decision-making process is far from straightforward [7].

Researchers are exploring new methods like attention visualisation and prompt engineering to shed light on these complex systems.

These new approaches aim to help us understand how LLMs work and explain their outputs. As AI continues to advance, finding ways to make it more transparent and explainable remains a key priority.

Real-World Impact: When Explainability Matters Most

Explainability isn’t just an academic exercise, it has real-world implications across various sectors:

1. Healthcare:

A recent study published in Nature Medicine [8] examined AI models used for analysing chest X-rays. The researchers discovered that these models were inadvertently developing the ability to predict demographic information such as race, gender, and age from the images, despite not being explicitly trained to do so.

This unintended capability led to “fairness gaps” in diagnostic accuracy across different demographic groups.

Without explainable AI techniques, such critical biases might go unnoticed, potentially leading to disparities in healthcare outcomes and misdiagnoses for certain populations.

Having personally worked in healthcare for 8 years, I’ve seen firsthand the importance of understanding how decisions are made. Explainable AI techniques not only help identify potential biases but also enable healthcare professionals to integrate AI with confidence and to ensure equitable care for all patients.

2. Finance:

In areas like credit scoring and fraud detection, understanding why an AI system made a particular decision is essential for regulatory compliance and customer trust [9].

3. Autonomous Vehicles:

Explainable AI techniques are vital for understanding and validating the decision-making processes of self-driving cars, enhancing safety and reliability.

Learning from AI’s Mistakes

Sometimes, the true value of explainability becomes apparent when it reveals flaws in model learning.

A classic example is the ‘Husky vs Wolf’ case study: researchers trained a model to differentiate between huskies and wolves, only to discover that the model was primarily focusing on the presence of snow in the background, rather than the features of the animals themselves [5].

This case highlighted how models can learn spurious correlations that don’t generalise well, emphasising the need for explainable AI to uncover such issues.

The Road Ahead: Challenges and Opportunities

While we’ve made significant progress in explainable AI, several challenges remain:

1. Balancing Accuracy and Explainability:

There’s often a perceived trade-off between model performance and explainability. However, recent advancements suggest that this gap may be narrowing.

2. Standardisation:

As the field evolves, there’s a growing need for standardised approaches and metrics for evaluating AI transparency [10].

3. Ethical Considerations:

Ensuring that explanations are accessible and meaningful to diverse user groups, including those without technical backgrounds, is an ongoing challenge.

Looking ahead, we can expect to see increased integration of explainable AI in critical sectors, advancements in natural language explanations, and the development of more comprehensive regulatory frameworks. As AI continues to evolve, so too will our methods for understanding and explaining these complex systems.

Conclusion: The Power of Understanding

Explainability in AI and ML is not just a technical challenge; it’s a crucial component of responsible AI development. As these technologies become more deeply integrated into our lives, the ability to understand, trust, and validate their decisions becomes paramount.

By continuing to advance the field of explainable AI, we can harness the full potential of these powerful technologies while ensuring transparency, fairness, and accountability. After all, the goal isn’t just to create smarter machines, but to create a smarter, fairer world for all of us.

If you’d like to find out more about me, please check out www.paulferguson.me, or connect with me on LinkedIn.
For more articles on AI in business, feel free to explore my Medium profile.

References

[1] F. Doshi-Velez and B. Kim, “Towards A Rigorous Science of Interpretable Machine Learning,” arXiv preprint arXiv:1702.08608, 2017.

[2] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nature Machine Intelligence, vol. 1, pp. 206–215, 2019.

[3] A. Vaswani et al., “Attention is All you Need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

[4] R. R. Selvaraju et al., “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 618–626.

[5] M. T. Ribeiro, S. Singh, and C. Guestrin, “Why Should I Trust You?: Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144.

[6] S. M. Lundberg and S. I. Lee, “A Unified Approach to Interpreting Model Predictions,” in Advances in Neural Information Processing Systems, 2017, pp. 4765–4774.

[7] T. Brown et al., “Language Models are Few-Shot Learners,” in Advances in Neural Information Processing Systems, 2020, pp. 1877–1901.

[8] M. Zhang et al., “The limits of fair medical imaging AI in real-world generalization,” Nature Medicine, vol. 30, no. 2, pp. 389–399, Feb. 2024.

[9] B. H. Misheva, “Explainable AI in Credit Risk Management,” arXiv preprint arXiv:2103.00949, Mar. 2021.

[10] D. V. Carvalho, E. M. Pereira, and J. S. Cardoso, “Machine Learning Interpretability: A Survey on Methods and Metrics,” Electronics, vol. 8, no. 8, p. 832, 2019.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI