Author(s): Elangoraj Thiruppandiaraj

Originally published on Towards AI.

This member-only story is on us. Upgrade to access all of Medium.

I’ve worked on anomaly detection problems for a while now, and one obstacle I consistently face is extreme imbalance in the data. When only a fraction of a percent of your records are “anomalies,” most standard methods — like oversampling or undersampling — just don’t cut it. In my experience, these approaches either lead to overfitting (repeating the same rare examples too often) or throw away valuable data. That’s where a newer technique I’ve been exploring comes in: using Large Language Model (LLM) embeddings to spot subtle irregularities.

Even though embeddings are typically associated with text (thanks to tools like BERT, GPT, or other transformer models), I’ve found that the same idea — representing data in a dense, meaningful vector space — works wonders for detecting outliers across various data types. Let me walk you through the thinking and process behind this method.

Those experienced in anomaly or rare event detection are keenly aware of the data’s extreme imbalance. In many scenarios:

Less than 1% of the dataset represents rare or critical events, forming the minority class.99% or more of the dataset falls under the normal or unknown category —… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI