Concept
Anomaly Detection
The task of identifying rare events or observations that deviate significantly from expected patterns - used to catch fraud, detect system failures, find manufacturing defects, and flag medical abnormalities.
Added May 18, 2026
Anomaly detection is the problem of identifying observations that are substantially different from the majority of the data - points that do not fit the expected patterns. Unlike most supervised learning, anomaly detection often operates in a one-class or semi-supervised setting: you have many examples of normal behaviour but few (or no) labelled examples of the anomalies you are trying to catch, because by definition, anomalies are rare.
Statistical anomaly detection uses probability distributions to identify low-probability events. Gaussian-based methods model each feature as normally distributed; observations more than k standard deviations from the mean are flagged. Multivariate Gaussian models capture correlations between features. Density estimation methods (kernel density estimation) flag points in low-density regions of the feature space. Statistical control charts (Shewhart charts, CUSUM) detect process deviations in sequential manufacturing data.
Machine learning anomaly detection learns a model of normal behaviour and flags observations that diverge. Isolation Forest works by building random decision trees and measuring how quickly a point is isolated (anomalies are isolated with fewer splits because they are in sparse regions). One-Class SVM learns a hyperplane that encloses the normal data in a high-dimensional feature space; points outside this boundary are anomalies. Autoencoders are trained to reconstruct normal examples; anomalies reconstruct poorly (high reconstruction error) because the latent space learned from normal examples does not represent the anomaly's patterns.
Deep learning approaches model complex, high-dimensional normal distributions. Deep autoencoders and Variational Autoencoders (VAEs) learn compressed representations of normal data; reconstruction error serves as an anomaly score. For time series, LSTM-based autoencoders model temporal dynamics and flag sequences that are hard to reconstruct. GANs have been used for anomaly detection: the discriminator learns to distinguish real from fake normal examples, and its score can indicate how normal a test example is.
Fraud detection is the highest-value anomaly detection application. Credit card fraud detection must identify fraudulent transactions (rare, typically 0.1-0.5% of all transactions) from billions of daily transactions in real time. Graph-based fraud detection identifies coordinated rings of accounts exhibiting suspicious interaction patterns. Industrial anomaly detection identifies manufacturing defects in product images or sensor readings before defective products ship. Network intrusion detection identifies unusual traffic patterns indicating cyberattacks.
The class imbalance problem is severe in anomaly detection: if 0.1% of transactions are fraudulent, a model that labels everything as normal achieves 99.9% accuracy but is useless. Precision and recall on the anomaly class, F1 score, AUC-ROC, and AUC-PR (area under the precision-recall curve) are more appropriate metrics. Oversampling the minority class (SMOTE), undersampling the majority, and cost-sensitive learning address the imbalance.
Analogy
A veteran quality control inspector at a factory who has spent years examining normal products and has developed an acute sense of what they should look and feel like. When a defective product comes through - subtly wrong texture, unusual weight, slight discolouration - the inspector immediately flags it, even without ever having seen that specific defect before. They are not pattern-matching to a catalogue of known defects; they are detecting deviation from the deeply internalised model of normality. Anomaly detection algorithms formalise this: learn a model of normal, flag what deviates.
Real-world example
PayPal's fraud detection system processes 40 billion transactions per year and detects fraud in sub-second real-time. Features include transaction amount, merchant category, geographic location, time of day, device fingerprint, and behavioural biometrics (how the user types, moves their mouse). A gradient boosted model and neural network ensemble assign fraud probability to each transaction. Transactions above threshold are declined, sent for step-up verification, or queued for human review. The system blocks billions of dollars of fraud annually while maintaining false positive rates low enough that legitimate customers are not excessively inconvenienced.
Why it matters
Anomaly detection is applied wherever the cost of missing rare important events is high: fraud costs financial systems billions annually, industrial defects cause recalls and safety incidents, network intrusions lead to data breaches, and medical anomalies cause misdiagnoses. Understanding anomaly detection - the one-class learning problem, the class imbalance challenge, and the appropriate evaluation metrics - is essential for building reliable systems in these high-stakes domains.
In the news
Related concepts