classification of unseen classes of image in open set classification

Open Set Image Classification: Tackling Unseen Classes

Traditional image classifiers are built under the closed‑set assumption: every test image belongs to one of the classes seen during training. In real‑world deployments, however, new objects appear constantly, and a model that forces every input into a known class will make confident yet erroneous predictions. Open set classification addresses this gap by explicitly recognizing when an image belongs to an unseen class.

Why Open Set Matters for Images

• Dynamic environments: Autonomous vehicles, surveillance systems, and e‑commerce platforms encounter novel products or obstacles.
• Safety and trust: Misclassifying a hazardous object as a benign one can have severe consequences.
• Scalability: Continuously retraining a closed‑set model for every new category is impractical.

Key Concepts

1. Known vs. Unknown Space

The feature space learned by a deep network can be divided into two regions:

• Known space – dense clusters formed by training classes.
• Open space – regions where training data is scarce or absent, likely containing unseen classes.

2. Open Space Risk

Open space risk quantifies the probability that a classifier will assign a label to a sample that lies far from any known class prototype. Minimizing this risk is central to open set design.

3. Thresholding & Calibration

Most open set methods rely on a confidence score (e.g., softmax probability, Mahalanobis distance, or energy score). A carefully chosen threshold separates “known” predictions from “unknown” rejections.

Popular Approaches

Softmax Calibration and Temperature Scaling

By raising the temperature of the softmax layer, the output distribution becomes softer, making high confidence less likely for out‑of‑distribution samples. Calibrated probabilities improve the reliability of the unknown detector.

OpenMax

OpenMax replaces the final softmax layer with a Weibull‑based model that estimates the probability of an input belonging to an unknown class. It leverages the activation vectors of the penultimate layer to compute a “distance” to known class centroids.

Mahalanobis Distance‑Based Detector

Feature vectors are assumed to follow a class‑conditional Gaussian distribution. The Mahalanobis distance between a test sample and each class mean serves as an uncertainty measure; large distances indicate potential novelty.

Energy‑Based Models

Recent work shows that the energy of a sample (negative log‑sum‑exp of logits) is a robust indicator of out‑of‑distribution data. A simple threshold on energy can separate known from unknown samples without extra parameters.

Generative Approaches

Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) learn the data manifold of known classes. Reconstruction error or discriminator confidence can flag images that lie outside this manifold as unseen.

Designing an Open Set Pipeline for Images

1. Feature extractor: Use a pretrained CNN (e.g., ResNet‑50) and fine‑tune on your known classes.
2. Class prototypes: Compute class means (or more sophisticated manifolds) in the feature space.
3. Uncertainty estimator: Choose a detector (OpenMax, Mahalanobis, energy, etc.) and train any required parameters on a validation split.
4. Threshold selection: Use a held‑out set of known and synthetic unknown images to set the operating point (e.g., maximizing F1‑score for known/unknown discrimination).
5. Post‑processing: Optionally route rejected images to a human reviewer or an incremental learning module that can create new class prototypes on the fly.

Evaluation Metrics

• AUROC (Area Under ROC): Measures the trade‑off between true‑positive rate for known classes and false‑positive rate for unknowns.
• OSCR (Open Set Classification Rate): Combines closed‑set accuracy with unknown detection performance.
• F‑measure for known/unknown: Harmonic mean of precision and recall on the binary unknown detection task.

Future Directions

• Continual Open Set Learning: Integrate mechanisms that not only detect unknowns but also adapt the model incrementally as new classes are labeled.
• Hybrid Generative‑Discriminative Models: Leverage the expressive power of diffusion models to better model the open space.
• Cross‑modal Open Set: Combine visual cues with textual descriptions (e.g., CLIP) to improve unknown detection.

Takeaway

Open set image classification shifts the paradigm from “assign every picture a label” to “understand when we don’t know.” By incorporating uncertainty estimation, calibrated scores, and robust detection thresholds, practitioners can build systems that remain reliable even as the visual world evolves.