How to reject boxes inside each other with Non Max Suppression

How to Reject Boxes Inside Each Other with Non‑Maximum Suppression

Object detection models such as YOLO, Faster R‑CNN, and SSD output a set of bounding boxes with associated confidence scores. In many cases, especially when objects are close together, the model predicts multiple overlapping boxes that actually refer to the same object. Non‑Maximum Suppression (NMS) is the standard post‑processing step that removes redundant boxes.

Why Standard NMS May Keep “Inner” Boxes

Classic NMS works by:

1. Sorting all detections by confidence.
2. Picking the highest‑scoring box as a keep candidate.
3. Removing every other box whose Intersection‑over‑Union (IoU) with the keep box exceeds a threshold t.

If the inner box is much smaller than the outer one, its IoU can be low even though it is completely contained. For example, a tiny box inside a large one may have an IoU of only 0.2, so standard NMS will keep both.

Strategy: Reject Boxes Fully Contained Within Another

To explicitly discard boxes that lie entirely inside another, we add an extra geometric check before the IoU comparison.

Algorithm Overview

def nms_with_containment(boxes, scores, iou_thresh=0.5):
    # boxes: Nx4 array (x1, y1, x2, y2)
    # scores: N array
    # Returns indices of boxes to keep
    order = scores.argsort()[::-1]          # high → low
    keep = []

    while order.size > 0:
        i = order[0]                        # current best
        keep.append(i)

        # Compute IoU of the best box with the rest
        ious = compute_iou(boxes[i], boxes[order[1:]])

        # Containment test: is the other box fully inside the best box?
        containment = is_fully_inside(boxes[i], boxes[order[1:]])

        # Remove boxes that either:
        #   - have IoU > iou_thresh  OR
        #   - are fully contained (containment == True)
        mask = (ious < iou_thresh) & (~containment)

        order = order[1:][mask]

    return keep

Helper Functions

def compute_iou(box_a, boxes_b):
    # Vectorized IoU between one box and many boxes
    x1 = np.maximum(box_a[0], boxes_b[:,0])
    y1 = np.maximum(box_a[1], boxes_b[:,1])
    x2 = np.minimum(box_a[2], boxes_b[:,2])
    y2 = np.minimum(box_a[3], boxes_b[:,3])

    inter = np.maximum(0, x2 - x1) * np.maximum(0, y2 - y1)
    area_a = (box_a[2]-box_a[0]) * (box_a[3]-box_a[1])
    area_b = (boxes_b[:,2]-boxes_b[:,0]) * (boxes_b[:,3]-boxes_b[:,1])
    union = area_a + area_b - inter
    return inter / union

def is_fully_inside(outer, inners):
    # Returns True if an inner box is completely inside the outer box
    inside_x1 = inners[:,0] >= outer[0]
    inside_y1 = inners[:,1] >= outer[1]
    inside_x2 = inners[:,2] <= outer[2]
    inside_y2 = inners[:,3] <= outer[3]
    return inside_x1 & inside_y1 & inside_x2 & inside_y2

Practical Tips for Using Containment‑Aware NMS

• Choose an appropriate IoU threshold. Typical values are 0.3–0.5. The containment check works independently of this value.
• Order matters. The algorithm always keeps the highest‑scoring box first, so a low‑confidence outer box will be removed if a higher‑confidence inner box exists.
• Batch processing. For large detection sets, vectorize the containment test to avoid Python loops.
• Edge cases. When two boxes have identical coordinates, treat them as duplicates and keep only the one with the highest score.

When to Use This Variant

This containment‑aware NMS is especially useful in:

• Detecting nested objects (e.g., a person holding a phone) where the model may output a small box for the phone inside the larger person box.
• Scenes with crowded objects where the detector tends to produce many tiny boxes inside a larger region.
• Applications that require a single, tight bounding box per object, such as tracking pipelines or downstream classification steps.

Conclusion

Standard Non‑Maximum Suppression removes overlapping detections based on IoU alone, which can leave fully contained boxes in the final output. By adding a simple geometric containment test, we can explicitly reject inner boxes, leading to cleaner detection results and more reliable downstream AI pipelines.