How to reject boxes inside each other with Non Max Suppression

How to Reject Boxes Inside Each Other with Non‑Maximum Suppression

Object detection models such as Faster R‑CNN, YOLO, and SSD output a set of bounding boxes with associated confidence scores. In many cases, especially when objects are close together, the model predicts multiple overlapping boxes that actually refer to the same object. Non‑Maximum Suppression (NMS) is the standard post‑processing step that removes redundant boxes.

Why Standard NMS May Keep “Inner” Boxes

Classic NMS works by:

1. Sorting all detections by confidence.
2. Picking the highest‑scoring box as a keep candidate.
3. Discarding every other box whose Intersection‑over‑Union (IoU) with the keep box exceeds a threshold t.

If two boxes are nested (one completely inside the other) but the inner box has a slightly higher score, the outer box will be removed, leaving the inner box in the final set. In many applications we actually want to keep the larger, more inclusive box and discard the inner one.

Modified NMS to Reject Inner Boxes

The idea is simple: after the usual IoU check, also compare the area ratio of the two boxes. If one box is significantly smaller and lies almost entirely inside the other, we treat it as redundant even if its IoU is below the usual threshold.

Algorithm Sketch

def nms_reject_inner(boxes, scores, iou_thresh=0.5, area_ratio=0.8):
    # boxes: Nx4 array (x1, y1, x2, y2)
    # scores: N array
    # iou_thresh: standard NMS IoU threshold
    # area_ratio: max allowed ratio of smaller/ larger area

    # 1. Sort by confidence
    order = scores.argsort()[::-1]
    keep = []

    while order.size > 0:
        i = order[0]
        keep.append(i)

        # Compute IoU of the keep box with the rest
        ious = compute_iou(boxes[i], boxes[order[1:]])

        # Compute area ratios
        area_i = box_area(boxes[i])
        area_rest = box_area(boxes[order[1:]])
        ratio = np.minimum(area_i, area_rest) / np.maximum(area_i, area_rest)

        # Condition to discard:
        #   - IoU > iou_thresh  OR
        #   - (ratio > area_ratio AND box_i is inside box_j)
        inside = (boxes[order[1:]][:,0] >= boxes[i][0]) &
                 (boxes[order[1:]][:,1] >= boxes[i][1]) &
                 (boxes[order[1:]][:,2] <= boxes[i][2]) &
                 (boxes[order[1:]][:,3] <= boxes[i][3])

        discard = (ious > iou_thresh) | ((ratio > area_ratio) & inside)

        # Keep only boxes that are not discarded
        order = order[1:][~discard]

    return keep

Explanation of the Extra Checks

• Area ratio: If the smaller box occupies more than area_ratio (e.g., 80 %) of the larger one, they are likely describing the same object.
• Inside test: Ensures the smaller box is truly nested (all four corners lie within the larger box). This prevents discarding a partially overlapping box that just happens to have a similar area.

Choosing Hyper‑Parameters

Parameter	Typical Range	Effect
iou_thresh	0.3 – 0.7	Higher values keep more overlapping boxes; lower values are stricter.
area_ratio	0.6 – 0.9	Higher values treat more nested boxes as duplicates.

For dense scenes (e.g., crowds), a lower iou_thresh (≈0.4) combined with an area_ratio of 0.8 works well. For sparse scenes, you can relax both thresholds.

Practical Tips for Implementation

• Vectorize the IoU and area‑ratio calculations with NumPy or PyTorch to keep the operation fast.
• If you are using a deep‑learning framework that already provides NMS (e.g., torchvision.ops.nms), run the standard NMS first, then apply the inner‑box filter on the remaining candidates.
• When deploying on edge devices, pre‑compute the inside mask using simple coordinate comparisons; it adds negligible overhead.
• Visualize the results on a few validation images to ensure the larger boxes are retained as expected.

Full Example with PyTorch

import torch
from torchvision.ops import nms

def nms_with_inner_rejection(boxes, scores, iou_thresh=0.5, area_ratio=0.8):
    # Standard NMS first
    keep = nms(boxes, scores, iou_thresh)

    boxes = boxes[keep]
    scores = scores[keep]

    # Apply inner‑box filter
    final_keep = nms_reject_inner(boxes, scores, iou_thresh, area_ratio)
    return keep[final_keep]

# Example usage
boxes = torch.tensor([[10, 20, 110, 120],
                      [15, 25, 105, 115],   # inner box
                      [200, 200, 300, 300]], dtype=torch.float32)
scores = torch.tensor([0.9, 0.85, 0.95])

selected = nms_with_inner_rejection(boxes, scores, iou_thresh=0.5, area_ratio=0.8)
print("Kept indices:", selected.tolist())

Conclusion

Standard Non‑Maximum Suppression removes duplicate detections based solely on IoU, which can leave smaller, nested boxes in the final output. By adding an area‑ratio check and an inside‑test, you can reliably reject boxes that are completely contained within larger ones, leading to cleaner detection results—especially in applications like autonomous driving, aerial imagery analysis, and retail shelf monitoring where the size of the bounding box carries semantic importance.