Faster R-CNN

Faster R-CNN (Ren et al., 2015) consolidated the modern era of Object Detection, evolving directly from fast-r-cnn to integrate all process steps into a single, unified end-to-end network.

The Evolutionary Context

Historically, the pipeline evolved by attempting to eliminate computational performance bottlenecks:

R-CNN: Passed $2,000$ image crops one by one through the neural network. (Infeasible and slow).
Fast R-CNN (and spp-net): Passed the image only once through the network, reusing the generated feature map using roi-pooling. This solved the network bottleneck but exposed a new problem: the generation of Region Proposals (Selective Search algorithm running on CPU) became the bottleneck, taking $\sim 2$ seconds per image.

The Trump Card: Region Proposal Network (RPN)

The stroke of genius in Faster R-CNN was to throw away Selective Search and replace it with a fully convolutional Region Proposal Network (RPN).

The RPN is a tiny network that slides a $3 \times 3$ window (sliding window) over the convolutional feature map generated by the backbone. Each window is encoded into a short vector ( $256$ -d for the ZF backbone, $512$ -d for VGG-16) that feeds two sibling layers. At each position, the network uses the concept of Anchor Boxes (pre-defined reference boxes at various scales and aspect ratios) to predict:

Objectness Score: A number indicating the probability of any object being there (background vs. object).
Bounding Box Regression: Precise adjustments (deltas) in $x, y, w, h$ to perfectly frame the anchor around the object’s mass.

Anchor Boxes and Translation Invariance

Anchors are reference boxes centered at each sliding-window position. The paper uses 3 scales (areas of $128^2$ , $256^2$ and $512^2$ pixels) and 3 aspect ratios ( $1{:}1$ , $1{:}2$ , $2{:}1$ ), totaling $k = 9$ anchors per position — roughly $20{,}000$ anchors per image. Thus, the classification layer has $2k$ outputs (object/background per anchor) and the regression layer has $4k$ (the deltas per anchor).

The design is translation invariant: if the object moves in the image, the same function predicts the proposal at the new location. This contrasts with earlier methods (MultiBox), whose k-means-fixed anchors required an output layer with an order of magnitude more parameters ( $27$ M vs. $2.4$ M with VGG-16) — and more overfitting risk on small datasets like PASCAL VOC.

⚠️ Practical pitfall: anchors that cross the image boundary must be ignored during training (they do not contribute to the loss). Without this, their large, hard-to-correct error terms prevent convergence. At test time, boundary-crossing boxes are simply clipped to the image limits.

Anchor Labeling

During training, each anchor receives a binary label:

Positive: the anchor with the highest IoU with a ground-truth box, or any anchor with IoU $> 0.7$ with any ground-truth box.
Negative: a non-positive anchor with IoU $< 0.3$ for all ground-truth boxes.
Ignored: everything in between does not contribute to the loss.

Since negatives dominate, each mini-batch samples $256$ anchors per image keeping up to a $1{:}1$ positive-to-negative ratio.

The loss function of the RPN is also multi-task, optimizing these two outputs simultaneously:

L(\{p_i\}, \{t_i\}) = \frac{1}{N_{cls}} \sum_i L_{cls}(p_i, p_i^*) + \lambda \frac{1}{N_{reg}} \sum_i p_i^* L_{reg}(t_i, t_i^*)

Eq. 1: Loss Function of the Region Proposal Network (RPN).

Where:

$i$ : index of the anchor being evaluated in a mini-batch.
$p_i$ : predicted probability that anchor $i$ is an object.
$p_i^*$ : binary ground truth ( $1$ if the anchor is an object, $0$ if it is background). An anchor is positive if it has a high IoU.
$t_i$ : vector representing the 4 predicted parameterized bounding box coordinates.
$t_i^*$ : ground truth vector containing the target coordinate adjustments.
$L_{cls}$ : classification loss function (log loss over two classes).
$L_{reg}$ : spatial regression loss function (Smooth L1).
$N_{cls}, N_{reg}$ : normalization terms.
$\lambda$ : balancing hyperparameter.
The term $p_i^* L_{reg}$ ensures that regression loss is activated only for positive anchors ( $p_i^* = 1$ ).

End-to-End Unification

The catch is that the RPN shares the same convolutional feature map as the final detection network. In other words, generating region proposals became a practically cost-free process (taking only $\sim 10$ milliseconds).

The proposals generated by the RPN go through Non-Maximum Suppression (NMS, with a fixed IoU of $0.7$ ) to eliminate redundancy, and only the top $300$ proposals ranked by objectness score move forward. They are then sent to an roi-pooling layer, which extracts fixed vectors to pass through the final layers, where the network classifies (is it a dog or a car?) and makes the final refinement of the bounding box.

4-Step Alternating Training

Training RPN and Fast R-CNN jointly from scratch does not converge trivially, because the detector depends on fixed proposals. The paper’s pragmatic solution is alternating optimization:

Train the RPN alone (initialized with ImageNet pre-trained weights).
Train Fast R-CNN separately, using the proposals from the step-1 RPN — the two networks do not share convolutions yet.
Reinitialize the RPN with the detector’s weights, freeze the shared convolutions and fine-tune only the RPN-specific layers. The networks now share features.
Fine-tune only Fast R-CNN’s fully-connected layers, keeping the convolutions frozen. The result: a single unified network.

Results and Ablation Lessons

With VGG-16, the system reaches 5 fps (198ms per image, versus 1830ms for the Selective Search pipeline) with $73.2\%$ mAP on PASCAL VOC 2007 — and 17 fps with the ZF backbone. The paper’s ablation experiments are worth as much as the final numbers:

Without the cls layer (no objectness ranking), mAP collapses from $56.8\%$ to $44.6\%$ with 100 proposals — the confidence score is what guarantees top-of-ranking quality.
Without the reg layer (raw anchors, no regression), mAP drops to $52.1\%$ — anchors alone are not enough; regression is what refines the positions.
Two-stage beats one-stage: emulating the OverFeat style (dense windows in a single stage) drops mAP by $\sim 4.8$ points, justifying the proposal → detection cascade.

References and tools

Ren, S., He, K., Girshick, R., Sun, J. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks” (2015). arXiv:1506.01497 — original paper.
github.com/ShaoqingRen/faster_rcnn — official implementation (Caffe).