Fast R-CNN

Fast R-CNN is the evolution of r-cnn, created to solve the extreme bottleneck of slowness when processing images by introducing convolutional computation sharing and the RoI Pooling technique.

How Fast R-CNN solved R-CNN’s problems

In the original R-CNN, the network independently processed ~2,000 proposed regions, which was highly redundant and slow. Fast R-CNN shifted this paradigm:

1. Feature Sharing: Instead of cropping the image 2,000 times, the entire image is passed through the CNN only once to produce a single giant feature map.

2. Proposal Projection: The Selective Search algorithm still proposes the ~2,000 regions, but now these coordinates are projected directly onto the already calculated feature map, rather than the original image.

3. RoI Pooling: For each region in the feature map (which varies in size), a Region of Interest Pooling layer is applied, converting the region to a fixed size (essential for the final dense layers). Note: RoI Pooling would later be improved by roi-align in Mask R-CNN due to rounding issues.

4. Multi-task Network: Slow SVMs were replaced by a Softmax layer, allowing classification and bounding box refinement to be trained together in the same network. This is done by optimizing a joint Loss function (Multi-task Loss):

   $$L(p, u, t^u, v) = L_{cls}(p, u) + \lambda [u \ge 1] L_{loc}(t^u, v)$$

   Eq. 1: Multi-task loss function unifying classification and regression.

   Where:

   • $p$: probability distribution calculated by Softmax over the network categories.
   • $u$: true class label (ground truth), where $u=0$ represents background.
   • $t^u$: predicted offsets (adjustments) by the bounding box regressor for class $u$.
   • $v$: true ground truth offsets of the bounding box.
   • $\lambda$: balancing hyperparameter between classification and localization.
   • $[u \ge 1]$: indicator function activating the regression loss only if the proposal is a true object (ignoring background).

Advantages and the New Limitation

• End-to-End Training: The classifier and regressor are now trained simultaneously.
• Much Faster: About 200x faster than classic R-CNN in inference time (processing an image in ~2 seconds).

⚠️ The Remaining Bottleneck: The only thing keeping Fast R-CNN from running in real-time was its reliance on Selective Search to propose regions (which ran on the CPU). This issue was later solved by faster-r-cnn.

Findings from the original paper (Girshick, 2015)

The paper proposes more than the architecture — it stress-tests design decisions that hold as reusable principles:

• Training all convolutional layers matters in very deep networks: freezing VGG16’s conv layers (mimicking SPP-net, which could not back-propagate through Spatial Pyramid Pooling) drops mAP from 66.9% to 61.4%. Updating from conv3_1 up already captures nearly all the gain.
• Softmax replaces SVMs without loss: the end-to-end softmax classifier matches or beats (+0.1 to +0.8 mAP) the post-hoc SVM pipeline with hard negative mining — eliminating r-cnn's multi-stage training.
• Multi-task loss improves even pure classification: training classification and bounding-box regression jointly improves mAP even when regression is disabled at test time (+0.8 to +1.1 points) — the tasks reinforce each other through the shared representation.
• Single scale is enough for deep networks: 5-scale image pyramids add marginal mAP at large compute cost; the network learns scale invariance directly from data (s = 600px).
• Truncated SVD compresses the FC layers almost for free: factorizing the fully connected weight matrices cuts detection time by over 30% with only a 0.3-point mAP drop — FCs account for ~45% of inference time when processing ~2000 RoIs.
• More proposals ≠ better detection: sweeping from 1k to 10k proposals per image makes mAP rise then fall; sparse proposals (Selective Search) beat dense boxes (45k/image), which drop mAP to ~53%. ⚠️ The paper also shows Average Recall does not correlate with mAP when varying the proposal count — proxy metrics must be used with care.

References and tools

• Girshick, R. “Fast R-CNN” (2015). arXiv:1504.08083 — original paper; the design evaluation in section 5 is an exemplary ablation study.

Related: r-cnn · faster-r-cnn · roi-align · ablation-study