Fast R-CNN

Fast R-CNN is the evolution of r-cnn, created to solve the extreme bottleneck of slowness when processing images by introducing convolutional computation sharing and the RoI Pooling technique.

How Fast R-CNN solved R-CNN’s problems

In the original R-CNN, the network independently processed ~2,000 proposed regions, which was highly redundant and slow. Fast R-CNN shifted this paradigm:

1. Feature Sharing: Instead of cropping the image 2,000 times, the entire image is passed through the CNN only once to produce a single giant feature map.

2. Proposal Projection: The Selective Search algorithm still proposes the ~2,000 regions, but now these coordinates are projected directly onto the already calculated feature map, rather than the original image.

3. RoI Pooling: For each region in the feature map (which varies in size), a Region of Interest Pooling layer is applied, converting the region to a fixed size (essential for the final dense layers). Note: RoI Pooling would later be improved by roi-align in Mask R-CNN due to rounding issues.

4. Multi-task Network: Slow SVMs were replaced by a Softmax layer, allowing classification and bounding box refinement to be trained together in the same network. This is done by optimizing a joint Loss function (Multi-task Loss):

   $$L(p, u, t^u, v) = L_{cls}(p, u) + \lambda [u \ge 1] L_{loc}(t^u, v)$$

   Eq. 1: Multi-task loss function unifying classification and regression.

   Where:

   • $p$: probability distribution calculated by Softmax over the network categories.
   • $u$: true class label (ground truth), where $u=0$ represents background.
   • $t^u$: predicted offsets (adjustments) by the bounding box regressor for class $u$.
   • $v$: true ground truth offsets of the bounding box.
   • $\lambda$: balancing hyperparameter between classification and localization.
   • $[u \ge 1]$: indicator function activating the regression loss only if the proposal is a true object (ignoring background).

Advantages and the New Limitation

• End-to-End Training: The classifier and regressor are now trained simultaneously.
• Much Faster: About 200x faster than classic R-CNN in inference time (processing an image in ~2 seconds).

⚠️ The Remaining Bottleneck: The only thing keeping Fast R-CNN from running in real-time was its reliance on Selective Search to propose regions (which ran on the CPU). This issue was later solved by faster-r-cnn.

Related: r-cnn · faster-r-cnn · roi-align