The Evolution to Mask R-CNN

The Mask R-CNN architecture (He et al., 2017) didn’t come out of nowhere; it is the pinnacle of years of iterative evolution in Region-Based Object Detection (Region-Based CNNs). This note maps the historical lineage from the slow, pioneering R-CNN to the elegant instance segmentation of Mask R-CNN.

1. R-CNN (2014)

The Deep Learning revolution began with AlexNet in 2012, but applying CNNs to object detection remained a challenge. The R-CNN (Regions with CNN features), by Ross Girshick, solved this by combining two ideas:

• Selective Search: To suggest ~2000 “regions of interest” (ROIs) in the image.
• Independent CNN: Each ROI was warped and passed individually through a CNN (AlexNet/VGG) for feature extraction, followed by an SVM classifier.
• Bottleneck: Complex multi-stage training and extremely slow inference (~47s per image), because the CNN ran thousands of times per image without sharing computation.

2. SPP-Net (2014)

Shortly after, SPP-Net (Spatial Pyramid Pooling) introduced a revolutionary concept: run the CNN only once on the entire image to generate a global feature map.

• The ROIs proposed by Selective Search were then projected onto this feature map.
• SPP pooled these projected areas into fixed-size vectors for the final dense network.
• Problem: SPP did not allow training the initial convolutional layers via backpropagation efficiently, limiting its accuracy.

3. Fast R-CNN (2015)

Girshick, now focused on the speed bottleneck, refined the SPP-Net idea and released Fast R-CNN.

• It ran the CNN on the entire image and extracted features via roi-pooling (a simplified 1-level version of SPP).
• Unified end-to-end training with a multi-task loss function (classification + bounding box regression).
• Remaining Bottleneck: Inference still relied on the slow Selective Search, which ran on the CPU and became the new speed limiter.

4. Faster R-CNN (2015)

The Microsoft Research team (Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun) killed the dependency on Selective Search by launching Faster R-CNN.

• RPN (Region Proposal Network): A tiny convolutional sub-network inserted right after feature extraction, responsible for “learning” to predict ROIs (using anchor boxes).
• The RPN shares the same feature backbone used for final detection, merging proposal generation and detection onto the same GPU, achieving near real-time rates.

5. Mask R-CNN (2017)

Kaiming He, Ross Girshick, and the Facebook AI Research (FAIR) team took the final step: adapting Faster R-CNN not just to detect bounding boxes, but to paint the exact pixel mask of each object (Instance Segmentation).

• They added a third branch to the network to predict the binary mask (using an FCN - Fully Convolutional Network).
• The big fix: Traditional RoI Pooling destroyed pixel-to-pixel alignment because of quantization roundings (which didn’t matter for coarse bounding boxes but ruined masks). They invented roi-align, using bilinear interpolation to perfectly align the features to the original regions.

Related: r-cnn · fast-r-cnn · faster-r-cnn · roi-pooling · roi-align