Spatial Pyramid Pooling

SPP-net

Spatial Pyramid Pooling (SPP) was an evolutionary milestone in neural networks that paved the architectural way for fast-r-cnn.

Introduced in the paper “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition” (He et al., 2014), it was revolutionary for eliminating the “fixed input size” restriction in classic Convolutional Neural Networks.

The Fixed Size Bottleneck

Before SPP-net, famous networks (like AlexNet and VGG) required the input image to always have strict dimensions (e.g., $224 \times 224$). This happened not because of the convolutional layers (which accept any size), but due to the Fully Connected Layers (FCs) at the end of the network, which require vectors of mathematically rigid size.

To circumvent this, cropping (cutting parts of the image) or warping (stretching and deforming) was used, which caused severe proportion distortions and geometry loss, degrading the accuracy of object detection models.

The Solution: SPP

SPP is a layer added right after the last convolutional layer, before the FC layers. Its operation consists of:

1. Receive a feature map of arbitrary size (let $a \times a$ be the size of the map).

2. Divide this map into several resolutions independent of the absolute size (e.g., a $1 \times 1$ grid, another $2 \times 2$, and another $4 \times 4$). For a pyramid level with an $n \times n$ grid of bins, the pooling window size (win) and stride (str) are calculated dynamically:

   $$win = \lceil a/n \rceil, \quad str = \lfloor a/n \rfloor$$

   Eq. 1: Dynamic calculation of window size and stride in SPP.

   Where:

   • $win$: size of the square pooling window.
   • $str$: step (stride) the window takes on the feature map.
   • $a$: current dimension (width or height) of the received feature map.
   • $n$: number of spatial bins at that specific pyramid level.
   • $\lceil \cdot \rceil$ and $\lfloor \cdot \rfloor$: ceiling (round up) and floor (round down) operations.

3. Apply Max Pooling within each of these bins and concatenate the results.

Regardless of the input image size, the SPP will always extract a fixed-size vector (in this example, $1 + 4 + 16 = 21$ features per convolutional filter), satisfying the strict requirement of the subsequent FC layers.

An important observation: the coarsest pyramid level ($1 \times 1$) is equivalent to a Global Pooling — an ancestor of the Global Average Pooling used today to replace entire FC layers in modern networks (e.g., ResNet, GoogLeNet).

Multi-size Training

In practice, the GPU implementations of the time (cuda-convnet, Caffe) required fixed-size batches. The paper works around this by approximating a variable-size network with several fixed-size networks that share all weights: train one full epoch with $224 \times 224$ images, switch to the $180 \times 180$ network (keeping the weights) on the next epoch, and so on alternately. Since SPP outputs always have the same length, the FC layer parameters work for any resolution.

This multi-size training acts as data augmentation along the scale dimension: it improves test accuracy (Overfeat-7 top-1 error drops from 30.36% to 29.68%) at no extra inference cost, and was, according to the authors, the first method to train a single network with images of multiple sizes.

Legacy in R-CNN

Instead of passing $2,000$ independent region proposals through the network (as in the original R-CNN), SPP-net allowed running the entire image through the CNN only once, and then extracting the feature pool of each proposed region directly from this giant convolutional map. This accelerated object detection by 24x–102x over R-CNN (with comparable or better accuracy on Pascal VOC 2007), paving the way for Fast R-CNN and faster-r-cnn.

RoI Pooling as a degenerate case of SPP

The Fast R-CNN’s RoI Pooling is a special case of SPP: a pyramid with a single level (e.g., only the $7 \times 7$ grid, without the $1 \times 1$ and $2 \times 2$ levels). SPP-net already solved in 2014 the two central problems Fast R-CNN (2015) became famous for addressing — projecting proposals onto the feature map and fixed-size pooling.

The decisive difference was not architectural, but about training: in SPP-net, fine-tuning only adjusted the FC layers (convolutions stayed frozen and features were pre-computed to disk); Fast R-CNN made end-to-end adjustment via SGD through the pooling layer, simplifying the pipeline into a single stage. The historical lesson: a technique can be mathematically ready years ahead, but only “catch on” once training becomes simple enough.

References and tools

• He, K., Zhang, X., Ren, S., Sun, J. “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition” (2014). arXiv:1406.4729 — original SPP-net paper (TPAMI 2015, preliminary at ECCV 2014).

Related: fast-r-cnn · faster-r-cnn · roi-pooling · ortogonalidade-arquitetural