Mask R-CNN

Mask R-CNN (He et al., 2017) is the natural extension of faster-r-cnn, designed not only to detect objects (draw boxes) but to perform Instance Segmentation — identifying the exact pixels that belong to each object individually.

How It Works

The brilliance of Mask R-CNN was maintaining the fast flow of Faster R-CNN and adding a small parallel network (branch) exclusively in charge of predicting a binary mask (pixel by pixel) for the object bounded by the proposed region.

1. Backbone and RPN: Identical to faster-r-cnn. Extracts features from the image and generates Region Proposals.
2. RoI Align: Where roi-pooling was previously used (which suffered from severe quantization errors for pixel-level alignment), Mask R-CNN introduced RoIAlign. RoIAlign uses spatial bilinear interpolation to compute the exact value on the feature map with sub-pixel accuracy, maintaining perfect alignment without destructive rounding.
3. Parallel Heads: The extracted region feeds two independent paths:
   • Main Branch: The classic path for classification and bounding box regression.
   • Mask Branch: A small fully convolutional network (FCN) that generates independent binary masks for each of the $K$ possible classes.

🧮 Mathematical Foundations (Tripartite Loss)

Training occurs by simultaneously optimizing three distinct tasks through a combined global loss function:

$$L = L_{cls} + L_{box} + L_{mask}$$

Eq. 1: Tripartite loss function of Mask R-CNN.

Where:

• $L$: total network loss.
• $L_{cls}$: ROI classification loss (what the object is).
• $L_{box}$: bounding box regression loss (where the strict limits of the box are).
• $L_{mask}$: average binary cross-entropy (BCE) loss specifically focused on predicting the correct mask for the region.

The Mathematics of the Mask Branch

The mask branch generates an output of dimension $K \times m \times m$, indicating $K$ binary masks of resolution $m \times m$ (one autonomous mask for each of the $K$ known classes).

Unlike traditional semantic segmentation (where all classes compete for the same pixel in a multinomial global Softmax), Mask R-CNN completely decouples mask classification, applying Sigmoid in isolation.

$$L_{mask} = - \frac{1}{m^2} \sum_{i,j} \left[ y_{i,j} \log \hat{y}_{i,j}^k + (1 - y_{i,j}) \log (1 - \hat{y}_{i,j}^k) \right]$$

Eq. 2: Isolated pixel-wise binary cross-entropy (BCE) for the true class.

Where:

• $k$: the true (ground truth) class of the object (identified in the training label).
• $y_{i,j}$: true binary label of the pixel ($1$ if the pixel is the object, $0$ if it is background).
• $\hat{y}_{i,j}^k$: probability (Sigmoid activator output) that pixel $(i,j)$ belongs to the object in the class $k$ mask.
• $m^2$: total number of pixels in the generated mask grid ($m \times m$).

Only the mask associated with the true class of the object ($k$) actively contributes to the $L_{mask}$ loss, totally ignoring the masks generated for the other $K-1$ classes. This decoupling prevents mathematical interference and was the secret to the model’s spectacular segmentation results.

Related: faster-r-cnn · roi-pooling · roi-align