ReLU

Rectified Linear Unit

ReLU is the default activation function in modern CNNs, applied after every convolutional layer to introduce non-linearity — without it, stacking layers as in mask-r-cnn or any deep network would mathematically collapse into a single linear transformation.

Practical Context

The idea is brutally simple: zero out any negative value, keep positive values untouched.

Why it works: convolutional and fully-connected layers are, in isolation, linear transformations ( $Wx + b$ ). Stacking $N$ linear transformations with no non-linear activation between them is still a single equivalent linear transformation — the network would never learn complex decision boundaries (curved edges, textures, irregular biological shapes).
Why ReLU over Sigmoid/Tanh: saturating activations (Sigmoid, Tanh) squash the output into a fixed range (0-1 or -1-1), making the gradient tend to zero at the extremes (vanishing gradient) and stalling learning in deep networks. ReLU has a constant gradient (1) for any positive input, keeping gradient flow stable during backpropagation.
⚠️ Dying ReLU: if a neuron receives persistently negative input (e.g., poorly initialized weights or too high a learning rate), its gradient becomes permanently zero and it never updates again — the neuron “dies” and stops contributing to the network.

import numpy as np

def relu(x: np.ndarray) -> np.ndarray:
    # zeroes negatives, keeps positives — non-linearity "for free" computationally
    return np.maximum(0, x)

def relu_grad(x: np.ndarray) -> np.ndarray:
    # gradient 1 where it fired, 0 where it was zeroed (undefined at x=0, conventionally 0)
    return (x > 0).astype(float)

Mathematical Foundation

f(x) = \max(0, x)

Eq. 1: ReLU activation function.

Where:

$f(x)$ : output signal after activation.
$x$ : scalar input value (weighted sum from a convolutional or fully-connected layer).

The derivative, needed for gradient backpropagation, is a step function:

f'(x) = \begin{cases} 1, & x > 0 \\ 0, & x \leq 0 \end{cases}

Eq. 2: ReLU derivative.

Where:

$f'(x)$ : local gradient propagated to the previous layer during backpropagation.

Variants That Fix Dying ReLU

Variant	Formula (negative region)	When to use
Leaky ReLU	$f(x) = \alpha x$ , with small fixed $\alpha$ (e.g. 0.01)	Minimal fix, near-zero extra compute cost
PReLU	$f(x) = \alpha x$ , with $\alpha$ learned via backpropagation	When the dataset is large enough to learn $\alpha$ without overfitting
GELU	probabilistic smoothing (uses the normal CDF)	Standard in Transformers (BERT, GPT); more compute-expensive than ReLU

No variant is strictly superior: plain ReLU remains the default choice in computer vision CNNs (as in mask-r-cnn, which uses ReLU across the four convolutional layers of the mask branch) precisely because of its minimal compute cost — a max(0, x) is orders of magnitude cheaper than any exponential-based function.

Interview Questions

Why is a network without a non-linear activation function, no matter how many layers, still equivalent to a single linear perceptron?
What causes the “dying ReLU” problem, and how does Leaky ReLU mitigate it?
Why does ReLU tend to train faster than Sigmoid in deep networks?

Related: cnn · mask-r-cnn · faster-r-cnn