U-Net

Image Segmentation

U-Net is the gold-standard architecture for Semantic Image Segmentation, evolving from traditional cnn.

U-Net (Ronneberger et al., 2015) is arguably the most influential convolutional neural network architecture in the field of Semantic Image Segmentation (dense pixel-by-pixel classification). Originally created for biomedical images, its brilliant design became the standard in all image processing and even Diffusion architectures (like Stable Diffusion).

The “U” Shaped Architecture

Unlike traditional CNNs that only reduce the image to a 1D vector to say what is in it (classification), segmentation requires returning an image (mask) of the same size as the original. U-Net solves this by being a Fully Convolutional Autoencoder (FCN) symmetrically designed like the letter U.

1. Contracting Path (Encoder / Downsampling)

It is the left half of the “U”. It follows the typical structure of a CNN (convolutions followed by Max Pooling). Its main role is to understand the context (WHAT) is in the image. With each descent, the spatial resolution of the image drops by half, but the number of feature channels (filters) doubles, allowing it to extract increasingly abstract high-level features.

2. Expansive Path (Decoder / Upsampling)

It is the right half of the “U”. The network needs to “rebuild” the image to its original size using Transposed Convolutions (or Up-convolutions). Here, the number of feature channels is progressively reduced, while the spatial resolution is doubled at each step to reconstruct the original details.

3. Skip Connections (The Magic of U-Net)

The big problem with the Encoder is that, by reducing the image, all high-frequency spatial information and precise edges (WHERE) are irreversibly destroyed. U-Net solves this by creating Skip Connections. It takes the high-resolution feature map straight from an Encoder level and concatenates it with the corresponding level in the Decoder. This gives the network the power to merge “What is this?” (Coming from the deep layers of the decoder) with “Where exactly are the edges of this?” (Coming straight from the shallow layers of the encoder), resulting in perfect millimeter-accurate masks, even with very few training samples.

Related: cnn · object-detection