Residual neural network

In a multilayer neural network model, consider a (non-residual) subnetwork with a certain number of stacked layers (e.g., 2 or 3). Let ${\displaystyle H(x;\alpha )}$ denote the subnetwork. Suppose ${\displaystyle H^{*}}$ is the desired optimal output of this subnetwork. Residual learning simply adds ${\displaystyle x}$ directly to the output, such that the optimal learned output now becomes ${\displaystyle H^{*}-x}$, which is interpreted as a "residual" with respect to ${\displaystyle x}$.

The operation of "adding ${\displaystyle x}$" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work.

Let ${\displaystyle F(x;\alpha )=H(x;a)+x}$. The function ${\displaystyle F}$ is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a "residual block".^[1] A deep residual network is constructed by simply stacking these blocks.

تحتوي الذاكرة طويلة المدى (LSTM) على آلية ذاكرة تعمل كوصلة متبقية. ^{[ 4 ]} في ذاكرة LSTM بدون بوابة نسيان ، يكون المدخل${\displaystyle x_{t}}$تتم معالجتها بواسطة دالة${\displaystyle F}$وأضيف إلى خلية ذاكرة${\displaystyle c_{t}}$، مما أدى إلى${\displaystyle c_{t+1}=c_{t}+F(x_{t})}$. تعمل شبكة LSTM المزودة ببوابة نسيان بشكل أساسي كشبكة طرق سريعة .

لتحقيق استقرار في تباين مدخلات الطبقات، يوصى باستبدال الوصلات المتبقية${\displaystyle x+f(x)}$مع${\displaystyle x/L+f(x)}$، أين${\displaystyle L}$يمثل العدد الإجمالي للطبقات المتبقية. ^{[ 5 ]}

إذا كانت الدالة${\displaystyle F}$من النوع${\displaystyle F:\mathbb {R} ^{n}\to \mathbb {R} ^{m}}$أين${\displaystyle n\neq m}$، ثم${\displaystyle F(x)+x}$غير مُعرَّف. لمعالجة هذه الحالة الخاصة، يتم استخدام اتصال إسقاطي:

$${\displaystyle y=F(x)+P(x)}$$

أين${\displaystyle P}$عادةً ما يكون إسقاطًا خطيًا، يتم تحديده بواسطة${\displaystyle P(x)=Mx}$أين${\displaystyle M}$هو${\displaystyle m\times n}$المصفوفة. يتم تدريب المصفوفة عبر الانتشار العكسي ، كما هو الحال مع أي معلمة أخرى للنموذج.

يُسهّل إدخال عمليات الربط المتطابقة انتشار الإشارة في كل من المسارات الأمامية والخلفية. ^{[ 6 ]}

A basic block is the simplest building block studied in the original ResNet.^[1] This block consists of two sequential 3x3 convolutional layers and a residual connection. The input and output dimensions of both layers are equal.

A bottleneck block^[1] consists of three sequential convolutional layers and a residual connection. The first layer in this block is a 1×1 convolution for dimension reduction (e.g., to 1/2 of the input dimension); the second layer performs a 3×3 convolution; the last layer is another 1×1 convolution for dimension restoration. The models of ResNet-50, ResNet-101, and ResNet-152 are all based on bottleneck blocks.^[1]

The pre-activation residual block^[6] applies activation functions before applying the residual function ${\displaystyle F}$. Formally, the computation of a pre-activation residual block can be written as:

$${\displaystyle x_{\ell +1}=F(\phi (x_{\ell }))+x_{\ell }}$$

where ${\displaystyle \phi }$ can be any activation (e.g. ReLU) or normalization (e.g. LayerNorm) operation. This design reduces the number of non-identity mappings between residual blocks, and allows an identity mapping directly from the input to the output. This design was used to train models with 200 to over 1000 layers, and was found to consistently outperform variants where the residual path is not an identity function. The pre-activation ResNet with 200 layers took 3 weeks to train for ImageNet on 8 GPUs in 2016.^[6]

Since GPT-2, transformer blocks have been mostly implemented as pre-activation blocks. This is often referred to as "pre-normalization" in the literature of transformer models.^[7]

Residual connections were noticed in neuroanatomy, such as Lorente de No (1938).^{[14]:Fig 3}McCulloch and Pitts (1943) proposed artificial neural networks and considered those with residual connections.^{[15]:Fig 1.h}

In 1961, Frank Rosenblatt described a three-layer multilayer perceptron (MLP) model with skip connections.^{[16]:313,Chapter 15} The model was referred to as a "cross-coupled system", and the skip connections were forms of cross-coupled connections.

During the late 1980s, "skip-layer" connections were sometimes used in neural networks. Examples include:^[17][18] Lang and Witbrock (1988)^[19] trained a fully connected feedforward network where each layer skip-connects to all subsequent layers, like the later DenseNet (2016). In this work, the residual connection was the form ${\displaystyle x\mapsto F(x)+P(x)}$, where ${\displaystyle P}$ is a randomly-initialized projection connection. They termed it a "short-cut connection". An early neural language model used residual connections and named them "direct connections".^[20]

Sepp Hochreiter discovered the vanishing gradient problem in 1991^[21] and argued that it explained why the then-prevalent forms of recurrent neural networks did not work for long sequences. He and Schmidhuber later designed the LSTM architecture to solve this problem,^[4][22] which has a "cell state" ${\displaystyle c_{t}}$ that can function as a generalized residual connection. The highway network (2015)^[23][24] applied the idea of an LSTM unfolded in time to feedforward neural networks, resulting in the highway network. ResNet is equivalent to an open-gated highway network.

During the early days of deep learning, there were attempts to train increasingly deep models. Notable examples included the AlexNet (2012), which had 8 layers, and the VGG-19 (2014), which had 19 layers.^[25] However, stacking too many layers led to a steep reduction in training accuracy,^[26] known as the "degradation" problem.^[1] In theory, adding additional layers to deepen a network should not result in a higher training loss, but this is what happened with VGGNet.^[1] If the extra layers can be set as identity mappings, however, then the deeper network would represent the same function as its shallower counterpart. There is some evidence that the optimizer is not able to approach identity mappings for the parameterized layers, and the benefit of residual connections was to allow identity mappings by default.^[6]

In 2014, the state of the art was training deep neural networks with 20 to 30 layers.^[25] The research team for ResNet attempted to train deeper ones by empirically testing various methods for training deeper networks, until they came upon the ResNet architecture.^[27]

Wide Residual Network (2016) found that using more channels and fewer layers than the original ResNet improves performance and GPU-computational efficiency, and that a block with two 3×3 convolutions is superior to other configurations of convolution blocks.^[28]

DenseNet (2016)^[29] connects the output of each layer to the input to each subsequent layer:

$${\displaystyle x_{\ell +1}=F(x_{1},x_{2},\dots ,x_{\ell -1},x_{\ell })}$$

Stochastic depth^[30] is a regularization method that randomly drops a subset of layers and lets the signal propagate through the identity skip connections. Also known as DropPath, this regularizes training for deep models, such as vision transformers.^[31]

ResNeXt (2017) combines the Inception module with ResNet.^[32][8]

Squeeze-and-Excitation Networks (2018) added squeeze-and-excitation (SE) modules to ResNet.^[33] An SE module is applied after a convolution, and takes a tensor of shape ${\displaystyle \mathbb {R} ^{H\times W\times C}}$ (height, width, channels) as input. Each channel is averaged, resulting in a vector of shape ${\displaystyle \mathbb {R} ^{C}}$. This is then passed through a multilayer perceptron (with an architecture such as linear-ReLU-linear-sigmoid) before it is multiplied with the original tensor. It won the ILSVRC in 2017.^[34]