CNN Architectures

In this post, I am going to summarize the key-ideas of the following CNN architectures:

1. AlexNet
2. VGG
3. Inception
4. Rethinking Inception
5. ResNet
6. Wide ResNet
7. ResNext
8. DenseNet
9. ShuffleNet
10. MobileNet-V2

1. AlexNet

Idea: ReLU, Dropout, GPU, ImageNet

This was one of the first network which started giving good result on ImageNet. Main changes incroporated in AlexNet that are sorted according to their importance accoding to authors are:

ReLU Non-Linearity: Earlier models used to face this issue of saturating neurons that resulted in vanishing gradients problem. Rectified Linear Unit, which is just max(0, x), solved this problem. ReLU led to faster convergence of the network.

Training on Multiple GPUs: Since we want to train bigger network, and at the time of this paper, there were not GPUs which could support this big network, authors went for multiple GPU training. Kerenls are split into two GPUs with one trick: the GPUs communicate only in certain layers.

Local Response Normalization: Even though ReLU does not have the problem of saturating neurons authors found improvement in generalization after applying input normalization. Activity of neruons is normalized over a set of adjacent kernel maps. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. This is applied in certain layers of the network.

Overlapping Pooling: Authors find that overlapping pooling generally find it difficult to overlap. Though I am not sure about this.

Filter size were of varying sizes which went on becoming smaller with layers: 11x11, 5x5, 3x3.

2. VGG

Idea: Smaller filter Authors were able to train a very deep network which was a challenge at that time. They used several techniques to do that.

Smaller filters: Suppose we want to get a receptive field of 7x7. One way to get that is using 7x7 filter which has 49 parameters and 7*7=49 operations to caluclate one activation. Other thing we can do is stack three 3x3 filters. This will have 3x3x3 = 27 parameters and 9+9+9=27 operation to calculate one activaion. This is the key idea of this paper. We can get big receptive field by using smaller filters. This has added advantage of incorporation of three non-linearity instead of single one which makes the decision function more discriminative.

1x1 Filters: Though this idea mainly comes from paper Network in Network, M. Lin, 2014, this paper mentions it and hence I put it here. Incorporation of 1x1 helps increase non-linearity without changing the receptive. Using two convolution back to back without non-linearity is pointless because two 3x3 is equivalent to 5x5 which is computationally expensive.

Removed LRN: Local response normalization is not used as it does not help in perfomance and only increases number of parameters.

VGG converges faster than AlexNet in spite of having more parameters due to possibly 2 reasons: (a) implicit regualarization due to greater depth and samller filters, (b) pre-initialization of certain layers.

Note: 4096 parameters in the last hidden layer has no particular reason. Earlier it started due to limitation of GPU size, and later researchers blindly used it without questioning it much.

3. Inception

Idea: Split, Transform, Aggregate

Main hallmark is improved utilization of computing resources inside the network. Use 1x1 filter for two purposes:

1. To increase non-linearity (aka discriminative power) without changing receptive field
2. Dimension reduction to remove computation bottlenecks

Increasing size of network helps. But computationally expensive and prone to overfitting. So, we move from fully connected to sparsely connected architectures even inside the convolutions. Correlated pixels will be together in lower layers (close to input). Also, there will be clusters which will be slightly far which might provide extra context. So, we need both 3x3 and 5x5 filters. Using them is expensive so we use 1x1 filter to reduce the computation. We bring features to lower dimension using 1x1 filter and then apply 3x3 and 5x5 filters.

Using inception modules only in higher layers (away from input) because of memory efficiency.

This aligns with the intuition that visual information should be processed at various scales and then aggregated so that the next stage can abstract features from different scales simultaneously.

4. Rethinking Inception

Idea: Factorized convoutions, aggressive regularization

General Design Principles:

• Avoid representational bottlenecks, especially early in the network. In general the representation size should gently decrease from the inputs to the outputs before reaching the final representation used for the task at hand.
• Higher dimensional representations are easier to process locally within a network. Increasing the activations per tile in a convolutional network allows for more disentangled features. The resulting networks will train faster.
• Spatial aggregation can be done over lower dimensional embeddings without much or any loss in representational power.  For example, before performing a more spread out (e.g. 3 × 3) convolution, one can reduce the dimension of the input representation before the spatial aggregation without expecting serious adverse effects. Authors hypothesize that the reason for that is the strong correlation between adjacent unit results in much less loss of information during dimension reduction, if the outputs are used in a spatial aggregation context.
• Optimal performance of the network can be reached by balancing the number of filters per stage and the depth of the network. The computational budget should therefore be distributed in a balanced way between the depth and width of the network.  -Factorize 5x5 into two 3x3 in inception modules. -Factorize nxn into nx1 and 1xn - This does not work in early layers. It works well in medium sized feature maps in between 12x12 to 20x20. On that level 1x7 and 7x1 works well.

-Auxiliary classification heads (used in original inception for training) only act as a regularizer but do not help with convergence.

5. ResNet

Idea: Skip connections, feature reuse, feature exploitation

We need do go deeper into the network, but…

When deeper networks are able to start converging, a degradation problem has been exposed: with the network depth increasing, accuracy gets saturated (which might be unsurprising) and then degrades rapidly. Unexpectedly, such degradation is not caused by overfitting,and adding more layers to a suitably deep model leads to higher training error.

How can a network enjoy increasing representational power (by going either deeper or wider) without sacrificing the feasibility of optimization using back-propagation with stochastic gradient descent (i.e., handling the vanishing and/or exploding gradient problem)?

To implement this, we do X + F(X) in each block.

This ensures that a deeper model should have training error no greater than its shallower counterpart. With the residual learning reformulation, if identity mappings are optimal, the solvers may simply drive the weights of the multiple nonlinear layers toward zero to approach identity mappings.

We might face an issue where the dimensions of X and F(X) don’t match. We do linear projections to fix this.

1. If the channels don’t match, we do 1x1 convolution on x to match them.
2. If the width, height don’t match, we do stride on x using 1x1 filter.

• Don’t destroy the original image by doing ReLU on identity shortcut. Instead, do ReLU on the CNN path.
• Feature Expolitation: We add results from identity mapping and CNN path.

6. Wide ResNet

Each percent increase in accuracy requires doubling the number of layers in a Resnet. So, it leads to diminishing feature use causing slower training.

Diminishing Feature Reuse: We note, however, that the residual block with identity mapping that allows to train very deep networks is at the same time a weakness of residual networks. As gradient flows through the network there is nothing to force it to go through residual block weights and it can avoid learning anything during training, so it is possible that there is either only a few blocks that learn useful representations, or many blocks share very little information with small contribution to the final goal.

Solutions

• Decrease depth and increase the width of ResNet.
• Dropout in identity has a negative effect. Dropout in convolution layers helps.

7. ResNext

Idea: cardinality, resnet + inception

Address the limitations of the Inception module: the filter numbers and sizes are tailored for each individual transformation, and the modules are customized stageby-stage (too much hand-crafting)

Cardinality is the number of splits like in inception. Increasing cardinality is more effective than going deeper or wider when we increase capacity.

• Identity shortcut added along with inception style splits.
• Depth-wise and separable convolution for efficiency.
• Residual connections are useful for optimizations, and aggregated transformations are helpful in stronger representations.

8. DenseNet

Idea: Combination of conacatenation

Explicitly differentiate between information that is added to the network (Exploration) and information that is preserved (Exploitation, especially by residual connection)

ResNets have problem of redundant features. Whereas traditional convolutional networks with L layers have L connections—one between each layer and its subsequent layer—our network has L(L+1) 2 direct connections. For each layer, the feature-maps of all preceding layers are used as inputs, and its own feature-maps are used as inputs into all subsequent layers. 

• Don’t sum features (unlike resnet), instead concatenate.

• Feature reuse in DenseNets as compared to ResNets: The number of parameters of ResNets is substantially larger because each layer has its own weights. You need to explicitly learn to preserve learnings from the previous layer and learn new features in ResNets. In DenseNets, you simple pass the learnings from previous layers as input so you can focus on only learning new information, hence they require less weights/

• DenseNet architecture explicitly differentiates between information that is added to the network and information that is preserved. - Each layer reads the state from its preceding layer and writes to the subsequent layer. It changes the state but also passes on information that needs to be preserved.
• Summation in ResNets might impede the information flow in the network. Concatenation doesn’t.
• DenseNets using pooling for downsampling in between two dense blocks (transition layers).

Advantages: 

1. they alleviate the vanishing-gradient problem
2. strengthen feature propagation
3. encourage feature reuse
4. substantially reduce the number of parameters - no need to learn redundant feature maps

9. ShuffleNet

Idea: channel shuffle operation

Two main ideas: 1. Pointwise group convolutions 2. Channel Shuffle operations

Desne 1x1 is expensive, even in efficient architectures like ResNext. Hence, group convolutions. Xception and ResNeXt introduce efficient depthwise separable convolutions or group convolutions. We noticed that both designs do not fully take the 1 × 1 convolutions (also called pointwise convolutions) into account, which requires considerable complexity. For example, in ResNeXt only 3 × 3 layers are equipped with group convolutions. To do channel shuffle, just take the transpose of the weight matrix.

10. MobileNet-v2

Main contribution is a novel layer module: the inverted residual with linear bottleneck. This module takes as an input a low-dimensional compressed representation which is first expanded to high dimension and filtered with a lightweight depthwise convolution. Features are subsequently projected back to a low-dimensional representation with a linear convolution.

When we look at all individual d-channel pixels of a deep convolutional layer, the information encoded in those values actually lie in some manifold, which in turn is embeddable into a low-dimensional subspace. At a first glance, such a fact could then be captured and exploited by simply reducing the dimensionality of a layer thus reducing the dimensionality of the operating space. Following that intuition, the width multiplier approach allows one to reduce the dimensionality of the activation space until the manifold of interest spans this entire space.

Simplified - Not all pixels in the d-channels are important. So, we can probably reduce the dimensions to the exact number where all the pixels have useful information. This is the main intuition for width multiplier approach.

Assuming the manifold of interest is low-dimensional we can capture this by inserting linear bottleneck layers into the convolutional blocks. Experimental evidence suggests that using linear layers is crucial as it prevents nonlinearities from destroying too much information. So, use 1x1 then 3x3 without relu in between.