Work done as final project for 6.5930 (Graduate Hardware Architecture for Deep Learning) at MIT.
AI accelerators have emerged as invaluable tools for achieving energy-efficient inference on deep convolutional networks (CNNs). However, large-scale CNN inference in modern AI systems brings challenges in energy efficiency to underlying hardware as they require significant data movement between on- and off-chip memory, which is often more energy-consuming than computation. Therefore, there has been great interest in developing structured dataflows which minimize total data movement cost. We show that existing accelertor architectures fail to exploit structured sparsity present in dilated convolutional workloads, and demonstrate using DeepLabv3 model architecture that incorporating dilated kernel decomposition into the computation mapping configuration achieves ~17% reduction in total access energy on the existing Eyeriss accelerator.
Atrous architectures, also known as dilated convolutions, are a type of convolution operation used in deep learning to expand the receptive field without increasing the number of parameters or reducing spatial resolution. A dilation rate introduces zero-padding between filter elements, allowing the convolution to capture broader signal context. Atrous convolutions are useful in tasks like semantic segmentation, where understanding long-range dependencies is critical. By preserving spatial resolution, they enable efficient feature extraction at multiple scales. In Einsum notation, an Atrous convolution is: $$ { \mathbf{O}\left[ n,m,x,y \right] = \text{ReLU} \left( \sum_{c,r,s}\mathbf{I}\left[ n,c,Ux+Dr,Yy+Ds \right] + \mathbf{W}\left[ m,c,r,s \right] \right) } $$
where lowercase indicates an index between $0$ inclusive and the corresponding uppercase definition exclusive. N is the number of inputs (batch size); C is the number of input channels; H and W are the height and width of the input feature map; M is the number of output channels (filters). The output feature map dimensions are X = (W - S + U) / U and Y = (H - R + U) / U. The effective kernel height is R = D(R_{original} - 1) + 1, and the effective kernel width is S = D(S_{original} - 1) + 1. Finally, D is the dilation factor and U is the stride size.DeepLabv3 is a popular architecture used for receptive field (context) broadening in semantic segmentation techniques. The model architecture composes ResNet50, Atrous Spatial Pyramid Pooling (ASPP) block, and a final vanilla convolution layer. To generate an equivalent architecture with dense convolutions, we decompose the ASPP layers into a series of dense vanilla convolutions involving only the dense kernel sub-tiles by omitting dilation padding and adjusting edge padding to preserve the padding correspondence with the operand data. Next, we analyze fJ/Compute metrics across various accelerator dataflows on the dilated and dense model variants.
On a pure WS accelerator, each kernel weight remains stationary in the register file to maximize weight reuse. Evaluating the dense and dilated workloads, we observe most energy consumption originates from shared global buffer I/O as it is used to store intermediate results (e.g. partial sums). Therefore, it quickly becomes a bottleneck during large model inference. However, in pointwise convolution step of ASPP, the DRAM access energy cost dominates in the dilated passthrough to over 1.5x the dense passthrough. As the WS architecture does not by default skip null computations introduced by kernel dilation, we attribute this to the increase in DRAM accesses. Similarly evaluating dilated and dense model workloads on a pure OS accelerator, we observe that DRAM accesses are the most significant contributor to energy usage due to suboptimal reuse patterns for inputs and weights. Additionally, in the case that the on-chip buffer is insufficient to hold all intermediates (e.g. for large outputs), they are spilled to DRAM, which throttles energy consumption.
Eyeriss addresses suboptimal reuse in the WS and OS schemes by proposing a row stationary dataflow, which keeps rows in registers and implicitly regularizes global buffer and DRAM usage. This reduces total energy usage by an order of magnitude. Nevertheless, for a dilated passthrough, we observe that DRAM accesses are still the main contributor to energy usage. However, with a row stationary dataflow, the dense workload consistently demonstrates significantly less DRAM access energy usage across all layers of ASPP. In fact, with our dilated kernel decomposition method, the dense variant achieves ~17% reduction in total energy usage, highlighting the promise of exploiting structured sparsity for increased energy efficiency on existing AI accelerators.
The most prominent application is dilated workloads. Atrous architectures are popular in semantic segmentation for enhancing network receptive field without introducing excessive computation overhead or losing information. More generally, we work toward reducing energy consumption in AI accelerators by introducing awareness of kernel structured-sparsity. While Eyeriss is specialized for CNN workloads, our approach is relevant to more generalized AI hardware, e.g. Tensor Processing Units (TPUs), and especially in edge scenarios where energy-efficiency is critical.