Towards Accurate and Compact Architectures via Neural Architecture Transformer

Abstract:

Deep neural networks succeed because of good architectures. Some Neural Architecture Search (NAS) methods search or manually design deep architectures. Even a well-designed/searched architecture may contain many nonsignificant or redundant modules/operations (e.g., intermediate convolution or pooling layers). Redundancy wastes memory, computational power, and performance. Thus, to improve performance without increasing computational cost, architecture operations must be optimized. We propose a Neural Architecture Transformer (NAT) method that turns the optimization problem into a Markov Decision Process (MDP) and replaces redundant operations with skip or null connection operations. NAT has a limited search space because it only considers a few replacements/transitions. Thus, a small search space may hinder architecture optimization. To improve architecture optimization, we propose a Neural Architecture Transformer++ (NAT++) method that expands the set of candidate transitions. We present a two-level transition rule to obtain valid transitions, allowing operations to have more efficient types (e.g., convolution ${\to }$ separable convolution) or smaller kernel sizes ($5{\times }5 {\to } 3{\times }3$). Valid transitions vary by operation. We suggest a Binary-Masked Softmax (BMSoftmax) layer to eliminate invalid transitions. Finally, we use policy gradient to learn an optimal policy from the MDP formulation to infer optimized architectures. The transformed architectures outperform both their original counterparts and those optimized by existing methods.

Note: Please discuss with our team before submitting this abstract to the college. This Abstract or Synopsis varies based on student project requirements.