A team from Facebook AI Research (FAIR) recently published paper Designing Network Design Spaces to introduce “a low-dimensional design space consisting of the simple, regular network..” called RegNet, which produces simple, fast and adaptable networks. In experiments, RegNet models have been five times faster than its contemporaries on the Graphics Processing Units (GPUs) and surpassed Google’s SOTA EfficientNet models. 

The researchers, Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár set out with the “Aim for interpretability and to discover general design principles that describe networks that are simple, work well, and generalize across settings.” As manual network design takes into considerations convolution, network and data size, depth, residuals, etc. manually identifying optimised networks, with increasing design choices, becomes difficult and inefficient. On the other hand, Neural architecture search (NAS) is a popular approach but it’s limitations in search space settings and lack of providing researchers insights into network design principles leaves much to be desired. 

Therefore, the team moved away from both the approaches and chose a middle-ground approach. They focused on designing actual network design spaces that would create infinite populations of model architects, “akin to manual network design, but elevated to the population level.”

Researchers start with an initial design space as input and gather model distributions via sampling and training. Design space quality is analyzed using error empirical distribution function (EDF). Various properties of the design space are visualized, and after an empirical bootstrap method predicts the likely range where the best models might fall, researchers use these insights to refine the design space.

The FAIR team then conducted controlled comparisons with EfficientNet sans taking-time enhancements, in the same training setup. With similar training settings and levels, and Flops (floating-point operations per second), RegNet models have performed better than EfficientNet models. 

Analysing the RegNet design space also provided researchers with other unexpected insights into network design. For example, they noticed that the depth of the best models is stable across compute regimes with an optimal depth of 20 blocks (60 layers). While it is a usual practice to use inverted bottlenecks methodology in modern mobile networks, researchers noticed that the methodology actually degraded performance. As per their findings, the best models did not employe neither the bottleneck nor the inverted bottleneck methodology.