Improving Saliency Metrics for Channel Pruning of Convolutional Neural Networks

Abstract

Channel pruning is an effective technique to reduce the size of Convolutional Neural Networks (CNNs). A decisive part of any pruning algorithm is its saliency metric. We propose different techniques to improve saliency metrics for channel pruning. Saliency metrics are encompassed in a larger pruning algorithm and are expressed in various forms. Without a standard form, it can be difficult to identify and compare these metrics. To facilitate the comparison of saliency metrics, we propose a taxonomy based on four independent components: base input, pointwise metric, reduction, and scaling. We classify existing saliency metrics according to our proposed taxonomy. We find that new channel saliency metrics can be created using the components of existing saliency metrics. We also propose a new scaling method. We evaluate the newly created saliency metrics (using existing components as well as our new scaling method) and find that some of the metrics outperform existing ones. We also provide some guidance for the construction of new saliency metrics. Specifically, we highlight the importance of the reduction and scaling methods. Pruning algorithms generally rely on a single saliency metric for pruning. Even if that chosen metric performs well on average, it can make poor decisions from time to time. Through the use of multiple saliency metrics, we can compensate the poor decisions of the single metric. We show that the combination of saliency metrics is possible and combine the decisions of multiple saliency metrics using a myopic oracle. We show that the decisions of the myopic oracle can lead to better pruning rates than the constituent metrics. When pruning one channel from CNNs with split and join connections, more pruning opportunities become apparent. Multiple channels can be pruned by transitively removing channel weights from other layers of the network. However, most saliency metrics do not factor in these extra structural constraints. We propose domino saliency metrics, built on top of existing channel saliency metrics, to factor in these constraints. We show that the use of domino saliency metrics can significantly improve pruning rates for networks with splits and joins.