FPANet: Feature pyramid aggregation network for real-time semantic segmentation

Semantic segmentation is used in many fields, and most fields not only require models with high-quality predictions but also require real-time speed in the forward inference phase. Therefore, our goal is to perform high-quality real-time semantic segmentation, thus proposing the feature pyramid aggregation network (FPANet). This network can be regarded as an encoder-decoder model. In the encoder stage, we use ResNet and atrous spatial pyramid pooling (ASPP) to extract more high-level semantic information. In the decoder stage, to simultaneously obtain the semantic and spatial information of the image, we propose a bilateral directional feature pyramid network for semantic segmentation to fuse features at different levels, it is named SeBiFPN. In SeBiFPN, we design a lightweight feature pyramid fusion module (FPFM) to fuse features from two different levels. In addition, when predicting the border region of an image, most real-time semantic segmentation models perform poorly; therefore, we propose a border refinement module (BRM) to improve the problem of inaccurate border segmentation. To reduce the computational complexity of the model, we redesign the ASPP module and reduce the number of feature channels during feature fusion. Our method achieves a better balance of speed and accuracy compared to the state-of-the-art methods on the Cityscapes and CamVid datasets.