Supervised Fine-Grained Cloud Detection and Recognition in Whole-Sky Images

The whole-sky imager has been increasingly used for ground-based cloud automatic observation. Many approaches based on image processing have been applied to detect or classify clouds in whole-sky images (WSIs). However, most of the studies only focus on image segmentation for cloud detection or image classification for cloud recognition separately. The cloud detection only does the binary segmentation (sky and cloud) without cloud types, while the cloud recognition only gives the single image-level label without cloud coverage. In this paper, a fine-grained cloud detection and recognition task with a solution is proposed to fill the gap, which can simultaneously detect and classify clouds in a WSI. It can be regarded as a pixel-level fine-grained dense prediction for images. First, a new data set is built with pixel-level annotation of nine different types. Then, a solution based on supervised learning is proposed, in which the pixel-level prediction problem is converted to a superpixel classification problem. Multiview features are extracted, including color, inside texture, neighbor texture, and global relation, to represent the superpixels. Moreover, a class-specific feature space transformation method based on metric learning and subspace alignment is proposed to overcome the challenge brought by the high similarity among cloud types and the feature shifting. Finally, several experiments have verified that our approach is effective to the challenging new task and also outperforms some other methods in the normal tasks of cloud detection and cloud classification, respectively.