Domain Balanced Sampling and Iterative Search for Product Identification

This paper introduces our solution to the 1st Workshop on Multimodal Product Identification in Livestreaming and Watch and Buy Challenge, a real-world task in a live stream scene and is challenging due to factors such as lighting, occlusion, and cross-domain. We model this task as the object detection and image retrieval problem. In the whole pipeline, we mainly focus on the problem of cross-domain retrieval and propose a domain-balanced sampling method, which enhances the robustness of multi-domain retrieval. Besides, to eliminate the influence of irrelevant clothing, we propose an iterative cross-search strategy, which greatly improves the accuracy of matching. In addition, we also experiment with the exploitation of text information, including multimodal product classification and multimodal intent recognition. With the aforementioned method, we achieved an F1 score of 69.2% and finally achieve first place in the competition.