A Compensation Method of Two-Stage Image Generation for Human-AI Collaborated In-Situ Fashion Design in Augmented Reality Environment

In this paper, we consider a human-AI collaboration task, fashion design, in augmented reality environment. In particular, we propose a compensation method of two-stage image generation neural network for generating fashion design with progressive users' inputs. Our work is based on a recent proposed deep learning model, pix2pix, that can successfully transform an image from one domain into another domain, such as from line drawings to color images. However, the pix2pix model relies on the condition that input images should come from the same distribution, which is usually hard for applying it to real humancomputer interaction tasks, where the input from users differs from individual to individual. To address the problem, we propose a compensation method of two-stage image generation. In the first stage, we ask users to indicate their design preference with an easy task, such as tuning clothing landmarks, and use the input to generate a compensation input. With the compensation input, in the second stage, we then concatenate it with the real sketch from users to generate a perceptual better result. In addition, to deploy the two-stage image generation neural network in augmented reality environment, we designed and implemented a mobile application where users can create fashion design referring to real world human models. With the augmented 2D screen and instant feedback from our system, users can design clothing by seamlessly mixing the real and virtual environment. Through an online experiment with 46 participants and an offline use case study, we showcase the capability and usability of our system. Finally, we discuss the limitations of our system and further works on human-AI collaborated design.