Using Deep Learning to Support the User Interface Design of Mobile Applications with App Inventor

Transforming a user interface sketch into a wireframe using App Inventor has been a common need both in computing education and the development of applications by end users. As this part of mobile application development is challenging and time-consuming, we present an approach that automates this process through the prototyping of user interfaces. The Sketch2aia approach employs deep learning to detect the user interface components and their position in a hand-drawn sketch, creating an intermediate representation of the user interface and automatically generating the App Inventor code for the respective wireframe. The approach achieves an average classification accuracy of the components of the user interface of 87.72% and results of a preliminary evaluation with users indicate that it generates wireframes that resemble sketches in terms of visual similarity. The approach is available as a web tool and can be used to support the teaching of user interface design effectively and efficiently, as well as the development of mobile applications by end users.