Tracking of Flexible Brush Tip on Real Canvas: Silhouette-based and Deep Ensemble Network-Based Approaches

Digital painting is a process of creating a digital artwork using modern human-computer interaction technologies. One of the core enabling technologies is the real-time tracking of user's strokes, which is generally supplied by a digital tablet with a stylus. While the digital tablet technology provides highly accurate tracking, the drawing should be done with a rigid stylus on a plastic surface. This sometimes destroys the realism of drawing, such as interaction with the digital tablet cannot provide the feedback of subtle texture, friction of the paper/fabric canvas and tension of soft painting brush. This becomes particularly problematic for traditional painting artists who are trained with and prefer real painting brush and paper/fabric canvas. Thus, the aim of this work is to present an alternative solution where the user's strokes can be tracked even when the actual brush and canvas are used. To this end, we proposed two approaches for digitally tracking the tip of flexible bristles of a soft brush, so that the painting can be created digitally on a computer. The first approach captures the silhouette of deforming bristles using a pair of well-aligned infra-red (IR) cameras, which extracts the tip from the silhouette, and reconstructs the 2D position of the tip. The second approach predicts the brush tip position through a deep ensemble network-based approach where the relationship between the brush tip position and brush handle pose are trained with our novel model comprising of Long-Short Term Memory Autoencoder and 1-D Convolutional Neural Network. The trained model is used to predict the brush tip position in realtime. Both approaches extensively evaluated through multiple tests. Furthermore, our model outperforms the state-of-the-art models.