Semantic assessment of shopping behavior using trajectories, shopping related actions, and context information

Automatic understanding of customers' shopping behavior and acting according to their needs is relevant in the marketing domain and is attracting a lot of attention lately. In this work, we propose a multi-level framework for the automatic assessment of customers' shopping behavior. The low level input to the framework is obtained from different types of cameras, which are synchronized, facilitating efficient processing of information. A fish-eye camera is used for tracking people, while a high-definition one serves for the action recognition task. The experiments are performed on both laboratory and real-life recordings in a supermarket. From the video recordings, we extract features related to the spatio-temporal behavior of trajectories, the dynamics and the time spent in each region of interest (ROI) in the shop and regarding the customer-products interaction patterns. Next we analyze the shopping sequences using a Hidden Markov Model (HMM). We conclude that it is possible to accurately classify trajectories (93%), discriminate between different shopping related actions (91.6%), and recognize shopping behavioral types by means of our proposed reasoning model in 95% of the cases.