Multimodal Interactions with an Instrumented Shelf

In this paper, we describe the initial implementation of our application demonstrator called 'ShopAssist'. This application will aid users in product queries within a shopping scenario context. We describe the wide range of input modalities that our application supports such as speech, handwriting, intra- and extra gestures, and the mixed modality combinations that promote advanced user interaction with real-world and virtual objects.