Êtes-vous un étudiant de l'EPFL à la recherche d'un projet de semestre?
Travaillez avec nous sur des projets en science des données et en visualisation, et déployez votre projet sous forme d'application sur Graph Search.
Sequence modeling for signs and gestures is an open research problem. In thatdirection, there is a sustained effort towards modeling signs and gestures as a se-quence of subunits. In this paper, we develop a novel approach to infer movementsubunits in a data-driven manner to model signs and gestures in the frameworkof hidden Markov models (HMM) given the skeleton information. This approachinvolves: (a) representation of position and movement information with measure-ment of hand positions relative to body parts (head, shoulders, hips); (b) modelingthese features to infer a sign-specific left-to-right HMM; and (c) clustering theHMM states to infer states or subunits that are shared across signs and updat-ing the HMM topology of signs. We investigate the application of the proposedapproach on sign and gesture recognition tasks, specifically on Turkish signs Hos-piSign database and Italian gestures Chalearn 2014 task. On both databases, ourstudies show that, while yielding competitive systems, the proposed approach leadsto a shared movement subunit representation that maintains discrimination acrosssigns and gestures.
Devis Tuia, Sylvain Lobry, Nicolas Courty