Dynamic Classifier Fusion for Multi-sensory Activity Recognition

Nowadays, many systems rely on fusing different sources of information to recognize human activities and gestures, speech, or brain activities for applications in areas such as clinical practice, and health care and Human Computer Interaction (HCI). Typically, related information comes from sensors mounted on the body, head and limbs and usually a classifier will be trained to recognize different patterns. For instance, in a human activity recognition task, a set of accelerometers placed on the body provides information suitable for gesture recognition. To increase the performance of recognition, more sensors, potentially of different types, can be deployed in order to gain more information (as a network of sensors). This brings the challenge of combining their information to reach the final decision, whilst considering that some of them may provide more information than others. Additionally, in long-term operation of a recognition system changes in the network pose a considerable problem. In real deployed systems, changes in the sensors and the network are inevitable. They include: degradation of the quality of the data, changes in the sensor readings, sensor failure, and communication problems. For example, in the human activity recognition scenario, one on-body sensor can slide and give abnormal data or the connection may be lost. To recover from changes in the sensors one approach is to stop the running system and to re-train it with the new structure, hence it may be costly in time and effort. The goal of this thesis is to develop methods that are able to handle on-line changes in the network of sensors. To satisfy this goal, the fusion system should support the selection of the most informative classifiers (or sensors) taking into account application constraints (e.g. limitation on power consumption). Furthermore, it needs to detect faults or changes in the system, in order to take compensatory actions (e.g. remove sensors, impute data). Our work is focused on classifier fusion (ensemble), giving more flexibility to fulfill the objectives of the thesis (especially sensor addition and removal). We developed approaches for ensemble creation and energy optimization, and for detecting anomalous classifiers in an ensemble and retraining them with new data. These methods aim to make the recognition task robust against dynamic changes in the network. We applied these online methods to real data (human activities) as well as to synthetic data, and we showed an improvement in the performance of the recognition system. Similarly, classifier fusion and anomaly detection can be applied to other domains such as Brain-Computer Interfaces (BCIs) and speech recognition. We proposed an approach to fuse information from classifiers of Electroencephalographic (EEG) and Electromyographic (EMG) signals to improve the Brain-Computer Interface (BCI) performance. Furthermore, an anomaly detector is proposed to spot the faulty Electroencephalography (EEG) electrode