Virtual reality based Brain-Machine-Interface for Sensori-Motor and Social experiments with Primates

As a result of improved understanding of brain mechanisms as well as unprecedented technical advancement in neural recording methods and computer technology, it is now possible to translate large-scale brain signals into movement intentions in real time. Such decoding of both actual and imagined movements of a subject allows for new paradigms of treatment for severely impaired patients, such as neural control of a prosthesis. The field of Brain Machine Interfaces (BMI) explores the tremendous potential of hybrid systems linking neural tissue to artificial devices. BMI operations involve a bidirectional learning process: the BMI system learns to decode brain signals by uncovering their relationship to voluntary movements, while the brain itself plastically adapts to the task. Proper BMI training is critical for its successful adoption by the patient. We believe that training a subject in a realistic virtual environment prior to the use of the physical prosthetic device is an efficient and safe method that can significantly facilitate design of practical neural prostheses for patients in need. In this dissertation we describe the control of a 3D virtual monkey (the avatar) as visual feedback for BMI with rhesus monkeys and address a number of key questions: • Monkeys’ interaction with the avatar. • Modulation of neurons in primary somatosensory (S1) and motor (M1) cortical areas during passive observation of the avatar being touched. • Modulation of neural responses by the observation of the avatar’s movements. • Changes in neural responses during long-term brain control of the avatar. We describe the plasticity of the body representation by the brain resulting from visual stimuli delivered via the avatar and tactile stimuli applied to the subject’s physical arm. We show how the avatar can be used for training rhesus monkeys to perform complex tasks. Behavioral evidence that rhesus monkeys respond to the avatar shape and motions and can even relate it with a representation of another monkey is presented. Finally two instances of novel brain controlled avatar are shown: a complete closed loop brain-machine-brain-interface with sensory feedback through direct cortical stimulation and the first successful attempt of a multi-limb BMI. We also study a simplified learning process of the BMI through the passive observation of the movements of the avatar arms.

A brain-spine interface alleviating gait deficits after spinal cord injury in primates

Quentin Barraud, Jocelyne Bloch, David Allenson Borton, Marco Capogrosso, Grégoire Courtine, Simone Ellen Joze Duis, Jérôme Gandar, Eduardo Martin Moraud, Silvestro Micera, Jean-Baptiste Mignardot, Tomislav Milekovic, Elodie Rey, Fabien Bertrand Paul Wagner

Spinal cord injury disrupts the communication between the brain and the spinal circuits that orchestrate movement. To bypass the lesion, brain-computer interfaces(1-3) have directly linked cortical activity to electrical stimulation of muscles, and have thus restored grasping abilities after hand paralysis(1,4). Theoretically, this strategy could also restore control over leg muscle activity for walking(5). However, replicating the complex sequence of individual muscle activation patterns underlying natural and adaptive locomotor movements poses formidable conceptual and technological challenges(6,7). Recently, it was shown in rats that epidural electrical stimulation of the lumbar spinal cord can reproduce the natural activation of synergistic muscle groups producing locomotion(8-10). Here we interface leg motor cortex activity with epidural electrical stimulation protocols to establish a brain-spine interface that alleviated gait deficits after a spinal cord injury in non-human primates. Rhesus monkeys (Macaca mulatta) were implanted with an intracortical microelectrode array in the leg area of the motor cortex and with a spinal cord stimulation system composed of a spatially selective epidural implant and a pulse generator with realtime triggering capabilities. We designed and implemented wireless control systems that linked online neural decoding of extension and flexion motor states with stimulation protocols promoting these movements. These systems allowed the monkeys to behave freely without any restrictions or constraining tethered electronics. After validation of the brain-spine interface in intact (uninjured) monkeys, we performed a unilateral corticospinal tract lesion at the thoracic level. As early as six days post-injury and without prior training of the monkeys, the brain-spine interface restored weight-bearing locomotion of the paralysed leg on a treadmill and overground. The implantable components integrated in the brain-spine interface have all been approved for investigational applications in similar human research, suggesting a practical translational pathway for proof-of-concept studies in people with spinal cord injury.

Nature Publishing Group2016

Active tactile exploration using a brain-machine-brain interface

Hannes Bleuler, Miguel Nicolelis, Solaiman Shokur

Brain–machine interfaces1, 2 use neuronal activity recorded from the brain to establish direct communication with external actuators, such as prosthetic arms. It is hoped that brain–machine interfaces can be used to restore the normal sensorimotor functions of the limbs, but so far they have lacked tactile sensation. Here we report the operation of a brain–machine–brain interface (BMBI) that both controls the exploratory reaching movements of an actuator and allows signalling of artificial tactile feedback through intracortical microstimulation (ICMS) of the primary somatosensory cortex. Monkeys performed an active exploration task in which an actuator (a computer cursor or a virtual-reality arm) was moved using a BMBI that derived motor commands from neuronal ensemble activity recorded in the primary motor cortex. ICMS feedback occurred whenever the actuator touched virtual objects. Temporal patterns of ICMS encoded the artificial tactile properties of each object. Neuronal recordings and ICMS epochs were temporally multiplexed to avoid interference. Two monkeys operated this BMBI to search for and distinguish one of three visually identical objects, using the virtual-reality arm to identify the unique artificial texture associated with each. These results suggest that clinical motor neuroprostheses might benefit from the addition of ICMS feedback to generate artificial somatic perceptions associated with mechanical, robotic or even virtual prostheses.

2011

Bimanual brain-machine interface

Zheng Li, Miguel Nicolelis, Solaiman Shokur

Brain-machine interfaces (BMIs) - devices that connect brain areas to external actuators - strive to restore limb mobility and sensation to patients suffering from paralysis or limb loss. Here we report a novel BMI that controls two virtual arms simultaneously. The development of BMIs for bimanual control is important because even the most basic daily movements such as opening a jar or buttoning a shirt require two arms. We for the first time have designed and implemented a bimanual BMI where activity of multiple cortical areas is translated in real-time into center-out reaching movements performed by two virtual arms. Eight multielectrode arrays, a total of 768 electrode channels, were implanted in the primary motor (M1), sensory (S1), supplementary motor (SMA), dorsal premotor (PMd), and posterior parietal (PP) cortices of both hemispheres of a rhesus monkey. Movement kinematics of each arm were extracted from the same ensemble of 400 neurons using a Wiener filter and an unscented Kalman filter (UKF). Typically, a single neuron contributed to the movements of both left and right arms. Movements were enacted by arms of a virtual rhesus monkey avatar on a computer screen presented in first-person to the monkey. On each trial, the virtual arms moved their central locations to peripheral targets presented simultaneously on the right and left sides of the computer screen. Peri-event time histograms and linear discriminant analysis revealed a highly distributed encoding scheme, with movement directions of both limbs represented by both ipsilateral and contralateral areas. Furthermore, movements were represented by multiple cortical regions, including both primary and non-primary motor areas which have been previously identified areas important for bimanual coordination. Over the course of several weeks of real-time BMI control, the monkey’s performance clearly improved both when the monkey continued to move the joystick and when the joystick was removed. These results support the feasibility of cortically-driven clinical neural prosthetics for bimanual operations.

2012

A brain-spine interface alleviating gait deficits after spinal cord injury in primates

Nature Publishing Group2016

Bimanual brain-machine interface

Zheng Li, Miguel Nicolelis, Solaiman Shokur

2012