Gesture Therapy (GT) is a virtual rehabilitation tool for the upper arm that has been in the making since 2008, and by now has successfully demonstrated therapeutic validity in two small clinical trials for stroke survivors.During this time, our group has published a number of contributions regarding different aspects of this platform ranging from hardware controllers to artificial intelligence algorithms guiding different aspects of the serious games behaviour, and clinical trial data from observable improvements in dexterity to changes in functional neuroreorganization.As we continue our research efforts in virtual rehabilitation and realising this knowledge in the GT platform, this paper presents an overview of the latest developments as well as a roadmap for future research.
Virtual rehabilitation environments may afford greater patient personalization if they could harness the patient's affective state. Four states: anxiety, pain, engagement and tiredness (either physical or psychological), were hypothesized to be inferable from observable metrics of hand location and gripping strength-relevant for rehabilitation. Contributions are; (a) multiresolution classifier built from Semi-Naïve Bayesian classifiers, and (b) establishing predictive relations for the considered states from the motor proxies capitalizing on the proposed classifier with recognition levels sufficient for exploitation. 3D hand locations and gripping strength streams were recorded from 5 post-stroke patients whilst undergoing motor rehabilitation therapy administered through virtual rehabilitation along 10 sessions over 4 weeks. Features from the streams characterized the motor dynamics, while spontaneous manifestations of the states were labelled from concomitant videos by experts for supervised classification. The new classifier was compared against baseline support vector machine (SVM) and random forest (RF) with all three exhibiting comparable performances. Inference of the aforementioned states departing from chosen motor surrogates appears feasible, expediting increased personalization of virtual motor neurorehabilitation therapies.
Virtual rehabilitation taps affective computing to personalize therapy. States of anxiety, pain and engagement (affective) and tiredness (physical or psychological) were studied to be inferable from metrics of 3D hand location-proxy of hand movement- and fingers' pressure relevant for upper limb motor recovery. Features from the data streams characterized the motor dynamics of 2 stroke patients attending 10 sessions of motor virtual rehabilitation. Experts tagged states manifestations from videos. We aid classification contributing with a marginalization mechanism whereby absent input is reconstructed. With the hand movement information absent, marginalization statistically outperformed a base model where such input is ignored. Marginalized classification performance was (Area below ROC curve: μ ± σ) 0.880 ± 0.173 and 0.738 ± 0.177 for each patient. Marginalization aid classification sustaining performance under input failure or permitting different sensing settings.
Computational systems that process multiple affective states may benefit from explicitly considering the interaction between the states to enhance their recognition performance. This work proposes the combination of a multi-label classifier, Circular Classifier Chain (CCC), with a multimodal classifier, Fusion using a Semi-Naive Bayesian classifier (FSNBC), to include explicitly the dependencies between multiple affective states during the automatic recognition process. This combination of classifiers is applied to a virtual rehabilitation context of post-stroke patients. We collected data from post-stroke patients, which include finger pressure, hand movements, and facial expressions during ten longitudinal sessions. Videos of the sessions were labelled by clinicians to recognize four states: tiredness, anxiety, pain, and engagement. Each state was modelled by the FSNBC receiving the information of finger pressure, hand movements, and facial expressions. The four FSNBCs were linked in the CCC to exploit the dependency relationships between the states. The convergence of CCC was reached by 5 iterations at most for all the patients. Results (ROC AUC) of CCC with the FSNBC are over $0.940 \pm 0.045$ ( $mean \pm std.\;deviation$ ) for the four states. Relationships of mutual exclusion between engagement and all the other states and co-occurrences between pain and anxiety were detected and discussed.
Background: Gesture Therapy is an upper limb virtual reality rehabilitation-based therapy for stroke survivors. It promotes motor rehabilitation by challenging patients with simple computer games representative of daily activities for self-support. This therapy has demonstrated clinical value, but the underlying functional neural reorganization changes associated with this therapy that are responsible for the behavioral improvements are not yet known. Objective: We sought to quantify the occurrence of neural reorganization strategies that underlie motor improvements as they occur during the practice of Gesture Therapy and to identify those strategies linked to a better prognosis. Methods: Functional magnetic resonance imaging (fMRI) neuroscans were longitudinally collected at 4 time points during Gesture Therapy administration to 8 patients. Behavioral improvements were monitored using the Fugl-Meyer scale and Motricity Index. Activation loci were anatomically labelled and translated to reorganization strategies. Strategies are quantified by counting the number of active clusters in brain regions tied to them. Results: All patients demonstrated significant behavioral improvements (P < .05). Contralesional activation of the unaffected motor cortex, cerebellar recruitment, and compensatory prefrontal cortex activation were the most prominent strategies evoked. A strong and significant correlation between motor dexterity upon commencing therapy and total recruited activity was found (r2 = 0.80; P < .05), and overall brain activity during therapy was inversely related to normalized behavioral improvements (r2 = 0.64; P < .05). Conclusions: Prefrontal cortex and cerebellar activity are the driving forces of the recovery associated with Gesture Therapy. The relation between behavioral and brain changes suggests that those with stronger impairment benefit the most from this paradigm.
Virtual rehabilitation supports motor training following stroke by means of tailored virtual environments. To optimize therapy outcome, virtual rehabilitation systems automatically adapt to the different patients’ changing needs. Adaptation decisions should ideally be guided by both the observable performance and the hidden mind state of the user. We hypothesize that some affective aspects can be inferred from observable metrics. Here we present preliminary results of a classification exercise to decide on 4 states; tiredness, tension, pain and satisfaction. Descriptors of 3D hand movement and finger pressure were collected from 2 post-stroke participants while they practice on a virtual rehabilitation platform. Linear Support Vector Machine models were learnt to unfold a predictive relation between observation and the affective states considered. Initial results are promising (ROC Area under the curve (mean+/-std): 0.713 +/- 0.137). Confirmation of these opens the door to incorporate surrogates of mind state into the algorithm deciding on therapy adaptation.
Robotic arm therapy devices that incorporate actuated assistance can enhance arm recovery, motivate patients to practice, and allow therapists to deliver semi-autonomous training. However, because such devices are often complex and actively apply forces, they have not achieved widespread use in rehabilitation clinics or at home. This paper describes the design and pilot testing of a simple, mechanically passive device that provides robot-like assistance for active arm training using the principle of mechanical resonance. The Resonating Arm Exerciser (RAE) consists of a lever that attaches to the push rim of a wheelchair, a forearm support, and an elastic band that stores energy. Patients push and pull on the lever to roll the wheelchair back and forth by about 20 cm around a neutral position. We performed two separate pilot studies of the device. In the first, we tested whether the predicted resonant properties of RAE amplified a user’s arm mobility by comparing his or her active range of motion (AROM) in the device achieved during a single, sustained push and pull to the AROM achieved during rocking. In a second pilot study designed to test the therapeutic potential of the device, eight participants with chronic stroke (35 ± 24 months since injury) and a mean, stable, initial upper extremity Fugl-Meyer (FM) score of 17 ± 8 / 66 exercised with RAE for eight 45 minute sessions over three weeks. The primary outcome measure was the average AROM measured with a tilt sensor during a one minute test, and the secondary outcome measures were the FM score and the visual analog scale for arm pain. In the first pilot study, we found people with a severe motor impairment after stroke intuitively found the resonant frequency of the chair, and the mechanical resonance of RAE amplified their arm AROM by a factor of about 2. In the second pilot study, AROM increased by 66% ± 20% (p = 0.003). The mean FM score increase was 8.5 ± 4 pts (p = 0.009). Subjects did not report discomfort or an increase in arm pain with rocking. Improvements were sustained at three months. These results demonstrate that a simple mechanical device that snaps onto a manual wheelchair can use resonance to assist arm training, and that such training shows potential for safely increasing arm movement ability for people with severe chronic hemiparetic stroke.