SocialCueSwitch: Towards Customizable Accessibility by Representing Social Cues in Multiple Senses
Jonathan Isaac SegalSamuel RodriguezAkshaya RaghavanH. Beau BaezCrescentia JungJazmin CollinsShiri AzenkotAndrea Stevenson Won
1
Citation
47
Reference
10
Related Paper
Citation Trend
Abstract:
In virtual environments, many social cues (e.g. gestures, eye contact, and proximity) are currently conveyed visually or auditorily. Indicating social cues in other modalities, such as haptic cues to complement visual or audio signals, will help to increase VR's accessibility and take advantage of the platform's inherent flexibility. However, accessibility implementations in social VR are often siloed by single sensory modalities. To broaden the accessibility of social virtual reality beyond replacing one sensory modality with another, we identified a subset of social cues and built tools to enhance them allowing users to switch between modalities to choose how these cues are represented. Because consumer VR uses primarily visual and auditory stimuli, we started with social cues that were not accessible for blind and low vision (BLV) and d/Deaf and hard of hearing (DHH) people, and expanded how they could be represented to accommodate a number of needs. We describe how these tools were designed around the principle of social cue switching, and a standard distribution method to amplify reach.Keywords:
Modalities
Sensory cue
Stimulus modality
Social cue
Modality (human–computer interaction)
Sensory Substitution
There has been considerable effort devoted towards understanding sensory substitution devices in terms of their relationship to canonical sensory modalities. The approach taken in this essay is rather different, although complementary, in that we seek to define a broad conceptual space of ‘sensory tools’ in which sensory substitution devices can be situated. Such devices range from telescopes, to cochlear implants, to attempts to create a magnetic sense. One feature of these devices is that they operate at the level of ‘raw’ sensory information. As such, systems such as Braille which operate at a symbolic/conceptual level do not count as a sensory tool (or a sensory substitution device) and nor would a device such as CCTV which, although capturing raw sensory information, would not meet a conventional definition of a tool. With this approach, we hope to avoid the circularity inherent in previous attempts at defining sensory substitution and provide a better starting point to explore the effects of sensory tools, more generally, on the functioning of the nervous system.
Sensory Substitution
Stimulus modality
Substitution (logic)
Braille
Cite
Citations (7)
Sensory Substitution
Modality (human–computer interaction)
Stimulus modality
Substitution (logic)
Modalities
Cite
Citations (2)
Various modes of sensory feedback to the user have the potential to enhance performance in robot-assisted surgery. In this paper, it is hypothesized that substituting or augmenting force feedback by visual representation of the force levels can potentially assist the user in limiting the amount of applied forces. In addition to confirming the above for a telemanipulated suturing task, the results indicate that there is a trade-off between the magnitudes of applied forces and the time required, to complete the task.
Sensory Substitution
Limiting
Representation
Visual feedback
Cite
Citations (17)
Sensory substitution devices were developed in the context of perceptual rehabilitation and they aim at compensating one or several functions of a deficient sensory modality by converting stimuli that are normally accessed through this deficient sensory modality into stimuli accessible by another sensory modality. For instance, they can convert visual information into sounds or tactile stimuli. In this article, we review those studies that investigated the individual differences at the behavioural, neural, and phenomenological levels when using a sensory substitution device. We highlight how taking into account individual differences has consequences for the optimization and learning of sensory substitution devices. We also discuss the extent to which these studies allow a better understanding of the experience with sensory substitution devices, and in particular how the resulting experience is not akin to a single sensory modality. Rather, it should be conceived as a multisensory experience, involving both perceptual and cognitive processes, and emerging on each user's pre-existing sensory and cognitive capacities.
Sensory Substitution
Stimulus modality
Modality (human–computer interaction)
Cite
Citations (21)
Sensory substitution refers to the use of one sensory modality (e.g., hearing) to supply environmental information normally gathered by another sense (e.g., vision) while still preserving some of the key functions of the original sense. For example, the use of auditory signals might give information about visual scenes. The development of sensory substitution devices has profoundly changed the classical definition of sensory modalities and contributed to the emergence of a form of “artificial synaesthesia”. In the last decade, our knowledge about cognitive and brain mechanisms involved in sensory substitution has grown considerably bringing new insights into human perception and neural plasticity. Thanks to technological advances and scientific achievements, sensory substitution has become a real alternative for restoring some functions of a defective sensory organ (e.g., sight in case of blindness or hearing in the case of deafness). This paper addresses some of the major questions raised by sensory substitution, demonstrates how the study of sensory substitution enhances our understanding of human perception and brain plasticity and provides an overview of rehabilitation potentialities.
Sensory Substitution
Stimulus modality
Substitution (logic)
Cite
Citations (4)
In this paper, we report two human factors studies performed to investigate human performance in tasks involving dextrous manipulations of virtual objects with haptic feedback. These studies were performed at the Man-Machine Interface Laboratory (Rutgers University, NJ, USA) and at the Laboratoire de Robotique de Paris (CRIIF-LRP). Different haptic devices were used : the Rutgers Master, the LRP Force Feedback Glove (LRP), and the ITR device (Interface Technology Research, UK). These experiments involved (1) accurate placements of virtual objects and (2) grasping-force regulation. In the former case, we compared the LRP FFG (that provides complex haptic cueing) and the ITR device(that provides binary haptic cueing : grasped/not grasped). In the latter case, we investigated the effect of sensory substitution (synesthetic representation of haptic informations) to convey virtual haptic cues using the visual and auditory sensory channels. Results of the experiments showed that (1) information redundancy (display of visual or auditory force feedback in addition to finger force feedback) allowed to increase performance compared to finger force feedback alone, and (2) simple tactual feedback (provided by the ITR device) allowed similar performance than complex force feedback (provided by the LRP FFG). This result led us to think that complex force feedback cues are not required in simple manipulation task (i.e. when grasping regulation forces is not crucial). In this case binary haptic information could be sufficient.
Sensory Substitution
Stereotaxy
Interface (matter)
Sensory cue
Cite
Citations (13)
Sensory substitution devices aim at compensating sensory deficits by converting stimuli coming from a deficient sensory modality (e.g., vision) into stimuli accessible through another modality (e.g., touch or audition). Studies conducted with these devices revealed the central nervous system to be very plastic. Various laboratories have conducted studies investigating such plasticity by means of behavioural and brain-imaging techniques. At the ISIR Laboratory, we focused on the factors underlying the learning of sensory substitution devices, their adequacy to the target population, and we explored ways of improving their design by the use of crossmodal correspondences and by taking into account individual differences in the used reference frames. We also investigated the nature of the experience with sensory substitution. In particular, we suggested moving beyond positions reducing experience to that of a single sensory modality. Rather, sensory substitution is considered as a multisensory experience, involving not only visual, but also auditory or tactile processes as well as cognitive processes. In this framework, individual differences do have an influence on the extent to which the different sensory modalities influence the experience with the devices.
Sensory Substitution
Crossmodal
Stimulus modality
Modality (human–computer interaction)
Multisensory Integration
Modalities
Substitution (logic)
Cite
Citations (11)
Sensory substitution devices make use of information in one sensory modality to deliver information usually provided by another. But when information usually presented visually is presented to a subject in an auditory or haptic way, is the resulting experience in any sense visual? Or does sensory substitution show that dimensions of experience—about the spatial layout of objects and properties in the environment—that were previously taken to be essentially visual can be experienced in other modalities too? I will consider this question by looking at whether a property such as the transparency of visual experience can be transferred to, and enhance, experience in other modalities.
Sensory Substitution
Stimulus modality
Modalities
Modality (human–computer interaction)
Substitution (logic)
Cite
Citations (2)
Sensory Substitution
Stimulus modality
Modality (human–computer interaction)
Artificial vision
Cite
Citations (5)
Sensory substitution refers to the use of one sensory modality (e.g., hearing) to supply environmental information normally gathered by another sense (e.g., vision) while still preserving some of the key functions of the original sense. For example, the use of auditory signals might give information about visual scenes. The development of sensory substitution devices has profoundly changed the classical definition of sensory modalities and contributed to the emergence of a form of
Sensory Substitution
Substitution (logic)
Stimulus modality
Modality (human–computer interaction)
Modalities
Cite
Citations (3)