Shruti Palaskar

Carnegie Mellon University

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Florian Metze

Carnegie Mellon University

Alan W. Black

Carnegie Mellon University

Ramon Sanabria

University of Edinburgh

Mark Hasegawa–Johnson

University of Illinois Urbana-Champaign

OÇ

Ozan Çağlayan

Imperial College London

Lucas Ondel

Brno University of Technology

Odette Scharenborg

Delft University of Technology

Pranava Madhyastha

City, University of London

Spandana Gella

Amazon (United States)

Amanda Duarte

Universitat Politècnica de Catalunya

Cooperative Institutions

Carnegie Mellon University

Johns Hopkins University

Apple (United Kingdom)

University of Sheffield

Centre National de la Recherche Scientifique

Imperial College London

Universitat Politècnica de Catalunya

IT University of Copenhagen

Tokyo Institute of Technology

Administration for Community Living

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the "Speaking Rosetta" JSALT 2017 Workshop

arXiv (Cornell University) (2018)

Odette Scharenborg Laurent Besacier Alan W. Black Mark Hasegawa–Johnson Florian Metze

We summarize the accomplishments of a multi-disciplinary workshop exploring the computational and scientific issues surrounding the discovery of linguistic units (subwords and words) in a language without orthography. We study the replacement of orthographic transcriptions by images and/or translated text in a well-resourced language to help unsupervised discovery from raw speech.

Orthography

10.48550/arxiv.1802.05092

Cite

Citations (2)

Multimodal Grounding for Sequence-to-Sequence Speech Recognition

arXiv (Cornell University) (2018)

Ozan Çağlayan Ramon Sanabria Shruti Palaskar Loïc Barrault Florian Metze

Humans are capable of processing speech by making use of multiple sensory modalities. For example, the environment where a conversation takes place generally provides semantic and/or acoustic context that helps us to resolve ambiguities or to recall named entities. Motivated by this, there have been many works studying the integration of visual information into the speech recognition pipeline. Specifically, in our previous work, we propose a multistep visual adaptive training approach which improves the accuracy of an audio-based Automatic Speech Recognition (ASR) system. This approach, however, is not end-to-end as it requires fine-tuning the whole model with an adaptation layer. In this paper, we propose novel end-to-end multimodal ASR systems and compare them to the adaptive approach by using a range of visual representations obtained from state-of-the-art convolutional neural networks. We show that adaptive training is effective for S2S models leading to an absolute improvement of 1.4% in word error rate. As for the end-to-end systems, although they perform better than baseline, the improvements are slightly less than adaptive training, 0.8 absolute WER reduction in single-best models. Using ensemble decoding, end-to-end models reach a WER of 15% which is the lowest score among all systems.

Word error rate

End-to-end principle

Pooling

10.48550/arxiv.1811.03865

Cite

Citations (0)

Speech Summarization using Restricted Self-Attention.

arXiv (Cornell University) (2021)

Roshan Sharma Shruti Palaskar Alan W. Black Florian Metze

Speech summarization is typically performed by using a cascade of speech recognition and text summarization models. End-to-end modeling of speech summarization models is challenging due to memory and compute constraints arising from long input audio sequences. Recent work in document summarization has inspired methods to reduce the complexity of self-attentions, which enables transformer models to handle long sequences. In this work, we introduce a single model optimized end-to-end for speech summarization. We apply the restricted self-attention technique from text-based models to speech models to address the memory and compute constraints. We demonstrate that the proposed model learns to directly summarize speech for the How-2 corpus of instructional videos. The proposed end-to-end model outperforms the previously proposed cascaded model by 3 points absolute on ROUGE. Further, we consider the spoken language understanding task of predicting concepts from speech inputs and show that the proposed end-to-end model outperforms the cascade model by 4 points absolute F-1.

Source

Cite

Citations (0)

How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language

arXiv (Cornell University) (2020)

Amanda Duarte Shruti Palaskar Deepti Ghadiyaram Kenneth DeHaan Florian Metze

Sign Language is the primary means of communication for the majority of the Deaf community. One of the factors that has hindered the progress in the areas of automatic sign language recognition, generation, and translation is the absence of large annotated datasets, especially continuous sign language datasets, i.e. datasets that are annotated and segmented at the sentence or utterance level. Towards this end, in this work we introduce How2Sign, a work-in-progress dataset collection. How2Sign consists of a parallel corpus of 80 hours of sign language videos (collected with multi-view RGB and depth sensor data) with corresponding speech transcriptions and gloss annotations. In addition, a three-hour subset was further recorded in a geodesic dome setup using hundreds of cameras and sensors, which enables detailed 3D reconstruction and pose estimation and paves the way for vision systems to understand the 3D geometry of sign language.

American Sign Language

Utterance

Source

Cite

Citations (2)

Transfer learning for multimodal dialog

Computer Speech & Language (2020)

Shruti Palaskar Ramon Sanabria Florian Metze

Audio-Visual Scene-Aware Dialog (AVSD) is best understood as an extension of Visual Question Answering, the task of generating a textual answer in response to a textual question on multi-media content. In AVSD, the answer-relevant "context" is expanded to include past dialog turns, which we view as a specialized form of extra textual knowledge (in addition to the standard video features). We have developed a framework that uses hierarchical attention to fuse contributions from different modalities, and had shown how it can be used to generate textual summaries from multi-modal sources, specifically videos with accompanying commentary. In this paper, we transfer the algorithmic approach, models, and data from this background corpus of 2000 h of how-to videos to the AVSD task, and report our findings. Our approach uses dialog context, but makes no assumption about the ordering of the history. Our system achieves the best performance in both automatic and human evaluations in the 7th Dialog State Tracking Challenge (AVSD).

Transfer of learning

10.1016/j.csl.2020.101093

Cite

Citations (6)

Learned In Speech Recognition: Contextual Acoustic Word Embeddings

arXiv (Cornell University) (2019)

Shruti Palaskar Vikas Raunak Florian Metze

End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon. In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simplified compared to phoneme, character or any other sort of sub-word units. In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution. On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions. In addition, we evaluate these embeddings on a spoken language understanding task, and observe that our embeddings match the performance of text-based embeddings in a pipeline of first performing speech recognition and then constructing word embeddings from transcriptions.

Word2vec

10.48550/arxiv.1902.06833

Cite

Citations (0)

Multimodal Grounding for Sequence-to-Sequence Speech Recognition

arXiv (Cornell University) (2018)

Ozan Çağlayan Ramon Sanabria Shruti Palaskar Loïc Barrault Florian Metze

Word error rate

End-to-end principle

Source

Cite

Citations (0)

How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

Amanda Duarte Shruti Palaskar Lucas Ventura Deepti Ghadiyaram Kenneth DeHaan

One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world impact, we conduct a study with ASL signers and show that synthesized videos using our dataset can indeed be understood. The study further gives insights on challenges that computer vision should address in order to make progress in this field.

American Sign Language

Modalities

10.1109/cvpr46437.2021.00276

Cite

Citations (108)

Multimodal Speech Summarization Through Semantic Concept Learning

Interspeech 2022 (2021)

Shruti Palaskar Ruslan Salakhutdinov Alan W. Black Florian Metze

10.21437/interspeech.2021-1923

Cite

Citations (8)

Linguistic unit discovery from multi-modal inputs in unwritten languages: Summary of the "Speaking Rosetta" JSALT 2017 Workshop