Mengxin Ren

East China Normal University

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Yuang Li

Harbin Engineering University

Xiaosong Qiao

Huawei Technologies (China)

Jiawei Yu

Third Affiliated Hospital of Harbin Medical University

Daimeng Wei

Huawei Technologies (China)

Miaomiao Ma

Huawei Technologies (China)

Xiaofeng Zhao

Hohai University

Jinsong Su

Xiamen University

Hao Yang

Huawei Technologies (China)

Shimin Tao

Huawei Technologies (China)

Hao Yang

Hunan University of Science and Technology

Cooperative Institutions

Huawei Technologies (China)

Chinese Academy of Sciences

East China Normal University

Shanghai Key Laboratory of Trustworthy Computing

Huawei Technologies (United States)

Harbin Institute of Technology

Tsinghua University

Peking University

Xiamen University

University of Chinese Academy of Sciences

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

Using Large Language Model for End-to-End Chinese ASR and NER

Interspeech 2022 (2024)

Yuang Li Jiawei Yu Min Zhang Mengxin Ren Yanqing Zhao

Mapping speech tokens to the same feature space as text tokens has become the paradigm for integrating speech modality into decoder-only large language models (LLMs). An alternative is to use an encoder-decoder architecture that incorporates speech features through cross-attention. In this work, we connect the Whisper encoder with ChatGLM3 and provide in-depth comparisons of these two approaches using Chinese automatic speech recognition (ASR) and named entity recognition (NER) tasks. We evaluate their performance using the F1 score and a fine-grained taxonomy of ASR-NER errors. Our experiments reveal that the encoder-decoder model outperforms the decoder-only model if the context is short, while the decoder-only model benefits from a long context as it fully exploits all layers of the LLM. Additionally, we obtain a state-of-the-art F1 score of 0.805 on the AISHELL-NER test set by using chain-of-thought NER which first infers long-form ASR transcriptions and then predicts NER labels.

End-to-end principle

10.21437/interspeech.2024-103

Cite

Citations (0)

Cross-Domain Audio Deepfake Detection: Dataset and Analysis

arXiv (Cornell University) (2024)

Yuang Li Min Zhang Mengxin Ren Miaomiao Ma Daimeng Wei

Audio deepfake detection (ADD) is essential for preventing the misuse of synthetic voices that may infringe on personal rights and privacy. Recent zero-shot text-to-speech (TTS) models pose higher risks as they can clone voices with a single utterance. However, the existing ADD datasets are outdated, leading to suboptimal generalization of detection models. In this paper, we construct a new cross-domain ADD dataset comprising over 300 hours of speech data that is generated by five advanced zero-shot TTS models. To simulate real-world scenarios, we employ diverse attack methods and audio prompts from different datasets. Experiments show that, through novel attack-augmented training, the Wav2Vec2-large and Whisper-medium models achieve equal error rates of 4.1\% and 6.5\% respectively. Additionally, we demonstrate our models' outstanding few-shot ADD ability by fine-tuning with just one minute of target-domain data. Nonetheless, neural codec compressors greatly affect the detection accuracy, necessitating further research.

10.48550/arxiv.2404.04904

Cite

Citations (0)

A Multitask Training Approach to Enhance Whisper with Open-Vocabulary Keyword Spotting

Interspeech 2022 (2024)

Yuang Li Min Zhang Chang Su Yinglu Li Xiaosong Qiao

The recognition of rare named entities, such as personal names and terminologies, is challenging for automatic speech recognition (ASR) systems, especially when they are not frequently observed in the training data. In this paper, we introduce keyword spotting enhanced Whisper (KWS-Whisper), a novel ASR system that leverages the Whisper model and performs open-vocabulary keyword spotting (OV-KWS) on the hidden states of the Whisper encoder to recognize user-defined named entities. These entities serve as prompts for the Whisper decoder. To optimize the model, we propose a multitask training approach that learns OV-KWS and contextual-ASR tasks. We evaluate our approach on Chinese Aishell hot word subsets and two internal code-switching test sets and show that it significantly improves the entity recall compared to the original Whisper model. Moreover, we demonstrate that the OV-KWS can be a plug-and-play module to enhance the ASR error correction methods and frozen Whisper models.

Keyword spotting

Spotting

10.21437/interspeech.2024-104

Cite

Citations (0)

Neural Barrier Certificates Synthesis of NN-Controlled Continuous Systems via Counterexample-Guided Learning

Hanrui Zhao Niuniu Qi Mengxin Ren Xia Zeng Zhenbing Zeng

Counterexample

10.1145/3649329.3658256

Cite

Citations (0)

Using Large Language Model for End-to-End Chinese ASR and NER

arXiv (Cornell University) (2024)

Yuang Li Jiawei Yu Yanqing Zhao Min Zhang Mengxin Ren

Mapping speech tokens to the same feature space as text tokens has become the paradigm for the integration of speech modality into decoder-only large language models (LLMs). An alternative approach is to use an encoder-decoder architecture that incorporates speech features through cross-attention. This approach, however, has received less attention in the literature. In this work, we connect the Whisper encoder with ChatGLM3 and provide in-depth comparisons of these two approaches using Chinese automatic speech recognition (ASR) and name entity recognition (NER) tasks. We evaluate them not only by conventional metrics like the F1 score but also by a novel fine-grained taxonomy of ASR-NER errors. Our experiments reveal that encoder-decoder architecture outperforms decoder-only architecture with a short context, while decoder-only architecture benefits from a long context as it fully exploits all layers of the LLM. By using LLM, we significantly reduced the entity omission errors and improved the entity ASR accuracy compared to the Conformer baseline. Additionally, we obtained a state-of-the-art (SOTA) F1 score of 0.805 on the AISHELL-NER test set by using chain-of-thought (CoT) NER which first infers long-form ASR transcriptions and then predicts NER labels.

Named Entity Recognition

End-to-end principle

10.48550/arxiv.2401.11382

Cite

Citations (0)

"I've Heard of You!": Generate Spoken Named Entity Recognition Data for Unseen Entities

arXiv (Cornell University) (2024)

Jiawei Yu Xiang Geng Yuang Li Mengxin Ren Wei Tang

Spoken named entity recognition (NER) aims to identify named entities from speech, playing an important role in speech processing. New named entities appear every day, however, annotating their Spoken NER data is costly. In this paper, we demonstrate that existing Spoken NER systems perform poorly when dealing with previously unseen named entities. To tackle this challenge, we propose a method for generating Spoken NER data based on a named entity dictionary (NED) to reduce costs. Specifically, we first use a large language model (LLM) to generate sentences from the sampled named entities and then use a text-to-speech (TTS) system to generate the speech. Furthermore, we introduce a noise metric to filter out noisy data. To evaluate our approach, we release a novel Spoken NER benchmark along with a corresponding NED containing 8,853 entities. Experiment results show that our method achieves state-of-the-art (SOTA) performance in the in-domain, zero-shot domain adaptation, and fully zero-shot settings. Our data will be available at https://github.com/DeepLearnXMU/HeardU.

10.48550/arxiv.2412.19102

Cite

Citations (0)

Cross-Domain Audio Deepfake Detection: Dataset and Analysis

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2024)

Yuang Li Min Zhang Mengxin Ren Xiaosong Qiao Miaomiao Ma

Audio analyzer

10.18653/v1/2024.emnlp-main.286

Cite

Citations (0)