Alan Ritter

Georgia Institute of Technology

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Wei Xu

Georgia Institute of Technology

Preslav Nakov

Mohamed bin Zayed University of Artificial Intelligence

Veselin Stoyanov

The University of Texas at Austin

Ashutosh Baheti

Allen Institute for Artificial Intelligence

Sara Rosenthal

IBM (United States)

Yang Chen

Simcere Pharmaceutical (China)

Jiwei Li

Zhejiang University

Jeniya Tabassum

The Ohio State University

Tarek Naous

American University of Beirut

Oren Etzioni

University of Washington

Cooperative Institutions

RIKEN Center for Advanced Intelligence Project

293

Mongolia International University

282

Tokyo Institute of Technology

108

IT University of Copenhagen

106

Administration for Community Living

106

American Jewish Committee

101

Carnegie Mellon University

University of Southern California

University of Washington

Google (United States)

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

Deep Reinforcement Learning for Dialogue Generation

arXiv (Cornell University) (2016)

Jiwei Li Will Monroe Alan Ritter Michel Galley Jianfeng Gao

Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes. Modeling the future direction of a dialogue is crucial to generating coherent, interesting dialogues, a need which led traditional NLP models of dialogue to draw on reinforcement learning. In this paper, we show how to integrate these goals, applying deep reinforcement learning to model future reward in chatbot dialogue. The model simulates dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity (non-repetitive turns), coherence, and ease of answering (related to forward-looking function). We evaluate our model on diversity, length as well as with human judges, showing that the proposed algorithm generates more interactive responses and manages to foster a more sustained conversation in dialogue simulation. This work marks a first step towards learning a neural conversational model based on the long-term success of dialogues.

Chatbot

10.48550/arxiv.1606.01541

Cite

Citations (431)

Pre-train or Annotate? Domain Adaptation with a Constrained Budget

arXiv (Cornell University) (2021)

Fan Bai Alan Ritter Wei Xu

Recent work has demonstrated that pre-training in-domain language models can boost performance when adapting to a new domain. However, the costs associated with pre-training raise an important question: given a fixed budget, what steps should an NLP practitioner take to maximize performance? In this paper, we view domain adaptation with a constrained budget as a consumer choice problem, where the goal is to select an optimal combination of data annotation and pre-training. We measure annotation costs of three procedural text datasets, along with the pre-training costs of several in-domain language models. The utility of different combinations of pre-training and data annotation are evaluated under varying budget constraints to assess which combination strategy works best. We find that for small budgets, spending all funds on annotation leads to the best performance; once the budget becomes large enough, however, a combination of data annotation and in-domain pre-training yields better performance. Our experiments suggest task-specific data annotation should be part of an economical strategy when adapting an NLP model to a new domain.

Domain Adaptation

Budget constraint

Labeled data

Training set

10.48550/arxiv.2109.04711

Cite

Citations (2)

Reducing Privacy Risks in Online Self-Disclosures with Language Models

arXiv (Cornell University) (2023)

Yao Dou Isadora Krsek Tarek Naous Anubha Kabra Sauvik Das

Self-disclosure, while being common and rewarding in social media interaction, also poses privacy risks. In this paper, we take the initiative to protect the user-side privacy associated with online self-disclosure through detection and abstraction. We develop a taxonomy of 19 self-disclosure categories and curate a large corpus consisting of 4.8K annotated disclosure spans. We then fine-tune a language model for detection, achieving over 65% partial span F$_1$. We further conduct an HCI user study, with 82% of participants viewing the model positively, highlighting its real-world applicability. Motivated by the user feedback, we introduce the task of self-disclosure abstraction, which is paraphrasing disclosures into less specific terms while preserving their utility, e.g., "Im 16F" to "I'm a teenage girl". We explore various fine-tuning strategies, and our best model can generate diverse abstractions that moderately reduce privacy risks while maintaining high utility according to human evaluation. To help users in deciding which disclosures to abstract, we present a task of rating their importance for context understanding. Our fine-tuned model achieves 80% accuracy, on-par with GPT-3.5. Given safety and privacy considerations, we will only release our corpus to researchers who agree to ethical guidelines.

10.48550/arxiv.2311.09538

Cite

Citations (0)

Human-in-the-loop Evaluation for Early Misinformation Detection: A Case Study of COVID-19 Treatments

Ethan Mendes Yang Chen Wei Xu Alan Ritter

We present a human-in-the-loop evaluation framework for fact-checking novel misinformation claims and identifying social media messages that support them. Our approach extracts check-worthy claims, which are aggregated and ranked for review. Stance classifiers are then used to identify tweets supporting novel misinformation claims, which are further reviewed to determine whether they violate relevant policies. To demonstrate the feasibility of our approach, we develop a baseline system based on modern NLP methods for human-in-the-loop fact-checking in the domain of COVID-19 treatments. We make our data and detailed annotation guidelines available to support the evaluation of human-in-the-loop systems that identify novel misinformation directly from raw user-generated content.

Misinformation

Human-in-the-loop

Baseline (sea)

10.18653/v1/2023.acl-long.881

Cite

Citations (2)

Code and Named Entity Recognition in StackOverflow

arXiv (Cornell University) (2020)

Jeniya Tabassum Mounica Maddela Wei Xu Alan Ritter

There is an increasing interest in studying natural language and computer code together, as large corpora of programming texts become readily available on the Internet. For example, StackOverflow currently has over 15 million programming related questions written by 8.5 million users. Meanwhile, there is still a lack of fundamental NLP techniques for identifying code tokens or software-related named entities that appear within natural language sentences. In this paper, we introduce a new named entity recognition (NER) corpus for the computer programming domain, consisting of 15,372 sentences annotated with 20 fine-grained entity types. We trained in-domain BERT representations (BERTOverflow) on 152 million sentences from StackOverflow, which lead to an absolute increase of +10 F-1 score over off-the-shelf BERT. We also present the SoftNER model which achieves an overall 79.10 F$_1$ score for code and named entity recognition on StackOverflow data. Our SoftNER model incorporates a context-independent code token classifier with corpus-level features to improve the BERT-based tagging model. Our code and data are available at: this https URL

Named Entity Recognition

Code (set theory)

Source

Cite

Citations (2)

Fluent Response Generation for Conversational Question Answering

arXiv (Cornell University) (2020)

Ashutosh Baheti Alan Ritter Kevin Small

Question answering (QA) is an important aspect of open-domain conversational agents, garnering specific research focus in the conversational QA (ConvQA) subtask. One notable limitation of recent ConvQA efforts is the response being answer span extraction from the target corpus, thus ignoring the natural language generation (NLG) aspect of high-quality conversational agents. In this work, we propose a method for situating QA responses within a SEQ2SEQ NLG approach to generate fluent grammatical answer responses while maintaining correctness. From a technical perspective, we use data augmentation to generate training data for an end-to-end system. Specifically, we develop Syntactic Transformations (STs) to produce question-specific candidate answer responses and rank them using a BERT-based classifier (Devlin et al., 2019). Human evaluation on SQuAD 2.0 data (Rajpurkar et al., 2018) demonstrate that the proposed model outperforms baseline CoQA and QuAC models in generating conversational responses. We further show our model's scalability by conducting tests on the CoQA dataset. The code and data are available at this https URL.

Natural Language Generation

Source

Cite

Citations (4)

Measuring Forecasting Skill from Text

Shi Zong Alan Ritter Eduard Hovy

People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier.

10.18653/v1/2020.acl-main.473

Cite

Citations (21)

Visual Exploration of Neural Document Embedding in Information Retrieval: Semantics and Feature Selection

IEEE Transactions on Visualization and Computer Graphics (2019)

Xiaonan Ji Han‐Wei Shen Alan Ritter Raghu Machiraju Po-Yin Yen

Neural embeddings are widely used in language modeling and feature generation with superior computational power. Particularly, neural document embedding - converting texts of variable-length to semantic vector representations - has shown to benefit widespread downstream applications, e.g., information retrieval (IR). However, the black-box nature makes it difficult to understand how the semantics are encoded and employed. We propose visual exploration of neural document embedding to gain insights into the underlying embedding space, and promote the utilization in prevalent IR applications. In this study, we take an IR application-driven view, which is further motivated by biomedical IR in healthcare decision-making, and collaborate with domain experts to design and develop a visual analytics system. This system visualizes neural document embeddings as a configurable document map and enables guidance and reasoning; facilitates to explore the neural embedding space and identify salient neural dimensions (semantic features) per task and domain interest; and supports advisable feature selection (semantic analysis) along with instant visual feedback to promote IR performance. We demonstrate the usefulness and effectiveness of this system and present inspiring findings in use cases. This work will help designers/developers of downstream applications gain insights and confidence in neural document embedding, and exploit that to achieve more favorable performance in application domains.

Feature vector

10.1109/tvcg.2019.2903946

Cite

Citations (41)

Machine Reading at the University of Washington

North American Chapter of the Association for Computational Linguistics (2010)

Hoifung Poon Janara Christensen Pedro Domingos Oren Etzioni Raphael Hoffmann

Machine reading is a long-standing goal of AI and NLP. In recent years, tremendous progress has been made in developing machine learning approaches for many of its subtasks such as parsing, information extraction, and question answering. However, existing end-to-end solutions typically require substantial amount of human efforts (e.g., labeled data and/or manual engineering), and are not well poised for Web-scale knowledge acquisition. In this paper, we propose a unifying approach for machine reading by bootstrapping from the easiest extractable knowledge and conquering the long tail via a self-supervised learning process. This self-supervision is powered by joint inference based on Markov logic, and is made scalable by leveraging hierarchical structures and coarse-to-fine inference. Researchers at the University of Washington have taken the first steps in this direction. Our existing work explores the wide spectrum of this vision and shows its promise.

Bootstrapping (finance)

Source

Cite

Citations (38)

Fluent Response Generation for Conversational Question Answering

Ashutosh Baheti Alan Ritter Kevin Small

Natural Language Generation

10.18653/v1/2020.acl-main.19

Cite

Citations (14)