Sugato Basu

Google (United States)

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Trends

Author Order

Document Type

Co-Authors

Pradyumna Narayana

Google (United States)

William Yang Wang

University of British Columbia

Raymond J. Mooney

The University of Texas at Austin

Kazoo Sone

Google (United States)

Arindam Banerjee

Indian Institute of Technology Kharagpur

Ian Davidson

University of Southampton

Varun Jampani

Google (United States)

Arjun Akula

IT University of Copenhagen

Tsu-Jui Fu

University of California, Santa Barbara

Wanrong Zhu

University of California, Santa Barbara

Cooperative Institutions

Google (United States)

The University of Texas at Austin

Carnegie Mellon University

University of California, Santa Cruz

Stanford University

University of California, Berkeley

Amazon (United States)

University of California, Davis

Chinese Academy of Sciences

University of California, Santa Barbara

Author Statistics

Papers

Citation

H-Index

i-10 index

Research Field

Semi-Supervised Learning

Encyclopedia of Database Systems (2009)

Sugato Basu

10.1007/978-0-387-39940-9_609

Cite

Citations (206)

iLink

Jeffrey Davitz J. Yu Sugato Basu David Gutelius Alexandra Harris

The growth of Web 2.0 and fundamental theoretical breakthroughs have led to an avalanche of interest in social networks. This paper focuses on the problem of modeling how social networks accomplish tasks through peer production style collaboration. We propose a general interaction model for the underlying social networks and then a specific model (iLink for social search and message routing. A key contribution here is the development of a general learning framework for making such online peer production systems work at scale. The iLink model has been used to develop a system for FAQ generation in a social network (FAQtory), and experience with its application in the context of a full-scale learning-driven workflow application (CALO) is reported. We also discuss methods of adapting iLink technology for use in military knowledge sharing portals and other message routing systems. Finally, the paper shows the connection of iLink to SQM, a theoretical model for social search that is a generalization of Markov Decision Processes and the popular Pagerank model.

PageRank

Social network (sociolinguistics)

10.1145/1281192.1281292

Cite

Citations (54)

CPL: Counterfactual Prompt Learning for Vision and Language Models

arXiv (Cornell University) (2022)

Xuehai He Diji Yang Weixi Feng Tsu-Jui Fu Arjun Akula

Prompt tuning is a new few-shot transfer learning technique that only tunes the learnable prompt for pre-trained vision and language models such as CLIP. However, existing prompt tuning methods tend to learn spurious or entangled representations, which leads to poor generalization to unseen concepts. Towards non-spurious and efficient prompt learning from limited examples, this paper presents a novel \underline{\textbf{C}}ounterfactual \underline{\textbf{P}}rompt \underline{\textbf{L}}earning (CPL) method for vision and language models, which simultaneously employs counterfactual generation and contrastive learning in a joint optimization framework. Particularly, CPL constructs counterfactual by identifying minimal non-spurious feature change between semantically-similar positive and negative samples that causes concept change, and learns more generalizable prompt representation from both factual and counterfactual examples via contrastive learning. Extensive experiments demonstrate that CPL can obtain superior few-shot performance on different vision and language tasks than previous prompt tuning methods on CLIP. On image classification, we achieve 3.55\% average relative improvement on unseen classes across seven datasets; on image-text retrieval and visual question answering, we gain up to 4.09\% and 25.08\% relative improvements across three few-shot scenarios on unseen test sets respectively.

Spurious relationship

Feature (linguistics)

Representation

10.48550/arxiv.2210.10362

Cite

Citations (1)

Semi-supervised Clustering: Learning with Limited User Feedback

Sugato Basu

Source

Cite

Citations (21)

Discriminative Diffusion Models as Few-shot Vision and Language Learners

arXiv (Cornell University) (2023)

Xuehai He Weixi Feng Tsu-Jui Fu Varun Jampani Arjun Akula

Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation. Since text-to-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in text prompts, can we leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such as image-text matching? To answer this question, we propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach mainly uses the cross-attention score of a Stable Diffusion model to capture the mutual influence between visual and textual information and fine-tune the model via efficient attention-based prompt learning to perform image-text matching. By comparing DSD with state-of-the-art methods on several benchmark datasets, we demonstrate the potential of using pre-trained diffusion models for discriminative tasks with superior results on few-shot image-text matching.

Discriminative model

Leverage (statistics)

Benchmark (surveying)

10.48550/arxiv.2305.10722

Cite

Citations (1)

MetaCLUE: Towards Comprehensive Visual Metaphors Research

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)

Arjun Akula Brendan Driscoll Pradyumna Narayana Soravit Changpinyo Zhiwei Jia

Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce Meta-CLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor classification, localization, understanding (retrieval, question answering, captioning) and generation (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities. Project page: https://metaclue.github.io

Closed captioning

Abstraction

Literal (mathematical logic)

Strengths and weaknesses

10.1109/cvpr52729.2023.02222

Cite

Citations (13)

Semi-Supervised Learning

Encyclopedia of Database Systems (2016)

Sugato Basu

10.1007/978-1-4899-7993-3_609-2

Cite

Citations (5)

Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

Wanrong Zhu Xin Wang Pradyumna Narayana Kazoo Sone Sugato Basu

A major challenge in visually grounded language generation is to build robust benchmark datasets and models that can generalize well in real-world settings. To do this, it is critical to ensure that our evaluation protocols are correct, and benchmarks are reliable. In this work, we set forth to design a set of experiments to understand an important but often ignored problem in visually grounded language generation: given that humans have different utilities and visual attention, how will the sample variance in multi-reference datasets affect the models' performance? Empirically, we study several multi-reference datasets and corresponding vision-and-language tasks. We show that it is of paramount importance to report variance in experiments; that human-generated references could vary drastically in different datasets/tasks, revealing the nature of each task; that metric-wise, CIDEr has shown systematically larger variances than others. Our evaluations on reference-per-instance shed light on the design of reliable datasets in the future.

Benchmark (surveying)

Sample (material)

10.18653/v1/2020.emnlp-main.708

Cite

Citations (7)

KAFA: Rethinking Image Ad Understanding with Knowledge-Augmented Feature Adaptation of Vision-Language Models

arXiv (Cornell University) (2023)

Zhiwei Jia Pradyumna Narayana Arjun Akula Garima Pruthi Hao Su

Image ad understanding is a crucial task with wide real-world applications. Although highly challenging with the involvement of diverse atypical scenes, real-world entities, and reasoning over scene-texts, how to interpret image ads is relatively under-explored, especially in the era of foundational vision-language models (VLMs) featuring impressive generalizability and adaptability. In this paper, we perform the first empirical study of image ad understanding through the lens of pre-trained VLMs. We benchmark and reveal practical challenges in adapting these VLMs to image ad understanding. We propose a simple feature adaptation strategy to effectively fuse multimodal information for image ads and further empower it with knowledge of real-world entities. We hope our study draws more attention to image ad understanding which is broadly relevant to the advertising industry.

Feature (linguistics)

Benchmark (surveying)

Fuse (electrical)

Adaptability

Closed captioning

10.48550/arxiv.2305.18373

Cite

Citations (0)

A Framework for Deep Constrained Clustering

arXiv (Cornell University) (2021)

Hongjing Zhang Tianyang Zhan Sugato Basu Ian Davidson

The area of constrained clustering has been extensively explored by researchers and used by practitioners. Constrained clustering formulations exist for popular algorithms such as k-means, mixture models, and spectral clustering but have several limitations. A fundamental strength of deep learning is its flexibility, and here we explore a deep learning framework for constrained clustering and in particular explore how it can extend the field of constrained clustering. We show that our framework can not only handle standard together/apart constraints (without the well documented negative effects reported earlier) generated from labeled side information but more complex constraints generated from new types of side information such as continuous values and high-level domain knowledge. Furthermore, we propose an efficient training paradigm that is generally applicable to these four types of constraints. We validate the effectiveness of our approach by empirical results on both image and text datasets. We also study the robustness of our framework when learning with noisy constraints and show how different components of our framework contribute to the final performance. Our source code is available at $\href{https://github.com/blueocean92/deep_constrained_clustering}{\text{URL}}$.

Robustness

Constrained clustering

Code (set theory)

Spectral Clustering

10.48550/arxiv.2101.02792

Cite

Citations (0)