Information Retriever (IR) aims to find the relevant documents (e.g. snippets, passages, and articles) to a given query at large scale. IR plays an important role in many tasks such as open domain question answering and dialogue systems, where external knowledge is needed. In the past, searching algorithms based on term matching have been widely used. Recently, neural-based algorithms (termed as neural retrievers) have gained more attention which can mitigate the limitations of traditional methods. Regardless of the success achieved by neural retrievers, they still face many challenges, e.g. suffering from a small amount of training data and failing to answer simple entity-centric questions. Furthermore, most of the existing neural retrievers are developed for pure-text query. This prevents them from handling multi-modality queries (i.e. the query is composed of textual description and images). This proposal has two goals. First, we introduce methods to address the abovementioned issues of neural retrievers from three angles, new model architectures, IR-oriented pretraining tasks, and generating large scale training data. Second, we identify the future research direction and propose potential corresponding solution.
Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models’ performance. To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and images and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experimental results show that a good retriever can significantly improve the reader’s performance on the OK-VQA challenge. The code and corpus are provided in https://github.com/luomancs/retriever_reader_for_okvqa.git.
In this chapter, we will learn about the modeling and learning techniques that drive multimodal applications. We will focus specifically on the recent advances in transformer-based modeling for natural language understanding, and image understanding, and how these approaches connect for jointly understanding combinations of language and image.
In today's rapidly evolving digital landscape, the wealth of available information has expanded beyond the boundaries of traditional text-based content. With the proliferation of multimedia platforms and data sources, we are constantly bombarded with a rich variety of images, videos, audio, and text. This vast array of heterogeneous data poses new challenges and opportunities for the field of Information Retrieval (IR). To address these challenges and harness the potential of multimodal information, researchers and practitioners have turned their attention toward the development of Multimodal Information Retrieval (MMIR) systems. We will begin by introducing the basic concept of IR systems which will lay the foundation for understanding the mechanism of IR. In this section, we will cover the concepts of query and target, indexing, and scoring functions. Then, we describe the state-of-the-art retrieval models for unimodal and multimodal IR systems. The unimodal retrieval is the foundation of multimodal IR including the text and the image IR. In the section on Multimodal IR, we will differentiate it with the cross-modal IR, and focus on multimodal-query IR. We will discuss two representative multimodal-query IR in detail. After this, we will discuss applications application of multimodal IR in crucial downstream tasks. Later, we will discuss the evaluation metrics spanning from traditional evaluation to advanced semantic-based measurement. Finally, we will discuss the broader impact of MMIR.
Purpose: Chronic pain is a growing problem among children and adolescents, and is more prevalent in low-income families. This observational study was conducted to describe the demographics and vari...
GQA~\citep{hudson2019gqa} is a dataset for real-world visual reasoning and compositional question answering. We found that many answers predicted by the best vision-language models on the GQA dataset do not match the ground-truth answer but still are semantically meaningful and correct in the given context. In fact, this is the case with most existing visual question answering (VQA) datasets where they assume only one ground-truth answer for each question. We propose Alternative Answer Sets (AAS) of ground-truth answers to address this limitation, which is created automatically using off-the-shelf NLP tools. We introduce a semantic metric based on AAS and modify top VQA solvers to support multiple plausible answers for a question. We implement this approach on the GQA dataset and show the performance improvements. Code and data are available in this link \url{https://github.com/luomancs/alternative_answer_set.git}.
This cross-sectional, descriptive study examined unmet social and economic needs and health information requests of low-income, expecting fathers who participated in the First 1000 Days program. The First 1000 Days is a systems-level intervention aiming to prevent obesity among low-income mothers and infants across 3 community health centers in Greater Boston, MA, USA. Fathers who attended their partner’s first prenatal care visit were invited to complete a program survey during early pregnancy. Among 131 fathers surveyed, 45% were white, 21% were Hispanic/Latino, 55% were foreign-born, and 69% reported an annual income under $50 000. Fathers reported elevated levels of food insecurity (18%) and 33% were unaware of someone that could provide a $50 loan; however, over 85% of fathers knew someone that could provide non-financial social support. Fathers requested information about pregnancy, birth preparation, and fatherhood. Findings support addressing fathers’ unmet needs during pregnancy and providing father-specific perinatal information.
Importance Adoption of primary care interventions to reduce childhood obesity is limited. Progress in reducing obesity prevalence and eliminating disparities can be achieved by implementing effective childhood obesity management interventions in primary care settings. Objective To examine the extent to which implementation strategies supported the uptake of research evidence and implementation of the Connect for Health pediatric weight management program. Design, Setting, and Participants This quality improvement study took place at 3 geographically and demographically diverse health care organizations with substantially high numbers of children living in low-income communities in Denver, Colorado; Boston, Massachusetts; and Greenville, South Carolina, from November 2019 to April 2022. Participants included pediatric primary care clinicians and staff and families with children aged 2 to 12 years with a body mass index (BMI) in the 85th percentile or higher. Exposures Pediatric weight management program with clinician-facing tools (ie, clinical decision support tools) and family-facing tools (ie, educational handouts, text messaging program, community resource guide) along with implementation strategies (ie, training and feedback, technical assistance, virtual learning community, aligning with hospital performance metrics) to support the uptake. Main Outcomes and Measures Primary outcomes were constructs from the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework examined through parent, clinician, and leadership surveys and electronic health record data to understand the number of children screened and identified, use of the clinical decision support tools, program acceptability, fidelity to the intervention and implementation strategies, and program sustainability. Results The program screened and identified 18 333 children across 3 organizations (Denver Health, 8480 children [46.3%]; mean [SD] age, 7.97 [3.31] years; 3863 [45.5%] female; Massachusetts General Hospital (MGH), 6190 children [33.8%]; mean [SD] age, 7.49 [3.19] years; 2920 [47.2%] female; Prisma Health, 3663 children [20.0%]; mean [SD] age, 7.33 [3.15] years; 1692 [46.2%] female) as having an elevated BMI. The actionable flagging system was used for 8718 children (48%). The reach was equitable, with 7843 children (92.4%) from Denver Health, 4071 children (65.8%) from MGH, and 1720 children (47%) from Prisma Health being from racially and ethnically minoritized groups. The sites had high fidelity to the program and 6 implementation strategies, with 4 strategies (67%) used consistently at Denver Health, 6 (100%) at MGH, and 5 (83%) at Prisma Health. A high program acceptability was found across the 3 health care organizations; for example, the mean (SD) Acceptability of Intervention Measure score was 3.72 (0.84) at Denver Health, 3.82 (0.86) at MGH, and 4.28 (0.68) at Prisma Health. The implementation strategies were associated with 7091 (39%) uses of the clinical decision support tool. The mean (SD) program sustainability scores were 4.46 (1.61) at Denver Health, 5.63 (1.28) at MGH, and 5.54 (0.92) at Prisma Health. Conclusions and Relevance These findings suggest that by understanding what strategies enable the adoption of scalable and implementation-ready programs by other health care organizations, it is feasible to improve the screening, identification, and management of children with overweight or obesity and mitigate existing disparities.
'/%3e%3cuse xlink:href='%23a' transform='rotate(23.036 .093 25.536)'/%3e%3cuse xlink:href='%23a' transform='rotate(45.87 1.273 16.18)'/%3e%3cuse xlink:href='%23a' transform='rotate(69.945 .996 12.078)'/%3e%3cuse xlink:href='%23a' transform='rotate(20.66 -19.689 31.932)'/%3e%3c/svg%3e)