Jointly leveraging spatial transcriptomics and deep learning models for pathology image annotation improves cell type identification over either approach alone.

The disorganization of cell types within tissues underlies many human diseases and has been studied for over a century using the conventional tools of pathology, including tissue-marking dyes such as the H&E stain. Recently, spatial transcriptomics technologies were developed that can measure spatially resolved gene expression directly in pathology-stained tissues sections, revealing cell types and their dysfunction in unprecedented detail. In parallel, artificial intelligence (AI) has approached pathologist-level performance in computationally annotating H&E images of tissue sections. However, spatial transcriptomics technologies are limited in their ability to separate transcriptionally similar cell types and AI-based pathology has performed less impressively outside their training datasets. Here, we describe a methodology that can computationally integrate AI-annotated pathology images with spatial transcriptomics data to markedly improve inferences of tissue cell type composition made over either class of data alone. We show that this methodology can identify regions of clinically relevant tumor immune cell infiltration, which is predictive of response to immunotherapy and was missed by an initial pathologist9s manual annotation. Thus, combining spatial transcriptomics and AI-based image annotation has the potential to exceed pathologist-level performance in clinical diagnostic applications and to improve the many applications of spatial transcriptomics that rely on accurate cell type annotations.