Keck spectroscopy and NGVS photometry in the direction of the Virgo cluster: Globular cluster satellites of dwarf ellipticals, Milky Way halo substructure, and large-scale structure in the background