Simultaneous high-probability bounds on the false discovery proportion in structured, regression, and online settings.

Many false discovery rate (FDR) controlling procedures construct a nested sequence, or path, of candidate rejection sets as well as an estimate of the false discovery proportion (FDP) for each set, and then use the FDP estimate to choose a single rejection set that controls the FDP in expectation. Instead, using novel martingale techniques, we use these FDP estimates to obtain closed form, high probability FDP upper bounds that hold simultaneously over all sets on the path. If desired, these bounds can be interpolated off the path to hold simultaneously for all subsets of hypotheses. We derive such simultaneous FDP bounds for the Benjamini-Hochberg procedure, ordered multiple testing (JASA'17), the knockoffs procedure for high-dimensional variable selection (AoS'15), arbitrary online multiple testing procedures (JRSS'08, AoS'19), and recent interactive methods (JRSS'18). The scientist may freely choose any rejection set in a post-hoc fashion after observing all bounds, and retain the promised error guarantee on the user-chosen set, as envisioned by the Goeman and Solari (2011) framework for exploratory multiple testing. In a variety of settings, our paper demonstrates strong ties between the parallel literatures of simultaneous FDP bounds and FDR control methods. Our closed-form expressions make the price of simultaneity explicit, unlike prior techniques using closed testing. We demonstrate their practical utility through a knockoffs analysis of the UK Biobank dataset.