On the Inability of the Higher Criticism to Detect Rare/Weak Departures.

Consider a multiple hypothesis testing setting involving rare/weak features: only few features, out of possibly many, deviate from their null hypothesis behavior. Summarizing the significance of each feature by a P-value, we construct a global test against the null using the Higher Criticism (HC) statistics of these P-values. We calibrate the rare/weak model using parameters controlling the asymptotic behavior of these P-values in their near-zero "tail". We derive a region in the parameter space where the HC test is asymptotically powerless. Our derivation involves very different tools than previously used to show the powerlessness of HC, relying on properties of the empirical processes underlying HC. In particular, our result applies to situations where HC is not asymptotically optimal, or when the asymptotically detectable region of the parameter space is unknown.