Higgs-boson-pair production H (→b b ̄)H (→γ γ ) from gluon fusion at the HL-LHC and HL-100 TeV hadron collider

We perform the most up-to-date comprehensive signal-background analysis for Higgs-pair production in $HH\ensuremath{\rightarrow}b\overline{b}\ensuremath{\gamma}\ensuremath{\gamma}$ channel at the HL-LHC and HL-100 TeV hadron collider, with the goal of probing the self-coupling ${\ensuremath{\lambda}}_{3H}$ of the Higgs boson which is normalized to its standard-model value of 1. We simulate all the standard-model signal and background processes and emphasize that the $ggH(\ensuremath{\rightarrow}\ensuremath{\gamma}\ensuremath{\gamma})$ background has been overlooked in previous studies. We find that, even for the most promising channel $HH\ensuremath{\rightarrow}b\overline{b}\ensuremath{\gamma}\ensuremath{\gamma}$ at the HL-LHC with a luminosity of $3000\text{ }\text{ }{\mathrm{fb}}^{\ensuremath{-}1}$, the significance is still not high enough to establish the Higgs self-coupling at the standard-model (SM) value. Instead, we can only constrain the self-coupling to $\ensuremath{-}1.0l{\ensuremath{\lambda}}_{3H}l7.6$ at 95% confidence level after considering the uncertainties associated with the top-Yukawa coupling and the estimation of backgrounds. Here we also extend the study to the HL-100 TeV hadron collider. With a luminosity of $3\text{ }\text{ }{\mathrm{ab}}^{\ensuremath{-}1}$, we find there exists a bulk region of $2.6\ensuremath{\lesssim}{\ensuremath{\lambda}}_{3H}\ensuremath{\lesssim}4.8$ in which one cannot pin down the trilinear coupling. Otherwise, one can measure the coupling with a high precision. At the SM value, for example, we show that the coupling can be measured with about 20% accuracy. While assuming $30\text{ }\text{ }{\mathrm{ab}}^{\ensuremath{-}1}$, the bulk region reduces to $3.1\ensuremath{\lesssim}{\ensuremath{\lambda}}_{3H}\ensuremath{\lesssim}4.3$, and the trilinear coupling can be measured with about 7% accuracy at the SM value.