Search for exotic decays of the Higgs boson to a pair of pseudoscalars in the μμbb and ττbb final states

Abstract

A search for exotic decays of the Higgs boson ($\text{H}$) with a mass of 125$\text{Ge}\text{V}$ to a pair of light pseudoscalars ${\text{a}}_1$ is performed in final states where one pseudoscalar decays to two $\text{b}$ quarks and the other to a pair of muons or $\tau$ leptons. A data sample of proton-proton collisions at $\sqrt{s}=13\text{Te}\text{V}$ corresponding to an integrated luminosity of 138${\text{fb}}^{-1}$ recorded with the CMS detector is analyzed. No statistically significant excess is observed over the standard model backgrounds. Upper limits are set at 95% confidence level ($\text{CL}$) on the Higgs boson branching fraction to $\mu \mu \text{b}\text{b}$ and to $\tau \tau \text{b}\text{b},$ via a pair of ${\text{a}}_1$s. The limits depend on the pseudoscalar mass $m_{{\text{a}}_1}$ and are observed to be in the range (0.17-3.3) $\times {10}^{-4}$ and (1.7-7.7) $\times {10}^{-2}$ in the $\mu \mu \text{b}\text{b}$ and $\tau \tau \text{b}\text{b}$ final states, respectively. In the framework of models with two Higgs doublets and a complex scalar singlet (2HDM+S), the results of the two final states are combined to determine upper limits on the branching fraction $B(\text{H}\to {\text{a}}_1{\text{a}}_1\to \ell \ell \text{b}\text{b})$ at 95% $\text{CL}$, with $\ell$ being a muon or a $\tau$ lepton. For different types of 2HDM+S, upper bounds on the branching fraction $B(\text{H}\to {\text{a}}_1{\text{a}}_1)$ are extracted from the combination of the two channels. In most of the Type II 2HDM+S parameter space, $B(\text{H}\to {\text{a}}_1{\text{a}}_1)$ values above 0.23 are excluded at 95% $\text{CL}$ for $m_{{\text{a}}_1}$ values between 15 and 60$\text{Ge}\text{V}$.