We determine the stability and instability of a sufficiently small and periodic travelling wave to long-wavelength perturbations, for a nonlinear dispersive equation which extends a Camassa-Holm equation to include all the dispersion of water waves and the Whitham equation to include nonlinearities of medium-amplitude waves. In the absence of the effects of surface tension, the result qualitatively agrees with the Benjamin-Feir instability of a Stokes wave. In the presence of the effects of surface tension, it qualitatively agrees with those from formal asymptotic expansions of the physical problem and improves upon that for the Whitham equation, predicting the critical wave number at the strong surface tension limit. We discuss the modulational stability and instability in the Camassa-Holm equation and other related models.
Keywords: Camassa–Holm; full dispersion; modulational instability; surface tension.