A linear stability analysis of the hydrodynamic equations with respect to the homogeneous cooling state is carried out to identify the conditions for stability of a granular gas of rough hard spheres. The description is based on the results for the transport coefficients derived from the Boltzmann equation for inelastic rough hard spheres [Phys. Rev. E 90, 022205 (2014)PLEEE81539-375510.1103/PhysRevE.90.022205], which take into account the complete nonlinear dependence of the transport coefficients and the cooling rate on the coefficients of normal and tangential restitution. As expected, linear stability analysis shows that a doubly degenerate transversal (shear) mode and a longitudinal ("heat") mode are unstable with respect to long enough wavelength excitations. The instability is driven by the shear mode above a certain inelasticity threshold; at larger inelasticity, however, the instability is driven by the heat mode for an inelasticity-dependent range of medium roughness. Comparison with the case of a granular gas of inelastic smooth spheres confirms previous simulation results about the dual role played by surface friction: while small and large levels of roughness make the system less unstable than the frictionless system, the opposite happens at medium roughness. On the other hand, such an intermediate window of roughness values shrinks as inelasticity increases and eventually disappears at a certain value, beyond which the rough-sphere gas is always less unstable than the smooth-sphere gas. A comparison with some preliminary simulation results shows a very good agreement for conditions of practical interest.