Friction is one of the leading causes of energy loss in moving parts, and understanding how roughness affects friction is of utmost importance. From creating surfaces with high friction to prevent slip and movement, to creating surfaces with low friction to minimize energy loss, roughness plays a key role. By measuring shear stresses of crosslinked elastomers on three rough surfaces of similar surface chemistry across nearly six decades of sliding velocity, we demonstrate the dominant role of adhesive frictional dissipation. Furthermore, while it was previously known that roughness-induced oscillations affected the viscoelastic dissipation, we show that these oscillations also control the molecular detachment process and the resulting adhesive dissipation. This contrasts with typical models of friction, where only the amount of contact area and the strength of interfacial bonding govern the adhesive dissipation. Finally, we show that all the data can be collapsed onto a universal curve when the shear stress is scaled by the square root of elastic modulus and the velocity is scaled by a critical velocity at which the system exhibits macroscopic buckling instabilities. Taken together, these results suggest a design principle broadly applicable to frictional systems ranging from tires to soft robotics.