In this paper a derivation of the attenuation factor in a waveguide with stochastic walls is presented. The perturbation method and Fourier analysis are employed to derive asymptotically consistent boundary-value problems at each asymptotic order. The derived approximation predicts the attenuation of the propagating mode in a rough waveguide through a correction to the eigenvalue corresponding to smooth walls. The proposed approach can be used to derive results that are consistent with those obtained by Bass et al. [IEEE Trans. Antennas Propag. 22, 278-288 (1974)]. The novelty of the method is that it does not involve the integral Dyson-type equation and, as a result, the large number of statistical moments included in the equation in the form of the mass operator of the volume scattering theory. The derived eigenvalue correction is described by the correlation function of the randomly rough surface. The averaged solution in the plane wave regime is approximated by the exponential function dependent on the derived eigenvalue correction. The approximations are compared with numerical results obtained using the finite element method (FEM). An approach to retrieve the correct deviation in roughness height and correlation length from multiple numerical realizations of the stochastic surface is proposed to account for the oversampling of the rough surface occurring in the FEM meshing procedure.