We study the relaxation of coherent acoustic phonon modes with frequencies up to 500 GHz in ultrathin free-standing silicon membranes. Using an ultrafast pump-probe technique of asynchronous optical sampling, we observe that the decay time of the first-order dilatational mode decreases significantly from ~4.7 ns to 5 ps with decreasing membrane thickness from ~194 to 8 nm. The experimental results are compared with theories considering both intrinsic phonon-phonon interactions and extrinsic surface roughness scattering including a wavelength-dependent specularity. Our results provide insight to understand some of the limits of nanomechanical resonators and thermal transport in nanostructures.