We develop a simple model, in the spirit of the Asakura-Oosawa theory, able to describe the effects of surface roughness on the depletion potential. The resulting explicit expressions are easily computed, without free parameters, for a wide range of physically interesting conditions. Comparison with recent numerical simulations [M. Kamp et al., Langmuir, 2016, 32, 1233] shows an encouraging agreement and allows predicting the onset of colloidal aggregation in dilute suspensions of rough particles. Furthermore, the model proves to be suitable to investigate the role of the geometry of the roughness.