A key difficulty to understanding friction is that many physical mechanisms contribute simultaneously. Here we investigate third-body frictional dynamics in a model experimental system that eliminates first-body interaction, wear, and fracture, and concentrates on the elastic interaction between sliding blocks and third bodies. We simultaneously visualize the particle motion and measure the global shear force. By systematically increasing the number of foreign particles, we find that the frictional dissipation depends only on the size ratio between surface asperities and the loose particles, irrespective of the particle's size or the surface's roughness. When the particles are comparable in size to the surface features, friction increases linearly with the number of particles. For particles smaller than the surface features, friction grows sublinearly with the number of particles. Our findings suggest that matching the size of surface features to the size of potential contaminants may be a good strategy for reliable lubrication.