In this paper we extend the theory of contact mechanics and rubber friction developed by one of us (B.N.J. Persson, J. Chem. Phys. 115, 3840 (2001)) to the case of surfaces with anisotropic surface roughness. As an application we calculate the viscoelastic contribution to the rubber friction. We show that the friction coefficient may depend significantly on the sliding direction, while the area of contact depends weakly on the sliding direction. We have carried out experiments for rubber blocks sliding on unidirectionally polished steel surfaces. The experimental data are in a good qualitative agreement with the theory.