In the standard diffusive picture for transport of cosmic rays (CRs), a gradient in the CR density induces a typically small, dipolar anisotropy in their arrival directions. This is being widely advertised as a tool for finding nearby sources. However, the predicted dipole amplitude at TeV and PeV energies exceeds the measured one by almost 2 orders of magnitude. Here, we critically examine the validity of this prediction, which is based on averaging over an ensemble of turbulent magnetic fields. We focus on (1) the deviations of the dipole in a particular random realization from the ensemble average, and (2) the possibility of a misalignment between the regular magnetic field and the CR gradient. We find that if the field direction and the gradient direction are close to ∼90°, the dipole amplitude is considerably suppressed and can be reconciled with observations, which sheds light on a long-standing problem. Furthermore, we show that the dipole direction in general does not coincide with the gradient direction, thus hampering the search for nearby sources.