The mechanism responsible for liquid crystal (LC) alignment on mechanically buffed or UV exposed polymer films is poorly understood. A comprehensive study of LC alignment on variously prepared substrates unequivocally shows that the anisotropy in the surface roughness of the substrate completely determines the direction of LC alignment. In all the cases studied, including those where an anchoring transition occurs with temperature, the LC director (re)aligns in the directions of low roughness.