Establishing reliable correspondences between two views is one of the most important components of various vision tasks. This paper proposes a novel sparse-to-local-dense (S2LD) matching method to conduct fully differentiable correspondence estimation with the prior from epipolar geometry. The sparse-to-local-dense matching asymmetrically establishes correspondences with consistent sub-pixel coordinates while reducing the computation of matching. The salient features are explicitly located, and the description is conditioned on both views with the global receptive field provided by the attention mechanism. The correspondences are progressively established in multiple levels to reduce the underlying re-projection error. We further propose a 3D noise-aware regularizer with differentiable triangulation. Additional guidance from 3D space is encoded by the regularizer in training to handle the supervision noise caused by the errors in camera poses and depth maps. The proposed method demonstrates outstanding matching accuracy and geometric estimation capability on multiple datasets and tasks.