Strong and narrow spectral feedback plays a key role in self-injection locking (SIL) single-frequency lasers, especially in the stabilization of single longitudinal mode (SLM) operation of a multi-mode laser. Here, we report on a narrow linewidth SIL fiber laser that adopts a fiber add-drop configuration composing of two tapered fibers and a high-Q MgF2 crystalline whispering-gallery-mode resonator. The feedback from the drop port could be controlled and optimized for stable SLM lasing of multi-mode fiber lasers. A stable single-frequency fiber laser with white frequency noise as low as ∼0.4 Hz2/Hz, corresponding to an instantaneous linewidth of ∼1.26 Hz, is demonstrated. Compact, controllable, and all-fiber configuration in this work to achieve an ultra-narrow linewidth laser will attract interest in many applications.