Optical vector network analyzers (OVNAs) based on swept-wavelength interferometry are applied widely in optical metrology and sensing to measure the complex transfer functions of optical components, devices, and fibers. Phase noise from laser sweep nonlinearities degrades the measurement quality as the distance increases and limits the usage of the OVNA in characterizing systems with long impulse responses as required in space-division multiplexing links with a high mode count or in the presence of large modal differential group delay (DGD). In this Letter, we use a densely distributed broadband ultra-weak fiber Bragg grating array to directly measure the distortion due to phase noise at a 5-m increment up to 400 m and use this measured data to directly eliminate the distortion. We experimentally extend the measurement range of the swept-wavelength OVNA over 400 m and successfully characterize a 2-km six-mode multimode fiber link with an accumulated impulse response as wide as 20 ns.