Phase is an inherent and important feature for coherent processes, which, unfortunately, has not been completely understood for surface plasmon polariton (SPP) and matter interactions. Here we propose a practical approach to extract the phase change dispersion during the interaction between free-space light, SPPs and nanogroove/slit based on far-field information only. Numerical simulation and experimental validation were both presented using nanoslit-groove plasmonic interferometers, agreeing well with theoretical near-field analysis. This approach is generally feasible to extract the intrinsic phase dispersion of other plasmonic nanostructures and can reveal more fundamental features of SPP-matter interactions.