We demonstrate a robust single-photon Ramsey interferometer based on a single Rydberg excitation, where the photon is stored as a Rydberg polariton in an ensemble of atoms. This coherent conversion of the photon to Rydberg polariton enables to split an incoming photon into a superposition state of two Rydberg states by applying microwave fields, which constructs two paths of interferometer. Ramsey interference fringes are demonstrated when we scan either the detuning of the microwave or the free evolution time, from which we can obtain the resonant transition frequency of two Rydberg states. We use the Ramsey-like sequence to demonstrate coherent manipulation of the stored single-photon to construct different interference patterns. In addition, the robustness of the Ramsey interferometer to the fluctuation of incoming photon numbers and optical depth (OD) of the atomic ensemble is tested, showing that the coherent of Ramsey interferometer is preserved for input photon number in a range of Rin < 15 and for OD varying from 1.0 to 4.0. The robust interferometer will find its applications in quantum precision measurement.