Vibrational wave-packet dynamics on the ground electronic state of the neutral silver pentamer (Ag5) are studied by femtosecond (fs) pump-probe spectroscopy using the 'negative ion - to neutral - to positive ion' (NeNePo) excitation scheme. A vibrational wave packet is prepared on the 2A1 state of Ag5via photodetachment of mass-selected, cryogenically cooled Ag5- anions using a fs pump pulse. The temporal evolution of the vibrational wave packet is then probed by an ultrafast probe pulse via resonant multiphoton ionization to Ag5+. Frequency analysis of the fs NeNePo transients for pump-probe delay times from 0.2 to 8 ps reveals three primary beating frequencies at 157 cm-1, 101 cm-1 and 56 cm-1 as well as four weaker features. A comparison of these experimentally obtained beating frequencies to harmonic normal mode frequencies calculated from electronic structure calculations confirms that Ag5 in the gas phase adopts a planar trapezoidal geometry, similar to that previously observed in solid argon. The dependence of the ionization yield on the laser polarization indicates a s-d wave electron photodetachment from a 'p-type' occupied molecular orbital of Ag5. Franck-Condon analysis shows that both processes, photodetachment and subsequent photoionization determine the beating frequencies probed in the time-dependent cation yield. The present study extends the applicability of fs NeNePo spectroscopy to characterize the vibrational spectra in the far-IR frequency range in the absence of perturbations from a medium or a messenger atom to mass-selected neutral metal clusters with more than three atoms in the ground electronic states.