In contrast to classical charge-based memories, the binary information in redox-based resistive switching devices is decoded by a change of the atomic configuration rather than changing the amount of stored electrons. This offers in principle a higher scaling potential as ions are not prone to tunneling and the information is not lost by tunneling. The switching speed, however, is potentially smaller since the ionic mass is much higher than the electron mass. In this work, the ultimate switching speed limit of redox-based resistive switching devices is discussed. Based on a theoretical analysis of the underlying physical processes, it is derived that the switching speed is limited by the phonon frequency. This limit is identical when considering the acceleration of the underlying processes by local Joule heating or by high electric fields. Electro-thermal simulations show that a small filamentary volume can be heated up in picoseconds. Likewise, the characteristic charging time of a nanocrossbar device can be even below ps. In principle, temperature and voltage can be brought fast enough to the device to reach the ultimate switching limit. In addition, the experimental route and the challenges towards reaching the ultimate switching speed limit are discussed. So far, the experimental switching speed is limited by the measurement setup.