Any physical system evolves at a finite speed that is constrained not only by the energetic cost but also by the topological structure of the underlying dynamics. In this Letter, by considering such structural information, we derive a unified topological speed limit for the evolution of physical states using an optimal transport approach. We prove that the minimum time required for changing states is lower bounded by the discrete Wasserstein distance, which encodes the topological information of the system, and the time-averaged velocity. The bound obtained is tight and applicable to a wide range of dynamics, from deterministic to stochastic, and classical to quantum systems. In addition, the bound provides insight into the design principles of the optimal process that attains the maximum speed. We demonstrate the application of our results to chemical reaction networks and interacting many-body quantum systems.