An echo state network (ESN) draws widespread attention and is applied in many scenarios. As the most typical approach for solving the ESN, the matrix inverse operation of high computational complexity is involved. However, in the modern big data era, addressing the heavy computational burden problem is necessary. In order to reduce the computational load, an inverse-free ESN (IFESN) is proposed for the first time in this article. Besides, an incremental IFESN is constructed to attain the network topology with theoretical proof on the training error's monotone decline property. Simulations and experiments are conducted on several numerical and real-world time-series benchmarks, and corresponding results indicate that the proposed model is superior to some existing models and possesses excellent practical application potential. The source code is publicly available at https://github.com/LongJin-lab/the-supplementary-file-for-CYB-E-2021-04-0944.