Retrieval of the optical phase information from measurement of intensity is of a high interest because this would facilitate simple and cost-efficient techniques and devices. In scientific and industrial applications that exploit multi-mode fibers, a prior knowledge of spatial mode structure of the fiber, in principle, makes it possible to recover phases using measured intensity distribution. However, current mode decomposition algorithms based on the analysis of the intensity distribution at the output of a few-mode fiber, such as optimization methods or neural networks, still have high computational costs and high latency that is a serious impediment for applications, such as telecommunications. Speed of signal processing is one of the key challenges in this approach. We present a high-performance mode decomposition algorithm with a processing time of tens of microseconds. The proposed mathematical algorithm that does not use any machine learning techniques, is several orders of magnitude faster than the state-of-the-art deep-learning-based methods. We anticipate that our results can stimulate further research on algorithms beyond popular machine learning methods and they can lead to the development of low-cost phase retrieval receivers for various applications of few-mode fibers ranging from imaging to telecommunications.