This study proposes and analyzes a novel methodology that can effectively detect multi-mode combustion instability (CI) in a gas turbine combustor. The experiment is conducted in a model gas turbine combustor, and dynamic pressure (DP) and flame images are examined during the transition from stable to unstable flame, which is driven by changing fuel compositions. As a powerful technique for early detection of CI in multi-mode as well as in single mode, a new filter bank (FB) method based on spectral analysis of DP is proposed. Sequential processing using a triangular filter with Mel-scaling and a Hamming window is applied to increase the accuracy of the FB method, and the instability criterion is determined by calculating the magnitude of FB components. The performance of the FB method is compared with that of two conventional methods that are based on the root-mean-squared DP and temporal kurtosis. From the results, the FB method shows comparable performance in detection speed, sensitivity, and accuracy with other parameters. In addition, the FB components enable the analysis of various frequencies and multi-mode frequencies. Therefore, the FB method can be considered as an additional prognosis tool to determine the multi-mode CI in a monitoring system for gas turbine combustors.