A new method is developed to decompose a physiological signal into a summation of transient and oscillatory components, referred to as mixed over-complete dictionary based sparse component decomposition algorithm (MOSCA). Based on the characteristics of the transient evoked potential (EP) and the background noise, the mixed dictionary is constructed with an over-complete wavelet dictionary and an over-complete discrete cosine (DC) function dictionary, and the signal is separated by learning in this mixed dictionary with a matching pursuit (MP) algorithm. MOSCA is designed specifically for the separation of a desired transient EP from the existing spontaneous EEG or other background noise. The method was evaluated with several simulation tests in which EPs or simulated EPs were deeply masked in different strong noise backgrounds, and the recovered signal is similar to the original assumed EP with a high and stable correlation coefficient (CC). The method was then applied to estimate event related potential (ERP) in the classical oddball experiment, and the results confirmed that the trial number for a reliable ERP estimation might be greatly reduced by MOSCA.