Electrocardiography (ECG) signals can be considered as multivariable time series (TS). The state-of-the-art ECG data classification approaches, based on either feature engineering or deep learning techniques, treat separately spectral and time domains in machine learning systems. No spectral-time domain communication mechanism inside the classifier model can be found in current approaches, leading to difficulties in identifying complex ECG forms. In this article, we proposed a novel deep learning model named spectral cross-domain neural network (SCDNN) with a new block called soft-adaptive threshold spectral enhancement (SATSE), to simultaneously reveal the key information embedded in spectral and time domains inside the neural network. More precisely, the domain-cross information is captured by a general convolutional neural network (CNN) backbone, and different information sources are merged by a self-adaptive mechanism to mine the connection between time and spectral domains. In SATSE, the knowledge from time and spectral domains is extracted via the fast Fourier transformation (FFT) with soft trainable thresholds in modified sigmoid functions. The proposed SCDNN is tested with several classification tasks implemented on the public ECG databases PTB-XL and CPSC2018. SCDNN outperforms the state-of-the-art approaches with a low computational cost regarding a variety of metrics in all classification tasks on both databases, by finding appropriate domains from the infinite spectral mapping. The convergence of the trainable thresholds in the spectral domain is also numerically investigated in this article. The robust performance of SCDNN provides a new perspective to exploit knowledge across deep learning models from time and spectral domains. The code repository can be found: https://github.com/DL-WG/SCDNN-TS.