This article studies whether heart sound signals can be used for emotion recognition. First, we built a small emotion heart sound database, and simultaneously recorded the participants' ECG for comparative analysis. Second, according to the characteristics of the heart sound signals, two emotion evaluation indicators were proposed: HRV of heart sounds (difference between successive heartbeats) and DSV of heart sounds (the ratio of diastolic to systolic duration variability). Then, we extracted linear and nonlinear features from two emotion evaluation indicators to recognize four kinds of emotions. Moreover, we used valence dimension, arousal dimension and valence-arousal synthesis as evaluation standards. The experimental results demonstrated that heart sound signals can be used for emotion recognition. It was more effective to achieve recognition results by combining the features of HRV and DSV of heart sounds. Finally, the average accuracy of four emotion recognitions on valence dimension, arousal dimension and valence-arousal synthesis was up to 96.875%, 88.5417% and 81.25%, respectively.