Affective state recognition based on multiple modalities of physiological signals has been a hot research topic. Traditional methods require designing hand-crafted features based on domain knowledge, which is time-consuming and has not achieved a satisfactory performance. On the other hand, conducting classification on raw signals directly can also cause some problems, such as the interference of noise and the curse of dimensionality. To address these problems, we propose a novel approach that encodes different modalities of data as images and use convolutional neural networks (CNN) to perform the affective state recognition task. We validate our aproach on the DECAF dataset in comparison with two state-of-the-art methods, i.e., the Support Vector Machines (SVM) and Random Forest (RF). Experimental results show that our aproach outperforms the baselines by 5% to 9%.