Object discrimination plays an important role in infrared (IR) imaging systems. However, at long observing distance, the presence of detector noise and absence of robust features make exo-atmospheric object classification difficult to tackle. In this paper, a recurrence-plots-based convolutional neural network (RP-CNN) is proposed for feature learning and classification. First, it uses recurrence plots (RPs) to transform time sequences of IR radiation into two-dimensional texture images. Then, a CNN model is adopted for classification. Different from previous object classification methods, RP representation has well-defined visual texture patterns, and their graphical nature exposes hidden patterns and structural changes in time sequences of IR signatures. In addition, it can process IR signatures of objects without the limitation of fixed length. Training data are generated from IR irradiation models considering micro-motion dynamics and geometrical shape of exo-atmospheric objects. Results based on time-evolving IR radiation data indicate that our method achieves significant improvement in accuracy and robustness of the exo-atmospheric IR objects classification.