Academic performance prediction aims to leverage student-related information to predict their future academic outcomes, which is beneficial to numerous educational applications, such as personalized teaching and academic early warning. In this article, we reveal the students' behavior trajectories by mining campus smartcard records, and capture the characteristics inherent in trajectories for academic performance prediction. Particularly, we carefully design a tri-branch convolutional neural network (CNN) architecture, which is equipped with rowwise, columnwise, and depthwise convolutions and attention operations, to effectively capture the persistence, regularity, and temporal distribution of student behavior in an end-to-end manner, respectively. However, different from existing works mainly targeting at improving the prediction performance for the whole students, we propose to cast academic performance prediction as a top-k ranking problem, and introduce a top-k focused loss to ensure the accuracy of identifying academically at-risk students. Extensive experiments were carried out on a large-scale real-world dataset, and we show that our approach substantially outperforms recently proposed methods for academic performance prediction. For the sake of reproducibility, our codes have been released at https://github.com/ZongJ1111/Academic-Performance-Prediction.