Prostate cancer (PCa) is one of the most prevalent cancers in men. Early diagnosis plays a pivotal role in reducing the mortality rate from clinically significant PCa (csPCa). In recent years, bi-parametric magnetic resonance imaging (bpMRI) has attracted great attention for the detection and diagnosis of csPCa. bpMRI is able to overcome some limitations of multi-parametric MRI (mpMRI) such as the use of contrast agents, the time-consuming for imaging and the costs, and achieve detection performance comparable to mpMRI. However, inter-reader agreements are currently low for prostate MRI. Advancements in artificial intelligence (AI) have propelled the development of deep learning (DL)-based computer-aided detection and diagnosis system (CAD). However, most of the existing DL models developed for csPCa identification are restricted by the scale of data and the scarcity in labels. In this paper, we propose a self-supervised pre-training scheme named SSPT-bpMRI with an image restoration pretext task integrating four different image transformations to improve the performance of DL algorithms. Specially, we explored the potential value of the self-supervised pre-training in fully supervised and weakly supervised situations. Experiments on the publicly available PI-CAI dataset demonstrate that our model outperforms the fully supervised or weakly supervised model alone.