Asbestos is a toxic ore widely used in construction and commercial products. Asbestos tends to dissolve into fibers and after years inhaling them, these fibers calcify and form plaques on the pleura. Despite being benign, pleural plaques may indicate an immunologic deficiency or dysfunctional lung areas. We propose a pipeline for asbestos-related pleural plaque detection in CT images of the human thorax based on the following operations: lung segmentation, 3D patch selection along the pleura, a convolutional neural network (CNN) for feature extraction, and classification by support vector machines (SVM). Due to the scarcity of publicly available and annotated datasets of pleural plaques, the proposed CNN relies on architecture learning with random weights obtained by a PCA-based approach instead of using traditional filter learning by backpropagation. Experiments show that the proposed CNN can outperform its counterparts based on backpropagation for small training sets.