Early detection of malignant thyroid nodules leading to patient-specific treatments can reduce morbidity and mortality rates. Currently, thyroid specialists use medical images to diagnose then follow the treatment protocols, which have limitations due to unreliable human false-positive diagnostic rates. With the emergence of deep learning, advances in computer-aided diagnosis techniques have yielded promising earlier detection and prediction accuracy; however, clinicians' adoption is far lacking. The present study adopts Xception neural network as the base structure and designs a practical framework, which comprises three adaptable multi-channel architectures that were positively evaluated using real-world data sets. The proposed architectures outperform existing statistical and machine learning techniques and reached a diagnostic accuracy rate of 0.989 with ultrasound images and 0.975 with computed tomography scans through the single input dual-channel architecture. Moreover, the patient-specific design was implemented for thyroid cancer detection and has obtained an accuracy of 0.95 for double inputs dual-channel architecture and 0.94 for four-channel architecture. Our evaluation suggests that ultrasound images and computed tomography (CT) scans yield comparable diagnostic results through computer-aided diagnosis applications. With ultrasound images obtained slightly higher results, CT, on the other hand, can achieve the patient-specific diagnostic design. Besides, with the proposed framework, clinicians can select the best fitting architecture when making decisions regarding a thyroid cancer diagnosis. The proposed framework also incorporates interpretable results as evidence, which potentially improves clinicians' trust and hence their adoption of the computer-aided diagnosis techniques proposed with increased efficiency and accuracy.