Pancreatic cancer is the emperor of all cancer maladies, mainly because there are no characteristic symptoms in the early stages, resulting in the absence of effective screening and early diagnosis methods in clinical practice. Non-contrast computerized tomography (CT) is widely used in routine check-ups and clinical examinations. Therefore, based on the accessibility of non-contrast CT, an automated early diagnosismethod for pancreatic cancer is proposed. Among this, we develop a novel causalitydriven graph neural network to solve the challenges of stability and generalization of early diagnosis, that is, the proposed method achieves stable performance for datasets from different hospitals, which highlights its clinical significance. Specifically, a multiple-instance-learning framework is designed to extract fine-grained pancreatic tumor features. Afterwards, to ensure the integrity and stability of the tumor features, we construct an adaptivemetric graph neural network that effectively encodes prior relationships of spatial proximity and feature similarity for multiple instances, and hence adaptively fuses the tumor features. Besides, a causal contrastivemechanism is developed to decouple the causality-driven and non-causal components of the discriminative features, suppress the non-causal ones, and hence improve the model stability and generalization. Extensive experiments demonstrated that the proposed method achieved the promising early diagnosis performance, and its stability and generalizability were independently verified on amulti-center dataset. Thus, the proposed method provides a valuable clinical tool for the early diagnosis of pancreatic cancer. Our source codes will be released at https://github.com/SJTUBME-QianLab/ CGNN-PC-Early-Diagnosis.