Purpose/Objective(s): With the development of deep learning, more convolutional neural networks (CNNs) are being introduced in automatic segmentation to reduce oncologists' labor requirement. However, it is still challenging for oncologists to spend considerable time evaluating the quality of the contours generated by the CNNs. Besides, all the evaluation criteria, such as Dice Similarity Coefficient (DSC), need a gold standard to assess the quality of the contours. To address these problems, we propose an automatic quality assurance (QA) method using isotropic and anisotropic methods to automatically analyze contour quality without a gold standard. Materials/Methods: We used 196 individuals with 18 different head-and-neck organs-at-risk. The overall process has the following 4 main steps. (1) Use CNN segmentation network to generate a series of contours, then use these contours as organ masks to erode and dilate to generate inner/outer shells for each 2D slice. (2) Thirty-eight radiomics features were extracted from these 2 shells, using the inner/outer shells' radiomics features ratios and DSCs as the input for 12 machine learning models. (3) Using the DSC threshold adaptively classified the passing/un-passing slices. (4) Through 2 different threshold analysis methods quantitatively evaluated the un-passing slices and obtained a series of location information of poor contours. Parts 1-3 were isotropic experiments, and part 4 was the anisotropic method. Result: From the isotropic experiments, almost all the predicted values were close to the labels. Through the anisotropic method, we obtained the contours' location information by assessing the thresholds of the peak-to-peak and area-to-area ratios. Conclusion: The proposed automatic segmentation QA method could predict the segmentation quality qualitatively. Moreover, the method can analyze the location information for un-passing slices.
Keywords: automatic segmentation; deep learning; head and neck; machine learning; quality assurance; radiotherapy.