Objective: To establish a deep learning algorithm that can accurately determine three-dimensional facial anatomical landmarks, multi-view stacked hourglass convolutional neural networks (MSH-CNN) and to construct three-dimensional facial midsagittal plane automatically based on MSH-CNN and weighted Procrustes analysis algorithm.
Methods: One hundred subjects with no obvious facial deformity were collected in our oral clinic. Three-dimensional facial data were scanned by three-dimensional facial scanner. Experts annotated twenty-one facial landmarks and midsagittal plane of each data. Eighty three-dimensional facial data were used as training set, to train the MSH-CNN in this study. The overview of MSH-CNN network architecture contained multi-view rendering and training the MSH-CNN network. The three-dimensional facial data were rendered from ninety-six views that were fed to MSH-CNN and the output was one heatmap per landmark. The result of the twenty-one landmarks was accurately placed on the three-dimensional facial data after a three-dimensional view ray voting process. The remaining twenty three-dimensional facial data were used as test set. The trained MSH-CNN automatically determined twenty-one three-dimensional facial anatomical landmarks of each case of data, and calculated the distance between each MSH-CNN landmark and the expert landmark, which was defined as position error. The midsagittal plane of the twenty subjects' could be automatically constructed, using the MSH-CNN and Procrustes analysis algorithm. To evaluate the effect of midsagittal plane by automatic method, the angle between the midsagittal plane constructed by the automatic method and the expert annotated plane was calculated, which was defined as angle error.
Results: For twenty subjects with no obvious facial deformity, the average angle error of the midsagittal plane constructed by MSH-CNN and weighted Procrustes analysis algorithm was 0.73°±0.50°, in which the average position error of the twenty-one facial landmarks automatically determined by MSH-CNN was (1.13±0.24) mm, the maximum position error of the orbital area was (1.31±0.54) mm, and the minimum position error of the nasal area was (0.79±0.36) mm.
Conclusion: This research combines deep learning algorithms and Procrustes analysis algorithms to realize the fully automated construction of the three-dimensional midsagittal plane, which initially achieves the construction effect of clinical experts. The obtained results constituted the basis for the independent intellectual property software development.
目的: 旨在建立一种可准确确定三维颜面解剖标志点的深度学习算法——多视图堆叠沙漏神经网络(multi-view stacked hourglass convolutional neural networks，MSH-CNN)，并结合赋权普氏分析算法实现三维颜面正中矢状平面的自动构建。
方法: 收集面部无明显畸形的受试者100例，获取三维颜面数据，由专家进行颜面标志点(21个)和正中矢状平面的标注。以上述其中80例受试者三维颜面数据作为训练集数据，训练并建立本研究的MSH-CNN算法模型。以其余20例作为测试集数据，由训练后的深度学习算法自动确定每例数据的三维颜面解剖标志点(21个)，并评价算法标点与专家标点间“定点误差”。将MSH-CNN自动确定的三维颜面解剖标志点应用于本课题组前期研究建立的赋权普氏分析算法，可自动构建出20例受试者的三维颜面正中矢状平面。计算MSH-CNN结合赋权普氏分析算法构建的正中矢状平面与专家正中矢状平面间“角度误差”，评价三维颜面正中矢状平面自动构建方法的效果。
结果: 针对20例面部无明显畸形的受试者，基于MSH-CNN和赋权普氏分析算法构建正中矢状平面与专家平面间的角度误差平均为0.73°±0.50°，其中MSH-CNN自动确定颜面21个解剖标志点的定点误差平均为(1.13±0.24) mm，眶区定点误差最大平均为(1.31±0.54) mm，鼻区定点误差最小平均为(0.79±0.36) mm。
结论: 将深度学习算法与赋权普氏分析算法结合应用，实现了三维颜面正中矢状平面的全自动构建，初步达到了临床专家的构建效果，为自主知识产权的软件开发奠定了基础。
Keywords: Deep learning; Midsagittal plane; Procrustes analysis.