Deep supervised learning with mixture of neural networks

Abstract

Deep Neural Network (DNN), as a deep architectures, has shown excellent performance in classification tasks. However, when the data has different distributions or contains some latent non-observed factors, it is difficult for DNN to train a single model to perform well on the classification tasks. In this paper, we propose mixture model based on DNNs (MoNNs), a supervised approach to perform classification tasks with a gating network and multiple local expert models. We use a neural network as a gating function and use DNNs as local expert models. The gating network split the heterogeneous data into several homogeneous components. DNNs are combined to perform classification tasks in each component. Moreover, we use EM (Expectation Maximization) as an optimization algorithm. Experiments proved that our MoNNs outperformed the other compared methods on determination of diabetes, determination of benign or malignant breast cancer, and handwriting recognition. Therefore, the MoNNs can solve the problem of data heterogeneity and have a good effect on classification tasks.

Keywords: Deep neural network; Diabetes determination; Expectation maximization; Mixture model.