Analysis of Information-Based Nonparametric Variable Selection Criteria

Małgorzata Łazęcka; Jan Mielniczuk

doi:10.3390/e22090974

Analysis of Information-Based Nonparametric Variable Selection Criteria

Entropy (Basel). 2020 Aug 31;22(9):974. doi: 10.3390/e22090974.

Authors

Małgorzata Łazęcka^{1

2}, Jan Mielniczuk^{1

2}

Affiliations

¹ Institute of Computer Science, Polish Academy of Sciences, Jana Kazimierza 5, 01-248 Warsaw, Poland.
² Faculty of Mathematics and Information Science, Warsaw University of Technology, Koszykowa 75, 00-662 Warsaw, Poland.

Abstract

We consider a nonparametric Generative Tree Model and discuss a problem of selecting active predictors for the response in such scenario. We investigated two popular information-based selection criteria: Conditional Infomax Feature Extraction (CIFE) and Joint Mutual information (JMI), which are both derived as approximations of Conditional Mutual Information (CMI) criterion. We show that both criteria CIFE and JMI may exhibit different behavior from CMI, resulting in different orders in which predictors are chosen in variable selection process. Explicit formulae for CMI and its two approximations in the generative tree model are obtained. As a byproduct, we establish expressions for an entropy of a multivariate gaussian mixture and its mutual information with mixing distribution.

Keywords: CIFE; CMI; JMI; Markov blanket; conditional infomax feature extraction; conditional mutual information; gaussian mixture; generative tree model; information measures; joint mutual information criterion; nonparametric variable selection criteria.