Two Measures of Dependence

Entropy (Basel). 2019 Aug 8;21(8):778. doi: 10.3390/e21080778.

Abstract

Two families of dependence measures between random variables are introduced. They are based on the Rényi divergence of order α and the relative α -entropy, respectively, and both dependence measures reduce to Shannon's mutual information when their order α is one. The first measure shares many properties with the mutual information, including the data-processing inequality, and can be related to the optimal error exponents in composite hypothesis testing. The second measure does not satisfy the data-processing inequality, but appears naturally in the context of distributed task encoding.

Keywords: Rényi divergence; Rényi entropy; data processing; dependence measure; relative α-entropy.