Rényi Entropy Estimation

Rényi Entropy Estimation#

The Rényi entropy is a generalization of the Shannon Entropy, incorporating a parameter \(\alpha\) that controls the sensitivity of the entropy to the probabilities of different states of the random variable. For \(\alpha < 1\), smaller probabilities \((p_i)\) are emphasized, highlighting rare events. For \(\alpha > 1\), higher probabilities are emphasized, focusing on common events. When \(\alpha = 1\), Rényi entropy reduces to Shannon entropy [Jiz04, Ren76].

Rényi Entropy

The Rényi entropy of order \(\alpha\) for a probability distribution with density function \(f\) is given by:

\[ H^*_{\alpha} = \frac{1}{1-\alpha} \log \int_{\mathbb{R}^m} f^{\alpha}(x) \, dx, \quad \alpha \neq 1, \]

As \(\alpha \to 1\), Rényi entropy converges to Shannon entropy:

\[ H_1 = - \int_{\mathbb{R}^m} f(x) \log f(x) \, dx. \]

Leonenko et al. [LPS06] introduce a class of estimators for Rényi entropy by extending the K-L entropy estimation technique, which is based on the \(k^{th}\)-Nearest Neighbour (kNN) search approach [LPS06, LPS08].

Let us suppose \(X\) has \(N\) data points. First, for each point \(X_i\), compute the distances \(\rho(X_i, X_j)\) to all other points \(X_j\) (where \(j \neq i\)) and record \(\rho_{k,N-1}^{(i)}\), the distance from \(X_i\) to its \(K^{th}\)-Nearest Neighbour.

Rényi entropy \(\hat{H}_{N,k,q}^*\) is estimated as follows:

For \(\alpha \neq 1\):

\[ \hat{H}_{N,k,\alpha}^* = \frac{\log \hat{I}_{N,k,\alpha}}{1 - \alpha}, \]

where:

\[ \hat{I}_{N,k,\alpha} = \frac{1}{N} \sum_{i=1}^N \left(\zeta_{N,i,k}\right)^{1-\alpha}, \]
\[ \zeta_{N,i,k} = (N-1) \, C_k \, V_m \, \left(\rho_{k,N-1}^{(i)}\right)^m, \]

  • \(V_m = \frac{\pi^{m/2}}{\Gamma(m/2 + 1)}\) is the volume of the unit ball in \(\mathbb{R}^m\),

  • \(C_k = \left[\frac{\Gamma(k)}{\Gamma(k+1-\alpha)}\right]^{1/(1-\alpha)}\),

  • \(\rho_{k,N-1}^{(i)}\) is the distance from the point \(X_i\) to its \(k^{th}\) nearest neighbor.

For \(\alpha = 1\):

\[ \hat{H}_{N,k,1} = \frac{1}{N} \sum_{i=1}^N \log \xi_{N,i,k}, \]

where:

\[ \xi_{N,i,k} = (N-1) \exp[-\Psi(k)] \, V_m \, \left(\rho_{k,N-1}^{(i)}\right)^m, \]

  • \(\Psi(z) = \frac{\Gamma'(z)}{\Gamma(z)}\) is the digamma function.

  • For \(k \geq 1\):

\[ \Psi(k) = -\gamma + A_{k-1}, \]

where:

  • \(\gamma\) is the Euler-Mascheroni constant,

  • \(A_j = \sum_{j=1}^j \frac{1}{j}\) is the sum of the harmonic series up to \(j\).

For demonstration, we generate a dataset of normally distributed values with mean \(0\) and standard deviation \(1\). We then calculate the entropy using the box kernel with a bandwidth of \(0.5\). The analytical expected values can be calculated with

\[ H(X) = \frac{1}{2} \log(2\pi e \sigma^2), \]

where \(\sigma^2\) is the variance of the data.

import infomeasure as im
import numpy as np
rng = np.random.default_rng(692475)

std = 1.0
data = rng.normal(loc=0, scale=std, size=2000)

h = im.entropy(data, approach="renyi", alpha=1)
h_expected = (1 / 2) * np.log(2 * np.pi * np.e * std ** 2)
h, h_expected

(np.float64(1.4230943269045966), np.float64(1.4189385332046727))

When \(\alpha \neq 1\), we can reweight the data points.

(im.entropy(data, approach="renyi", alpha=0.8),
 im.entropy(data, approach="renyi", alpha=1.2))

(np.float64(1.4737116494050049), np.float64(1.3816100667645845))

For a 2D point cloud, it is as easy to calculate the entropy:

im.entropy(
    rng.normal(loc=0, scale=1, size=(2000, 2)),
    approach="renyi", alpha=2
)

np.float64(2.5847180849949165)

Local values are not supported.

The estimator is implemented in the RenyiEntropyEstimator class, which is part of the im.measures.entropy module.

class infomeasure.estimators.entropy.renyi.RenyiEntropyEstimator(*data, k: int = 4, alpha: float | int = None, base: int | float | str = 'e')[source]

Bases: EntropyEstimator

Estimator for the Rényi entropy.

Attributes:
*dataarray_like

The data used to estimate the entropy.

kint

The number of nearest neighbors used in the estimation.

alphafloat | int

The Rényi parameter, order or exponent. Sometimes denoted as \(\alpha\) or \(q\).

Raises:
ValueError

If the Renyi parameter is not a positive number.

ValueError

If the number of nearest neighbors is not a positive integer.

Notes

The Rényi entropy is a generalization of Shannon entropy, where the small values of probabilities are emphasized for \(\alpha < 1\), and higher probabilities are emphasized for \(\alpha > 1\). For \(\alpha = 1\), it reduces to Shannon entropy. The Rényi-Entropy class can be particularly interesting for systems where additivity (in Shannon sense) is not always preserved, especially in nonlinear complex systems, such as when dealing with long-range forces.