Rényi & Tsallis MI Estimation

Rényi & Tsallis MI Estimation#

Mutual Information (MI) quantifies the information shared between two random variables \(X\) and \(Y\). For our purpose, let us write the expression of MI in between the two times series \(X_t\) and \(Y_t\) as:

\[ I(X_{t}; Y_t) = \sum_{x_{t}, y_t} p(x_{t}, y_t) \log \frac{p(x_{t}, y_t)}{p(x_{t}) p(y_t)} \]

where

\(p(x_t,y_t)\) is the joint probability distribution (probability density function, pdf),
\(p(x_t)\) and \(p(y_t)\) are the marginal probabilities (pdf) of \(X_t\) and \(Y_t\).

MI can be further expressed in terms of entropy and joint entropy as follows [CT12, Khi57]:

\[ I(X; Y) = H(X) + H(Y) - H(X, Y) \]

where

\(H(X)\) is the entropy of \(X\),
\(H(Y)\) is the entropy of \(Y\),
\(H(X, Y)\) is the joint entropy of \(X\) and \(Y\).

Rényi MI estimate is computed by plugging-in the entropies and the joint entropy estimates by using the estimation method explained in the Rényi Entropy Estimation.

Tsallis MI estimate is computed by plugging-in the entropies and the joint entropy estimates by using the estimation method explained in the Tsallis Entropy Estimation.

To demonstrate these approaches of MI, we generate a multivariate Gaussian distribution with two dimensions. The data is centred around the origin and has a correlation coefficient of \(\rho = 0.5\). For Gaussian random variables, we know the analytical MI is given by:

\[ I(X; Y) = -\frac{1}{2} \log(1 - \rho^2) \]

where \(\rho\) is the Pearson correlation coefficient between \(X\) and \(Y\). We then compare this analytical value with the estimated MI using infomeasure.

import infomeasure as im
import numpy as np
rng = np.random.default_rng(692475)

rho = 0.5
data = rng.multivariate_normal([0, 0], [[1, rho], [rho, 1]], size=2000)
x, y = data[:, 0], data[:, 1]

(im.mutual_information(x, y, approach="renyi", alpha=1.0),
 im.mutual_information(x, y, approach="tsallis", q=1.0),
 -0.5 * np.log(1 - rho**2))  # analytical value

(np.float64(0.15155747010586973),
 np.float64(0.15155789103368322),
 np.float64(0.14384103622589045))

Introducing the offset:

(im.mutual_information(x, y, approach="renyi", alpha=1.0, offset=1),
 im.mutual_information(x, y, approach="tsallis", q=1.0, offset=1))

(np.float64(0.009448024167262314), np.float64(0.009447890998474762))

The MI decreases greatly because the offset unmatched the pairs of the generated data.

For three or more variables, add them as positional parameters.

data = rng.multivariate_normal([0, 0, 0], [[1, rho, 0], [rho, 1, 0], [0, 0, 1]], size=1000)
data_x, data_y, data_z = data[:, 0], data[:, 1], data[:, 2]
im.mutual_information(data_x, data_y, data_z, approach="renyi", alpha=1.0)

np.float64(0.19847977253312532)

Local Mutual Information is not supported for the RMI and TMI, but Hypothesis testing is.

est = im.estimator(data_x, data_y, measure="mi", approach="renyi", alpha=1.0)
stat_test = est.statistical_test(n_tests=50, method="permutation_test")
stat_test.p_value, stat_test.t_score, stat_test.confidence_interval(90), stat_test.percentile(50)

(np.float64(0.0),
 np.float64(7.62783577626735),
 array([-0.03996,  0.02262]),
 np.float64(-0.005330034657894167))

The estimators are implemented in the RenyiMIEstimator and TsallisMIEstimator class, which is part of the im.measures.mutual_information module.

class infomeasure.estimators.mutual_information.renyi.RenyiMIEstimator(*data, cond=None, k: int = 4, alpha: float | int = None, noise_level=1e-08, offset: int = 0, normalize: bool = False, base: int | float | str = 'e', **kwargs)[source]

Bases: BaseRenyiMIEstimator, MutualInformationEstimator

Estimator for the Renyi mutual information.

Attributes:

*dataarray_like, shape (n_samples,): The data used to estimate the mutual information. You can pass an arbitrary number of data arrays as positional arguments.
kint: The number of nearest neighbors used in the estimation.
alphafloat | int: The Rényi parameter, order or exponent. Sometimes denoted as \(\alpha\) or \(q\).
noise_levelfloat: The standard deviation of the Gaussian noise to add to the data to avoid issues with zero distances.
offsetint, optional: Number of positions to shift the data arrays relative to each other. Delay/lag/shift between the variables. Default is no shift.
normalizebool, optional: If True, normalize the data before analysis.

Notes

The Rényi entropy is a generalization of Shannon entropy, where the small values of probabilities are emphasized for \(\alpha < 1\), and higher probabilities are emphasized for \(\alpha > 1\). For \(\alpha = 1\), it reduces to Shannon entropy. The Rényi-Entropy class can be particularly interesting for systems where additivity (in Shannon sense) is not always preserved, especially in nonlinear complex systems, such as when dealing with long-range forces.

class infomeasure.estimators.mutual_information.tsallis.TsallisMIEstimator(*data, cond=None, k: int = 4, q: float | int = None, noise_level: float = 1e-08, offset: int = 0, normalize: bool = False, base: int | float | str = 'e', **kwargs)[source]

Bases: BaseTsallisMIEstimator, MutualInformationEstimator

Estimator for the Tsallis mutual information.

Attributes:

*dataarray_like, shape (n_samples,): The data used to estimate the mutual information. You can pass an arbitrary number of data arrays as positional arguments.
kint: The number of nearest neighbors used in the estimation.
qfloat: The Tsallis parameter, order or exponent. Sometimes denoted as \(q\), analogous to the Rényi parameter \(\alpha\).
noise_levelfloat: The standard deviation of the Gaussian noise to add to the data to avoid issues with zero distances.
offsetint, optional: Number of positions to shift the data arrays relative to each other. Delay/lag/shift between the variables. Default is no shift.
normalizebool, optional: If True, normalize the data before analysis.

Notes

In the \(q \to 1\) limit, the Jackson sum (q-additivity) reduces to ordinary summation, and the Tallis entropy reduces to Shannon Entropy. This class of entropy measure is in particularly useful in the study in connection with long-range correlated systems and with non-equilibrium phenomena.