Rényi & Tsallis TE Estimation

Rényi & Tsallis TE Estimation#

Let us suppose two RVs \(X\) and \(Y\) represent time series processes as:

\[\begin{split} \begin{aligned} X_{t_n}^{(k)} &= (X_{t_n}, X_{t_n-1}, \ldots, X_{t_n-k+1}),\\ Y_{t_n}^{(l)} &= (Y_{t_n}, Y_{t_n-1}, \ldots, Y_{t_n-l+1}), \end{aligned} \end{split}\]

The Transfer Entropy (TE) from \(X \to Y\) can be expressed using the formulation of entropy and joint entropy as follows [CT12, Khi57]:

\[ T_{X \rightarrow Y} = I(Y_{t_{n+1}} ; X_{t_n}^{(k)} \,|\, Y_{t_n}^{(l)}) = H(Y_{t_{n+1}}, Y_{t_n}^{(l)}) + H(X_{t_n}^{(k)}, Y_{t_n}^{(l)}) - H(Y_{t_{n+1}}, X_{t_n}^{(k)}, Y_{t_n}^{(l)}) - H(Y_{t_n}^{(l)}) \]

where

\(H(Y_{t_{n+1}}, Y_{t_n}^{(l)})\) is the joint entropy of \(Y_{t_{n+1}}\) and its history \(Y_{t_n}^{(l)}\).
\(H(X_{t_n}^{(k)}, Y_{t_n}^{(l)})\) is the joint entropy of the histories \(X_{t_n}^{(k)}\) and \(Y_{t_n}^{(l)}\).
\(H(Y_{t_{n+1}}, X_{t_n}^{(k)}, Y_{t_n}^{(l)})\) is the joint entropy of \(Y_{t_{n+1}}\), \(X_{t_n}^{(k)}\), and \(Y_{t_n}^{(l)}\).
\(H(Y_{t_n}^{(l)})\) is the entropy of the history \(Y_{t_n}^{(l)}\).

Rényi TE estimate is computed by plugging-in the entropies and the join entropy estimates by using the estimation method explained in the Rényi Entropy Estimation.

Tsallis TE estimate is computed by plugging-in the entropies and the join entropy estimates by using the estimation method explained in the Tsallis Entropy Estimation.

import infomeasure as im
import numpy as np
rng = np.random.default_rng(5673267189)

data_x = rng.normal(size=1000)
data_y = np.roll(data_x, 1)
data_control = rng.normal(size=1000)

(im.transfer_entropy(
    data_x,  # source
    data_y,  # target
    approach="renyi", alpha=1.2,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
),
 im.transfer_entropy(
    data_x,  # source
    data_control,  # target
    approach="renyi", alpha=1.2,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
),
 im.transfer_entropy(
    data_x,  # source
    data_y,  # target
    approach="tsallis", q=1.2,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
),
 im.transfer_entropy(
    data_x,  # source
    data_control,  # target
    approach="tsallis", q=1.2,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
))

(np.float64(2.6534426628661683),
 np.float64(0.01183131882659505),
 np.float64(1.7558438763821946),
 np.float64(0.22372587831787505))

For further methods, create an instance of the estimator.

est = im.estimator(
    data_x,  # source
    data_y,  # target
    measure='te',  # or 'transfer_entropy'
    approach="renyi", alpha=1.2,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
)
est.effective_val()

np.float64(2.6728621335253004)

Like this, Effective Transfer Entropy (eTE) can be accessed. Local Transfer Entropy is not supported for the Renyi and Tsallis TE.

Hypothesis testing can be conducted, with either a permutation test or bootstrapping.

stat_test = est.statistical_test(n_tests=50, method="permutation_test")
stat_test.p_value, stat_test.t_score, stat_test.confidence_interval(90), stat_test.percentile(50)

(np.float64(0.0),
 np.float64(71.88667937881381),
 array([-0.08639,  0.02764]),
 np.float64(-0.039828913392277454))

Data of higher dimension can easily be digested.

data_x = rng.normal(size=(1000, 5))  # 5d data
data_y = rng.normal(size=(1000, 3))  # 3d data
im.te(data_x, data_y, approach="tsallis", q=0.9)

np.float64(-9.488085514677545)

The estimator is implemented in the RenyiTEEstimator class, which is part of the im.measures.mutual_information module.

class infomeasure.estimators.transfer_entropy.renyi.RenyiTEEstimator(source, dest, *, cond=None, k: int = 4, alpha: float | int = None, noise_level=1e-08, prop_time: int = 0, step_size: int = 1, src_hist_len: int = 1, dest_hist_len: int = 1, cond_hist_len: int = 1, offset: int = None, base: int | float | str = 'e', **kwargs)[source]

Bases: BaseRenyiTEEstimator, TransferEntropyEstimator

Estimator for the Renyi transfer entropy.

Attributes:

source, destarray_like: The source (X) and dest (Y) data used to estimate the transfer entropy.
kint: The number of nearest neighbors used in the estimation.
alphafloat | int: The Rényi parameter, order or exponent. Sometimes denoted as \(\alpha\) or \(q\).
noise_levelfloat: The standard deviation of the Gaussian noise to add to the data to avoid issues with zero distances.
prop_timeint, optional: Number of positions to shift the data arrays relative to each other (multiple of step_size). Delay/lag/shift between the variables, representing propagation time. Assumed time taken by info to transfer from source to destination. Alternatively called offset.
step_sizeint, optional: Step size between elements for the state space reconstruction.
src_hist_len, dest_hist_lenint, optional: Number of past observations to consider for the source and destination data.

Raises:

ValueError: If the Renyi parameter is not a positive number.
ValueError: If the number of nearest neighbors is not a positive integer.
ValueError: If the step_size is not a non-negative integer.

Notes

The Rényi entropy is a generalization of Shannon entropy, where the small values of probabilities are emphasized for \(\alpha < 1\), and higher probabilities are emphasized for \(\alpha > 1\). For \(\alpha = 1\), it reduces to Shannon entropy. The Rényi-Entropy class can be particularly interesting for systems where additivity (in Shannon sense) is not always preserved, especially in nonlinear complex systems, such as when dealing with long-range forces.