Kernel TE Estimation

Kernel TE Estimation#

The Transfer Entropy (TE) from the source process \(X(x_n)\) to the target process \(Y(y_n)\) in terms of probabilities is written as:

\[ T_{x \rightarrow y}(k, l) = -\sum_{y_{n+1}, \mathbf{y}_n^{(l)}, \mathbf{x}_n^{(k)}} p(y_{n+1}, \mathbf{y}_n^{(l)}, \mathbf{x}_n^{(k)}) \log \left( \frac{p(y_{n+1} \mid \mathbf{y}_n^{(l)}, \mathbf{x}_n^{(k)})} {p(y_{n+1} \mid \mathbf{y}_n^{(l)})} \right) \]

where

\(y_{n+1}\) is the next state of \(Y\) at time \(n\),
\( \mathbf{y}_n^{(l)} = \{y_n, \dots, y_{n-l+1}\} \) is the embedding vector of \(Y\) considering the \( l \) previous states (history length),
\( \mathbf{x}_n^{(k)} = \{x_n, \dots, x_{n-k+1}\} \) embedding vector of \(X\) considering the \( k \) previous states (history length),
\(p(y_{n+1}, \mathbf{y}_n^{(l)}, \mathbf{x}_n^{(k)})\) is the joint probability of the next state of \(Y\), its history, and the history of \(X\),
\(p(y_{n+1} \mid \mathbf{y}_n^{(l)}, \mathbf{x}_n^{(k)})\) is the conditional probability of next state of \(Y\) given the histories of \(X\) and \(Y\),
\(p(y_{n+1} \mid \mathbf{y}_n^{(l)})\) is the conditional probability of next state of \(Y\) given only the history of \(Y\).

Kernel TE estimates the required probability density function (pdf) via kernel density estimation (KDE), which provides the probability values to be plugged into the above formula [KS02, MK02, Sch00]. KDE estimates density at a reference point by weighting all samples based on their distance from it, using a kernel function \((K)\) [Sil86]. For more detail on pdf-estimation and available kernel functions check the Kernel Entropy Estimation section.

Note

This TE estimaton techinque offers two different kernel functions: box kernel and gaussian kernel.

import infomeasure as im
import numpy as np
rng = np.random.default_rng(5673267189)

data_x = rng.normal(size=1000)
data_y = np.roll(data_x, 1)
data_control = rng.normal(size=1000)

(im.transfer_entropy(
    data_x,  # source
    data_y,  # target
    approach="kernel", kernel="box", bandwidth=0.7,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
),
 im.transfer_entropy(
    data_x,  # source
    data_control,  # target
    approach="kernel", kernel="box", bandwidth=0.7,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
))

(np.float64(1.7485852766974523), np.float64(0.14903367505918047))

For further methods, create an instance of the estimator.

est = im.estimator(
    data_x,  # source
    data_y,  # target
    measure='te',  # or 'transfer_entropy'
    approach="kernel", kernel="box", bandwidth=0.7,
    step_size = 1, prop_time = 0, src_hist_len = 1, dest_hist_len = 1,
)
est.local_vals()

array([0.93827, 1.48638, 1.45593, ..., 1.88162, 2.84781, 1.31219],
      shape=(999,))

The Effective Transfer Entropy (eTE) method can be accessed like so:

est.effective_val()

np.float64(1.6006462617843744)

Hypothesis testing can also be conducted, with either a permutation test or bootstrapping.

stat_test = est.statistical_test(n_tests=50, method="permutation_test")
stat_test.p_value, stat_test.t_score, stat_test.confidence_interval(90), stat_test.percentile(50)

(np.float64(0.0),
 np.float64(112.41872514403875),
 array([0.13893, 0.18344]),
 np.float64(0.1633511588849929))

Data of higher dimension can easily be digested.

data_x = rng.normal(size=(1000, 5))  # 5d data
data_y = rng.normal(size=(1000, 3))  # 3d data
im.te(data_x, data_y, approach="kernel", kernel="gaussian", bandwidth=0.5)

np.float64(2.508893427085009)

The estimator is implemented in the KernelTEEstimator class, which is part of the im.measures.mutual_information module.

class infomeasure.estimators.transfer_entropy.kernel.KernelTEEstimator(source, dest, *, cond=None, bandwidth: float | int = None, kernel: str = None, prop_time: int = 0, step_size: int = 1, src_hist_len: int = 1, dest_hist_len: int = 1, cond_hist_len: int = 1, workers: int = 1, offset: int = None, base: int | float | str = 'e', **kwargs)[source]

Bases: BaseKernelTEEstimator, TransferEntropyEstimator

Estimator for transfer entropy using Kernel Density Estimation (KDE).

Attributes:

source, destarray_like: The source (X) and destination (Y) data used to estimate the transfer entropy.
bandwidthfloat | int: The bandwidth for the kernel.
kernelstr: Type of kernel to use, compatible with the KDE implementation kde_probability_density_function().
prop_timeint, optional: Number of positions to shift the data arrays relative to each other (multiple of step_size). Delay/lag/shift between the variables, representing propagation time. Assumed time taken by info to transfer from source to destination. Alternatively called offset.
step_sizeint: Step size between elements for the state space reconstruction.
src_hist_len, dest_hist_lenint: Number of past observations to consider for the source and destination data.

Notes

A small bandwidth can lead to under-sampling, while a large bandwidth may over-smooth the data, obscuring details.