kullback_leiber_divergence

infomeasure.kullback_leiber_divergence(data_p, data_q, approach: str = '', **kwargs)

Calculate the Kullback-Leibler Divergence between two distributions.

The Kullback-Leibler Divergence is a measure of the difference between two probability distributions. It is calculated as the expectation of the logarithm of the ratio of the probability of two events. To calculate, we use the identity of combining the joint and marginal entropies:

\[KL(P \| Q) = \sum_{x \in X} P(x) \log \left( \frac{P(x)}{Q(x)} \right) = H_Q(P) - H(P)\]

Parameters:

data_parray_like

The first data.

data_qarray_like

The second data.

approachstr

The name of the entropy estimator to use.

**kwargsdict

Additional keyword arguments to pass to the entropy estimator.

Returns:

float

The Kullback-Leibler Divergence.

Raises:

ValueError

If the approach is not supported or the entropy estimator is not compatible with the Kullback-Leibler Divergence.