Conditional H

Conditional entropy is defined as the remaining uncertainty of a variable after considering the information from another variable. In other words, it is the remaining unique information of the first variable after having the knowledge of the conditional variable.

Let \(X\) and \(Y\) be two RVs with the marginal probability as \(p(x)\) and \(p(y)\), and conditional probability distribution of \(X\) conditioned on \(Y\) denoted as \(p(x|y)\) (or \(p(y|x)\) vice versa), then the conditional entropy is calculated by as:

\[ H(X \mid Y) = - \sum_{x,y} p(x, y) \log p(x \mid y) \]

One can use the chain rule and express the above expression in terms of Joint Entropy \(H(X,Y)\) and marginal entropy (eg: \(H(X)\) and \(H(Y)\)) as follows:

\[ H(X \mid Y) = H(X,Y) - H(Y) \]

Joint Entropy

The joint entropy represents the amount of information gained by jointly observing two RVs.

\[ H(X, Y) = - \sum_{x,y} p(x, y) \log p(x, y) \]

This package does not offer methods to compute conditional entropy, as both simple Entropy and Joint Entropy are offered, and can be combined. For further information measures, e.g., mutual information and transfer entropy, this package offers dedicated, probabilistic implementations which minimize bias compared to entropy combinations.

Local Conditional H

Similar to shannon Local Entropy \(h(x)\), one can also define local or point-wise conditional entropy \(h(x \mid y)\) as follows:

\[ h(x \mid y) = - \log p(x \mid y) \]

This local conditional entropy also satisfies the chain rule as its average counterparts, hence one can express the local conditional entropy as:

\[ h(x \mid y) = h(x,y) - h(y) \]

Joint entropy can be accessed from the usual entropy an estimator interface. To signal that the random variables should be considered jointly, the random variables should be passed as a tuple .

import infomeasure as im

x = [0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0]
y = [1, 1, 0, 0, 2, 2, 1, 1, 0, 2, 0, 0, 2, 0, 0]
h_x = im.entropy(x, approach="discrete")        # x marginal
h_y = im.entropy(y, approach="discrete")        # y marginal
h_xy = im.entropy((x, y), approach="discrete")  # (x, y) joint
h_x, h_y, h_xy

(np.float64(0.6909233093138181),
 np.float64(1.0606018056124555),
 np.float64(1.7489707507713135))

The number of RVs is arbitrary:

z = [2, 1, 1, 3, 2, 1, 3, 2, 2, 3, 2, 1, 3, 2, 3]
im.entropy((x, y, z), approach="discrete")  # (x, y, z) joint

np.float64(2.2110688711446103)

The local values need to, again, be accessed via an estimator class instance.

est_xy = im.estimator((x, y), measure="h", approach="discrete")  # (x, y) joint estimator
est_xy.result(), est_xy.local_vals()

(np.float64(1.7489707507713135),
 array([2.01490302, 2.01490302, 1.60943791, 1.60943791, 2.01490302,
        2.01490302, 2.01490302, 2.01490302, 1.32175584, 2.01490302,
        1.32175584, 1.60943791, 2.01490302, 1.32175584, 1.32175584]))

Joint entropy works for all approaches in im.entropy().