BayesEntropyEstimator

class infomeasure.estimators.entropy.BayesEntropyEstimator(*data, alpha: float | str, K: int = None, base: int | float | str = 'e')

Bases: DiscreteHEstimator

Bayesian entropy estimator.

Computes an estimate of Shannon entropy using Bayesian probability estimates with a Dirichlet prior characterized by concentration parameter α. This approach provides a principled way to handle sparse data and incorporate prior knowledge about the probability distribution.

The Bayesian probabilities are calculated as:

\[p_k^{\text{Bayes}} = \frac{n_k + \alpha}{N + K \alpha}\]

where \(n_k\) is the count of symbol \(k\), \(N\) is the total number of observations, \(K\) is the support size (number of unique symbols), and \(\alpha\) is the concentration parameter of the Dirichlet prior.

The entropy is then \(-\sum p_k^{\text{Bayes}} \log p_k^{\text{Bayes}}\), same as the maximum likelihood entropy estimator, also supporting local entropy values.

Concentration Parameter Choices

The concentration parameter α controls the strength of the prior belief in uniform distribution. Several well-established choices are available:

Jeffreys Prior (α = 0.5 = "jeffrey")

Non-informative prior that is invariant under reparameterization. Provides good performance for most applications [KT81].

Laplace Prior (α = 1.0 = "laplace")

Uniform prior that adds one pseudocount to each symbol [BP63]. Simple and widely used, equivalent to add-one smoothing.

Schürmann-Grassberger Prior (α = 1/K = "sch-grass")

Adaptive prior that scales with the alphabet size. Particularly effective for large alphabets.

Minimax Prior (α = √N/K = "min-max")

Minimises the maximum expected loss. Balances between sample size and alphabet size.

Attributes:

*dataarray_like

The data used to estimate the entropy.

alphafloat

The concentration parameter α of the Dirichlet prior.

Kint, optional

The support size. If not provided, uses the observed support size.

Attributes Summary

bayes_probs
dist_dict	Return the Bayesian distribution dictionary for JSD.

Attributes Documentation

bayes_probs

dist_dict

Return the Bayesian distribution dictionary for JSD.