Bibliography#
Kishor Acharya, Felipe Olivares, and Massimiliano Zanin. How representative are air transport functional complex networks? A quantitative validation. Chaos: An Interdisciplinary Journal of Nonlinear Science, 34(4):043133, April 2024. doi:10.1063/5.0189642.
Christoph Bandt and Bernd Pompe. Permutation entropy: a natural complexity measure for time series. Phys. Rev. Lett., 88:174102, Apr 2002. URL: https://link.aps.org/doi/10.1103/PhysRevLett.88.174102, doi:10.1103/PhysRevLett.88.174102.
David Barber and Felix Agakov. The IM Algorithm: A variational approach to Information Maximization. In Proceedings of the 17th International Conference on Neural Information Processing Systems, NIPS'03, 201–208. Cambridge, MA, USA, December 2003. MIT Press.
Mr. Bayes and Mr. Price. An Essay towards Solving a Problem in the Doctrine of Chances. By the Late Rev. Mr. Bayes, F. R. S. Communicated by Mr. Price, in a Letter to John Canton, A. M. F. R. S. Philosophical Transactions (1683-1775), 53:370–418, 1763. arXiv:105741.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual Information Neural Estimation. In Proceedings of the 35th International Conference on Machine Learning, 531–540. PMLR, July 2018.
Juan A. Bonachela, Haye Hinrichsen, and Miguel A. Munoz. Entropy estimates of small data sets. Journal of Physics A: Mathematical and Theoretical, 41(20):202001, May 2008. arXiv:0804.4561, doi:10.1088/1751-8113/41/20/202001.
Anne Chao and Tsung-Jen Shen. Nonparametric estimation of Shannon's index of diversity when there are unseen species in sample. Environmental and Ecological Statistics, 10(4):429–443, December 2003. doi:10.1023/A:1026096204727.
Anne Chao, Y. T. Wang, and Lou Jost. Entropy and the species accumulation curve: a novel entropy estimator via discovery rates of new species. Methods in Ecology and Evolution, 4(11):1091–1100, 2013. doi:10.1111/2041-210X.12108.
T.M. Cover and J.A. Thomas. Elements of Information Theory. Wiley, 2012. ISBN 9781118585771. URL: https://books.google.ee/books?id=VWq5GG6ycxMC.
Juan De Gregorio, David Sánchez, and Raúl Toral. Entropy Estimators for Markovian Sequences: A Comparative Analysis. Entropy, 26(1):79, January 2024. doi:10.3390/e26010079.
M. D. Donsker and S. R. S. Varadhan. Asymptotic evaluation of certain markov process expectations for large time, I. Communications on Pure and Applied Mathematics, 28(1):1–47, 1975. doi:10.1002/cpa.3160280102.
D.M. Endres and J.E. Schindelin. A new metric for probability distributions. IEEE Transactions on Information Theory, 49(7):1858–1860, 2003. doi:10.1109/TIT.2003.813506.
R. M. Fano. Transmission of Information: A Statistical Theory of Communications. M.I.T. Press, Cambridge, MA, USA, 1961. See Chapter 2.
Stefan Frenzel and Bernd Pompe. Partial mutual information for coupling analysis of multivariate time series. Phys. Rev. Lett., 99:204101, Nov 2007. URL: https://link.aps.org/doi/10.1103/PhysRevLett.99.204101, doi:10.1103/PhysRevLett.99.204101.
Eduardo García-Portugués. Chapter 2 Kernel density estimation. In Notes for Nonparametric Statistics. 6.12.1 edition, 2025. URL: https://bookdown.org/egarpor/NP-UC3M/kde-i.html (visited on 2025-05-31).
German Gomez-Herrero, Wei Wu, Kalle Rutanen, Miguel Soriano, Gordon Pipa, and Raul Vicente. Assessing coupling dynamics from an ensemble of time series. Entropy, 17:, 08 2010. doi:10.3390/e17041958.
M. Grabchak, Z. Zhang, and D. T. Zhang. Authorship Attribution Using Entropy. Journal of Quantitative Linguistics, 20(4):301–313, November 2013. doi:10.1080/09296174.2013.830551.
P. Grassberger. Entropy Estimates from Insufficient Samplings. January 2008. arXiv:physics/0307138, doi:10.48550/arXiv.physics/0307138.
Peter Grassberger. Finite sample corrections to entropy and dimension estimates. Physics Letters A, 128(6):369–373, April 1988. doi:10.1016/0375-9601(88)90193-4.
Qing Guo, Junya Chen, Dong Wang, Yuewei Yang, Xinwei Deng, Lawrence Carin, Fan Li, Jing Huang, and Chenyang Tao. Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimization. October 2022. arXiv:2107.01131, doi:10.48550/arXiv.2107.01131.
Michael U. Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 13(null):307–361, February 2012.
R. V. L. Hartley. Transmission of information. Bell System Technical Journal, 7:535–563, 1928.
Jean Hausser and Korbinian Strimmer. Entropy Inference and the James-Stein Estimator, with Application to Nonlinear Gene Association Networks. J. Mach. Learn. Res., 10:1469–1484, December 2009.
R. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations. September 2018.
Katerina Hlavackova-Schindler, Milan Palus, Martin Vejmelka, and Joydeep Bhattacharya. Causality detection based on information-theoretic approaches in time series analysis. Physics Reports, 441 (2007) 1 – 46:, 02 2007.
Matt Hoffman, David M. Blei, Chong Wang, and John Paisley. Stochastic Variational Inference. April 2013. arXiv:1206.7051, doi:10.48550/arXiv.1206.7051.
Petr Jizba. Information Theory and Generalized Statistics. In Hans-Thomas Elze, editor, Decoherence and Entropy in Complex Systems: Selected Lectures from DICE 2002, pages 362–376. Springer, Berlin, Heidelberg, 2004. doi:10.1007/978-3-540-40968-7_26.
A. Kaiser and Thomas Schreiber. Information transfer in continuous processes. Physica D, v.166, 43-62 (2002), 166:, 06 2002. doi:10.1016/S0167-2789(02)00432-3.
David A. Kelly and Ilaria Pia La Torre. DiscreteEntropy.jl: Entropy Estimation of Discrete Random Variables with Julia. Journal of Open Source Software, 9(103):7334, November 2024. doi:10.21105/joss.07334.
A.I. Khinchin. Mathematical Foundations of Information Theory. Dover, New York, 1957.
A. N. Kolmogoroff. Grundbegriffe Der Wahrscheinlichkeitsrechnung. Berlin, 1933.
L.F. Kozachenko and N.N. Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23:95–100, 1987.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Erratum: Estimating mutual information [phys. Rev. E 69, 066138 (2004)]. Physical Review E, January 2011. doi:10.1103/PhysRevE.83.019903.
R. Krichevsky and V. Trofimov. The performance of universal encoding. IEEE Transactions on Information Theory, 27(2):199–207, March 1981. doi:10.1109/TIT.1981.1056331.
C.- A. Laisant. Sur la numération factorielle, application aux permutations. Bulletin de la Société Mathématique de France, 2:176–183, 1888. doi:10.24033/bsmf.378.
D. H. Lehmer. Teaching combinatorial tricks to a computer. In Proceedings of Symposia in Applied Mathematics, volume 10 of Proceedings of Symposia in Applied Mathematics. Providence, Rhode Island, 1960. American Mathematical Society. doi:10.1090/psapm/010.
Nikolai Leonenko, Luc Pronzato, and Vippal Savani. Estimation of entropies and divergences via nearest neighbors. In ProbaStat 2006, volume 39, 265–273. Smolenice, Slovakia, June 2006.
Nikolai Leonenko, Luc Pronzato, and Vippal Savani. A class of Rényi information estimators for multidimensional densities. The Annals of Statistics, 36(5):2153–2182, 2008. doi:10.1214/07-AOS539.
J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information Theory, 37(1):145–151, 1991. doi:10.1109/18.61115.
Joseph T. Lizier. Measuring the Dynamics of Information Processing on a Local Scale in Time and Space, pages 161–193. Springer Berlin Heidelberg, Berlin, Heidelberg, 2014. URL: https://doi.org/10.1007/978-3-642-54474-3_7, doi:10.1007/978-3-642-54474-3_7.
Joseph T. Lizier. JIDT: An Information-Theoretic Toolkit for Studying the Dynamics of Complex Systems. Frontiers in Robotics and AI, December 2014. doi:10.3389/frobt.2014.00011.
Joseph T. Lizier, Mikhail Prokopenko, and Albert Y. Zomaya. Local information transfer as a spatiotemporal filter for complex systems. Phys. Rev. E, 77:026110, Feb 2008. URL: https://link.aps.org/doi/10.1103/PhysRevE.77.026110, doi:10.1103/PhysRevE.77.026110.
Antoni Lozano, Bernardino Casas, Chris Bentz, and Ramon Ferrer-i-Cancho. Fast calculation of entropy with Zhang's estimator. July 2017. arXiv:1707.08290, doi:10.48550/arXiv.1707.08290.
Zhuang Ma and Michael Collins. Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3698–3707. Brussels, Belgium, October 2018. Association for Computational Linguistics. doi:10.18653/v1/D18-1405.
C. Manning and H. Schutze. Foundations of Statistical Natural Language Processing. Foundations of Statistical Natural Language Processing. MIT Press, 1999. ISBN 9780262133609. URL: https://books.google.ee/books?id=YiFDxbEX3SUC.
Eric Marcon and Bruno Hérault. entropart: An R package to measure and partition diversity. Journal of Statistical Software, 67(8):1–26, 2015. doi:10.18637/jss.v067.i08.
R. Marschinski and H. Kantz. Analysing the information flow between financial time series . an improved estimator for transfer entropy. European Physical Journal B, 30:275–281, 11 2002. doi:10.1140/epjb/e2002-00379-2.
Sina Molavipour, Germán Bassi, and Mikael Skoglund. Conditional Mutual Information Neural Estimator. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5025–5029. May 2020. arXiv:1911.02277, doi:10.1109/ICASSP40776.2020.9053422.
Ilya Nemenman, William Bialek, and Rob de Ruyter van Steveninck. Entropy and information in neural spike trains: Progress on the sampling problem. Physical Review E, 69(5):056111, May 2004. arXiv:physics/0306063, doi:10.1103/PhysRevE.69.056111.
Ilya Nemenman, Fariel Shafee, and William Bialek. Entropy and inference, revisited. January 2002. arXiv:physics/0108025, doi:10.48550/arXiv.physics/0108025.
XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847–5861, November 2010. arXiv:0809.0853, doi:10.1109/TIT.2010.2068870.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On Variational Bounds of Mutual Information. In Proceedings of the 36th International Conference on Machine Learning, 5171–5180. PMLR, May 2019.
A. Rényi. Selected Papers of Alfred Rényi, Vol. 2. Akadémia Kiado, Budapest, 1976.
Feras Saad, Marco Cusumano-Towner, and Vikash Mansinghka. Estimators of Entropy and Information via Inference in Probabilistic Models. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 5604–5621. PMLR, May 2022.
Thomas Schreiber. Measuring information transfer. Phys. Rev. Lett., 85:461–464, Jul 2000. URL: https://link.aps.org/doi/10.1103/PhysRevLett.85.461, doi:10.1103/PhysRevLett.85.461.
Payam Shahsavari Baboukani, Carina Graversen, Emina Alickovic, and Jan Østergaard. Estimating conditional transfer entropy in time series using mutual information and nonlinear prediction. Entropy, 2020. URL: https://www.mdpi.com/1099-4300/22/10/1124, doi:10.3390/e22101124.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423, 1948. doi:10.1002/j.1538-7305.1948.tb01338.x.
B.W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman and Hall, London, 1986. URL: http://dx.doi.org/10.1007/978-1-4899-3324-9.
Matthäus Staniek and Klaus Lehnertz. Symbolic transfer entropy. Phys. Rev. Lett., 100:158101, Apr 2008. URL: https://link.aps.org/doi/10.1103/PhysRevLett.100.158101, doi:10.1103/PhysRevLett.100.158101.
C. Tsallis. Nonextensive statistics: theoretical, experimental and computational evidences and connections. Braz. J. Phys., 29:1, 1999.
C. Tsallis, R.S. Mandes, and A.R. Plastino. The role of constraints within generalized nonextensive statistics. Physica A, 261:534, 1998.
Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics, 52:479–487, 07 1988. doi:10.1007/BF01016429.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. January 2019. arXiv:1807.03748, doi:10.48550/arXiv.1807.03748.
Greg Ver Steeg and Aram Galstyan. Information-theoretic measures of influence based on content dynamics. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, WSDM '13, 3–12. New York, NY, USA, February 2013. Association for Computing Machinery. doi:10.1145/2433396.2433400.
Paul L. Williams and Randall D. Beer. Generalized Measures of Information Transfer. February 2011. arXiv:1102.1507, doi:10.48550/arXiv.1102.1507.