舉個例說明[15][16],想像家陣有個黑盒,個盒裝住啲嘢,A 君唔知個盒裝咗乜(不確定),但佢識 B 君,而 B 君睇過個盒嘅內容;想像家陣 B 君用把口講俾 A 君聽個盒裝咗乜,喺呢個過程當中,B 君要將自己嘅所思所想轉化講嘢嘅聲(訊號),而啲聲會由佢把口傳過去 A 君對耳仔嗰度,A 君個腦就會由聽到嘅聲嗰度解讀 B 君想表達嘅內容(重新建構);假設 B 君係靠得住嘅,A 君就可以透過收訊號嚟判斷個盒嘅內容係乜(減少不確定)-B 君講嘢向 A 君傳遞咗資訊;喺現實,呢種做法好多時都係唔完全靠得住嘅(有雜音),例如可能佢哋兩個周圍嘅環境好嘈,搞到 A 君聽錯。
當中 係指「 同時 嘅概率」,成條式道理同基本嗰條資訊熵式道理一樣,都係諗嗮所有可能性,將每個可能性嘅概率乘以個概率嘅對數,跟手再將所有可能性嘅呢個數加埋一齊。假如 X 同 Y 係獨立[歐 8](其中一個數值係乜唔會影響另一個數值係乜)嘅,佢哋嘅聯合熵會等同佢哋各自嘅資訊熵相加,噉係因為當兩件事件係獨立嗰陣,以下呢條式會成立[23]:
當中 係指「根據 , 嘅機率」。相對熵可以按照「平均嚟講知道個真相會做成幾大嘅驚訝」嚟理解:想像有一個帶有隨機性嘅變數 ,佢真實嘅概率分佈係 ,而家係噉抽 嘅數值,如果 A 君知 嘅樣,而 B 君心目中嘅分佈係 ,噉平均嚟講 B 君會有更多嘅驚訝(更加常會估錯 嘅數值)。所以相對熵條式可以話係反映 B 君嘅驚訝減 A 君嘅驚訝[27]。
J. L. Kelly, Jr., Betbubbles.com, "A New Interpretation of Information Rate" Bell System Technical Journal, Vol. 35, July 1956, pp. 917-26.
Andrey Kolmogorov (1968), "Three approaches to the quantitative definition of information" in International Journal of Computer Mathematics.
R. Landauer, IEEE.org, "Information is Physical" Proc. Workshop on Physics and Computation PhysComp'92 (IEEE Comp. Sci.Press, Los Alamitos, 1993) pp. 1-4.
R. Landauer, IBM.com, "Irreversibility and Heat Generation in the Computing Process" IBM J. Res. Dev. Vol. 5, No. 3, 1961.
Shannon, C.E. (1948), "A Mathematical Theory of Communication" (PDF), Bell System Technical Journal, 27, pp. 379–423 & 623–656, July & October, 1948. Notes and other formats.
Timme, N., Alford, W., Flecker, B., & Beggs, J. M. (2011). Multivariate information measures: an experimentalist's perspective. arXiv preprint arXiv:1111.6857.
Arndt, C. Information Measures, Information and its Description in Science and Engineering (Springer Series: Signals and Communication Technology), 2004, ISBN 978-3-540-40855-0
Stone, JV. Chapter 1 of book "Information Theory: A Tutorial Introduction", University of Sheffield, England, 2014. ISBN 978-0956372857.
Yeung, RW. A First Course in Information Theory. Kluwer Academic/Plenum Publishers, 2002. ISBN 0-306-46791-7.
Yeung, RW. Information Theory and Network Coding. Springer 2008, 2002. ISBN 978-0-387-79233-0.
第啲講資訊理論嘅書:
Leon Brillouin, Science and Information Theory. Mineola, N.Y.: Dover, [1956, 1962] 2004. ISBN 0-486-43918-6
James Gleick, The Information: A History, a Theory, a Flood. New York: Pantheon, 2011. ISBN 978-0-375-42372-7
A. I. Khinchin, Mathematical Foundations of Information Theory. New York: Dover, 1957. ISBN 0-486-60434-9
H. S. Leff and A. F. Rex, Editors, Maxwell's Demon: Entropy, Information, Computing. Princeton University Press, Princeton, New Jersey (1990). ISBN 0-691-08727-X
Robert K. Logan. What is Information? - Propagating Organization in the Biosphere, the Symbolosphere, the Technosphere and the Econosphere. Toronto: DEMO Publishing.
Tom Siegfried, The Bit and the Pendulum, Wiley, 2000. ISBN 0-471-32174-5
Huelsenbeck, J. P.; Ronquist, F.; Nielsen, R.; Bollback, J. P. (2001). "Bayesian inference of phylogeny and its impact on evolutionary biology". Science. 294 (5550): 2310-2314.
Burnham, K. P. and Anderson D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition (Springer Science, New York).
Thorton, C. (2013, January). A new way of linking information theory with cognitive science. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 35, No. 35).
Allikmets, Rando; Wasserman, Wyeth W.; Hutchinson, Amy; Smallwood, Philip; Nathans, Jeremy; Rogan, Peter K. (1998). "Thomas D. Schneider], Michael Dean (1998) Organization of the ABCR gene: analysis of promoter and splice junction sequences". Gene. 215 (1): 111-122.
Weary, G., & Jacobson, J. A. (1997). Causal uncertainty beliefs and diagnostic information seeking. Journal of Personality and Social Psychology, 73(4), 839.
Chen, A. H., Zhou, Y., Gong, H. Q., & Liang, P. J. (2004). Firing rates and dynamic correlated activities of ganglion cells both contribute to retinal information processing. Brain research, 1017(1-2), 13-20.
Arndt, C. Information Measures, Information and its Description in Science and Engineering (Springer Series: Signals and Communication Technology), 2004.
Einicke, G. A. (2018). "Maximum-Entropy Rate Selection of Features for Classifying Changes in Knee and Ankle Dynamics During Running". IEEE Journal of Biomedical and Health Informatics. 28 (4): 1097-1103.
Saleem Bhatti. "Channel capacity". Lecture notes for M.Sc. Data Communication Networks and Distributed Systems D51 -- Basic Communications and Networks.
Hazewinkel, Michiel, ed. (2001) [1994], "Transmission rate of a channel", Encyclopedia of Mathematics, Springer Science+Business Media B.V. / Kluwer Academic Publishers, ISBN 978-1-55608-010-4.
Cherkaoui, A., Fischer, V., Fesquet, L., & Aubert, A. (2013, August). A very high speed true random number generator with entropy assessment. In International Workshop on Cryptographic Hardware and Embedded Systems (pp. 179-196). Springer, Berlin, Heidelberg.
Varchola, M., & Drutarovsky, M. (2010, August). New high entropy element for FPGA based true random number generators. In International Workshop on Cryptographic Hardware and Embedded Systems (pp. 351-365). Springer, Berlin, Heidelberg.
Wang, Z., Alahmadi, A., Zhu, D., & Li, T. (2015, December). Brain functional connectivity analysis using mutual information. In 2015 IEEE Global Conference on Signal and Information Processing (GlobalSIP) (pp. 542-546). IEEE.
Zhang, W., Muravina, V., Azencott, R., Chu, Z. D., & Paldino, M. J. (2018). Mutual Information Better Quantifies Brain Network Architecture in Children with Epilepsy. Computational and mathematical methods in medicine, 2018.