A Mathematical Theory of Communication¶
Why this mattered¶
Shannon’s paper turned communication from an engineering craft organized around particular media into a mathematical theory of information itself. Its central shift was to separate the meaning of a message from the problem of transmitting symbols reliably: information could be measured statistically, in bits, by the uncertainty reduced when a message is received. This made it possible to reason uniformly about telegraphy, telephony, radio, and later digital computers, storage devices, and networks. Concepts such as entropy, redundancy, channel capacity, and coding gave engineers a vocabulary for the fundamental limits of communication rather than only rules for improving specific systems.
The paper’s most consequential result was the noisy-channel coding theorem: below a channel’s capacity, arbitrarily reliable communication is possible in principle by suitable coding; above it, no coding scheme can overcome the noise. This was a paradigm shift because it showed that noise did not merely impose a gradual engineering penalty but defined a precise boundary between possible and impossible transmission. It also transformed “error correction” from a collection of ad hoc techniques into a central mathematical problem: find practical codes that approach Shannon’s limit. Much of modern coding theory, from Hamming codes and convolutional codes to turbo codes, LDPC codes, and polar codes, can be understood as a decades-long attempt to realize the existence proof Shannon gave in 1948.
The influence of the paper extended well beyond telecommunications. By formalizing compression through source entropy, Shannon provided the theoretical basis for lossless data compression and for understanding the value of exploiting statistical structure in messages. By treating communication systems abstractly as sources, encoders, channels, decoders, and destinations, he supplied a model that later underpinned digital signal processing, computer networking, cryptography, statistical inference, and parts of machine learning. Subsequent breakthroughs in the internet, mobile communication, deep-space telemetry, compact storage, and multimedia compression all rely, directly or indirectly, on the limits and abstractions introduced in this paper.
Abstract¶
The recent development of various methods of modulation such as PCM and PPM which exchange bandwidth for signal-to-noise ratio has intensified the interest in a general theory of communication. A basis for such a theory is contained in the important papers of Nyquist 1 and Hartley 2 on this subject. In the present paper we will extend the theory to include a number of new factors, in particular the effect of noise in the channel, and the savings possible due to the statistical structure of the original message and due to the nature of the final destination of the information.
Related¶
- enables → Maximum entropy modeling of species geographic distributions — Shannon entropy provided the information-theoretic quantity maximized in Maxent species distribution modeling.
- enables → Information Theory and Statistical Mechanics — Shannon's entropy formalism gave Jaynes the information measure used to derive statistical mechanics by maximum entropy inference.
- cite ← Maximum entropy modeling of species geographic distributions — Maximum entropy species-distribution modeling draws on Shannon entropy as the information-theoretic quantity maximized under environmental constraints.
- cite ← Coefficient Alpha and the Internal Structure of Tests — Cronbach's alpha cites information theory to connect test reliability with information transmission and measurement error.
- cite ← Information Theory and Statistical Mechanics — Jaynes derives statistical-mechanical entropy by applying Shannon's information-theoretic entropy and maximum-uncertainty reasoning.