Shannon's source coding theorem

• Entropy
• Differential entropy
• Conditional entropy
• Joint entropy
• Cross-entropy
• Mutual information
• Directed information
• Conditional mutual information
• Relative entropy
• Entropy rate
• Limiting density of discrete points

• Asymptotic equipartition property
• Rate–distortion theory

• Shannon's source coding theorem
• Channel capacity
• Noisy-channel coding theorem
• Shannon–Hartley theorem

The coded sequence of length Σ
1 and Σ1 and let Σ
2 where 2</sub>{{!}} {{=}} ''a''"}},"i":0}}]}'/>|Σ2| = a. Let  f  is optimal in the sense that it has the minimal expected word length for 𝔼 denotes the expected value operator.

Given H(X) in the discrete-valued case and differential entropy in the continuous-valued case. The Source coding theorem states that for any 0"}},"i":0}}]}'/>ε > 0, i.e. for any rate n i.i.d. repetition of the source, 1:''n''</sup>"}},"i":0}}]}'/>X, and maps it to 1 − ε.

Proof of Achievability. Fix some 0"}},"i":0}}]}'/>ε > 0, and let

Anε={(x1,⋯,xn) : |−1nlog⁡p(x1,⋯,xn)−Hn(X)|<ε}.

The asymptotic equipartition property (AEP) shows that for large enough n, 2−n(H(X)+ε)≤p(x1,⋯,xn)≤2−n(H(X)−ε)

• The probability of a sequence 1 − ε.
• |Anε|≥(1−ε)2n(H(X)−ε), which follows from upper bound for |Anε|≤2n(H(X)+ε),n(H(X)+ε) bits are enough to point to any string in this set.

  The encoding algorithm: the encoder checks if the input sequence lies within the typical set; if yes, it outputs the index of the input sequence within the typical set; if not, the encoder outputs an arbitrary ε.

  Proof of converse: the converse is proved by showing that any set of size smaller than 1 ≤ i ≤ n let i</sub>''"}},"i":0}}]}'/>si denote the word length of each possible i</sub>''"}},"i":0}}]}'/>xi. Define q1 + ... + qn = 1. Then

  log C ≤ 0.

  For the second inequality we may set

  a−si≤pi

  and

  𝔼S=∑pisi<∑pi(−logapi+1)=∑−pilog2pilog2a+1=H(X)log2a+1

  Extension to non-stationary independent sources

  edit

  Fixed rate lossless source coding for discrete time non-stationary independent sources

  edit

  Define typical set δ > 0, for 2n(Hn‾(X)+ε). Thus, on an average, n</sub>''}}(''X'') + ''ε''"}},"i":0}}]}'/>Hn(X) + ε bits suffice for encoding with probability greater than δ can be made arbitrarily small, by making

  1. ↑ Shen, A. and Uspensky, V.A. and Vereshchagin, N. (2017). "Chapter 7.3. : Complexity and entropy". Kolmogorov Complexity and Algorithmic Randomness. American Mathematical Society. p. 226. ISBN 9781470431822.{{cite book}}: CS1 maint: multiple names: authors list (link)
  2. ↑ C.E. Shannon, "A Mathematical Theory of Communication Archived 2009-02-16 at the Wayback Machine", Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, July, October, 1948
  3. ↑ David J. C. MacKay. Information Theory, Inference, and Learning Algorithms Cambridge: Cambridge University Press, 2003.

     ---

     View original on Wikipedia | Back | FazBrowse Home