ECE 587: Information Theory (FA24)

Course Information

• Lecture: Teer 203 MW 10:05 - 11:20 AM

  • TAs: Avijit Mandal, Minxue Tang

  • Dr. Pfister Office Hour: Th 2:00 - 3:00 PM

  • Duke Course Rules and Information (updated 08/13/20)

• Syllabus (updated 10/22/24)

• Textbooks

  • Elements of Information Theory, 2nd Ed. by Cover and Thomas
    

  • Information Theory (Lecture Notes) V6.16 by Stefan Moser

  • Advanced Topics in Information Theory (Lecture Notes) V5.7 by Stefan Moser

  • Information Theory by Yury Polyanskiy and Yihong Wu

  • Information Theory and Reliable Communication by Robert Gallager

• Lecture Notes

  • Lecture 1 - Introduction

  • Lecture 2 - Measures of Information

  • Lecture 3 - Convergence and Typical Sets

  • Lecture 4 - Entropy Rates & Markov Chains

  • Lecture 5 - Lossless Compression

  • Lecture 6 - Channel Coding

  • Lecture 7 - Differential Entropy

  • Lecture 8 - Gaussian Channel

  • Lecture 9 - Rate Distortion

  • Lecture 11 - f Divergence

  • Capacity Notes

  • Wide-Sense Stationary Processes

Handouts

• Homework 0
  

• Homework 1
  

• Homework 2
  

• Homework 3
  

• Homework 4
  

• Homework 5
  

• Homework 6
  

• Homework 7
  

Web Resources

• Historical References

• • Guglielmo Marconi

  • Claude Shannon

  • Shannon's 1948 Paper: A Mathematical Theory of Communication

• Applications

• Mathematical References

  • A Tutorial on Convex Optimization (brief)

  • Convex Optimization for Machine Learning (complete)

  • Toeplitz and circulant matrices: A review by Robert Gray

  • Nice discussion and proof of Pinsker's inequality and extensions

• Machine Learning

  • Information-theoretic analysis of generalization capability of learning algorithms

  • Generalization bounds: Perspectives from information theory and PAC-Bayes

• Channel Coding References

  • Coding for two noisy channels by Peter Elias

  • On exponential error bounds for random codes on the BSC by David Forney

  • A simple derivation of the coding theorem and some applications by Robert Gallager

  • Capacity of a burst-noise channel by E. N. Gilbert

  • The capacity of the band-limited Gaussian channel by A. Wyner

  • Computation of Channel Capacity and Rate-Distortion Functions by R. E. Blahut

  • Hypothesis Testing and Information Theory by R. E. Blahut

  • Capacity and coding for Gilbert-Elliott channels by M. Mushkin and I. Bar-David

  • Capacity of multi-antenna Gaussian channels by Emre Telatar