Information Theory: Entropy and Capacity

Related lectures (28)

Page 3 of 3

Covers source coding theorems, entropy, and various source models in information theory.

Covers quantifying information in messages, source entropy, common information, and communication channel capacity.

Explores entropy, KL divergence, and maximum entropy principle in probability models for data science.

Explores entropy bounds, conditional entropy theorems, and the chain rule for entropies, illustrating their application through examples.

Introduces the Random-Subcube Model (RSM) for constraint satisfaction problems, exploring its structure, phase transitions, and variable freezing.

On entropy and mutual information explores quantifying information in data science through probability distributions.

Explores random walks, Moran model, bacterial chemotaxis, entropy, information theory, and coevolving sites in proteins.

Covers generative models with a focus on self-attention and transformers, discussing sampling methods and empirical means.