Publications de Michael Christoph Gastpar

Characterising high-order interdependence via entropic conjugation

High-order phenomena are pervasive across complex systems, yet their formal characterisation remains a formidable challenge. The literature provides various information-theoretic quantities that capture high-order interdependencies, but their conceptual fo ...

2025

Attention with Markov: A Curious Case of Single-layer Transformers

Michael Christoph Gastpar, Martin Jaggi, Marco Bondaschi

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to ...

ICLR2025

Sibson α-Mutual Information and Its Variational Representations

Michael Christoph Gastpar

Institute of Electrical and Electronics Engineers (IEEE)2025

Lower Bounds on the Bayesian Risk via Information Measures

Michael Christoph Gastpar, Amedeo Roberto Esposito, Adrien Vandenbroucque

This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually any information measure, including Rényi's α, φ-divergences, and Sibson's α-Mutual Information. The appr ...

2024

Local to global: Learning dynamics and effect of initialization for transformers

Michael Christoph Gastpar, Marco Bondaschi

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in using Markov input processes to study transformers. However, our current un ...

Curran Associates, Inc.2024

Shannon Bounds for Quadratic Rate-Distortion Problems

Michael Christoph Gastpar, Erixhen Sula

The Shannon lower bound has been the subject of several important contributions by Berger. This paper surveys Shannon bounds on rate-distortion problems under mean-squared error distortion with a particular emphasis on Berger’s techniques. Moreover, as a n ...

Institute of Electrical and Electronics Engineers (IEEE)2024

Characterising directed and undirected metrics of high-order interdependence

Michael Christoph Gastpar

Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active develo ...

Institute of Electrical and Electronics Engineers Inc.2024

Fundamental Limits of Prompt Compression: A Rate- Distortion Framework for Black-Box Language Models

Michael Christoph Gastpar

We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts for black-box models. We derive the distortion-rate function for this setup as ...

Curran Associates, Inc.2024

Batch Universal Prediction

Michael Christoph Gastpar, Marco Bondaschi

Large language models (LLMs) have recently gained much popularity due to their surprising ability at generating human-like English sentences. LLMs are essentially predictors, estimating the probability of a sequence of words given the past. Therefore, it i ...

IEEE2024

The Persuasion Bottleneck

Michael Christoph Gastpar

When a delivery agency persuades a bicycle courier worker to accept a certain job, it does so by revealing the right amount and kind of information. The goal is to reward both the agent as well as the worker accordingly. In this paper, we model the act of ...

Institute of Electrical and Electronics Engineers Inc.2024