Our preprint “Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms” is now available on arXiv.
The study
For a function represented by d two-entry heads using 2d entries in total, exact representation by a single head requires 2^d entries, even with arbitrary fixed keys, scalar values, and positive masses. The same lower bound holds for uniform error below 1/2 over the full query space.
Paper and release details
- Paper
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
- Authors
- Hiroyasu Usami, Keisuke Hara, Ayato Tsuboi, Naohiko Matsuda
- Public version
- arXiv:2611.02943 [cs.LG] (preprint; not peer reviewed)
- arXiv v1 submitted
- 2026-11-02 (UTC)