Attention to Mamba: A Recipe for Cross-Architecture Distillation

The authors propose a two-stage cross-architecture distillation recipe: first distilling a Transformer into a linearized version of Attention using an adaptation of the kernel trick, then distilling that linearized version into an adapted Mamba model with no Attention block. The distilled Mamba model retains most of the original Pythia-1B Transformer's performance, reaching a perplexity of 14.01 versus the teacher's 13.86, with results largely carrying over to downstream tasks though some gaps appear on tasks relying more on in-context recall. They conduct ablations at 1B scale with 10B tokens, scaling analysis on model sizes and distillation tokens, and a sensitivity analysis on token allocation between stages.