TinyCeNN-LM Tests Quality-Gated Attention Conversion in Language Models

Summary

TinyCeNN-LM presents a quality-gated post-training framework for converting attention layers in pretrained language models into CeNN-inspired cellular-recurrent layers. The design combines bounded local processing, compact recurrent memory, routing, fusion, and an accept-or-rollback validation step. The authors study Integrated Memory, MemoryFusion, and PDelta3-GDN2-CLVR+Local32 implementations. Under strict PDelta3 criteria, layers 0-2 of SmolLM2-135M are accepted with cumulative ΔNLL of +0.01209, while layer 3 is rejected because representation fidelity fails despite acceptable NLL. On Qwen3.5-0.8B, full-attention layers 3, 7, and 11 are accepted, with final ΔNLL of +0.02073. Integrated Memory keeps perplexity between -0.07% and +0.93% while reducing total cache by up to 6.01%. A sampled 200-item downstream check reports 28.5%-32.0% overall accuracy for converted Qwen releases. The results support conservative, quality-gated structural conversion, rather than universal attention replacement or a demonstrated speedup.