Tensorizing Engram

Tensorizing Engram: Sharing Latents Across N-Gram Embeddings is Beneficial in LLMs

arXiv:2606.08347v1 Announce Type: new Abstract: Modern language models represent text using discrete token-level embeddings, which forces recurring multi-token patterns to be learned implicitly across Transformer layers. Both Over-tokenized Transformers and Engram attempt to address this limitation by explicitly incorporating multi-token (n-gram) memories. However, they rely on separate hash tables for each n-gram order, which introduces hash collisions and prevents nested n-grams from...