I'm making LLMs work on small and old devices

Every device around you is more capable than it looks. TV boxes, old phones, single-board computers and routers all ship with CPUs, RAM and flash that sit idle most of the day. smolmaxxing is about unlocking that capability: finding the biggest language model each device can really run, then pushing it further with custom kernels, smarter memory use and weight streaming. Every claim is measured on the hardware itself. Device #1 is a Xiaomi TV Box S (3rd Gen): Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable), eMMC flash, Android 14, no root. On it, this repo runs models from 260 K to 16 B parameters: - Qwen3.5-0.8B: 16.8 tok/s (2.2× the stock 7.5; measured in the evening of 11-05, see §17). - Qwen3.5-2B: 7.7 tok/s in real chat. The box's speed drifts by ~10 % over a day, so every step is measured back to back: §16 gave +11 %, §17 another +7 %. - An 8B mixture-of-experts model (Moon-Beam-8B-A1B): 7.2 tok/s in real chat, its experts paged from flash on demand and prefetched one layer ahead (§19). - A dense 1-bit 8B model (Bonsai-8B): 5.2 tok/s in real chat, up from 1.2: fully resident in a "headless" mode plus a lookup-table kernel (§19). - A 16B mixture-of-experts model (Ling-mini-2.0): 5.1 tok/s in real chat, coherent over 600-word answers, the same expert pager applied unchanged (§20). Twice Moon-Beam's size, but not better per byte: bigger only helps on this box when the model is also better. - A dense 9B model: 0.27 tok/s, streaming most of its weights from flash. Device #2 is a first-generation iPad Air (2013): Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM, iOS 12.5.7, jailbroken, CPU only. A 13-year-old tablet runs a 1.7B-parameter model at 24-26 tok/s (18-22 in long chats), a 4B one at about 10 tok/s, an 8B mixture of experts at about 3.5 tok/s (5 with half its experts, at a quality cost), and a 16B one at 2.6-4.5 tok/s, with their experts paged from flash, from a chat page in its own Safari. See Results on device #2. Every step is written up experiment by experiment in EXPERIMENT.md (E1-E160). | Device | Status | |---|---| | Xiaomi TV Box S (3rd Gen): Amlogic S7D, Cortex-A55, 2 GB | ✅ done: 260K → 16B parameters, 8B at 5-7 tok/s, 16B at 5 tok/s | iPad Air (2013): Apple A7, 2 cores, 1 GB, iOS 12.5.7, jailbroken (ipad-air/ ) | ✅ running: Bonsai-1.7B (1-bit) at 24-26 tok/s, 18-22 in long chats; Bonsai-4B at about 10 tok/s; Moon-Beam-8B-A1B (MoE) at about 3.5 tok/s; Ling-mini-2.0 (16B MoE) at 2.6-4.5 tok/s; CPU only | | More TV boxes, phones, single-board computers, routers… | 🔜 coming soon | The techniques aren't tied to one box. The 1-bit lookup-table kernel needs only base NEON, as the iPad shows; any ARM chip with dotprod gets the other kernels; and any Linux or Android device with more model than RAM can use the streaming engine. | Class | Best model | First run → now (tok/s) | Gain | Bits/byte | What got it there | Download | |---|---|---|---|---|---|---| | 0.8B | Qwen3.5-0.8B | 7.5 → 16.8 bench | ×2.2 | ≈1.00 | all-Q4_K, padded 65K head, fp16 DeltaNet state | HF | | 2B | Qwen3.5-2B | 0.1 → 7.7 chat | ×77 | 0.909 | all-Q4_K, padded 65K head, fused DeltaNet | HF | | 4B | Ternary-Bonsai-4B | 2.2 → 4.3 chat | ×2.0 | ≈0.965 | ternary weights in 2-bit fields, merged scales | HF | | 8B dense | Bonsai-8B (1-bit) | 1.21 → 5.2 chat | ×4.3 | 0.974 | 1-bit LUT kernel, fully resident in headless mode | HF | | 8B MoE | Moon-Beam-8B-A1B | 1.87 → 7.2 chat | ×3.9 | 0.996 | Q2_K experts paged from flash, cache-aware routing, prefetch | HF | | 9B | Qwen3.5-9B | 0.16 → 0.27 chat | ×1.7 | 0.885 | weight streaming plus its own MTP head | rebuild (q9 ) | | 16B MoE | Ling-mini-2.0 | 4.1 → 5.1 chat | ×1.2 | 1.054 | the Moon-Beam recipe, unchanged | HF | "First run" is the first time each model ran on the box: - 0.8B: the stock Q4_K_M file. - 2B: the stock file, streamed from flash because it didn't fit in RAM. Its first build that fit ran at 4.8, so from there the gain is ×1.6. - 4B: the stock file, which thrashed. - 8B dense: the last 0.1 GB streamed from flash. - 8B MoE: the first expert-pager run, without routing bias. - 9B: streamed. - 16B: started on the Moon-Beam recipe, so its gain is small. On this box, the 2B is the best speed-quality balance. Ling-mini-2.0 shows that bigger isn't automatically better: it is twice Moon-Beam's size, but scores worse per byte and runs slower. What we built, grouped by the bottleneck it attacks (details and numbers in What's in here): | Bottleneck | Innovations | Biggest wins | |---|---|---| | Compute (in-order A55, 10 items) | Q2_K and ternary repacks for vdotq_laneq , 1-bit bit planes, a 1-bit lookup-table kernel, ternary in 2-bit fields, Q8_0 attention with a skip threshold, fused Gated-DeltaNet, DeltaNet state read by id / fp16, the LM head on the fast kernel, guided matmul scheduling | Q2_K 7.8 → 13.9 G weights/s; Bonsai-8B 3.77 → 5.1-5.4; 4B 3.2 → 4.3; long answers no longer halve the speed | | RAM (~1 GB, 5 items) | pruned LM head, lossless 1.875-bit ternary, headless mode, a fork-free memory guard, a repack cache on flash | −0.2 to −0.5 GB per model; free RAM ~1,000 → ~1,300 MB; Bonsai-8B 1.2 → 3.6 before any kernel work | | Flash (278 MB/s, 5 items) | weight streaming with O_DIRECT, MTP speculation when flash-bound, expert pager, cache-aware routing, pre-gated prefetch | 9B +69 %; expert misses 13 % → ~2 %; Moon-Beam 4.4 → 7.2 | | Output quality (2 items) | a long-answer degeneration test, a repeat penalty of 1.15 over 256 tokens | degenerate long answers 12/13 → 1/13 | Quality is English perplexity on a fixed 22 KB text, converted to bits per byte so models with different tokenizers compare fairly (lower is better). Speed is generation on the TV, measured either through llama-server with real chat requests ("chat") or with llama-bench ("bench"). | Model | Size on the box | Quality (bits/byte) | Generation | Prompt | Notes | |---|---|---|---|---|---| | stories260K (llama2.c) | 1 MB | n/a | ~650 tok/s | n/a | the first thing that ran (§1-8) | | Gemma 3 270M Q4_K_M | 253 MB | n/a | 22 tok/s | 73 tok/s | | | Qwen3.5-0.8B Q4_K_M (stock) | 497 MB | n/a | 7.5 bench | 28 tok/s | | | Qwen3.5-0.8B, all-Q4_K + 65K head | 442 MiB (566 MB RSS) | ≈1.00* | 16.8 bench | 33 tok/s | best speed for its quality so far (§15-17) | | Qwen3.5-2B, all-Q4_K + 65K head | 0.99 GB | 0.909 | 7.7 chat | 17.8 | the default: best overall (§11, §16, §17) | | Ternary-Bonsai-4B, merged scales + 2-bit fields | 1.03 GB | ≈0.965 | 4.3 chat (4.8 bench) | 9.6 | best 4B: was 3.2 chat / 7.0 prompt on the same build (§18) | | Moon-Beam-8B-A1B (MoE), Q2_K experts + 60K head | 0.48 GB GGUF + 2.6 GB repack cache on flash, ~1.1 GB RSS | 0.996 | 7.2 chat | n/a | fastest 8B: 128 experts, 6 used; expert pager, cache-aware routing, pre-gated prefetch, repeat penalty against long-answer loops (§19, headless mode) | | Ling-mini-2.0 (MoE, 16.3 B), Q2_K experts + 60K head | 0.47 GB GGUF + 5.3 GB repack cache on flash, ~1.13 GB RSS | 1.054 | 5.1 chat | n/a | biggest model at a usable speed: 256 experts of 1 MB, 8 used, pool 43 per layer, bias 0.3 (§20) | | Bonsai-8B, 1-bit, LUT kernel | 0.96 GB, all resident | 0.991 (0.974 on the head) | 5.2 chat (5.1-5.4 bench), 4.0 at ~900 tokens of context | 5.8-6.1 | best dense 8B: headless mode, Q8_0 attention (§19) | | Bonsai-8B, 1-bit + compact scales | 0.84 GB + 0.12 GB streamed | 0.991 | 1.21 chat (2.2 bench) | 4.9 | §14, before headless mode | | Ternary-Bonsai-8B | 0.89 GB + 0.80 GB streamed | 0.919 | 0.31 bench | 3.45 | §14 | | Qwen3.5-9B UD-IQ2_XXS + MTP | 0.63 GB + 1.6 GB streamed | 0.885 | 0.16 chat, 0.27 with MTP | 0.8 | best quality; flash-bound (§14) | * measured on the pruned head's positions (1.3 % of tokens fall outside it), so not exactly comparable; the stock full-vocabulary Q4_0 file scores 1.024. Other small models, stock files (screening, §15 E54): | Model | Generation (bench) | Prompt | Bits/byte | |---|---|---|---| | LFM2.5-350M (QAD Q4_0) | 23.7 tok/s | 104 tok/s | 1.805 | | LFM2-700M (Q4_0) | 11.5 tok/s | 52.8 tok/s | 1.169 | | Qwen3-0.6B (Q8_0) | 9.35 tok/s | 50.6 tok/s | 1.059 | | LFM2.5-1.2B (QAD Q4_0) | 8.3 tok/s | 31.6 tok/s | 1.133 | | Llama 3.2 1B (Q4_0) | 7.0 tok/s | 29.5 tok/s | 0.998 | | Ternary-Bonsai-2-2B (Q2_0) | 5.9 tok/s | 15.8 tok/s | 1.124 | | Gemma 3 1B (Q4_K_M) | 5.7 tok/s | 11.1 tok/s | 1.260 | | Granite 4.0 H 1B (Q4_0, ~1.5 B) | too big as shipped | 0.916 | | | ↳ rebuilt: all-Q4_K, Q8_0 embedding, 48K head (E62) | 5.29 tok/s (fits only after freeing RAM, 950 MB RSS) | 0.919 | | | LFM3-1.7B (QAD Q4_0) | doesn't fit | Granite 4.0 H 1B was the one model close to the 2B's quality. Rebuilt our way (E62), it runs at 5.29 tok/s, below the 2B's 6.55 at slightly lower quality: it carries about twice the 0.8B's weight bytes per token. Physics of the box: one process can hold ~0.96 GB before Android starts killing things (~1.15 GB in headless mode, with the launchers and Google services force-stopped), and flash reads top out at 278 MB/s, with a lone ~1 MB random read costing ~6 ms. A model that has to stream X MB per token can never beat 278 / X tokens per second. That, not the CPU, is what limits everything above 2B, and why a mixture-of-experts model, which touches ~1 B of its 8 B weights per token, is the way past it. 10 tok/s at 8 B? Measured, not guessed (E98): Moon-Beam reaches 10.4 tok/s with 2 experts per token, but the text turns to gibberish; the best coherent setting is ~7-8 tok/s. Dense Qwen3-8B is Bonsai-8B at 1 bit, and the smallest Qwen MoE (35B-A3B) touches 3× Moon-Beam's weights per token. A 2013 iPad Air: Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM, iOS 12.5.7, jailbroken with Amethyst. Everything runs on the CPU with 2 threads and a 1,024-token context, cross-compiled from a Mac. The write-up is §21 of EXPERIMENT.md , and build steps are in ipad-air/README.md . | Model | File | Generation (tok/s) | Peak RSS | Fixed three-question test | |---|---|---|---|---| | Bonsai-1.7B Q1_0 (1-bit) | 237 MiB | 23.7-25.8, 18-22 through long chats | 605 MB | all three right, and writes correct code: the one to use | Bonsai-4B Q1_0 (1-bit), -c 512 -b 64 | 546 MiB | 9.4-11.4, 9.2-10.3 through long chats; as a server, iOS resets it within minutes | 615-682 MB | right on France, tides mostly right, fails recall every run; better per byte (1.073 vs 1.158 bits/byte) | Moon-Beam-8B-A1B (MoE, Q2_K experts), expert pager, -c 512 -b 64 | 2.6 GiB (474 MB mapped) | 3.5-3.6 on longer answers; 4.9-5.1 with --experts 4 (+27 % perplexity) | ~600-625 MB | all three right; a session model, too big for the always-on server | | SmolLM2-360M-Instruct Q4_K_M | 258 MiB | 10.9-12.5 | 269-316 MB | all three right; the lightest good option | | Gemma 3 270M-it Q4_K_M | 241 MiB | 15.4-19.0 | 248-294 MB | wrong on tides, empty reply on recall | | SmolLM2-135M-Instruct Q4_K_M | 101 MB | 18.6-29.3 | 141 MB | right on France and recall, weak on tides | | Qwen3.5-0.8B Q4_K_M | 508 MiB | about 0.2 | 506 MB | thrashes the page cache, stopped by hand | Only the Bonsai and Moon-Beam rows were measured on the latest build. The other rows predate the attention and sampler work below. | Class | Best model | First run → now (tok/s) | Gain | What got it there | |---|---|---|---|---| | 1.7B | Bonsai-1.7B (1-bit) | 2.75 → 23.7-25.8 | ×8.6 | multiplication-free and lookup-table 1-bit kernels without dotprod, the LM head on the fast kernel, row-order attention with pruning, a one-pass sampler | | 4B | Bonsai-4B (1-bit) | 9.4-11.4 | the 1.7B's kernels, attention for 4 q heads per K head, -c 512 -b 64 to stay under the ~700 MB ceiling | | | 8B dense | Bonsai-8B (1-bit) | 0.08 | doesn't fit in RAM; most of the file is re-read from flash every token | | | 8B MoE | Moon-Beam-8B-A1B | can't load → 1.6 → 3.5 (5.0 with 4 experts, +27 % perplexity) | ×2.2 | expert pager, Q2_K lookup-table kernel, prefetch, cache bias 0.03, frequency-based eviction | | 16B MoE | Ling-mini-2.0 | can't load → 2.6 (3.3 at +2.6 %, 4.5 with bias 0.3) | new | the Moon-Beam recipe, and a model file assembled over HTTP and streamed straight to the iPad | "Can't load": iOS 12 lets a process map at most about 1.7 GB, so stock llama.cpp fails on the 2.6 and 5.5 GB files before running a token. "First run" for Moon-Beam is the first expert-pager run (pool 8, no bias). What we built on the iPad, grouped by the bottleneck it attacks: | Bottleneck | Innovations | Biggest wins | |---|---|---| | Compute (2-core A7, no dotprod, 6 items) | a multiplication-free 1-bit kernel, the TV's 1-bit lookup-table kernel ported to base NEON, a lookup-table copy of the tied LM head, row-order attention with pruning (2 or 4 q heads per K head), a one-pass sampler with a 40-entry heap, a Q2_K lookup-table kernel for experts | Bonsai-1.7B 2.75 → 3.45 → ~11 → 23.7-25.8 tok/s; sampling 3 → 0.23 ms per token; Moon-Beam 2.94 → 3.06 from the Q2_K kernel alone | | Address space and RAM (~1.7 GB mappable, ~700 MB usable, 4 items) | an expert pager that maps only the non-expert part (experts-last files), pruned LM heads for bailingmoe2 and bailingmoe3 , small batches (-c 512 -b 64 ), std::filesystem compiled out for iOS 12 | 8B and 16B MoE models load at all, in 556-625 MB; Bonsai-4B fits | | Flash (~260 MB/s, 5 items) | uncached pread with 4 readers, reads queued by when they're needed, pre-gated prefetch, cache-aware routing bias, decayed-frequency eviction | Moon-Beam 1.6 → 3.5 tok/s at no quality cost; misses 42 % → 38 % from eviction alone | | Tooling (5 items) | an op profiler with page-fault counts, an expert trace and cache simulator, on-device perplexity (tinyppl ), model assembly over HTTP range requests, a bench that kills orphaned runs | found the LM head that never reached the fast kernel (45 of 72 ms); ruled out the soft-fault theory; built a 5.5 GB model with 1.4 GB of Mac disk | How Bonsai-1.7B got from 2.75 to about 25 tok/s (same model file throughout): | Step | tok/s | What changed | |---|---|---| | Stock llama.cpp, 2 threads | 2.75 | emulates dotprod with widening multiplies | | Multiplication-free NEON kernel | 3.45 | weights are ±1, so no multiply is needed; output identical | | Lookup-table kernel from device #1, ported to base NEON | about 11 | its dotprod requirement turned out to be only an assumption | | The output head on that kernel | 18-22 | 151,669 rows isn't a multiple of 32, so the tied head had never reached it: 45 of 72 ms per token | | Row-order attention with pruning, and a sampler that doesn't sort the vocabulary | 23.7-25.8 | the KV cache stores all heads of a row side by side, so one head at a time missed the cache constantly | In long chats (400-token answers), the three turns went from 15.4 / 7.6 / 8.2 to 21.8 / 17.9 / 18.4 tok/s through the attention work alone. The pruning threshold that holds long chats above 17 costs +0.42 % perplexity at 1,024 tokens of context, near the ±0.3 % noise floor; GGML_FA_EPS=0.0001 trades that back for about 15 tok/s. An 8B model, through an expert pager. iOS 12 caps a process at about 1.7 GB of mapped address space (1.3 GB for the web server's user), so Moon-Beam-8B's 2.6 GB file can't even be mapped, and keeping fewer experts to make it fit more than doubles its perplexity (§21.7). Dense Bonsai-8B runs at 0.08 tok/s from flash. So the model file is rewritten with its experts last, only the 474 MB before them is mapped, and each layer keeps 14 experts in RAM, reading the others from flash with pread when the router picks them. Results match the unpaged model exactly. A small routing bias toward experts already in RAM (+0.3 % perplexity, inside the noise) brought it to 2.4-2.8 tok/s in about 600 MB (§21.8). Round 2 (§21.9) queues flash reads by when they're needed, prefetches the next layer's likely experts, and runs the experts through a lookup-table kernel for Q2_K: 3.4 tok/s, 40× the dense 8B, at +0.7 % perplexity on the full eval text. The flash is still the limit: each token reads about 53 MB at 266 MB/s. Wiring weights in RAM kernel-panicked the iPad, like a pool of 16 did before. Round 3 (§21.10) evicts by decayed use frequency instead of recency, for 3.5 tok/s at no quality cost. 5 tok/s takes 4 experts per token instead of 8 (+27 % perplexity); choosing the count per token adaptively turned out no better than a fixed count. A 16B model (§21.11): Ling-mini-2.0, 16.3B parameters with 256 small experts per layer, runs through the same pager at 2.6 tok/s with unchanged routing, 3.3 at +2.6 % perplexity and 4.5 with a strong cache bias, in under 580 MB. Its 5.5 GB file was assembled from two published quantizations over HTTP range requests and streamed straight to the iPad, since the Mac had no room for it. It is twice Moon-Beam's size at the same speed, but not a better model (+3 % perplexity). Limits worth knowing. Bonsai is a 1-bit model and still gets facts wrong (in one test it called Mercury the closest planet to Earth). About 700 MB is the practical memory ceiling: 1-bit weights don't compress, so iOS starts killing processes past it, which is why the 4B needs a 512-token context. The jailbreak doesn't survive a reboot. The fast output head adds 2-3 s to the first prompt after each launch. Kernels for Cortex-A55 (ARMv8.2 dotprod, no i8mm), as a patch on llama.cpp v0.8.0, each with an exactness test against a scalar reference: | Kernel | Before → after | Where | |---|---|---| Q2_K, repacked 8 rows × 4 bytes for vdotq_laneq (upstream repacks Q2_K only on x86 / RISC-V) | 7.8 → 13.9 G weights/s; generation +41 %, prompt 2.4× | E34 | | TQ1_0 / TQ2_0 ternary, repacked | 9.5 → 14.4 / 11.9 → 19.1 G weights/s | E38 | | Ternary Q2_0 → lossless 1.875 bits/weight in RAM (base-3 packing that keeps the per-64 scales) | Ternary-Bonsai-4B from "thrashes" to 3.2 tok/s | E39, E42 | Q1_0 bit planes: vtst turns a bit plane straight into a mask for vdot , no table lookups; 8-bit row-affine scales (−5.6 %, 0.07 % error) | Bonsai-4B 3.02 → 4.22 tok/s; Bonsai-8B fits in 0.96 GB | E44, E52 | | Fused Gated-DeltaNet step with direct state write (Qwen3.5) | 2B 6.0 → 6.3-6.7 tok/s | E28 | DeltaNet state read in place by id instead of a 1 MB/layer gather every token; fp16 state (LLAMA_STATE_F16=1 , on by default in the UI) | 0.8B 11.9 → 12.9 tok/s; 2B +2.5 % for +0.15 % perplexity | E56, E61, E65 | | The LM head on the fast kernel: the 65,668-row head was never repacked (ggml repacks only row counts divisible by 8), so 4 token rows were added; embedding rows are then rebuilt from the repacked head (inverse 8-row interleave) instead of read from flash; the one-token DeltaNet conv step fused from 4 nodes into one NEON op | 2B +7 % (bench and chat); 0.8B +9.5 % | E69-E72 | | Ternary weights in 2-bit fields, with scales merged per 128: on the in-order A55, one base-3 digit costs ~5 NEON ops per 16 weights and a 2-bit field costs 2, so spending 13 % more RAM buys speed. Merging adjacent scales costs +0.18 % perplexity | Ternary-Bonsai-4B 3.2 → 4.3 tok/s in chat, prompt 7.0 → 9.6 | E76, E77 | | 1-bit lookup-table kernel (Four-Russians / T-MAC style): 32 rows per group, one 16-entry table per 4 activations built straight from the float activations, ~8 NEON ops + 2 loads per 128 multiply-adds vs 16 + 4 | Bonsai-8B 3.77 → 5.1-5.4 tok/s (with E82, E88) | E81 | Q8_0 attention with a threshold: all K dots first with sdot , so the softmax maximum is known up front, then expf and V work skipped for rows below max + ln(1e-4) | a 1,000-token answer no longer halves the speed (2.50 → ~4 tok/s at the end) | E86 | | Guided matmul scheduling: each thread grabs remaining / (2 × threads) rows, so a slow or preempted thread leaves a short tail instead of ~28 ms per token of idle cores; single-row element-wise ops split by columns (ggml left them all on thread 0) | 2B 6.26 → 6.54 tok/s; 0.8B +1.4 % | E63, E64 | Memory tricks: - Pruned LM head: keep only the ~49-66 K most useful vocabulary rows, chosen by token frequency on English, code and 7 other languages. The other logits are scattered to −inf. Saves 0.2-0.5 GB and misses ~1 % of English tokens. - Repack cache on flash ( GGML_REPACK_CACHE ): repacked weights are written to a file once and mapped afterwards. Models load in seconds. Headless mode (tools/tv-llm-mode.sh on|off ): a small foreground activity, then the launchers, Google services and other heavy apps are force-stopped, and svc power stayon keeps the animated screensaver (17 % of a core) away. Nothing persistent except stay-awake, which off restores. MemAvailable ~1000 → ~1300 MB. A fork-free guard (tools/tv-watch.c ) replaced a shell loop that started ~25 processes a second and cost 10 % of the speed. Expert pager for mixture-of-experts models (ggml-cpu/moe-pager-tv.inc ): - Experts live on flash: each expert tensor is overlaid with empty anonymous memory, and right after a layer's routing the missing experts are read with parallel O_DIRECT straight into their own addresses; a per-layer LRU pool (43 of 128 experts) keeps the rest. Kernels see an ordinary tensor. The GGUF's own expert bytes are hole-punched once the repack cache holds them, so the model costs one copy of its weights on flash. - Cache-aware routing ( GGML_MOE_BIAS ): a small bonus for experts already in RAM cuts misses from 13 % to ~2 % for a measured quality cost (arXiv 2412.00099 has the idea on other models). Fewer experts per token (LLAMA_MOE_USED=6 ) is the other dial. - Warm start from a list of each layer's most used experts, and pre-gated prefetch: layer L+2's router applied to layer L's hidden state, predicting what the router will actually pick (sigmoid, the model's own selection bias and the cache bonus), loads ~60 % of misses before they are needed. - Why answers looped, and the fix: 2-bit experts break long-form coherence after ~250-300 words, and routing bias amplifies it. DRY sampling only hides it (the model paraphrases its loop); a repeat penalty of 1.15 over 256 tokens brings degenerate long answers from 12/13 to 1/13 on the TV (E99, E100). Weight streaming for models bigger than RAM (ggml-cpu/stream-tv.inc ): - What stays in RAM: a chosen part of the weights stays resident. The rest lives in "stream zones". - Prefetch: a hook on every matmul records the order the weights are used. Reader threads then fetch zone weights ahead of time with parallel 8 MB O_DIRECT reads straight into the weights' own addresses, at the flash's full 278 MB/s. - Correctness: every compute thread waits until its weight is completely loaded, so a wrong prediction costs time, never correctness. - Drop-behind: each zone weight is freed right after use. - Verify mode: GGML_STREAM_VERIFY=1 compares every streamed tensor with the file. It found two silent-corruption bugs, which are fixed and documented in E48 and E50. - Speculative decoding: with the model's own MTP head, verifying 2 tokens costs about one pass over the weights when the pass is flash-bound, so it pays off (+69 %). On the in-RAM 2B it lost (E31). Profiling (env switches in the patch): GGML_OPPROF=1 per-op table, GGML_OPPROF=2 a node-by-node, per-thread trace of one graph (tools/optrace.py ), LLAMA_UBATCH_PROF / GGML_CPU_PROF phase timings. They showed where a small model's time really goes (E57): ~71 % Q4_K matmuls, and only ~2 ms per token of barriers and scheduling. Tooling: - Streaming GGUF rewriter: adds a pruned head, extra tensors and metadata in one sequential pass, so Hugging Face → Mac → TV pipelines never store the model on the Mac. - MTP fetcher: range-reads only the MTP tensors from safetensors. - Memory guard: wraps every heavy run on the TV. - Benchmark and perplexity harnesses, a model screener that streams any Hugging Face GGUF to the TV and scores it ( tools/screen-small.sh ), and a live token-stream web UI. You need: the TV box (or another ARMv8.2 Android device) with network ADB, adb , zig (for static aarch64 musl builds), cmake + ninja , python3 , and uv (for numpy). export ADB_SERIAL=:5555 # every script talks to the box through this adb connect $ADB_SERIAL # 1. llama.cpp v0.8.0 + the patch, cross-compiled for the TV git clone https://github.com/ggml-org/llama.cpp third_party/llama.cpp cd third_party/llama.cpp && git checkout 8642541 && git apply ../../patches/llama.cpp-v0.8.0-tvbox.patch cmake -S . -B build-tv -G Ninja -DCMAKE_SYSTEM_NAME=Linux -DCMAKE_SYSTEM_PROCESSOR=aarch64 \ -DCMAKE_C_COMPILER=$PWD/../../zigcc/cc -DCMAKE_CXX_COMPILER=$PWD/../../zigcc/c++ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_EXE_LINKER_FLAGS=-static -DBUILD_SHARED_LIBS=OFF -DGGML_NATIVE=OFF \ -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16 -DGGML_OPENMP=OFF -DLLAMA_OPENSSL=OFF -DLLAMA_BUILD_TESTS=OFF ninja -C build-tv llama-server llama-cli llama-bench llama-perplexity llama-quantize cd ../.. adb -s $ADB_SERIAL shell mkdir -p /data/local/tmp/llm/patched25 /data/local/tmp/llm/patched adb -s $ADB_SERIAL push third_party/llama.cpp/build-tv/bin/llama-{server,cli,bench,perplexity} /data/local/tmp/llm/patched25/ adb -s $ADB_SERIAL push third_party/llama.cpp/build-tv/bin/llama-quantize /data/local/tmp/llm/patched/ # used by tools/tv-quant.sh zig cc -target aarch64-linux-musl -O2 -static -o gguf_recut tools/gguf-recut/gguf_recut.c && adb -s $ADB_SERIAL push gguf_recut /data/local/tmp/llm/ # 2. models: rebuilt from their Hugging Face sources straight onto the TV (see below) tools/rebuild-models.sh 2b # also: 08 4b b8 tb8 q9 mb lm caches # 3. the live chat page (default: the 2B on patched25 with fp16 state), then open http://:8090 from any device on the network tv-llm-ui/start.sh tv-llm-ui/start.sh stop Other models: # Moon-Beam-8B-A1B MoE, ~7.2 tok/s (first: `tools/rebuild-models.sh mb`, needs a native Mac build and ~9 GB on the Mac while it runs). # start.sh turns on headless mode, the expert pager, routing bias, prefetch and the repeat penalty; POOL / BIAS / EXPERTS override them. MODEL=mb-q2-h60.gguf tv-llm-ui/start.sh # Ling-mini-2.0 16B MoE, ~5.1 tok/s (first: `tools/rebuild-models.sh lm`) MODEL=lm2-q2-h60.gguf tv-llm-ui/start.sh # Bonsai-8B 1-bit fully resident, ~5.2 tok/s (first: `tools/rebuild-models.sh b8`; the repack cache is written on first start) MODEL=b8-q1-hml.gguf tv-llm-ui/start.sh sh /data/local/tmp/llm/tv-llm-mode.sh off # on the TV, afterwards: launcher and screensaver back # Qwen3.5-0.8B, ~16 tok/s (first: `tools/rebuild-models.sh 08`, needs a native Mac build of llama-quantize) MODEL=q08-q4k-h65p8.gguf tv-llm-ui/start.sh # The 8B / 9B numbers below were measured with the build of that time (patched16); the current build has the same # code plus §15-17. GGML_CHUNK_GUIDED=0 in SENV restores the old matmul scheduling. No fp16 state with MTP. # Bonsai-8B 1-bit with streaming and no headless mode, ~1.2 tok/s (first run: `tools/rebuild-models.sh b8 caches`) MODEL=b8-q1-hml.gguf BIN=patched25 EXTRA="-ctk q8_0 -ctv q8_0 -fa on -cram 0" \ SENV="GGML_REPACK_CACHE=/data/local/tmp/llm/cache/b8c GGML_REPACK_RESIDENT_MB=800 GGML_STREAM_MODE=cache GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh # Qwen3.5-9B with MTP, ~0.27 tok/s MODEL=q9-iq2-h65-mtp.gguf BIN=patched25 EXTRA="-nr -ctxcp 0 -cram 0 -ub 16 -b 64 --spec-type draft-mtp --spec-draft-n-max 1" \ SENV="LLAMA_STREAM_RESIDENT_MB=600 LLAMA_STREAM_PIN=output.,blk.32. GGML_STREAM_WINDOW_MB=64" tv-llm-ui/start.sh Everything on the box lives in /data/local/tmp/llm and runs as the adb shell user. The only system setting touched is stay-awake in headless mode (svc power stayon ), which tv-llm-mode.sh off turns back off. adb shell rm -rf /data/local/tmp/llm removes it all. The model files (0.5-4 GB each) are not in this repo. tools/rebuild-models.sh recreates every one from public Hugging Face files, streaming them through tools/gguf_build.py onto the TV. The SHA-256s below are of the files used in the experiments. Rebuilds of the 4B, Bonsai-8B and 9B files were checked byte-identical. Ready-made downloads. The best file of each size class is on Hugging Face exactly as the TV runs it. They need the patched llama.cpp above; stock llama.cpp can't load the pruned head. Each card has the run command and the measured numbers: | Model | Hugging Face | File | |---|---|---| | Qwen3.5-0.8B, 16.8 tok/s | hyperionaidev/Qwen3.5-0.8B-tvbox-GGUF | q08-q4k-h65p8.gguf | | Qwen3.5-2B, 7.7 tok/s | hyperionaidev/Qwen3.5-2B-tvbox-GGUF | w-q4k-h65p8.gguf | | Ternary-Bonsai-4B, 4.3 tok/s | hyperionaidev/Ternary-Bonsai-4B-tvbox-GGUF | tb4-q20-hml.gguf | | Moon-Beam-8B-A1B (MoE), 7.2 tok/s | hyperionaidev/Moon-Beam-8B-A1B-tvbox-GGUF | mb-q2-h60.gguf + moe-warm-moonbeam.txt | | Bonsai-8B (1-bit), 5.2 tok/s | hyperionaidev/Bonsai-8B-tvbox-GGUF | b8-q1-hml.gguf | | Ling-mini-2.0 (16B MoE), 5.1 tok/s | hyperionaidev/Ling-mini-2.0-tvbox-GGUF | lm2-q2-h60.gguf + moe-warm-ling.txt | Download with hf download hyperionaidev/ --local-dir models/ , then adb push the files to /data/local/tmp/llm/ , with moe-warm-*.txt next to them. That replaces the 08 / 2b / 4b / mb / b8 / lm targets. For the 8B and 16B models, the first start writes the repack cache. iPad Air downloads. Every model run on the iPad, where to get it, and its speed there. The three MoE files are our own builds, laid out for the iPad's expert pager and lookup-table kernel: they need the iOS 12 patch (ipad-air/ ), and any other llama.cpp loads them but gets scrambled experts. The rest are used exactly as published; the Bonsai models' fast layouts are built in RAM at load time. 53 experiments in all (E108-E160, §21): | Model | Download | iPad speed (tok/s) | Peak RSS | Experiments | |---|---|---|---|---| | Ling-mini-2.0 (16B MoE) | hyperionaidev/Ling-mini-2.0-ipad-GGUF: lm2-q2-h60-lut.gguf (SHA-256 3e464310…4e50bd ) | 2.6 unbiased; 3.3 with bias 0.1 (+2.6 % perplexity); 4.5 with bias 0.3; 5.2 adding --experts 6 | 556-576 MB | 4 (E157-E160) | | Moon-Beam-8B-A1B (MoE) | hyperionaidev/Moon-Beam-8B-A1B-ipad-GGUF: mb-q2-h60-lut.gguf (SHA-256 acf1cb56…94b976 ); the experts-last file without the repack, about 4 % slower, is hyperionaidev/Moon-Beam-8B-A1B-ipad-xl-GGUF: mb-q2-h60-xl.gguf (aa919400…871d7e ) | 3.5-3.6; 3.7-4.0 with 7 experts (+0.4-1.9 %); 5.0 with 4 experts (+27 %) | 600-625 MB | 19 (E138-E156) | | Bonsai-8B (1-bit, dense) | prism-ml/Bonsai-8B-gguf | 0.08: doesn't fit in RAM, read from flash | 629 MB | 4 (E134-E137, with the first Moon-Beam attempts) | Bonsai-4B (1-bit), -c 512 -b 64 | prism-ml/Bonsai-4B-gguf | 9.4-11.4; 9.2-10.3 through long chats | 615-682 MB | 5 (E129-E133) | | Bonsai-1.7B (1-bit), the default | prism-ml/Bonsai-1.7B-gguf | 23.7-25.8; 18-22 through long chats | 605 MB | 11 (E118-E128) | | SmolLM2-360M-Instruct Q4_K_M | bartowski/SmolLM2-360M-Instruct-GGUF | 10.9-12.5 | 269-316 MB | 10 (E108-E117: first runs, the small models and the web chat) | | Gemma 3 270M-it Q4_K_M | unsloth/gemma-3-270m-it-GGUF | 15.4-19.0 | 248-294 MB | | | SmolLM2-135M-Instruct Q4_K_M | stock Q4_K_M | 18.6-29.3 | 141 MB | | | Qwen3.5-0.8B Q4_K_M | unsloth/Qwen3.5-0.8B-GGUF | about 0.2: thrashes | 506 MB | | | stories15M q4_0 | stock | about 290 in llama-simple | | File on the TV | Target | Built from | SHA-256 | |---|---|---|---| w-q4k-h65.gguf (Qwen3.5-2B) | 2b | unsloth/Qwen3.5-2B-GGUF BF16 + imatrix_unsloth , MTP block from Qwen/Qwen3.5-2B , quantized on the TV, head data/head-ids-ml60k.txt | 16057be3…3285b | w-q4k-h65p8.gguf (Qwen3.5-2B, default) | 2b | the file above re-cut on the TV with data/head-ids-ml60k-p8.txt (65,672 rows, so the head is repacked, §17) | 631a27ae…54fd11 | w-q4k-h45.gguf (option) | 2b45 | the same, 45,654-row head data/head-ids-ml40k.txt (byte-identical to E29's H45 file) | a50eadc5…664d5b | q08-q4k-h65.gguf (Qwen3.5-0.8B) | 08 | unsloth/Qwen3.5-0.8B-GGUF BF16 + its imatrix_unsloth , pure Q4_K, head data/head-ids-ml60k.txt , quantized on the Mac | 901f3838…8523a6b | q08-q4k-h65p8.gguf | 08 | the file above re-cut on the Mac with data/head-ids-ml60k-p8.txt (§17) | 7dd2aac5…52fc232 | tb4-q20-hml.gguf | 4b | prism-ml/Ternary-Bonsai-4B-gguf Q2_0_g64, head data/head-ids-qwen3-ml64k.txt | a02d805f…3fe24 | b8-q1-hml.gguf | b8 | prism-ml/Bonsai-8B-gguf Q1_0, same head | fc297cb3…0d42a | tb8-q20-hml.gguf | tb8 | prism-ml/Ternary-Bonsai-8B-gguf Q2_0_g64, same head | 01b1e0ef…93e06 | mb-q2-h60.gguf | mb | tablinum/Moon-Beam-8B-A1B-GGUF-Preview Q8_0, requantized on the Mac: experts Q2_K, the rest Q4_K, own imatrix from data/corpus , head data/head-ids-ling-60k.txt ; expert bytes hole-punched on the TV after the repack cache (cache/mbq2.* ) is written | 2d78fcdd…1c34ee | lm2-q2-h60.gguf | lm | bartowski/inclusionAI_Ling-mini-2.0-GGUF Q4_K_M + its imatrix: experts requantized to Q2_K on the Mac, the rest Q4_K, head data/head-ids-ling-60k.txt ; punched on the TV after the repack cache (cache/lmq2.* ) is written | c089aeb0…e8d7 | q9-iq2-h65-mtp.gguf | q9 | unsloth/Qwen3.5-9B-GGUF UD-IQ2_XXS, MTP block from Qwen/Qwen3.5-9B (Q4_0), 65K head | 7766478e…8dcbc | cache/b8c.* , cache/tb8.* | caches | written by the first run with GGML_REPACK_CACHE | The head-id lists in data/ were made by tools/vocab_select.py (Qwen3.5 tokenizer) and tools/head_select_tok.py (any tokenizer, via llama-tokenize ), from the corpus tools/fetch_corpus.py downloads (Wikipedia + Project Gutenberg). | Path | What | |---|---| EXPERIMENT.md | the full write-up: §1-8 the first tiny model, §9-10 llama.cpp and the first flash streaming, §11 the 2B push, §13 the 4B push, §14 the 9B push, §15 the 0.8B speed push, §16-17 the 2B rounds 3-4, §18 the 4B round 2 (arXiv / GitHub scan, 2-bit ternary layouts), §19 the 8B push (headless mode, LUT kernel, Moon-Beam MoE with an expert pager, why long answers loop), §20 bigger than 9B (Ling-mini-2.0 16B, and why LFM2.5-24B and Qwen3.6-35B don't fit) | patches/llama.cpp-v0.8.0-tvbox.patch | every change to llama.cpp: pruned head (output_ids ) for Qwen3 / Qwen3.5 / Granite 4.0 H, the kernels above (ggml-cpu/repack-*-tv.inc ), fused DeltaNet (state read by id, optional fp16 state, one-token conv step), the 1-bit LUT kernel, Q8_0 attention, the MoE expert pager with routing bias and prefetch (ggml-cpu/moe-pager-tv.inc ), a pruned head for bailingmoe3 , embedding rows from the resident head, repack cache and streaming engine (ggml-cpu/stream-tv.inc , loader hooks), padded small batches, op profiler, pruned-head-aware perplexity, Vulkan tuning for Mali | tools/ | GGUF rewriter / head pruner / re-cutter, vocabulary selection, corpus and MTP fetchers, model rebuild script, TV memory guard and RAM helper, benchmark sweeps, kernel benches and exactness checks (kbench/ ), flash benchmark | tv-llm-ui/ | the live token-stream page for llama-server (confidence colours, top-3 alternatives, speed, box stats), its launcher and the benchmark harness | tiny-llm-web/ | the llama2.c story server + TV page from the first experiment | bench/ | results.jsonl (every chat benchmark: prompts, outputs, speeds) and raw/ (logs of every experiment) | data/ | head-id lists and the two evaluation texts | zigcc/ | zig wrappers for cross-compiling static aarch64 binaries | ipad-air/ | device #2: cross-compile for iOS 12, smolchat (terminal chat and Safari web page), deploy and benchmark scripts (README, §21) | screenshots/ | the web pages running on the TV | - On this box the 2B at 4 bits beats every bigger model at 1-2 bits. Post-training quantization below ~2.5 bits breaks normal models (E22, E23, E35). Models trained for 1-bit or ternary hold up, but at 4-8 B they don't fit without streaming. - Past RAM, flash bandwidth sets the speed. The kernel's LRU is the worst possible policy for a cyclic scan bigger than the cache: it re-reads everything every token. Keep a fixed resident set and stream the rest yourself. - Batching is free when you are flash-bound. Prompts and speculative verification cost about one weight pass, whatever their size. - Measure, then verify. Two streaming bugs gave plausible speeds and wrong numbers; only comparing tensors with the file, and perplexity with a Mac reference, caught them. - Small models are mostly vocabulary. Qwen3.5-0.8B's 248K-row head is a third of its weights; pruning it to 65K rows plus all-Q4_K took it from 7.5 to 12.1 tok/s (E55). Small models also lose more to 4 bits (+7 % perplexity vs +2.3 % for the 2B). - Simulate before you write a kernel. Activation sparsity (E58) and a low-rank head (E60) both looked promising on paper; a quality simulation on the Mac killed each in under an hour. - On an in-order core, count NEON ops as well as bytes. Tighter packing saved 20 % of the bytes and cost 2.5× the ops; undoing it made the 4B 33 % faster in chat (E76, E77). The 4-bit kernel is the opposite case, limited by RAM (E27), so the trick does not help the 2B or 0.8B. - Check that every big matrix reaches the fast kernel. The 2B's head (65,668 rows, ~10 % of each token's weights) silently ran on the generic kernel for weeks because ggml only repacks row counts divisible by 8; four extra rows gave +5.8 % (E71, E72). The load log's repack buffer size gives it away. - Most of the RAM was Android's, not the model's. Force-stopping the launchers and Google services for the duration (nothing persistent) let Bonsai-8B sit fully in RAM: 1.2 → 3.6 tok/s before any kernel work (E80). An animated screensaver and a guard that forked 25 times a second each cost another 10-15 % (E82, E88). - For a mixture-of-experts model the flash is a cache miss, not a stream. A ~6 ms miss per expert means the pool hit rate decides the speed; routing toward cached experts and prefetching what the next router will pick turned 4.4 into 7.2 tok/s. - Perplexity can't see loops. It scores one token at a time on clean text; the 2-bit model scored fine and still fell into loops in 12 of 13 long answers. Score long free-running generations too ( tools/loop_stats.py ), and compare models with different tokenizers in bits per byte, never in perplexity. - Past 9B, count resident bytes, not parameters. The ceiling for a mixture-of-experts model here is the non-expert weights that must stay in RAM plus a useful expert pool. Ling-mini-2.0 (16B) keeps 279 MB resident and runs at 5 tok/s; Qwen3.6-35B-A3B would need ~870 MB before caching a single expert (§20). - Guard the box. After two early kernel panics, every heavy run goes through a wrapper that kills it below 180 MB of free memory. MIT. The llama.cpp patch is a derivative of llama.cpp (MIT). Model weights keep their own licenses, and the evaluation texts are Wikipedia excerpts (CC BY-SA 4.0). Built on llama.cpp and llama2.c. Models from Qwen, unsloth (GGUFs and importance matrices), PrismML (Bonsai), Google (Gemma) and stories260K. The evaluation texts are excerpts from Wikipedia (CC BY-SA 4.0). The 🤏 logo is the pinching-hand emoji from Google's Noto Emoji (Apache 2.0).