Why harnesses matter for energy (and token) efficiency

Why harnesses matter for energy efficiency By Chad Gibson, Co-founder & CEO Key findings - Harnesses cache differently. We saw 1.6% to 11.1% uncached input across clients. "Cached" is the share of input tokens served from the prompt cache instead of computed again. At 92% cached, 8 tokens per 100 are uncached, versus 3 at 97%: about 2.7 times the prefill work. - A cache miss roughly doubles a request's energy. Here, a miss means a request with little or none of its prompt cached (under 10%). On GLM-5.3, it's about 2.5 times as much. - Deep sessions make misses expensive. A re-read means recomputing that part of the prompt from scratch. Past 128k tokens, one turn that re-reads more than half its prompt costs as much prefill as about 600 normal turns at that depth. - Most of this is fixable. Every cache break in our lab tests came from a harness editing what it had already sent, forcing a re-read from the first difference onward. At Neuralwatt, we can see how the agent harness calling a model affects the energy it uses. I've been looking through a month of traffic on our cloud, and the harness seems to decide a surprising share of how much work our GPUs do. We usually talk about inference efficiency in terms of the model (how many active parameters, its attention mechanism) or the hardware (H200 vs B200 vs B300). The client deserves a closer look. Almost all of our traffic is agentic, with far more input than output. A typical coding-agent turn sends 100k tokens or more of conversation and gets back a few hundred tokens. Most of that input is the previous turn's prompt with a little added at the end. If the earlier text is unchanged, the server already has its key-value (KV) cache and only has to process the new part. This is prompt caching, our biggest source of efficiency. Serving a cached token is far cheaper than computing it again. Caching needs an exact prefix. A cache break happens when the harness changes something it already sent. We then recompute everything from the first difference onward. A change near the beginning, even a timestamp, means recomputing nearly the whole prompt. A change near the end costs little. The harness controls how the prompt is built. Same model, same price, different cache rates Every request tells us which client sent it. In the 30 days from September 8 to October 7, 2026, we saw hundreds of different clients, including open-source coding agents such as OpenCode, Oh My Pi (omp) and Pi. The mix also included editor integrations, the OpenAI SDKs and custom apps. OpenCode alone is used by about one in four of our customers. For the most widely used clients, here is the share of input tokens we could not serve from cache: Every client here caches the large majority of its input. Those small differences still matter: a client at 97% cached sends us three uncached tokens per hundred; a client at 92% sends eight. That is 2.7 times as much prefill work for the same conversation. Hit rate also depends on the workload, not only the harness. Longer sessions score higher, because each turn's new text is a smaller share of the prompt, and about half of deepseek-harness's lead comes from users who run very long sessions. A cleaner measure of the harness is how often a turn breaks the cache: about 1 turn in 100 for deepseek-harness, against about 3 in 100 for OpenCode and Pi. We see both OpenCode 1.x and 2.x in our traffic, so the bar combines them. 2.x looks better alone (2.2% uncached against 3.0%), mostly because of the model mix. On the same model, they cache about equally well. 2.x fixes most of the 1.x cache breaks we found in the lab. That difference doesn't show up in production, and we don't yet know why. What a miss costs in energy We bill on measured energy, so we can see the cost directly. Over the week of October 1 to 7, 2026, we compared requests on the same model with the same prompt size (about 50k tokens) and similar output length. Here is the extra energy each request used compared with a fully cached one: A mostly uncached request takes roughly twice the energy of a cached one of the same size. On GLM-5.3 it is closer to two and a half times. Prefill also competes with generation for the same GPUs, so wasted prefill slows down everyone else's responses on that server. Cache misses get expensive deep in a session We followed individual sessions turn by turn on our three busiest models for the week of October 6 to 13, 2026. We looked at every turn that landed on the same server within five minutes of the previous one, where the cache should have been warm. Most cache breaks were small: a few thousand tokens near the end of the prompt got recomputed. Past 128k tokens of context, about 3 turns in every 1,000 re-read more than half of the prompt from scratch. Those few turns account for 75% of all the re-read tokens at that depth. Each one re-reads about 230k tokens on average. A normal turn at that depth adds a median of about 370 new tokens, so a single one of these costs as much prefill as roughly 600 ordinary turns. The delay is noticeable too. Here is the median time to first token for a normal turn compared with a turn that had to re-read most of its prompt: I pay more attention to how deep into a session the cache breaks than how often it breaks. Changing the beginning of the prompt once, early on, costs little. Do it at turn 200 of a long session and you get a large, visible pause and a large energy cost. Long agent sessions account for most of our tokens, so this adds up. We can't always tell from the server side whether a harness edited the prompt or the cache was evicted under memory pressure. These numbers include both. Our lab tests below show that harnesses cause the early-prompt edits we could reproduce; eviction likely accounts for some of the rest. What breaks the cache I looked for the causes in two places: the lab, where we could see every token, and production, where we only see request metadata. In the lab We ran four popular open-source agents (OpenCode, Pi, omp and Kilo Code) through a recording proxy against DeepSeek V4.1 Flash and GLM-5.3. We used the latest releases as of October 6, with default settings: OpenCode 1.18.34, Pi 1.0.4, omp 18.6.3 and Kilo Code 7.8.3. We then repeated the session on OpenCode 2.0.29 from the 2.x line. Each worked through the same scripted session on a small Python project: plan, then build, switch reasoning effort and back, spawn a sub-agent, sit idle, get restarted, connect and disconnect an MCP server, add a skill, and run long enough to compact. We compared every request with the one before it, both as JSON and as the exact tokens the server renders. Of 539 requests that followed an earlier one in the same conversation, 88% were pure appends: the previous prompt was an exact prefix of the new one. The other 65 broke the cache. Every break came from the harness changing something it had already sent; none came from our chat templates. Idle time caused no breaks in any of the four. Neither did restarting the harness and resuming the session, or using sub-agents. Two causes break the cache without any action from the user: - OpenCode's plan-mode reminder. Once you've used plan mode, OpenCode attaches a reminder ("Your operational mode has changed from plan to build") to each new user message. It removes that reminder from the message on the following turn. Every turn then re-reads the whole previous turn, including its tool calls and results. In our short sessions, that was 330 to 2,340 tokens a turn. The cost grows with the size of the turn. OpenCode 2.x keeps the reminder in its own message, so it no longer triggers a re-read. - omp's pruning of re-read files. When a file is read a second time, omp replaces the earlier copy in history with [Superseded by a newer read of this file] . In our run, that one edit landed 62% of the way into the prompt, forcing a re-read of 5,846 of 15,520 tokens. omp also prunes "uneventful" tool results, and after 90 minutes idle it prunes at any depth. Settingcompaction.supersedeReads: false andcompaction.dropUseless: false keeps the cache. Pi and Kilo Code had no breaks of this kind. The other causes depend on what the user does. How much they cost depends on where the prompt changes: - Switching reasoning effort. Both templates write the effort level in the first few tokens of the prompt: Reasoning Effort: Max on GLM-5.3, a number from 1 to 100 on DeepSeek V4.1. The first difference was at token 8 on both models, so a switch re-reads everything. Only that one value changes; its length and everything after it stay identical. - Connecting or disconnecting an MCP server. OpenCode 1.x and Kilo Code insert new tools in alphabetical order, in the middle of the tool list. OpenCode 2.x leaves the tool list alone and appends a note, so both connecting and disconnecting were pure appends. Pi adds a tool and edits its system prompt. omp appends a section to its system message. On GLM-5.3, the tool list is rendered before the system prompt, so the change lands very early: about 1,300 tokens in for OpenCode 1.x. - Adding a skill. All four list skills in the system prompt. The first difference landed between token 461 and token 2,948 on DeepSeek V4.1, and later on GLM-5.3, where the tools come first. - The date changing. OpenCode 1.x and Kilo Code put today's date in the system prompt, about 2,000 tokens in. OpenCode 2.x keeps the session's original date and appends a note when the date changes. omp puts it in the first user message. Pi doesn't send a date. For clients that overwrite the date, crossing local midnight means re-reading nearly everything once. - Compaction. Each harness does it differently. Pi keeps the last ~20k tokens word for word but rewrites the summary near the beginning. The prompt hardly shrinks, and most of it gets re-read anyway. With the same 40k-token window, OpenCode 2.x compacted three times and re-read 67,850 tokens in total. 1.x compacted once and re-read 29,776. Switching back to a setting we'd used earlier in the session was cheaper than I expected. The server still holds that earlier version of the prompt. Switching effort back hit 10.6k to 27.5k cached tokens in our runs. A toggle costs one full re-read the first time each new state is used, not on every flip. In production We don't log or keep the text of customers' prompts or responses. Everything in this section comes from request metadata. The token-level analysis above comes from our own scripted test sessions, not customer traffic. That metadata records the reasoning effort each request asked for and how many tools it sent. We used it to check two of these causes against real sessions from October 6 to 13, 2026, on DeepSeek V4.1 Flash, GLM-5.3 and GLM-5.3 Flash. We used consecutive turns of at least 8k tokens on the same server within five minutes. To keep a few heavy accounts from driving the result, we left out the three customers with the most re-read tokens in each group. Changing the tool list or effort level makes a cache break more than 20 times as likely and costs, on average, about 40% of the previous prompt. Tool-list changes happen on about 0.5% of turns, yet account for 19% of all re-read tokens on these models. That share is concentrated: three accounts make up about two thirds of it. The per-turn effect holds across everyone else. The remaining three quarters of re-read tokens come on turns where neither the effort nor the tool count changed. That "nothing we can see" bucket holds the causes we can't see from the server: the reminder in OpenCode 1.x, omp's pruning, skills, the date, compaction and cache eviction. Next, I'd like us to record where each prompt first differs from the previous one. That would let us split the bucket up without reading any prompt text. How deepseek-harness keeps the cache intact In our lab capture, every one of deepseek-harness's follow-up requests was a pure append to the previous one, even after the process was restarted and the session resumed. Reading its source, I found a few practices worth looking at: - Requests are rebuilt from an append-only session log. Retries and resumes produce byte-identical prompts. - The system prompt is fixed and minimal. It leaves out the date, git status and OS. It is written once and left alone. - Changing state goes after the history. When the approval policy or available skills change, the harness appends a short snapshot saying it supersedes earlier ones. It leaves the system prompt and older messages alone. - The tool list stays sorted and identical in every mode. Prose instructions enforce modes without adding and removing tools. - Compaction happens late. The summarizing call reuses the existing prefix, so even compaction is mostly a cache hit. Summarizers that re-serialize the transcript into a single new message start cold. In the largest such workload we saw, those calls were only 22% cached. - Every change gets reviewed for cache impact. Each package in the repo carries a short "KV cache effect" section. What you can do If you build or configure a harness: - Keep the start of the prompt identical every turn. Add new messages and results from tools at the end of the conversation. Describe changes to settings there too, leaving the earlier text untouched. - Don't put the date, git status or anything else that changes into the system prompt. - Keep the tool list stable and sorted. Add new tools at the end. - Compact late, and have the summarizer call reuse the conversation's exact prefix. - Send a stable session id. On Neuralwatt, X-NW-Conversation-ID lets us keep a session on the server that already holds its cache, even across compaction. If you use a harness: - Pick a reasoning effort per session rather than switching mid-session. - For omp, consider compaction.supersedeReads: false . - Settle settings like effort, MCP servers and skills before the session gets long. That is when a cache break hurts most. See it in your own sessions Everything in this post comes from data we record for every request, and you can see the same data for your own work. Session Insights in the Neuralwatt dashboard shows each agent session turn by turn: how much of each prompt came from cache, how much had to be computed cold, the time to first token, and the measured energy and cost. It never shows prompt or completion text. This is one of my own sessions from September 26, 2026: 35 turns of a coding agent on DeepSeek V4.1 Flash, growing from 29k to 257k tokens of context. 95% of its prompt tokens came from cache, and the small red caps are the new text each turn added. A cache break shows up as a tall red bar, so if you switch reasoning effort or connect an MCP server halfway through a session, you can see what it cost. You can use it to compare harnesses or settings on your own workload, and to check that a change actually kept the cache warm. What we're doing on our side We route each session back to the server that holds its cache. On our busiest models, that already keeps 88–96% of each session's tokens on its home server. When a session has to move, our shared KV cache pool lets the new server load the prefix instead of recomputing it. We recently measured how much more a smarter prefix-aware router would recover on top of that: a tenth to two tenths of a percentage point of hit rate. Most of the remaining waste comes from how prompts are built. Next, we plan to send fixes and suggested defaults upstream to the open-source harnesses our customers use most. We also want to research ways for the server to reuse cached work after a small edit near the beginning of the prompt, such as the effort line or the date. That would help with clients we can't change ourselves. Why this is worth measuring A cache break doesn't show up as an error. You still get your answer, just later and with more energy spent on it. Measuring the energy of every request showed us how much that waste adds up. Most of it turned out to be fixable with a setting or a small change to how a harness builds its prompt. AI runs on energy. The more of it we can see, the more of it we can stop wasting, whether that's a single developer's session or the industry's next data center. Methodology notes. We identify clients by the name in each request's User-Agent header. SDK and runtime labels (Node.js apps, OpenAI SDK apps) cover many different applications. We only show clients used by many customers and never identify individual customers. Servers cache prompts in fixed-size blocks, so even a pure append can leave a few tokens at the end of the previous turn to re-read. We count those as uncached input, adding under 0.1 percentage points for the coding agents in Figure 1. Cache rates are the share of prompt tokens served from cache across all models from September 8 to October 7, 2026. Energy is measured GPU energy apportioned to each request; the cached vs uncached comparison holds prompt size and output length fixed and covers October 1 to 7, 2026. The deep-session and cause analyses cover October 6 to 13, 2026, on DeepSeek V4.1 Flash, GLM-5.3 and GLM-5.3 Flash, following each session's consecutive turns on the same server within five minutes. Chad Gibson is the co-founder and CEO of Neuralwatt. He spent more than 20 years at Microsoft before investing in early-stage startups and co-founding Neuralwatt in 2024.