Intel-backed AI chip startup SambaNova breathes new life into aging Nvidia GPUs in latest benchmarks

Intel's big bet on SambaNova appears to be paying off in a big way. This week, the AI chip startup shared benchmark results showing its latest generation of AI acceleration, which combines Nvidia GPUs and the company's accelerators, beating GPU-only inference platforms by a wide margin. The testing, conducted by the AI benchmarking gurus at Artificial Analysis, showed SambaNova's SN50-series accelerators, announced in February, churning out 763 tokens a second in MiniMax M2.7 at short context lengths (10,000 input tokens) — several times faster than competing inference providers running on GPUs alone. Meanwhile, for longer context lengths, the company says that its platform is able to sustain more than 450 tokens a second. This feat was accomplished by combining Nvidia GPUs with SambaNova Reconfigurable Dataflow Units (RDUs) to form a heterogeneous inference platform. Specifically, the computationally intensive prefill phase of the inference pipeline, during which prompts are processed and key value caches are generated, was handled by four Nvidia H200 GPUs. Meanwhile, memory-bandwidth-bound decode operations, where output tokens are generated, were done on a single SambaNova rack containing 16 SN50 accelerators. Disaggregating prefill from decode has become a key lever for reducing token costs for long-running AI agents, like code assistants. Nvidia initially demonstrated this with its NVL72 rack systems, by varying the ratio of GPUs used for prefill versus decode. The company further disaggregated this with its Groq-based LPX racks revealed at GTC this spring. Since then, just about everyone from AMD to AWS and Cerebras has announced some kind of disaggregated or heterogeneous inference platform using one or more accelerators. With SambaNova's latest performance figures, the startup hopes to demonstrate how customers can breathe new life into their aging GPU fleets by using its systems as decode accelerators. And because its systems are air-cooled, they can be deployed in existing datacenters — something that can't be said of Nvidia's latest generation of Rubin GPUs, which absolutely need liquid cooling. SambaNova plans to show off even more powerful inference configs, with 128 and eventually 256 accelerators to demonstrate its ability to maintain high token generation rates at high throughput. As we’ve previously explored, this is something that GPUs alone have historically struggled with and one of the key drivers behind Nvidia’s Groq acquihire late last year. The results come just a month after SambaNova and Intel announced Vector Core Compute would be among the first to deploy the combined GPU + RDU offering with TogetherAI as their first large-scale customer. Ramping production of any chip isn’t a cheap prospect, but for its fifth-gen part, capital shouldn’t be an issue. On Wednesday, SambaNova completed the first close of a $1 billion Series F funding round led by General Atlantic, giving the AI chip startup an $11 billion valuation. ® [Image text:] sambanova sambanova sambanova