AMD's Mext buy shows how AI could solve the RAM shortage it created

With no end in sight to the memory crunch, AMD thinks that AI, the main cause of the shortage, could be part of the solution. This week, the House of Zen acquired predictive memory startup Mext for an undisclosed sum, setting the stage for a world where bots decide which data to put into RAM and which to store in less-expensive flash. Founded in 2023, the Mext proactive memory platform uses machine learning algorithms and learned heuristics to proactively offload "cold" memory to flash storage, and, based on data access patterns, restore it before its needed again. Modern flash arrays are already approaching main memory in terms of aggregate bandwidth, but swapping to disk still imposes a stiff latency penalty. Mext claims it can expand the effective memory of a system by 2 to 4x using flash, which gig for gig is still vastly less expensive than DRAM. This flash memory is exposed to the operating system like regular memory simply by running the Mextd daemon. Memory tiering is nothing new and has seen various reincarnations over the years with some being software based and others, like Intel Optane persistent memory, using special 3D XPoint memory tech co-developed by Micron. Mext stands out for its use of machine learning to migrate data from hot memory to cold storage almost like a branch predictor — something AMD has an awful lot of experience with. Mext isn't using one model to decide when to shuffle your data. Instead it uses a series of heuristics, long short term memory, and modern transformer architectures depending on which combination renders the best results. “This approach has the potential to reduce infrastructure costs, improve resource utilization, and help customers more effectively scale general-purpose and AI workloads,” Dan McNamara SVP of AMD’s compute and enterprise AI biz wrote in a blog post this week. Beyond enterprise applications, the technology could have implications for AI serving. Modern mixture of experts (MoE) models are, as their name suggests, comprised of multiple sub-models. For each token predicted, a different selection of experts may be used. In practice an LLM may use some experts more frequently and others rarely. We wouldn't be surprised to see AMD use Mext's prediction algorithms to offload infrequently utilized experts from HBM to slower system memory, enabling enterprises to take advantage of larger more capable models with fewer resources. That’s just speculation of course, but we've reached out to AMD for comment; we'll let you know if we hear anything back. ® [Image text:] KOREA 622X SEC M416C2560J-6