Cerebras and the engineering of disaggregated AI inference

Two engines for one inference workflow

AMD and Cerebras announced a technical partnership on Jul 23, 2026 to combine AMD Helios infrastructure with Cerebras Wafer-Scale Engine technology. Helios, with EPYC CPUs and Instinct MI400-series accelerators, would process prompts and large context windows; the WSE would generate subsequent tokens. Cerebras plans to install Helios systems in its data centres, with initial availability through Cerebras Cloud expected in the second half of 2026. The companies have not disclosed the interconnection or additional performance measurements.

Their projected gain of up to five times more tokens per second per watt comes from July 2026 modelling by AMD and Cerebras. That comparison uses the Kimi 2.6 model with one trillion parameters at comparable interactivity, comparing the combined infrastructure with a Cerebras WSE-only configuration. Actual configurations can differ. It is a forecast for the proposed combination, rather than a measurement of an already deployed joint service.

Prompt latency and the pace of generated tokens

Large language model serving has two separately timed operations. Prefill processes the prompt; decoding produces the continuing response. DistServe, a research serving system, assigns those operations to different GPUs. It optimizes resource allocation and parallelism for each phase while placing the work according to the bandwidth available in the cluster. Physical separation introduces communication between the two groups of processors, which its placement strategy seeks to minimize. DistServe addresses that transfer alongside the computation, rather than treating processor allocation as the only scheduling decision.

Its two latency constraints are time to first token, or TTFT, and time per output token, or TPOT. Colocated execution can couple the resource allocations and create interference between phases. In the researchers' evaluations across models, applications and latency requirements, DistServe served up to 7.4 times as many requests, or supported up to 12.6 times tighter service objectives, than comparison systems. More than 90% of requests remained within the latency constraints in those evaluations.

Moving request state between machines

Splitwise separates prompt computation and token generation onto different machines. Its characterization distinguishes compute-intensive prompt processing from memory-intensive generation, including their different latency, throughput, memory and power characteristics. The generation phase can leave computing resources underused even with batching and scheduling. Splitwise therefore permits independently provisioned machines for the two phases, using hardware suited to each operation. Its designs can use either the same type of machine in both groups or different types, rather than requiring a uniform accelerator fleet for every stage.

The transition requires request state to move from the prompt machine to the generation machine. Splitwise implements and optimizes that transfer through fast backplane interconnects in GPU clusters. The researchers designed clusters against throughput, cost and power objectives. One evaluated configuration provided 1.4 times higher throughput at 20% lower cost than existing designs; another provided 2.35 times the throughput with unchanged cost and power budgets. Those are alternative configurations, rather than benefits that can be added together.

Scheduling within a shared serving system

Sarathi-Serve handles the same two-phase workload through a different mechanism: dividing a prompt into nearly equal chunks and constructing schedules that admit new requests without pausing ongoing decoding. Prompt iterations process multiple input tokens in parallel and can saturate GPU computation, whereas a decode iteration processes one token per request. Batching helps utilize that decoding capacity, but interleaving whole prompt iterations with decoding creates a throughput and latency tradeoff. Sarathi-Serve changes the size of the scheduled prompt work to address that interference.

Its more uniform batches also reduce imbalances between iterations and the empty intervals known as pipeline bubbles. Under tail-latency constraints, the researchers measured 2.6 times higher serving capacity than vLLM for Mistral-7B on one A100 GPU. Their Yi-34B evaluation used two A100 GPUs and reached up to 3.7 times higher capacity. With pipeline parallelism on Falcon-180B, the gain reached up to 5.6 times. Each result belongs to its particular model and hardware configuration, not to the proposed AMD–Cerebras installation.

Keeping the request's memory available

The key-value cache, or KV cache, expands and contracts with requests during language model serving. Fragmentation and duplicated allocations can consume memory that would otherwise accommodate more requests in a batch. PagedAttention applies ideas from operating-system paging and virtual memory to attention computation. The vLLM serving system built around it supports flexible sharing of cached state within a request and between requests. Its memory management addresses the changing request state, rather than assuming that every request needs a permanently fixed allocation.

In the original researchers' evaluations, vLLM achieved two to four times higher throughput than systems including FasterTransformer and Orca at the same latency level. Gains were more pronounced for longer sequences, larger models and more complex decoding algorithms. The design combines near-zero wasted KV-cache memory with sharing that reduces redundant storage. These are GPU-serving results for memory management and batching; they do not establish the size of a cache in the AMD–Cerebras workflow or identify the software that the two companies will use.

A cache distributed across the cluster

Mooncake, the serving platform for Moonshot AI's Kimi service, separates prefill and decoding clusters while distributing cached state across additional resources. Its architecture uses otherwise underused CPU, DRAM and SSD capacity in the GPU cluster. A cache-centred scheduler balances effective throughput against latency-related service objectives. That arrangement makes request-state storage a cluster resource, alongside the separate processor groups doing the prompt and generation work. It also addresses overloaded operation, where processing every incoming request is not an assumption.

Mooncake uses prediction-based early rejection to manage such overload. Its September 2025 version reports up to a 525% throughput increase in certain simulated scenarios while meeting service objectives, and 75% more requests handled under real workloads. The simulated and operational comparisons are distinct. They concern Mooncake's architecture and comparison baseline, rather than a measured gain for every separated inference deployment.

Streaming, swapping and recovery

DéjàVu combines three functions around a KV-cache streaming library called DéjàVuLib. Prompt-token disaggregation reduces pipeline bubbles; microbatch swapping manages GPU memory; and state replication supports fault tolerance. Its researchers evaluated the mechanisms on large models across cloud deployments. The problems include memory overprovisioning and lengthy recovery after failures, as well as unequal prompt and token-processing times. State transfer therefore serves several different operations in this system. Moving a microbatch for memory management and replicating state for recovery are separate functions, even though both use the streaming library.

Placing work on unequal processors

Helix, a research system distinct from AMD's Helios infrastructure, represents heterogeneous GPU serving as a maximum-flow problem. A directed, weighted graph represents the infrastructure: nodes correspond to GPU instances, while capacities account for both processor and network differences. It uses mixed integer linear programming to select serving strategies. Model placement and request scheduling are optimized together, rather than chosen as unrelated decisions. That approach explicitly includes variation in the network connections as well as variation among the GPUs carrying the model.

Across the researchers' heterogeneous clusters of 24 to 42 GPU nodes, Helix increased serving throughput by up to 3.3 times relative to existing approaches. Prompting latency fell by up to 66%, and decoding latency by up to 24%. These are separate metrics from those evaluations. The similarly named research system does not supply an announced component of the AMD–Cerebras collaboration.

Programming a wafer instead of a shared-memory GPU

Wafer-scale accelerators distribute local memory among very large numbers of cores connected in a mesh. WaferLLM targets that arrangement, with tens of gigabytes of on-chip memory and memory bandwidth measured in tens of petabytes per second. Its PLMR device model captures characteristics that differ from the shared-memory assumptions common in GPU inference software. The system introduces wafer-scale parallel execution and two matrix-operation implementations, MeshGEMM and MeshGEMV. Those operations provide software designed for the mesh rather than simply placing an existing GPU serving system on it.

In its researchers' evaluations on a Cerebras WSE2, WaferLLM achieved up to 200 times higher accelerator utilization than comparison methods. Matrix-vector operations were 606 times faster and 16 times more energy-efficient than on an Nvidia A100 GPU. Full language model inference, a different comparison, was 10–20 times faster than A100 clusters running SGLang and vLLM. The primitive-operation ratios are not full-model speedups, and the tested WSE2 is not an unspecified future partnership configuration.

Local memory, routing and computational intensity

The Cerebras wafer-scale engine evaluated in September 2024 had 2.6 trillion transistors, 850,000 cores and 40 GB of on-chip memory. These specifications identify the older hardware used for that benchmarking. Its Sparse Linear Algebra Compute cores omit multiply-by-zero operations, while the memory is uniformly distributed across cores for local access to model parameters. Cerebras software configures routes between cores at runtime. Multi-headed self-attention in the evaluated models captures long-range relationships between input words or image patches.

Language model training and inference were benchmarked on this engine, including the memory constraints of computational tasks. Performance was analysed against computational intensity using a roofline model, allowing the researchers to examine scaling against available computation and bandwidth. The specified memory bandwidth is 20 petabytes per second for this hardware. Its transistor count, core count, memory capacity and bandwidth belong together as one historical specification. The hardware parameters refer to the engine used for these evaluations, while the model considers performance scaling across different computational intensities.

Communication algorithms within the wafer

Reduce and AllReduce collectives are another part of wafer-scale programming. Researchers investigating the Cerebras WSE built a performance model for those communication operations and checked predicted execution times experimentally across input sizes. They implemented new algorithms tailored to the architecture alongside existing implementations. Their work also established a lower bound on the runtime of Reduce. The resulting code-generation method selects implementations using the model, targeting near-optimal execution across the range of input sizes rather than selecting one fixed implementation for every case.

Their new Reduce and AllReduce algorithms outperformed the vendor implementation by up to 3.27 times in the experiments, while the model's prediction error was below 4%. The reference is the existing communication solution, not an entire language model running on a GPU cluster. These results concern the runtime of collectives and the accuracy of their performance model. They do not supply a token-generation speed or a throughput-per-watt forecast for the proposed joint inference platform.

Attention computation and memory traffic

GPU attention kernels address another layer of the memory problem. FlashAttention computes exact attention with tiling that reduces reads and writes between high-bandwidth memory and on-chip SRAM. It accounts for movement between memory levels as well as arithmetic work. Its researchers analysed the number of memory accesses and found the method optimal for a range of SRAM sizes. They also developed a block-sparse extension, which is approximate attention; that extension is distinct from the exact computation in the original tiled algorithm.

The original evaluations reported training improvements: a 15% end-to-end speedup for BERT-large with sequences of 512 tokens relative to the MLPerf 1.1 training record, three times faster GPT-2 training at a sequence length of 1K, and a 2.4-fold gain on Long Range Arena tasks with sequences of 1K–4K. These comparisons concern training, not the latency of the proposed Cerebras decode service. FlashAttention and its block-sparse extension also enabled longer-context experiments, including Path-X at 16K sequence length with 61.4% accuracy and Path-256 at 64K with 63.1% accuracy.

Dividing the GPU's attention work

FlashAttention-2 changes the partitioning of attention work across GPU thread blocks and warps. Its design reduces arithmetic outside matrix multiplication, distributes even a single attention head across multiple thread blocks, and reduces shared-memory communication among warps within a block. These are three distinct changes to the kernel's execution:

• Less non-matrix arithmetic reduces work outside the main multiplication operations.
• More parallel work across thread blocks increases occupancy.
• Redistribution within blocks reduces reads and writes through shared memory.

In the researchers' A100 evaluations, the revised kernel reached 50–73% of theoretical maximum floating-point throughput, compared with 25–40% for the preceding FlashAttention implementation, and delivered approximately twice its speed. End-to-end GPT-style model training reached up to 225 teraflops per second per A100, or 72% model FLOPs utilization. Kernel throughput and training utilization are different measures from requests served within latency constraints. The changes operate inside an attention implementation, alongside the separate questions of cache management, scheduling and where inference phases execute.

The physical system around the processor

Wafer-scale integration also has manufacturing, thermal-management and reliability requirements. WSE-3, Nvidia H100 and B200 architectures were compared in March 2025 across these topics, memory bandwidth, latency and scalability. The researchers identify cost-effectiveness and long-term viability as areas requiring further work. Their architectural comparison is distinct from a production measurement of the proposed AMD–Cerebras service. The hardware generations also differ across these evaluations: WSE-3 features in the architectural comparison, while the WaferLLM experiments use WSE2.