AI Inference Is Splitting in Two—and Cerebras CS-4 Shows Why

Related Daily Briefing

August 19: CS-4 arrives while AI infrastructure follows power and frontier safety constrains development

The August 19 Daily Briefing records the CS-4 launch as one of the day’s verified hardware developments. This article examines the systems architecture behind it.

Open the August 19 report

The immediate development

The CS-4 launch matters less as a chip announcement than as evidence that inference is fragmenting

Cerebras introduced CS-4 this week as the next generation of its inference system. Reuters reports that the rack uses three of the company’s wafer-scale processors in pluggable modules, adds new networking, uses a WSE-3 Turbo chip fabricated on TSMC’s 5-nanometer process and is designed with fewer components to simplify deployment. Cerebras says the system will become available in the third quarter.

Those details are news. They are not yet the most interesting part of the story.

The deeper change is that the industry is beginning to treat large-language-model inference as a pipeline whose stages may deserve different hardware. Cerebras is not pursuing this idea alone. Amazon Web Services now supports separate prefill and decode pools in SageMaker HyperPod. AMD and Cerebras are building a heterogeneous system in which AMD Helios handles prompt processing while Cerebras hardware handles token generation. Cerebras has separately described a similar strategy with AWS Trainium for prefill and its own systems for decode.

That is a different competitive model from the familiar question, “Which accelerator is fastest?” The new question is more operational: Which accelerator is best for this phase of this workload, and what does it cost to move the state between phases?

One LLM request is really two different compute jobs

When a user sends a prompt to a large language model, the system does not perform one uniform calculation from start to finish. It first processes the input, then generates the output. Those stages have different computational shapes.

Prefill reads the prompt and processes many input tokens in parallel. During this phase, the model builds the key-value cache, usually called the KV cache: intermediate attention data that lets later token generation reuse what has already been computed instead of reprocessing the entire context every time. Long prompts make prefill heavier because more input tokens have to be transformed and stored.

Decode begins after that context has been prepared. The model generates the response one token at a time. Each new token depends on the tokens already produced, so decode is inherently more sequential. The hardware repeatedly reads model weights and the growing KV cache. That makes memory bandwidth and memory movement unusually important.

A useful shorthand is that prefill is often more compute-bound, while decode is often more memory-bandwidth-bound. The distinction is not absolute—model size, batch size, quantization, sequence length and serving software can change the balance—but it is strong enough to shape modern inference systems.

This matters outside computer engineering because the two stages affect different parts of the user experience. Prefill influences how quickly a system can absorb a long document, codebase or conversation history. Decode influences how fast the answer appears once generation begins. A system can be excellent at one and mediocre at the other.

Running both phases on the same accelerator creates a resource compromise

A conventional serving stack often runs prefill and decode on the same class of GPU. That has obvious operational advantages: one hardware fleet, one software environment, fewer network transfers and less scheduling complexity.

The disadvantage is contention. AWS explains the problem directly in its HyperPod documentation: a large prefill job can occupy the same accelerator resources needed by concurrent decode work. A long-context request may therefore stall token generation for other users, damaging the latency of interactive applications even when total system throughput remains acceptable.

Disaggregated inference separates the two phases onto different worker pools. A homogeneous version can still use GPUs on both sides—one pool optimized operationally for prefill, another for decode. That is the model AWS now supports in HyperPod, using Elastic Fabric Adapter and GPU-Direct RDMA to transfer the KV cache between pools.

The more radical version is heterogeneous disaggregation: use different accelerator architectures for different phases. AMD and Cerebras say their planned system will use Helios as the high-throughput prompt engine and the Cerebras Wafer-Scale Engine for low-latency decode. The companies claim that the combination can improve tokens per second per watt, but that figure remains a vendor projection until the joint system is available and independently measured.

This is the architectural context in which CS-4 becomes interesting. It is not trying to make every other accelerator irrelevant. It is increasingly positioned as a specialist inside a larger inference pipeline.

CS-4 turns Cerebras’s low-latency argument into a rack-scale systems bet

Cerebras’s core architectural claim is that a wafer-scale processor reduces the penalty of moving model state across many separate chips. Instead of building one logical accelerator out of large numbers of conventional packages connected by external networking, it uses a processor that occupies most of a silicon wafer and places a large amount of memory bandwidth close to compute.

That makes intuitive sense for decode. Token generation repeatedly touches model weights while advancing through a sequential loop. If the required data can be accessed with less off-chip movement and lower communication overhead, latency can fall.

CS-4 extends that approach at system level. According to Reuters, the rack contains three wafer-scale chips, new networking components and a simplified component count. Cerebras says those changes improve performance and deployment speed. It also says its future roadmap will produce large increases in throughput through 2027.

Those future numbers should be treated as plans, not measurements. No independent apples-to-apples CS-4 benchmark package was available at publication time that compares the new system with current GPU platforms under equivalent models, quantization, batching, context lengths, concurrency and power limits. A launch specification can establish what Cerebras built. It cannot by itself establish the cost or performance advantage across real workloads.

That distinction is particularly important because AI inference is easy to benchmark badly. A very high per-user token rate at low concurrency can coexist with weak aggregate economics. A system optimized for massive batching may look efficient overall while feeling slow to an individual user. Comparing only one number hides the trade-off.

Once prefill and decode separate, the KV cache becomes a network problem

Disaggregation removes one bottleneck by creating another.

The decode worker cannot simply start from nothing. It needs the KV cache created during prefill. That cache can be large, especially for long contexts, and it has to move from the prefill worker to the decode worker quickly enough that the transfer does not erase the latency benefit of specialization.

This is now an active research problem. SmartGen, a July 2026 preprint, describes selective KV-cache transfer because sending the full cache between nodes can saturate limited interconnect bandwidth. The authors report that selective and speculative transfer reduced time to the second token by as much as 4.3 times in their tested setup while preserving later decoding performance. That is promising evidence on a research system, not a universal production result.

AWS’s production architecture attacks the same problem from the infrastructure side with high-speed EFA networking and RDMA. Other research explores compression, mixed precision, topology-aware routing and overlapping cache movement with computation.

The engineering implication is easy to miss: the accelerator with the best isolated decode performance may not produce the best end-to-end service if the handoff is slow, the scheduler is imbalanced or the network becomes congested. Once inference is split, data movement becomes part of model-serving performance.

Faster token generation does not automatically mean cheaper intelligence

Inference economics contain at least three separate questions: how quickly one user receives output, how many total requests the infrastructure can serve, and how much useful work is produced per dollar or per watt.

These objectives can conflict. Serving one request at exceptional speed may underuse expensive hardware. Maximizing batch throughput can increase queueing or per-user latency. Splitting prefill and decode can improve utilization, but it also adds networking, scheduling and operational complexity.

Independent work is valuable here because accelerator vendors naturally emphasize the workload in which their designs look strongest. The 2026 xPU-athalon study compared Cerebras CS-3, SambaNova, Intel Gaudi, Google TPU, Nvidia and AMD systems across different workload shapes. Its central finding was not that one architecture won universally. The best platform varied with batch size, model size and sequence length. The researchers also measured 10% to 60% higher idle power on Cerebras, SambaNova and Gaudi than on the tested Nvidia and AMD GPUs, making utilization important to real efficiency.

That result should temper any simple “specialized accelerator beats GPU” conclusion. A system with excellent active performance can still have weak economics if demand is too bursty to keep it busy. Conversely, a slower device can be commercially attractive if software maturity, availability, utilization and purchase economics are better.

This is why AIUpdateWatch treats published token prices and infrastructure finance as separate but connected layers. The price paid by an API customer is the end of a chain that includes accelerator efficiency, power, capital cost, utilization and the provider’s margin strategy.

Agents make low latency more valuable because delay compounds across model calls

For a simple chatbot, shaving a few hundred milliseconds from one response may improve the experience without changing the economics of the task very much.

Agentic systems are different. A coding agent, research agent or operational assistant may call a model repeatedly while it plans, uses tools, reads results, revises its plan and verifies work. Latency is then multiplied across a chain of dependent calls. A five-second delay repeated 40 times becomes more than three minutes of waiting even before tool execution is counted.

Cerebras uses this argument heavily in its commercial positioning, including its work with software-generation platform Lovable. The vendor’s conclusion—that faster inference enables more interactive multi-step workflows—is plausible. The unanswered question is how much of that speed survives realistic agent stacks once tool latency, retrieval, verification, queueing and network transfers are included.

This also connects to model routing. If an agent can choose different models for different steps, infrastructure may eventually route not only by model quality and price but by inference phase, latency target and hardware type. The orchestration layer becomes responsible for deciding both which model and which compute path should handle the next operation.

The strongest evidence points to specialization, not to the death of the GPU

There is a temptation to frame every new accelerator as a direct replacement for Nvidia. That produces a clean headline and a poor mental model.

Current evidence points instead toward a more heterogeneous serving market. AWS supports disaggregation using GPU pools. AMD and Cerebras plan to combine different architectures. Cerebras and AWS have described a Trainium-plus-Cerebras design. Research benchmarks show that optimal hardware varies substantially with workload shape. New papers focus on the network and cache-transfer layer because disaggregation itself creates new systems constraints.

That looks less like one winner taking the entire inference market and more like specialization inside a distributed system.

The strategic consequence is that software matters more, not less. A heterogeneous stack needs schedulers, routing policies, observability, failure handling, compatible serving runtimes and fast interconnects. Procurement also becomes more complicated: the buyer is no longer comparing only accelerator price and headline throughput but the end-to-end cost of a serving architecture.

For technically curious professionals outside infrastructure engineering, this is the practical point to retain. A model does not “run on a chip” in the simple sense implied by many product announcements. At production scale, it runs through a system. As models become longer-running, more agentic and more latency-sensitive, the boundaries inside that system increasingly determine cost and responsiveness.

The next useful evidence is end-to-end, workload-specific and power-aware

CS-4 is an important launch because it arrives while the serving stack is being reorganized around different inference phases. Whether that architecture becomes dominant will depend on evidence that goes beyond peak token speed.

Equivalent workloads: CS-4 should be compared with current GPU and alternative-accelerator systems using the same model version, quantization, prompt lengths, output lengths and concurrency.
Time to first useful output: tests should separate prefill time, time to first token, decode rate and complete task latency.
KV-cache transfer: heterogeneous systems need published measurements showing how much handoff time is added as contexts grow.
Utilization: tokens per watt and tokens per dollar should be measured under realistic load rather than only at a favorable saturation point.
Agent completion time: multi-step workloads can show whether lower decode latency materially changes end-to-end productivity.
Operational complexity: independent users need evidence about software maturity, failure recovery, scheduling and the cost of running two accelerator pools instead of one.

If those measurements favor disaggregation, the AI hardware race will be harder to summarize with a single chip leaderboard. The winning system may be the one that assigns each phase to the right hardware and moves state between them with the least friction.

CS-4 is therefore better read as a marker of architectural change than as another entry in a faster-chip contest.

Sources