The Execution Physics
The Autoregressive Decode Bottleneck
Large Language Models (LLMs) operate across two fundamentally distinct computational regimes: the prefill phase and the decode phase.
During prefill, the model processes the user prompt in parallel. Because thousands of input tokens are evaluated simultaneously, matrix multiplication units (such as Tensor Cores) achieve high arithmetic intensity—performing hundreds of floating-point operations for every byte of memory transferred from DRAM. In this phase, modern accelerators easily reach 80% to 90% of their peak theoretical TFLOPs.
The decode phase, however, is entirely serial. Each new token generated requires reading every parameter in the model (e.g., 70 billion to 400 billion weights) plus the accumulated Key-Value (KV) cache for all previous context tokens out of High Bandwidth Memory into on-chip SRAM registers. For a single token generation step, the arithmetic intensity drops to nearly 1 FLOP per byte transferred.
Under these conditions, matrix compute engines sit idle waiting for memory retrieval. The inference latency of frontier models is therefore dictated not by compute TFLOPs, but strictly by sustained memory bandwidth.
Interconnect Architecture
Breaking the 1,024-Bit Barrier: The 2,048-Bit Bus
From the original HBM standard ratified in 2013 through HBM2, HBM2E, HBM3, and HBM3e, every generation of High Bandwidth Memory adhered to a rigid physical constraint: a 1,024-bit wide interface per memory cube.
To increase aggregate memory bandwidth, memory manufacturers (SK hynix, Samsung, Micron) were forced to increase the per-pin transfer frequency. HBM3e pushed transfer speeds up to 9.6 Gbps per pin, delivering roughly 1.15 to 1.2 TB/s per stack. However, pushing beyond 10 Gbps across micro-bumps and Through-Silicon Vias (TSVs) introduced exponential signal integrity degradation, crosstalk, and unsustainable power dissipation.
With the ratification of the JESD238 HBM4 standard by JEDEC, the industry made the bold decision to double the interface to 2,048 bits.
- Reduced Frequency Strain: By doubling the parallel data paths, HBM4 delivers over 2.05 TB/s per stack at more conservative clock frequencies (~8.0 Gbps per pin), cutting electrical switching energy per bit transferred.
- Massive Aggregate Throughput: An 8-stack accelerator configuration (such as next-generation NVIDIA Rubin Ultra or AMD Instinct MI400) reaches an astonishing 16.4 TB/s of aggregate memory bandwidth, doubling the throughput of current HBM3e configurations.
Silicon Manufacturing
The Logic Base Die Revolution: Sub-7nm Foundry Nodes
The most profound structural disruption introduced by HBM4 is not in the DRAM layers, but at the foundation: the base (interface) die.
In all previous HBM generations, the base die was manufactured directly by memory makers on mature, low-cost DRAM manufacturing processes (~15nm to 20nm). These processes were adequate for simple routing and basic test logic.
In HBM4, routing 2,048 high-speed signal lines plus thousands of power and ground lines within a 11mm × 11mm footprint creates insurmountable interconnect congestion on DRAM nodes. Consequently, JEDEC standardizes the transition of the HBM4 base die to sub-7nm pure-play logic foundry nodes (primarily TSMC N5/N3 and Samsung SF4).
Moving to advanced logic nodes provides three critical architectural superpowers:
- Interconnect Density: Advanced CMOS nodes offer 15+ tight-pitch metal layers, easily routing 2,048 TSV interconnects without signal interference.
- Custom Interfacing and Built-In Self-Test (BIST): Accelerators can now embed custom PHY controllers, advanced error correction code (ECC), and dynamic voltage-frequency scaling directly into the base die.
- Near-Memory Logic and Compute-in-Memory (CiM): The logic base die can perform pre-processing, sparsity filtering, and KV-cache compression directly underneath the memory stack before data ever crosses the interposer to the host GPU.
Packaging Physics
Stacking Physics: Micro-Bumps vs. Direct Cu-Cu Hybrid Bonding
HBM4 standardizes vertical stacking for 12-high (12Hi) and 16-high (16Hi) DRAM dies, supporting capacities of 36 GB, 48 GB, and up to 64 GB per stack.
Packaging 16 active silicon dies within a standard package height profile requires mechanical thinning of each individual DRAM wafer to under 30 micrometers. At this scale, traditional solder micro-bumps encounter physical limits:
- Micro-Bump Pitch Limit: Solder micro-bumps cannot be scaled below ~18–25 micrometer pitch without risk of solder bridging and short circuits.
- Direct Copper-to-Copper (Cu-Cu) Hybrid Bonding: Pioneered by TSMC (SoIC) and adapted for HBM4, Cu-Cu hybrid bonding fuses polished copper pads and dielectric surfaces at atomic tolerances without solder balls. This compresses interconnect pitch below 1 micrometer, eliminates inter-die parasitic capacitance, and lowers thermal resistance by over 30%.
Comparative Specifications
HBM2E to HBM4: Architectural Progression
| Generation | Bus Width | Pin Speed | Bandwidth / Stack | 8-Stack GPU Bandwidth | Base Die Node |
|---|---|---|---|---|---|
| HBM2E (A100) | 1,024 bits | 3.2 Gbps | 410 GB/s | ~2.0 TB/s (6-stack) | DRAM Node (~20nm) |
| HBM3 (H100) | 1,024 bits | 6.4 Gbps | 819 GB/s | ~3.35 TB/s (5-stack) | DRAM Node (~14nm) |
| HBM3e (B200 / MI325X) | 1,024 bits | 9.6 Gbps | 1,228 GB/s | 8.0–9.6 TB/s (8-stack) | DRAM Node (~12nm) |
| HBM4 (Rubin / MI400) | 2,048 bits | 8.0+ Gbps | >2,048 GB/s | >16.4 TB/s (8-stack) | Sub-7nm Logic (TSMC N3/N5) |
Data Center Impact
Impact on 2026–2027 Frontier Accelerators
The arrival of HBM4 transforms the economics of serving frontier models.
When running dense 400B+ parameter models or mixture-of-experts (MoE) architectures with extended 128k context windows, memory-bandwidth starvation previously limited inference concurrency. Accelerators were forced to split models across multi-node tensor parallel clusters purely to aggregate enough memory bandwidth to achieve acceptable generation speeds.
With 16.4 TB/s of bandwidth available within a single 8-stack package, individual server nodes can serve 2x to 3x more concurrent inference streams without decode throttling. This directly compresses the cost-per-token for reasoning-heavy workloads like Claude 3.7 Sonnet and OpenAI o3-mini.
Checklist
Operational Checklist for Hardware Buyers
- Audit Silicon Foundry Dependency: Note that HBM4 couples DRAM memory makers directly to leading logic foundries (TSMC, Samsung Foundry), making advanced packaging capacity (CoWoS-L) an even more centralized supply chain risk.
- Model Advanced Packaging CapEx: Because HBM4 base dies require advanced logic wafers and hybrid bonding tooling, expect initial HBM4 memory module ASPs to be 35% to 40% higher per gigabyte than mature HBM3e.
- Prepare Liquid-Cooled Infrastructure: Dissipating heat through a 16-high DRAM stack sitting atop an active sub-7nm logic base die demands tight junction thermal management; air cooling is virtually non-viable for HBM4-equipped AI servers.