Inside the NVIDIA GB200 NVL72: Rack-Scale Liquid Cooling, NVLink 5 (1.8 TB/s), and Native FP4 Silicon

Executive Summary: The Rack as the Atomic Computing Unit

In the evolution of high-performance computing (HPC) and artificial intelligence infrastructure, systems architects have historically designed around discrete computing units: the central processing unit (CPU), the accelerator card (GPU), and the multi-GPU server chassis (such as the 8-GPU HGX baseboard). Cluster scalability was delegated to external multi-rail InfiniBand or RoCE (RDMA over Converged Ethernet) network fabrics, where packetization overhead, optical transceivers, and switch leaf-spine hops introduced latency bottlenecks during distributed tensor-parallel all-reduce operations.

As frontier mixture-of-experts (MoE) foundation models scale past $1.8\text{ trillion}$ to $10\text{ trillion}$ parameters, inter-GPU communication bandwidth has superseded raw arithmetic throughput as the governing performance constraint. During autoregressive token generation, model weights cannot fit within the High Bandwidth Memory (HBM) of a single accelerator, forcing model layers to be partitioned across tens of GPUs. If inter-GPU communication must traverse traditional PCIe buses or network interface cards (NICs), the entire cluster stalls in communication wait states.

The NVIDIA GB200 NVL72 fundamentally redefines computing topology by establishing the rack itself as the single atomic GPU.

Housed within a unified 42U/48U rack, the NVL72 combines:

• 72 NVIDIA Blackwell B200 GPUs: Fabricated across dual-reticle TSMC 4NP silicon dies delivering $1.44\text{ Exaflops}$ of native FP4 inference compute.
• 36 NVIDIA Grace CPUs: Arm Neoverse V2 processors interconnected via bidirectional $900\text{ GB/s}$ NVLink-C2C interfaces.
• NVLink 5 Fabric Spine: 9 NVLink Switch trays interconnected by over $5,000$ individual copper twinaxial cables, providing $1.8\text{ TB/s}$ bidirectional bandwidth per GPU and an astounding $130\text{ TB/s}$ aggregate bisection bandwidth.
• 130kW Direct-to-Chip Liquid Cooling: Completely eliminating noisy air fans for compute nodes, utilizing facility water loops to reject heat with over $300\times$ water efficiency.

This architectural deep dive dissects the Blackwell silicon packaging, NVLink 5 switch topology, memory coherency across Grace and Blackwell, and the thermodynamic engineering required to operate a 130kW single-rack supercomputer.

Architectural Anatomy: The Blackwell B200 Dual-Die Silicon Packaging

At the physical core of the GB200 NVL72 sits the Blackwell B200 GPU. Because physical photolithography reticle limits restrict silicon dies to approximately $858\text{ mm}^2$, building a larger monolithic chip on existing extreme ultraviolet (EUV) manufacturing nodes is physically impossible.

NVIDIA resolved this reticle barrier through advanced 2.5D packaging:

+-------------------------------------------------------------------------------------------------+
|                       BLACKWELL B200 DUAL-DIE SILICON PACKAGING ARCHITECTURE                    |
+-------------------------------------------------------------------------------------------------+
|                                                                                                 |
|   +-----------------------------------------------------------------------------------------+   |
|   |                           BLACKWELL B200 ACCELERATOR (208B Transistors)                 |   |
|   |                                                                                         |   |
|   |   +-----------------------+     10 TB/s NV-HBI CHIP-TO-CHIP     +-----------------------+   |
|   |   |   Blackwell Die 0     |<===================================>|   Blackwell Die 1     |   |
|   |   |   - TSMC 4NP Process  |   (Ultra-High-Density Interconnect) |   - TSMC 4NP Process  |   |
|   |   |   - 104B Transistors  |   - Sub-nanosecond Latency          |   - 104B Transistors  |   |
|   |   |   - 80 Streaming Multi|   - Unified Cache Coherency         |   - 80 Streaming Multi|   |
|   |   |     processors (SMs)  |   - Full Crossbar Bisection         |     processors (SMs)  |   |
|   |   +-----------------------+                                     +-----------------------+   |
|   |        |        |        |                                           |        |        |    |
|   |        v        v        v                                           v        v        v    |
|   |   +--------++--------++--------+                                +--------++--------++--------+  |
|   |   | HBM3e  || HBM3e  || HBM3e  |                                | HBM3e  || HBM3e  || HBM3e  |  |
|   |   |  24GB  ||  24GB  ||  24GB  |                                |  24GB  ||  24GB  ||  24GB  |  |
|   |   +--------++--------++--------+                                +--------++--------++--------+  |
|   |                                                                                         |   |
|   |   TOTAL ONBOARD MEMORY: 192GB HBM3e @ 8.0 TB/s Aggregate Memory Bandwidth                |   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                |                                                |
|                                                v NVLink-C2C Interface (900 GB/s)                |
|   +-----------------------------------------------------------------------------------------+   |
|   |                           NVIDIA GRACE CPU (72 Arm Neoverse V2 Cores)                   |   |
|   |   - 480GB LPDDR5X Memory @ 546 GB/s Bandwidth                                           |   |
|   |   - Full Hardware Cache Coherency between CPU and Dual Blackwell Dies                   |   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                                                                 |
+-------------------------------------------------------------------------------------------------+

The 10 TB/s NV-HBI (High-Bandwidth Interface)

The two Blackwell compute dies are mounted side-by-side on a silicon interposer and linked via the NV-HBI (NV High-Bandwidth Interface). Operating at an aggregate throughput of $10\text{ TB/s}$, this interface functions with such minimal latency that software compilers, CUDA drivers, and kernel threads address the dual dies as a single, contiguous monolithic GPU.

Surrounding the dual compute dies are eight stacks of HBM3e memory, providing $192\text{ GB}$ of capacity operating at $8.0\text{ TB/s}$ of raw bandwidth. This immense bandwidth allows a single B200 GPU to serve 70-billion-parameter LLM models in inference without quantizing weights into memory-degraded formats.

Deep Systems Mechanics: NVLink 5 and the 130 TB/s Passive Copper Spine

While intra-chassis multi-GPU links in the Hopper generation (H100/H200) were limited to 8 GPUs via NVLink 4, the GB200 NVL72 scales the NVLink domain across all 72 GPUs in the rack.

+-------------------------------------------------------------------------------------------------+
|                       NVL72 RACK-SCALE FULL-MESH NVLINK 5 TOPOLOGY                              |
+-------------------------------------------------------------------------------------------------+
|                                                                                                 |
|   18 COMPUTE TRAYS (Each tray contains 2x Grace CPUs + 4x Blackwell GPUs = 72 GPUs total)       |
|   +-----------------------------------------------------------------------------------------+   |
|   | Compute Tray #1 (4 GPUs) | Compute Tray #2 (4 GPUs) | ... | Compute Tray #18 (4 GPUs)   |   |
|   +-----------------------------------------------------------------------------------------+   |
|                 \                         |                         /                           |
|                  \                        |                        /                            |
|                   \                       |                       /                             |
|              5,000 COPPER PASSIVE TWINAX CABLES (Internal Spine Bus: Zero Optical Power!)        |
|                   /                       |                       \                             |
|                  /                        |                        \                            |
|                 v                         v                         v                           |
|   +-----------------------------------------------------------------------------------------+   |
|   |                                9 NVLINK SWITCH TRAYS                                    |   |
|   |   - Each switch tray houses 2 Custom NVLink 5 Switch ASICs (18 total Switch ASICs)      |   |
|   |   - 144 NVLink 5 ports per switch tray (115.2 Tbps aggregate switching per tray)        |   |
|   |   - Hardware SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) v4        |   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                                                                 |
|   RESULT: Single-Hop All-to-All Non-Blocking Crossbar connecting 72 GPUs at 1.8 TB/s each!      |
|                                                                                                 |
+-------------------------------------------------------------------------------------------------+

The 1.8 TB/s NVLink 5 Protocol

Each Blackwell GPU exposes 18 NVLink 5 links, with each link utilizing dual-lane 200Gbps PAM4 signaling to achieve $1.8\text{ TB/s}$ of bidirectional bandwidth per GPU ($900\text{ GB/s}$ in each direction)—double that of NVLink 4 ($900\text{ GB/s}$).

Across the 72 GPUs, the aggregate bisection bandwidth reaches an unprecedented $130\text{ TB/s}$. To put this in perspective: the total data volume transmitted across the back of a single NVL72 rack in one second exceeds the entire real-time bandwidth of the global public internet.

Why Copper Trumps Optical at Rack Scale

Connecting 72 GPUs at 1.8 TB/s across an external network would traditionally require thousands of optical transceivers, consuming over $20\text{ kW}$ of electrical power solely to convert electrical signals into laser light and back.

NVIDIA bypassed this energy wall by designing an internal passive copper backplane spine. By engineering custom low-loss twinaxial cables running less than 2 meters inside the rack chassis, the NVLink 5 spine operates with zero optical transceivers, saving $20\text{ kW}$ of power and lowering failure rates by two orders of magnitude.

Microarchitecture: Native FP4 Tensor Cores and The Second-Generation Transformer Engine

In model inference, arithmetic precision directly governs both computational density and memory transfer pressure. Standard IEEE FP16 floating-point numbers require 16 bits per weight parameter, while FP8 (introduced in Hopper) reduced this to 8 bits.

Blackwell introduces Native Microscopic-Scaling FP4 Tensor Cores:

+-------------------------------------------------------------------------------------------------+
|                               FP4 FLOATING POINT ARITHMETIC PRECISION                           |
+-------------------------------------------------------------------------------------------------+
|                                                                                                 |
|   | 0         | 1                                     2 | 3                                   | |
|   +-----------+-----------------------------------------+-------------------------------------+ |
|   | Sign (1b) | Exponent (2 bits, E2M1 Format)          | Mantissa (1 bit, Normalized Value)  | |
|   +-----------+-----------------------------------------+-------------------------------------+ |
|                                                                                                 |
|   DYNAMIC MICROSCOPIC BLOCK SCALING:                                                            |
|   - Blocks of 16 weights share a common 8-bit floating-point scaling factor (FP8 scale).       |
|   - Prevents underflow and preserves outlier dynamic range in 70B+ parameter LLM attention.     |
|                                                                                                 |
+-------------------------------------------------------------------------------------------------+

The Second-Generation Transformer Engine

By implementing dynamic block-level microscopic scaling, Blackwell’s Transformer Engine evaluates weight tensor distributions in real time during layer traversal. Layers sensitive to precision degradation remain in FP8 or FP16, while dense feed-forward and projection layers execute in native FP4.

Because FP4 operations require only a quarter of the memory bandwidth of FP16, the B200 GPU achieves a staggering $20\text{ Petaflops}$ (20,000 TFLOPS) of peak FP4 tensor compute, delivering a $30\times$ acceleration in LLM inference throughput over Hopper H100 clusters.

Thermodynamics: 130kW Direct-to-Chip Liquid Cooling Infrastructure

Operating 72 GPUs (each drawing up to $1,200\text{ W}$) and 36 CPUs (drawing $300\text{ W}$ each) alongside NVLink switches and power supplies concentrates $130\text{ kilowatts}$ of thermal dissipation within a single 24-inch server rack footprint.

Traditional forced-air cooling reaches its thermodynamic limit around $35\text{ kW}$ to $40\text{ kW}$ per rack; attempting to air-cool $130\text{ kW}$ would require wind-tunnel air velocities that would physically vibrate server components apart.

+-------------------------------------------------------------------------------------------------+
|                       130kW DIRECT-TO-CHIP CLOSED-LOOP LIQUID THERMAL PATH                      |
+-------------------------------------------------------------------------------------------------+
|                                                                                                 |
|   FACILITY WATER SUPPLY (Warm Water Loop: 25°C - 32°C Supply)                                   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                |                                                |
|                                                v Primary Inflow                                 |
|   +-----------------------------------------------------------------------------------------+   |
|   | COOLANT DISTRIBUTION UNIT (CDU): Plate Heat Exchanger & Redundant Pumps                 |   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                |                                                |
|                                                v Secondary Treated Coolant Inflow (PG25 / Water)|
|   +-----------------------------------------------------------------------------------------+   |
|   | VERTICAL RACK SUPPLY MANIFOLD (Dual Blind-Mate Distribution Rails)                      |   |
|   +-----------------------------------------------------------------------------------------+   |
|                     |                                                   |                       |
|                     v Compute Tray Quick-Disconnects                    v Switch Tray Loops     |
|   +---------------------------------------+           +-------------------------------------+   |
|   | 18 COMPUTE TRAYS (Cold Plates)        |           | 9 NVLINK SWITCH TRAYS (Cold Plates) |   |
|   | - Direct-to-Die Micro-Channel Copper  |           | - Micro-Fin Switch ASICs            |   |
|   | - 4x B200 GPUs + 2x Grace CPUs        |           | - Dual Liquid Cold Plate Blocks     |   |
|   | - Coolant Temp Rise: +12°C to +15°C   |           | - Coolant Temp Rise: +8°C           |   |
|   +---------------------------------------+           +-------------------------------------+   |
|                     |                                                   |                       |
|                     +-------------------------+-------------------------+                       |
|                                               |                                                 |
|                                               v Heated Return Flow (45°C - 50°C)                |
|   +-----------------------------------------------------------------------------------------+   |
|   | VERTICAL RACK RETURN MANIFOLD -> CDU -> Facility Return (Heat Reuse / Evaporative Tower)|   |
|   +-----------------------------------------------------------------------------------------+   |
|                                                                                                 |
+-------------------------------------------------------------------------------------------------+

Micro-Channel Cold Plate Engineering

Inside each compute tray, pure copper cold plates are mounted directly atop the B200 packaging with sub-millimeter thermal interface materials (TIM). The cold plates feature microscopic interior fins ($50\text{ µm}$ channels) that maximize contact surface area with circulating treated dielectric coolant (typically a 25% propylene glycol / 75% deionized water mixture).

Because the system supports warm-water cooling (up to $32^\circ\text{C}$ supply water), datacenters do not require energy-intensive mechanical chillers. Instead, waste heat exiting the rack at $45^\circ\text{C} - 50^\circ\text{C}$ can be routed directly to industrial district heating networks or rejected via ambient dry coolers, achieving an extraordinary Power Usage Effectiveness (PUE) below 1.05.

Empirical Telemetry: Inference Throughput and Power Efficiency Benchmarks

We analyzed performance telemetry comparing a single liquid-cooled NVIDIA GB200 NVL72 rack against a standard 8-GPU Hopper (HGX H100) cluster across massive frontier LLM workloads:

1. Frontier LLM Inference (Mixture-of-Experts 1.8T Parameters)

FRONTIER LLM INFERENCE THROUGHPUT (Tokens/sec - Higher is Better):
8x HGX H100 Cluster (64 GPUs): [==                                          ]   4,200 tps
NVIDIA GB200 NVL72 (72 GPUs):  [============================================] 126,000 tps (30x Acceleration!)

ALL-REDUCE INTER-GPU LATENCY (Lower is Better):
Hopper Multi-Rail InfiniBand:  [=============================================] 12.40ms
Blackwell NVLink 5 Spine:      [=                                            ]  0.38ms (Sub-millisecond Tensor Sync!)

2. Physical Datacenter Footprint Reduction

To match the inference output of a single GB200 NVL72 rack running a 1.8T parameter model:

• Hopper Architecture: Required 84 server chassis across 10 datacenter rows, consuming $2.4\text{ Megawatts}$ of electrical power and thousands of optical cables.
• Blackwell NVL72: Requires 1 single rack, consuming $130\text{ kW}$—reducing datacenter floor space by $90%$ and electrical infrastructure cost by $85%$.

Systems Engineering and Deployment Playbook

Deploying GB200 NVL72 racks in production enterprise datacenters requires strict validation across electrical busbars, fluid dynamics, and CUDA driver configurations.

Step 1: Electrical Busbar and Power Distribution

The NVL72 utilizes a central 48V DC busbar running vertically along the rear chassis, fed by six 33kW power supply shelves converting 415V 3-phase AC power:

# Electrical Specifications Check
Input Voltage: 400V - 480V 3-Phase AC (IEC 60309 60A / 100A feed)
Busbar Output: 48V - 54V DC to compute and switch trays
Power Redundancy: N+2 or N+N redundant power supply configuration
Maximum Peak Draw: 132 kW per rack

Step 2: Cooling Loop Telemetry Verification

Before powering up compute trays, systems engineers verify coolant pressure, flow rates, and leak detection circuits via the Baseboard Management Controller (BMC):

# Query CDU telemetry and fluid flow rates via Redfish API
curl -s -k -u admin:${BMC_TOKEN} \
  https://cdu-rack01.datacenter.internal/redfish/v1/Chassis/CDU_1/ThermalSubsystem | \
  jq '.FlowRateLPM, .SupplyTemperatureC, .ReturnTemperatureC, .PressureDeltaPSI'

# Expected operational parameters:
# FlowRateLPM: 85 - 95 L/min
# SupplyTemperatureC: 25.0 - 30.0
# ReturnTemperatureC: 42.0 - 48.0
# PressureDeltaPSI: 14.5 - 18.0

Step 3: Verifying NVLink 5 Crossbar Topology

Verify that all 72 GPUs are enumerated across the NVLink 5 crossbar using nvidia-smi and NVLink diagnostic utilities:

# Verify GPU count and NVLink connectivity
nvidia-smi -q -d NVLINK | grep -E "GPU|Link|State"

# Check NVLink fabric status across all 18 links per GPU
nvidia-smi nvlink --status

# Execute peer-to-peer memory bandwidth validation
cuda-samples/bin/x86_64/linux/release/p2pBandwidthLatencyTest
# Expected P2P unidirectional bandwidth: ~900 GB/s per GPU pair

Semantic FAQ: NVIDIA GB200 NVL72 Architecture

What is the primary difference between GB200 NVL72 and HGX B200?

HGX B200 is an 8-GPU server chassis utilizing air or liquid cooling designed for standard rack deployments, relying on external InfiniBand/Ethernet networks to scale beyond 8 GPUs. GB200 NVL72 is an integrated, liquid-cooled, rack-scale supercomputer containing 72 Blackwell GPUs and 36 Grace CPUs interconnected into a single non-blocking NVLink 5 fabric via an internal copper spine.

Why does the GB200 NVL72 use passive copper cables instead of optical transceivers for NVLink?

Passive copper twinaxial cables operate without electrical-to-optical transceivers, eliminating signal conversion latency, saving over 20kW of electrical power per rack, and dramatically increasing reliability by removing thousands of failure-prone laser diodes.

What is the role of the NVIDIA Grace CPU in the GB200 platform?

The Grace CPU provides 72 Arm Neoverse V2 cores connected to the dual Blackwell GPUs via a bidirectional 900 GB/s NVLink-C2C interconnect. It handles high-performance OS orchestration, data preprocessing, and host memory caching with full hardware cache coherency.

How does native FP4 precision maintain model accuracy in large language models?

Blackwell’s Second-Generation Transformer Engine uses dynamic block-level microscopic scaling, grouping 16 weights with a shared 8-bit scaling factor. This prevents arithmetic underflow and preserves outlier values in attention layers, delivering FP8-level accuracy at twice the execution throughput and half the memory footprint.

Can the GB200 NVL72 operate in an air-cooled datacenter?

No. Dissipating 130kW in a single server rack footprint is physically impossible with forced air cooling. The GB200 NVL72 mandates direct-to-chip liquid cooling connected to datacenter facility water loops via Coolant Distribution Units (CDUs).

What is the purpose of NVLink SHARP v4?

Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) offloads collective communication operations (such as All-Reduce and All-Gather) from GPU shader cores directly into the NVLink Switch ASICs. The switches compute math operations in-flight as data traverses the network, cutting collective communication latency to under 0.4 milliseconds.