I wanted to know how AI data centers are built

I use frontier models every day (mostly Claude, mostly through Claude Code), and at some point I realized I had no real picture of what's on the other end of the request. I know "a lot of GPUs in a data center." That's about it. Words like SXM, NVSwitch, InfiniBand and CDU kept showing up in things I was reading, and I couldn't have told you how they fit together.

So I tried to build the picture myself. I asked Claude Code to draw the hardware it would take to serve a single copy of a model like Opus 5.5, for inference only (not training), and to explain every part along the way. This post is what came out of that.

A big caveat up front. Nothing about how Anthropic serves its models is public, and Anthropic runs on a mix of Google TPUs, AWS Trainium and NVIDIA GPUs. So everything here is built on a representative machine, a rack in the style of NVIDIA's GB300 NVL72, and every number about the model is an estimate from stated assumptions. The hardware figures are NVIDIA's published ones. I'm doing this to learn, so "a reasonable guess with the math shown" was good enough for me (and honestly it's about as good as anyone outside these companies can do).

The rack

Here's the figure. Hover or tap any part to light it up and read what it does. Press O to slide a compute tray out and see what's inside it. Space pauses serving (watch the return temperature on the coolant gauge drop), S switches to slow motion, and the arrow keys step through the parts.

Fig 1. One rack, one frontier model · open it full size

Huge shout-out to Anatomy for this. It's an open-source Claude skill that explains an idea by building it as a real, working object, and it's the reason this is a machine you can poke at instead of a boxes-and-arrows diagram. Every pipe and cable in that drawing is checked so nothing passes through anything, and the numbers in the readouts come from the same model as the text below. I couldn't have drawn any of it myself (check out the Anatomy site for what else it can do).

The rest of this post walks through the rack, part by part. I also tried to work out what all of it costs, which turned into its own post: What one copy of a frontier model costs.

Fitting the model into the rack

A frontier model today is very likely a mixture-of-experts (MoE) model. Each layer holds many "expert" sub-networks, and each token only uses a few of them. Here's what I assumed:

Total parameters: ~5 trillion
Active per token: ~250 billion
Layers: 120
Hidden size: 16,384
Experts per layer: 288, with each token routed to 8
Weight format: NVFP4, about 4.5 bits per parameter
Weights in memory: 2.8 TB

The rack has 72 GPUs × 288 GB = 20.7 TB of HBM (the very fast memory stacked right next to each GPU). The weights take 2.8 TB, about 1 TB goes to activations and workspace, and the remaining 16.9 TB holds the KV cache: the stored keys and values of every token the model has already read, in every conversation it's serving. At about 240 KB per token (120 layers × 8 KV heads × 128 dimensions × K and V × 1 byte), that's room for about 69 million tokens of context.

How the rack's 20.7 TB of HBM divides between weights, workspace and KV cache

The weights are the small part. Most of a serving rack's memory holds conversations in progress.

This surprised me the most. I'd assumed the model itself would be what fills the hardware, and it's the other way around. It also explains why one rack is a natural unit for one copy of the model: the whole thing plus a big KV cache fits inside a single NVLink domain (more on that below), so no layer ever waits on the slower network between racks.

SXM and OAM: GPUs as mezzanine boards

A data-center GPU isn't a PCIe card like the one in a gaming PC. It's a mezzanine module: a small, thick board carrying the GPU package and its power stages, with high-density connectors on the underside that plug flat into a baseboard or motherboard. Lying flat lets a cold plate press down on top, and those connectors carry what a card edge can't: more than a kilowatt of power, plus the high-speed links to the other GPUs.

SXM is NVIDIA's version. On HGX baseboards, eight SXM modules sit on one board with NVSwitch chips between them. In GB300 the B300 GPUs sit on SXM-style modules on the compute tray's motherboard, next to the Grace CPUs.
OAM (OCP Accelerator Module) is the Open Compute Project's open standard for the same idea, used by AMD's Instinct MI300 and MI355 and others. Eight OAMs mount on a Universal Baseboard (UBB) that carries the links between them.

The open case on the floor in the figure has both modules, connector side up, plus a bare B300 package: two reticle-sized dies joined at their edge, ringed by eight HBM3e stacks. Each stack is 12 DRAM layers holding 36 GB, which is where 288 GB per GPU comes from.

Inside a compute tray

Each of the 18 compute trays is one rack unit (1U, 4.4 cm) tall and holds:

4 B300 GPUs, each with 288 GB of HBM3e at 8 TB/s, drawing about 1.4 kW under its cold plate.
2 Grace CPUs, each with 72 Arm cores and up to 480 GB of LPDDR5X. Each one talks to its GPUs over NVLink-C2C at 900 GB/s, so the GPUs can use that memory too.
4 ConnectX-8 SuperNICs, one 800 Gb/s port per GPU, for the network between racks.
1 BlueField-3 DPU for the "front-end" network: user requests, storage and management.
8 E1.S NVMe drives at the front.
A coolant rail with short hoses to every cold plate, and two quick disconnects at the back.

Everything at the back blind-mates. The NVLink connectors slide into the spine, the power clip slides onto the busbar, and the coolant couplings slide into the manifolds, so a technician can pull a tray out of the front without touching anything at the back. That was a neat detail I didn't know about.

Splitting every layer across GPUs

No single GPU holds the model, so every layer runs on many GPUs at once, split two ways.

Tensor parallelism (TP) splits the big matrix multiplications inside attention across a group of GPUs, here 8. Each GPU computes a slice of the output, then the group adds the slices together in an all-reduce. With 256 conversations decoding at once, that's about 8.4 MB per layer.
Expert parallelism (EP) spreads the 288 experts across all 72 GPUs, 4 per GPU. Once the router has picked each token's 8 experts, an all-to-all sends each token to the GPUs holding its experts (dispatch), and a second all-to-all brings the results back (combine).

One layer of decoding: attention, all-reduce, routing, dispatch, experts, combine

One layer of decoding. Three of the six steps are traffic between GPUs, and it repeats 120 times per token.

So every layer ends in three rounds of GPU-to-GPU traffic over NVLink, 120 layers in a row, for every token the model writes. That's the traffic you see running up and down the spine at the back of the rack in the figure.

NVLink switch trays and the spine

Each B300 has 18 NVLink links of 100 GB/s, 1.8 TB/s in all. The rack's 9 switch trays hold 2 NVLink switch chips each, 18 chips in total, and each GPU runs exactly one link to every chip. So any GPU is a single switch hop from any other, at full bandwidth, whichever pair it is: 130 TB/s across the rack.

Six of the 72 GPUs, each with one link to each of the 18 NVLink switch chips

Every GPU sends one of its 18 links to each switch chip, so every pair of GPUs is one hop apart.

Those links don't run on circuit boards. They run on about 5,000 copper cables bundled into cartridges at the back of the rack, called the NVLink spine. Copper needs no optical transceivers, which saves power and money over the short distances inside one rack.

One shared memory pool

Because every GPU can read and write every other GPU's memory over NVLink at close to local speed, the rack behaves like one giant accelerator:

HBM3e on the GPUs: 20.7 TB, at 8 TB/s per GPU (576 TB/s total)
LPDDR5X on the Grace CPUs: 17.3 TB, at 900 GB/s per CPU link

This is what people mean by scale-up: growing the machine that shares one memory space, instead of adding more machines that send each other messages. The CPU memory is a second tier, slower but big, and a good place to park the KV cache of conversations that are waiting for their next turn.

Where a decode step's time goes

Writing one token for each of the 256 conversations takes one pass through all 120 layers. And it turns out decoding is limited by reading memory, not by doing math:

Reading each GPU's slice of the weights (2.8 TB ÷ 72 GPUs at 8 TB/s): 4.9 ms
Reading the KV cache (256 conversations × 32k tokens of context): 3.6 ms
360 NVLink collectives at ~8 µs each: 2.9 ms
Total per step: 11.3 ms

An 11.3 ms decode step: 4.9 ms weights, 3.6 ms KV cache, 2.9 ms NVLink, about 0.1 ms arithmetic

About three quarters of each step is reading HBM. The math itself would fit in a sliver.

That works out to about 88 tokens a second per conversation and about 22,600 tokens a second for the whole replica. The actual arithmetic (about 128 TFLOP per step) would take the rack's 1.08 exaFLOPS of FP4 compute only about 0.1 ms. I had always pictured these chips as compute engines first, and for serving they're closer to very fast memory readers.

Between racks: InfiniBand and RoCE

The NVLink domain ends at the edge of the rack. Everything beyond it goes over the scale-out network. Each GPU has its own 800 Gb/s NIC (57.6 Tb/s per rack), cabled in fiber to leaf switches, which connect to spine switches that join hundreds of racks.

For serving one replica this network carries less than it does in training, but it still matters. Requests arrive and tokens leave through it, KV caches move between racks when the prompt-reading and answer-writing steps run on different machines, and model weights load over it.

InfiniBand is a network built for HPC (NVIDIA's Quantum switches, like the Quantum-X800). It's lossless by design, thanks to credit-based flow control, and has the lowest latency plus mature in-network reductions (SHARP).
Ethernet with RoCE (RDMA over Converged Ethernet) carries InfiniBand's transport inside ordinary Ethernet frames (RoCEv2 runs over UDP/IP). To stay lossless it needs priority flow control (PFC) and ECN tuning, or one of the newer adaptive transports. The upside is normal Ethernet tooling, lots of vendors and very large scale (NVIDIA's Spectrum-X and Ultra Ethernet-style fabrics).

Both give the GPU RDMA: one NIC writes straight into another machine's GPU memory without either CPU copying the data. In a rail-optimized layout, GPU 0 of every tray connects to the same leaf switch, GPU 1 to the next, and so on, so the most common traffic only crosses one switch.

Rail-optimized scale-out: GPU 0 of every rack on leaf 0, GPU 1 on leaf 1, both leaves to the spine

Same-numbered GPUs in different racks share a leaf, so most traffic is one hop.

There's also a separate front-end network, through each tray's BlueField-3, for user requests, storage traffic and management, so it never competes with GPU-to-GPU traffic. And a third, slower out-of-band network reaches every tray's management controller.

Liquid cooling

At about 135 kW, this one rack draws more than ten typical enterprise racks, way more than air can carry away. About 85% of the heat leaves in liquid:

Cold plates sit directly on every GPU and CPU, with coolant running through fine copper channels.
Hoses in the tray join the plates to a rail, and two quick disconnects at the back mate with the rack's supply and return manifolds.
The manifolds connect over the top of the rack to a coolant distribution unit (CDU), which pumps the loop: PG25, 25% propylene glycol in water.
Inside the CDU a plate heat exchanger hands the heat to the building's facility water, which carries it out to dry coolers or cooling towers.

The two cooling loops: cold plates, manifolds, CDU, facility water

Two loops. The rack's coolant never leaves the row, and the CDU passes its heat to the building's water.

To carry about 115 kW with a 10 °C rise, the loop needs about 173 liters a minute: in at 30 °C, out at 40 °C. Water that warm needs no chillers for most of the year. The other ~15% of the heat (drives, NICs, power supplies) still leaves in air. The CDU's two gauges in the figure show the supply and return temperatures, and if you pause serving you can watch the return line fall toward 32 °C.

Power

Power comes from an overhead busway, through a tap-off box and two armored cables, into the rack's 8 power shelves of 33 kW each (enough for N+N redundancy). The shelves convert AC to DC and feed a busbar that runs down the back of the rack, and every tray clips onto it.

What a real deployment adds

One rack is one copy of the model. A real service runs many copies behind a load balancer, often splits prefill (reading the prompt) and decode (writing the answer) onto different racks, adds storage clusters for weights and logs, and spreads all of it across rows of racks joined by the scale-out fabric.

About the guesses

I want to be clear about what's real here and what isn't. The hardware numbers (GPU count, HBM size and bandwidth, NVLink speeds, NIC speeds, power shelf ratings) come from NVIDIA's published GB300 NVL72 specs. Everything about the model (parameter counts, layer shape, the expert layout, batch size, context length, step time) is my estimate, and so are the coolant and power figures, worked out from the assumptions above.

None of it is a claim about how Anthropic actually runs Opus 5.5. They haven't published that, and they spread their models across TPUs, Trainium and GPUs anyway. I was trying to understand how these systems fit together, and for that, a well-reasoned guess with the math shown got me a lot further than I expected. If you know this stuff firsthand and something here is off, I'd love to hear about it.

And if you're wondering what a rack like this costs to buy and run, that's the companion post.

How I made this

I built the interactive figure with Claude Code and Anatomy, which draws machines as isometric figures and audits every part (nothing passes through anything, every line ends on something). Thanks again to the Anatomy project for making something this fun possible. I've also submitted the figure to the Anatomy community gallery. The diagrams come from an explainer I generated with html-anything.

Sources

NVIDIA, GB300 NVL72: the rack's GPUs, memory, NVLink bandwidth and networking.
Glenn K. Lockwood, B300 notes: per-GPU specs for Blackwell Ultra.
SemiAnalysis, GB300 NVL72 on InferenceX: per-chip memory, bandwidth and FP4 compute.
Meta Engineering, Sharing a common form factor for accelerator modules: where OAM came from.
Open Compute Project, OCP Accelerator Module design specification v1.5: the OAM form factor and Universal Baseboard.
The Register, Google and Anthropic's TPU deal, and Maginative, Anthropic secures 1M Google TPUs while keeping Amazon as primary training partner: Anthropic's mix of TPUs, Trainium and GPUs.