Player wiki

Networking and clusters

Scale-up versus scale-out, fabrics and oversubscription, and how the Supercomputer screen turns racks into one machine.

A cluster is racks plus a network fabric. Racks that aren't clustered can still serve inference straight from the Customers screen, but a training customer rents one big coherent machine — that's what you build here.

Two kinds of network

Wrong balance between the two means idle GPUs. Training needs both; inference mostly needs memory and tokens per second.

The Supercomputer screen

  1. Pick racks — any online racks not already in a cluster, across all halls.
  2. Wire the network — the easy way is Auto-design for TRAINING (finds the cheapest fabric that meets the year's expected GB/s per GPU) or Auto-design for INFERENCE (cheaper still). The advanced panel exposes everything: scale-out tech (InfiniBand NDR/XDR, Spectrum-X, 400G/800G Ethernet, RoCE, Ultra Ethernet), topology (fat-tree rail-optimized, fat-tree, single-domain, dragonfly, torus-ish), leaf and spine switches, node→leaf and leaf→spine cables, and an oversubscription slider.
  3. Read the stats — accelerators, scale-up domain, FP8 compute, memory, power, effective GB/s per GPU, bisection bandwidth and network capex.
  4. What can this machine do? — a per-model table showing whether the training state fits (tokens/s and MFU) and whether weights fit for serving (tokens/s, time-to-first-token, tensor-parallel size).
  5. BUILD CLUSTER.

Existing machines can be unclustered and re-clustered at any time — dissolve one, adjust the rack selection, rebuild. Clusters show in the hall as colored markers, and each rack's inspector says which cluster and customer it belongs to.

Supercomputer builder
Pick racks, wire the fabric, and see what the machine can train or serve.

Cabling you can see

Zoom the hall camera down to a rack and its rear shows the cabling: one cable per network port from each chassis to the top-of-rack switch, run in vertical cable managers, held by ties and tagged at the port end. Scale-out links alternate green and gold; a management cable is grey. Rack-scale systems add a thick purple NVLink spine. A failed or still-burning-in rack shows its cables dimmed. The count follows the design, so a rack with more NICs shows more cables. At a distance the cables thin out to one loom per chassis, and on the Low and Potato presets they are not drawn in the hall at all.

Rear cabling at close range
The rear of a rack row at close range.

Oversubscription

Spine uplinks can carry less than the sum of leaf downlinks — that ratio is oversubscription. 2:1 cuts your spine cost while slowing collective operations. Pick 1:1 for training (big all-reduce traffic punishes congestion) and 2–4:1 for inference, where it is usually fine and much cheaper. The stats panel flags effective GB/s per GPU below the year's training tier.

At scale, cables fail. Every month each cluster rolls for dead links: optics flap and get dirty most often, active optical cables less, passive copper (DAC) almost never. Bigger fabrics lose links more often, and a short-staffed crew makes it worse. While links are down, the cluster's scale-out bandwidth drops by more than the dead links' own share, because a collective operation is only as fast as its slowest rail. Training contracts that need a minimum scale-out bandwidth per GPU can be missed this way.

You can see network health at all times: the NETWORK box in the top bar, the Network card on the Overview, and red overhead runs on the cluster in the hall.

Design rules of thumb

Internet transit

Customer-facing bandwidth lives on the Build screen: buy IP transit circuits, watch demand (Gb/s) against your uplink capacity, and — before signing SLA-heavy contracts — buy a second carrier on a physically diverse path. One circuit is one backhoe away from an outage (see Incidents and resilience).

Related: Customers and contracts for assigning clusters, Hardware roadmap for how fabrics evolve.

Found a mistake or something missing? Edit this page on GitHub or ask in Discussions.