Diagram showing an InfiniBand switch and an Ethernet switch connected to the same MPO fiber patch panel row using identical yellow OS2 MPO trunk cables and patch cords

TABLE OF CONTENTS

InfiniBand and Ethernet can both carry the RDMA traffic that ties GPU servers together in an AI cluster, and both run large AI workloads today. InfiniBand is a dedicated fabric with its own switches and management model, while Ethernet carries RDMA through RoCE on Ethernet switches and has been gaining ground in AI back-end networks.

For the fiber plant, the difference between InfiniBand and Ethernet is smaller than many buyers expect. In NVIDIA’s 400G generation, the original transceivers work with either protocol, and the MPO-12/APC connectors, fiber cables and patch panels are the same. Not every part number is dual-protocol, though. Below we compare the two fabrics, then go layer by layer from the network card to the patch panel to show what changes when you switch and what carries over.

InfiniBand and Ethernet (RoCE): What Each One Is

InfiniBand is a networking architecture built for high-performance computing and AI. It has its own switches, adapters and cables, and a single trade association writes the specification. Ethernet is the general-purpose network technology found in almost every data center. For AI training, it carries the same RDMA traffic through RoCE, which stands for RDMA over Converged Ethernet. RDMA lets one server read or write another server’s memory without involving that server’s CPU, which speeds up data transfer between GPUs.

Two different bodies define them. The InfiniBand Trade Association (IBTA) defines and maintains both the InfiniBand and the RoCE specifications. The physical layer and media access control of wired Ethernet are defined by the IEEE 802.3 working group, which writes the Ethernet standards. RoCE is an industry standard transport that lets Remote Direct Memory Access (RDMA) run on ordinary Ethernet layer 2 and 3 networks, and the IBTA released its first specification in 2010.

InfiniBand speeds move in generations. According to the IBTA roadmap, NDR 400 Gb/s InfiniBand is shipping today and GDR 1.6 Tb/s products are planned for the 2028 timeframe.[1]

One point of scope: we compare the scale-out network that links servers. NVLink is a scale-up interconnect for GPU-to-GPU communication inside a rack-scale system, so it sits alongside InfiniBand or Ethernet rather than replacing either. Our GPU networking guide explains how the scale-up and scale-out layers fit together.

ai cluster network scale up vs scale out

InfiniBand vs Ethernet at a Glance

InfiniBand Ethernet with RoCE
Defined by IBTA IEEE 802.3 for Ethernet; IBTA for the RoCE transport
RDMA Native Through RoCE
Packet loss Designed for lossless I/O fabrics Lossless behavior depends on flow control and congestion settings across the fabric
Fabric management A subnet manager handles routing Standard Ethernet switching and operations tools
Switches NVIDIA Quantum InfiniBand switches Several vendors; in 2024 Celestica, NVIDIA and Huawei led AI back-end Ethernet switching
Published scale Tens of thousands of nodes in one subnet (IBTA) RoCE cluster designs for up to 32,000 GPUs (Meta)
Network cards ConnectX-7 runs either protocol ConnectX-7 runs either protocol
Optics and fiber MPO-12/APC, single-mode or multimode Same connectors and fiber in the 400G generation
A few rows need context. The IBTA describes InfiniBand as enabling fully redundant and lossless I/O fabrics, with a subnet manager providing traffic routing. On Ethernet, a lossless network has to be configured. Meta, which runs RoCE for production AI training, reported that setting buffers took more careful balancing than on InfiniBand because of RoCE’s more coarse-grained flow control. The scale figures are of different kinds. The IBTA figure describes what the architecture can support, while Meta’s figures come from production: designs for RoCE clusters of up to 32,000 GPUs, and a large Llama 3 variant trained on 16,000 GPUs of a 24,000-GPU RoCE cluster.[2] Dell’Oro Group named Celestica, NVIDIA and Huawei as the leaders of the Ethernet segment of the AI back-end switch market in 2024.

Latency and Performance: Why the Numbers Disagree

InfiniBand is often chosen for low latency, but claims that one fabric has lower latency than the other rarely compare like with like. Some are port-to-port through one switch, some are end-to-end between hosts, and few say whether the fabric was loaded or congested. The IBTA cites ultra-low latency for InfiniBand, with measured delays of 600 ns end-to-end,[1] but it does not state the test conditions, and Ethernet results depend on the switch, the NIC, the configuration and the traffic pattern.

For AI training, a single latency number says little. Training jobs synchronize GPUs over and over, so the whole job waits for the slowest transfer, and one congested link can become the bottleneck. Throughput under load matters as much as latency. In Meta’s RoCE network, flows that collided after a link failure slowed down the whole training job, and the stated goal of its congestion work was predictable tail latency. The fair comparison is job completion time on your own topology, with your own workload. Our GPU networking guide covers the common causes of slow training traffic.

Ethernet vs InfiniBand: Which One Fits Your AI Cluster?

Both fabrics serve the same use cases, large-scale AI training and HPC, so in the Ethernet vs InfiniBand decision neither wins in every case. NVIDIA itself publishes reference designs for both. Its DGX SuperPOD H100 architecture uses InfiniBand for the compute fabric and an InfiniBand network fabric for storage too, while its enterprise reference architecture for GB300 NVL72 uses Spectrum-X Ethernet for the compute fabric.

If this describes your project It points toward Why
You are deploying a vendor reference design as published Whatever that design specifies Switches, cables and lengths are validated together
Your team already runs Ethernet at scale and buys from several vendors Ethernet with RoCE Same tools, skills and supply chain as the rest of the network
You want a fabric designed for lossless RDMA from the start InfiniBand Less tuning of flow control and congestion settings
The cluster will share switching with front-end or storage traffic Ethernet with RoCE One switching technology across networks
Several thousand GPUs or more Either Both have shown scalability in published deployments or designs

The market is moving toward Ethernet. Dell’Oro Group reported in July 2025 that InfiniBand held over 80 percent of AI back-end networks when its coverage began in late 2023, and that Ethernet is now winning in that segment.[3] That is a market forecast, not a ruling on any one cluster. Both product lines are still sold and supported.

infiniband vs ethernet decision flowchart

Does Switching Fabrics Mean New Fiber Cabling?

Mostly not, as long as you stay within the same speed generation and optical interface. The protocol lives in the switches and, for some parts, in the transceiver firmware. The connectors, fiber and patch panels are passive: they handle data transmission without knowing which protocol they carry. Port bandwidth is also the same on both sides in this generation, 400 Gb/s per port. Here is the picture layer by layer for NVIDIA’s 400G (100G-PAM4) generation:

LayerChanges with the protocol?What the vendor documentation says
SwitchesYesNVIDIA Quantum-2 NDR InfiniBand and Spectrum-4 SN5600 Ethernet are separate switch lines, but both use twin-port OSFP cages holding two 400 Gb/s ports
Network cardsUsually notConnectX-7 supports both protocols in the same card, activated by the InfiniBand or Ethernet switch it connects to; some ConnectX-6 and BlueField-2 adapters are protocol-specific
TransceiversCheck the part numberThe original NDR/400GbE transceivers support both protocols, and their firmware enables the protocol of the switch; newer Ethernet-only versions end in “-T”[4]
ConnectorsNoMPO-12/APC: eight fibers in use, four positions blank
Fiber cablesNoNVIDIA states that the same fiber cables are used across its NDR and newer optics
Patch panels and trunksNoReach figures for these optics assume two optical patch panels in the link
DAC and AOC cablesCheck the part numberCheck that each cable part is listed for the target switch and speed; some older adapters need protocol-specific cables

The optical interface is set by the transceiver, not the protocol. These NVIDIA 400G-generation optics show the pattern:

PartForm factorConnectorFiberReach
MMS4X00-NSTwin-port OSFP (2 × 400G)2 × MPO-12/APCSingle-mode, 1310 nmUp to 100 m
MMS4X00-NMTwin-port OSFP (2 × 400G)2 × MPOSingle-mode, 1310 nmUp to 500 m
MMA4Z00-NSTwin-port OSFP (2 × 400G)2 × MPO-12/APCMultimode, 850 nmUp to 50 m
MMA1Z00-NS400QSFP112 (400G)MPO-12/APCMultimode, 850 nmUp to 50 m

What carries over and what does not

Within one generation and one optical interface, the structured fiber plant (MPO-12/APC trunks, patch panels and patch cords) can stay in place. The switches change. Transceivers, DAC and AOC cables need a part-number check, and cable lengths should be checked against the new reference design. NVIDIA’s DGX SuperPOD H100 design, for example, limits the cable path of any InfiniBand cable to 50 m.

What does change the fiber is a change of optical interface. A single-mode DR4-type link and a multimode SR4-type link both use MPO-12/APC, but single-mode and multimode fiber cannot be swapped. Our 400G network guide lists the fiber each 400G optic needs, and our MPO/MTP patch cables cover both fiber types.

infiniband to ethernet cabling what changes

Migrating or Running Both: A Cabling Checklist

Use this list when moving a cluster from one fabric to the other, or when one data hall runs both:
  1. Check every transceiver part number. Confirm it supports the target protocol; Ethernet-only versions exist.
  2. Keep the optical interface. Reuse fiber only where the new optic uses the same connector and fiber type (single-mode or multimode).
  3. Keep APC throughout. APC and UPC connectors should not be mated.
  4. Keep one polarity method. NVIDIA’s direct optic-to-optic cables are straight crossover, type-B MPO-12/APC cables.
  5. Retest the trunks you keep. Measure insertion loss and compare it with the channel limit, for example 3.0 dB for 400GBASE-DR4.[5]
  6. Recheck lengths against the new reference design. Length limits belong to each design.
  7. Plan for the next speed. NVIDIA’s XDR switch cages also accept NDR cables and transceivers for backward compatibility. A 16-fiber interface such as 800GBASE-DR8 needs 16 fibers per port.
For zoning, fiber counts, labeling and acceptance testing of the plant itself, see our guide to data center structured cabling for AI clusters.

Where Ultra Ethernet Fits

The Ultra Ethernet Consortium (UEC) is a consensus-based standards organization operating under the Linux Foundation, and it released Ultra Ethernet Specification 1.0 on June 11, 2025.[6] It aims to make Ethernet better suited to AI and HPC traffic. It is a new specification in AI networking, not yet an alternative to InfiniBand or RoCE, so for now AI infrastructure is still built on one of those two. If you plan a fabric today, ask your switch and NIC vendors which UEC features their products support and whether existing optics and cabling carry over.

MPO Fiber Links for InfiniBand and Ethernet Clusters

MPO/MTP Patch Cables: MPO/MTP trunks and patch cords in single-mode and multimode for 400G and 800G links. View MPO/MTP cables

The passive links described above work the same under either fabric, and they are what YingFeng supplies. Our MPO/MTP patch cables come in OS2 single-mode and OM3 or OM4 multimode, with APC or UPC end faces, in 8-, 12-, 16- and 24-fiber versions; low-loss versions are available. MPO/MTP assemblies are produced in a cleanroom, and samples are available on request. For other passive parts, see our data center fiber optics. To get a quote, tell us the transceiver part numbers, link lengths and polarity method.

Frequently Asked Questions

Is InfiniBand copper or fiber?

Both. Short links inside a rack often use copper direct-attach cables, while longer links use active optical cables or transceivers with fiber. In the 400G generation, the transceivers use MPO-12/APC connectors over single-mode or multimode fiber.

Both. NVIDIA sells Quantum InfiniBand switches and Spectrum Ethernet switches, publishes reference designs for each, and its ConnectX-7 network cards run either protocol.

Yes, within the same generation and optical interface. The MPO-12/APC connectors and fiber are the same; what needs checking is whether each transceiver supports the target protocol. See Does Switching Fabrics Mean New Fiber Cabling? for the full breakdown.

RoCE (RDMA over Converged Ethernet) is an industry standard transport, defined by the IBTA, that lets RDMA run on ordinary Ethernet layer 2 and 3 networks. The first specification was released in 2010.

In AI back-end networks, Ethernet has been gaining share quickly since late 2023, according to Dell’Oro Group’s July 2025 report. Ethernet and InfiniBand are both still sold and deployed, and NVIDIA publishes reference designs for both.

External references

  1. InfiniBand Trade Association: About InfiniBand
  2. Gangidi et al., RDMA over Ethernet for Distributed AI Training at Meta Scale, ACM SIGCOMM 2024: ACM Digital Library
  3. Dell’Oro Group: Ethernet is Winning the War Against InfiniBand in AI Back-End Networks (press release, July 15, 2025)
  4. NVIDIA: LinkX 400Gb/s Cables and Transceivers Main Deck (product documentation)
  5. TIA Fiber Optics Technology Consortium: 400GBASE-DR4 Application Overview
  6. Ultra Ethernet Consortium: Ultra Ethernet Specification v1.0 (June 11, 2025)