rocelet RoCE v2 reference design for the 2x QSFP28 FMC

An open-source, lightweight RoCE v2 (RDMA over Converged Ethernet) engine, rocelet, on the Opsero 2x QSFP28 FMC (OP120): 100 Gb/s on a KCU116 with AMD’s integrated 100G MAC, and 40 Gb/s on a ZCU106 with the open-source omac Ethernet MACs, entirely license-free (a 10 Gb/s build option uses the same board). The engine has its own documentation (architecture, interface, register map, integration and host rules) at rocelet.ethernetfmc.com; this guide covers the reference design built around it.

Important

Status: version 0.5.0, rocelet v0.3.0, opsero-ethernet-mac v0.1.0. Both targets are validated on hardware against an NVIDIA ConnectX-6 Dx. kcu116 (100G): line-rate RDMA in both directions, loss recovery, link loss including a real cable pull, the built-in self-test, the Holoscan Sensor Bridge device mode with NVIDIA’s HSB 3.0.0 host software and the beamforming demo. zcu106 (new in 0.5.0, 40G): a Zynq UltraScale+ design in which Linux on the Cortex-A53 runs the control plane (daemon, command line, web UI, a network interface on the 40G port), host RDMA into on-chip memory and into PS DDR at 40G line rate, 802.3x PAUSE in both directions, the self-test, the HSB device mode and the beamforming demo at up to 40 Gb/s. Measurements and known limitations: Resources, timing and measurements, Revision history.

The test bench: KCU116 with the 2x QSFP28 FMC, a 100G AOC to the ConnectX-6 Dx in the GPU host, the beamforming demo on the host's monitor

The design connects a RoCE v2 capable host NIC (validated with an NVIDIA ConnectX-6 Dx) point-to-point to QSFP28 port 0 of the FMC. The host and the FPGA exchange RDMA traffic in both directions:

  • FPGA to host: RDMA WRITE (with or without immediate) and SEND into host memory, reliable (RC, go-back-N retransmission from an on-chip ring) or unreliable (UC, including the Holoscan Sensor Bridge packet format).

  • Host to FPGA: RDMA WRITE and SEND into the FPGA’s memory windows, and RDMA READ of them.

The connection is set up out of band: the host helper rocelet_host (libibverbs) and the FPGA console exchange one-line parameter records (QPN, PSN, addresses, keys); there is no RDMA connection manager on the FPGA. On the ZCU106 the records travel over SSH or the web UI.

Two targets, one engine

kcu116

zcu106

Board

KCU116 (Kintex UltraScale+ XCKU5P), FMC on HPC

ZCU106 (Zynq UltraScale+ XCZU7EV), FMC on HPC0

Link

100GBASE-CR4 / -SR4 with RS-FEC, QSFP28 port 0

40GBASE-CR4 (no FEC), QSFP28 port 0; 10GBASE-R as a build option

Ethernet MAC

AMD integrated 100G Ethernet (CMAC) hard block, 512 bits at 322.27 MHz (no-cost license)

open-source omac_40g (MIT, from scratch) on four GTH transceivers, 256 bits at 156.25 MHz (omac_10g, 64 bits, for the 10G option); no license of any kind

Control plane

MicroBlaze console on the UART

Linux (AMD EDF Yocto image): daemon roceletd, command line rl, web UI, network interface rl0; or a bare-metal Cortex-A53 console (same commands)

FPGA memory for RDMA

256 KB UltraRAM window

256 KB UltraRAM window + a PS DDR window (a 256 MB reserved region, two HP ports per direction)

Other traffic (ARP, ICMP, IP, HSB control)

firmware over a frame FIFO

Linux network interface rl0 through a DMA channel (SSH and ping over the 40G link)

Boot

bitstream over JTAG or from the configuration flash

SD card (Linux image, or the bare-metal BOOT.BIN), or JTAG

Everything else is shared: the rocelet engine with the same parameters, the self-test logic with its 19-test suite, the Holoscan Sensor Bridge frame producer and its sources (IQ16 test signal, counter, the beam-array generator), and the console command set. The MAC sits behind one generic MAC boundary, so the system logic does not know which MAC it talks to.

The Zynq UltraScale+ architecture

The ZCU106 target is designed for the Zynq UltraScale+ processing system rather than ported from the MicroBlaze design, so that it carries over to other Zynq UltraScale+ boards and to the RFSoC boards (same PS; see Porting to other Zynq UltraScale+ and RFSoC boards):

  • PS control, PL data path. The engine, the MAC and everything that must keep line rate are in the PL; the A53 configures them through one 1 MB register window. A clock guard answers every CPU access, so software can never hang on, or take an SError from, the MAC clock domain while the link is reset; an always-on register block holds the GT and MAC resets, the link status, the QSFP module lines and one aggregated interrupt.

  • Linux owns the port’s IP address. A small kernel driver turns the engine’s non-RoCE traffic into the network interface rl0 (ARP, ICMP, TCP/UDP, the HSB control plane), while RoCE v2 (UDP 4791) stays in hardware. The daemon roceletd runs the same console sources as the bare-metal builds; rl and the web UI talk to it.

  • Memory windows. Host RDMA reaches 256 KB of UltraRAM or a 256 MB region of PS DDR reserved in the device tree and mapped to user space through UIO (PYNQ-compatible); the PS DDR window spreads its traffic over two HP ports per direction to keep up with 40G.

  • Streaming source boundary. The HSB frame producer takes its samples from one AXI-Stream input with a source multiplexer: the place where an RFSoC design plugs in its RF-ADC stream.

Details: Hardware overview (the targets and the two design families), Zynq UltraScale+ design: hardware reference (Zynq UltraScale+ hardware and registers), Zynq UltraScale+ design: Linux (the Linux stack), MicroBlaze design: hardware reference (the MicroBlaze design).

Supported boards

FPGA boards

Carrier board

Target design

FMC slot

QSFP28 ports

MAC

FEC

Linux

KCU116

kcu116

HPC

1x 100G

CMAC (hard)

RS-FEC (CL91)

no

Zynq UltraScale+ boards

Carrier board

Target design

FMC slot

QSFP28 ports

MAC

FEC

Linux

ZCU106

zcu106

HPC0

1x 40G (10G option)

omac_40g (open source)

none

yes (Yocto)

Licensing

The design, the rocelet engine and the omac Ethernet MACs are released under the MIT license.

  • kcu116: the 100G Ethernet MAC is AMD’s UltraScale+ Integrated 100G Ethernet (CMAC) hard block, whose IP license is free of charge (add it to your account on the AMD licensing site).

  • zcu106: no IP license at all. The MAC is open source (omac, MIT); the transceivers use AMD’s free transceiver wizard (gtwizard_ultrascale), the rest is standard Vivado IP (Zynq UltraScale+ PS, AXI DMA, SmartConnect, AXI IIC, Processor System Reset, AXI4-Stream Data Width Converter, and the inline HDL cells for constants, the interrupt concatenation and a reset OR). Vivado Standard edition (free) builds it.

Contents

User Guide