NetTLP: Adapter

Overview

NetTLP adapter is a PCIe device that bridges a PCIe link and an Ethernet link. When the NetTLP adapter receives TLPs from the PCIe link, the NetTLP adapter encapsulates each TLP in Ethernet, IP, UDP, and NetTLP header for sequencing and timestamping and sends the packets to the device host via the Ethernet link. When the NetTLP adapter receives a UDP packet from the Ethernet link, the NetTLP adapter checks whether the packet’s payload is a TLP, decapsulates the packet, and sends the inner TLP to the PCIe link. As a result, from the perspective of the adapter host, all TLPs sent from the device host by the software are recognized as TLPs generated by the NetTLP adapter.

Figure 1: NetTLP adapter overview

The current NetTLP adapter (v0.17) has three Base Address Registers (BAR). BAR0 is used to store adapter specific information (i.e., MAC address and IP address), and BAR2 is used to store MSI-X table. BAR4 is quite special: Transaction Layer Packets (TLPs) to the BAR are encapsulated in Ethernet, IP, and UDP headers, and transmitted to the device host via the 10Gbps Ethernet Link. In the above example, memory accesses to 0xb0000000-0xbfffffff in the MMIO space are delivered to the device host by Ethernet/IP/UDP encapsulation.

Build the adapter bit file

Build environment

All source files can be found at https://github.com/NetTLP/adapter/. And the bit file can be found at https://github.com/NetTLP/adapter/releases. We use the files created by Xilinx for the si5324 configurations. So you need to get the rdf0285-vc709-connectivity-trd-2014-3.zip file from the Xilinx web page to build the bit image. 

$ git clone https://github.com/NetTLP/adapter.git

$ unzip rdf0285-vc709-connectivity-trd-2014-3.zip
$ cp v7_xt_conn_trd/hardware/sources/hdl/clock_control/kcpsm6.v adapter/boards/kc705/rtl/clock_control/
$ cp v7_xt_conn_trd/hardware/sources/hdl/clock_control/clock_control.v adapter/boards/kc705/rtl/clock_control/
$ cp v7_xt_conn_trd/hardware/sources/hdl/clock_control/clock_control_program.v adapter/boards/kc705/rtl/clock_control/

$ source /tools/Xilinx/Vivado/2019.2/settings64.sh
$ cd adapter/boards/kc705

# lint check
$ sudo apt install verilator
$ make lint

# verilog simulation
$ make sim

# build bit file
$ make
$ ls build/nettlp-adapter.bit

NetTLP header

Overview

All TLPs on an Ethernet link are encapsulated in Ethernet, IP, UDP, and NetTLP header. The NetTLP header constructs the packet sequence number field and the timestamp field, and all values are generated by the NetTLP adapter. The NetTLP header length is fixed 6-byte. And current value of the timestamp field is generated by the Ethernet clock and the timestamp resolution is 6.4ns (we will replace the timestamp source from the Ethernet clock to the PCIe clock in a future release).

Packet format 


 2               1               0B
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |           Sequence            |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |           Timestamp           |
 |                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |              TLP              |
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Port number: 0x3000 + (TLP_tag & 0xF)

Header fields

PCIe configuration packet

Overview

The device host can access the PCIe configuration space of the NetTLP adapter by using the PCIe configuration packet. The current adapter implementation can read all PCIe config spaces by specifying an address. However, many PCIe configuration fields cannot be written due to limitations of the PCIe IP Core. The R/W permission depends on the PCIe base specification. If you want to modify the PCIe configuration space, please use PCIe Endpoint IP Core settings in the Xilinx vivado. This PCIe configuration space API will be improved in the future.

LibTLP API

TBD

Packet format 


 2               1               0B
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |CMD| Mask  |       DWaddr      |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |             Data              |
 |                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Port number: 0x4001

Header fields

NetTLP adapter command packet

Overview

The NetTLP adapter settings such as the value of the Ethernet and IP packet header used for encapsulation can be modified by using the NetTLP adapter command. In addition, the PCIe requester ID of the NetTLP adapter allocated by a root complex can be gotten with this API. The device host should get the requester ID using this API and use the ID to generate the TLP header.

LibTLP API

TBD 


 2               1               0B
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |    opcode     |     DWaddr    |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 |             Data              |
 |                               |
 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Port number: 0x4002

Header fields