Overview
PCIe topologies and their communication models are flexible: CPU-to-Device and Device-to-Device. By exploiting Device-to-Device PCIe communications, as known as Peer-to-Peer DMA (P2P DMA), on a NetTLP platform, we can observe TLPs sent from commercial peripherals.
Capturing TLPs in NetTLP
For capturing TLPs sent from peripherals by NetTLP, it is necessary to change destinations of DMAs from the peripheral: from main memory to BAR4 of the NetTLP adapter. It requires modifications for device drivers, but Linux provides helpers for such purposes (PCI Peer-to-Peer DMA Support — The Linux Kernel documentation)
Once TLPs go out into IP networks, we can capture the TLPs by tcpdump. Figure 1 shows an overview of a setup for capturing TLPs. An easy way to capture TLPs encapsulated in IP is to run tcpdump on a device host. Capturing TLPs on the device host is sufficient to observe the contents of the TLPs. On the other hand, using port mirroring and another box enables observing them without affecting the performance of software PCIe devices on the device host from the software view. Moreover, using hardware RX time stamping on the box brings microsecond-scale observation for PCIe transactions.
Figure 1: An overview of capturing TLPs.
Examples
Figure 2 shows captured TLPs of two types of NICs: Intel X520 10Gbps NIC and Intel XL710 40 Gbps NIC. They are captured by port mirroring and hardware RX time stamping at a Linux box. Both NICs tried to send 32 1500-byte packets with DMA Read, and the packets were on the BAR4 of the NetTLP adapter where psmem at the device host was associated.
In this experiment, we prepared Linux kernel 4.20.2 with the PCI_P2PDMA option that enables the P2P DMA support. Moreover, we used netmap and modified ixgbe and i40e netmap drivers for changing destinations of DMA reads from the NICs from the main memory to the BAR4 on the NetTLP adapter,
Figure 2: Captured TLPs of X520 and XL710 Ethernet NICs