Overview
This section shows how to play example applications of LibTLP on a NetTLP platform. To build a NetTLP platform, please see the setup.
Example applications described here are (1) dma_read, which is a very simple DMA read issued from software to hardware, and (2) psmem, which is a pseudo memory software device. Both applications can be found at apps directory in LibTLP repository.
tcpdump
Before starting LibTLP applications, we recommend to compile and install a modified tcpdump, which can distinguish and display TLPs encapsulated in Ethernet/IP by NetTLP.
$ sudo apt install libpcap-dev
$ git clone https://github.com/nettlp/tcpdump
$ cd tcpdump
$ ./configure
$ make
$ sudo make install
# then, the modified tcpdump is installed in /usr/local/sbin/tcpdump
wireshark
We use the modified tcpdump to describe the section. wireshark-nettlp is other option for displaying TLPs encapsulated in ethernet/ip by NetTLP.
$ git clone https://github.com/nettlp/wireshark-nettlp.git
$ mkdir -p ~/.local/lib/wireshark/
$ cd ~/.local/lib/wireshark
$ ln -s ~/wireshark-nettlp/plugins
Figure: Wireshark image
dma_read
This application simply sends MRd to the adapter host through the NetTLP adapter, receives CplD, and dumps data.
$ cd libtlp/apps/
$ ./dma_read -h
./dma_read: invalid option -- 'h'
usage
-r remote addr at NetTLP link
-l local addr at NetTLP link
-b bus number of NetTLP adapter, XX:XX
-t tag
-a target address
-s transfer size (default 4-byte)
-m MaxReadRequestSize
-r and -l options specify remote and local addresses on the Ethernet link between adapter and device hosts from the viewpoint of LibTLP: remote means NetTLP adapter and local means LibTLP (device host's IP address). Namely, -r 192.168.10.1 and -l 192.168.10.3 at the example setup. -b is bus number where the adapter host is installed at the adapter host. This value is used for requester ID of TLP(s).
For the first trial of DMA read on NetTLP, clone and run a simple demo program at adapter host. This program just periodically shows a physical address of a char* buffer that holds "test_string".
# At adapter host
$ git clone https://github.com/NetTLP/demo
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 9 (delta 1), reused 9 (delta 1), pack-reused 0
Unpacking objects: 100% (9/9), done.
$ cd demo
$ make
gcc -Wall -o demo demo.c
$ sudo ./demo
PHY ADDR 0x838de1dc0 test_string
PHY ADDR 0x838de1dc0 test_string
PHY ADDR 0x838de1dc0 test_string
^C
By using the dma_read application, you can read the memory region of the char* on the adapter host through DMA read from the device host as shown below:
# 1. run the demo program at adapter host.
$ sudo ./demo
PHY ADDR 0x82159c720 test_string
PHY ADDR 0x82159c720 test_string
PHY ADDR 0x82159c720 test_string
# 2. run dma_read at device host
$ ./dma_read -r 192.168.10.1 -l 192.168.10.3 -b 1b:00 -a 0x82159c720 -s 12
======== struct nettlp ========
port: 12288
remote_addr: 192.168.10.1
local_addr: 192.168.10.3
requester: 1b:00
sockfd: 3
===============================
dma_read to 0x82159c720 returns 12
Hex dump
7465 7374 5f73 7472 696e 6700
ASCII dump
test _str ing
The ASCII dump shows the DMAed memory region that is "test_string".
At that moment, dma_read (actually LibTLP) made a MRd TLP and sent it to the NetTLP adapter over the Ethernet link, and the adapter decapsulated it and delivered the original (inner) TLP to the root complex at the adapter host over the PCIe links. Afterward, the root complex sent a corresponding CplD TLP to the adapter, and then the adapter encapsulated the TLP and sent it to the device host.
As expected, the TLPs can be tcpdumped at the device host as shown below:
# 3. capture the TLPs by the modified tcpdump at device host
$ sudo tcpdump -ni eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:47:10.315276 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: MRd, 4DW, tc 0, flags [none], attrs [none], len 3, requester 1b:00, tag 0x00, last 0xf, first 0xf, Addr 0x000000082159c720
20:47:10.315307 IP 192.168.10.1.12288 > 192.168.10.3.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 3, completer 00:00, success, byte count 12, requester 1b:00, tag 0x00, lowaddr 0x20
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
In addition, DMA on NetTLP would read any memory regions as shown:
$ ./dma_read -r 192.168.10.1 -l 192.168.10.3 -b 1b:00 -a 0x0 -s 256
======== struct nettlp ========
port: 12288
remote_addr: 192.168.10.1
local_addr: 192.168.10.3
requester: 1b:00
sockfd: 3
===============================
dma_read to 0x0 returns 256
Hex dump
f3ee 00f0 f3ee 00f0 c3e2 00f0 f3ee 00f0 f3ee 00f0 54ff 00f0 8732 00f0 2f32 00f0
a5fe 00f0 87e9 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 57ef 00f0 53ff 00f0
dc17 00c0 4df8 00f0 41f8 00f0 59ec 00f0 39e7 00f0 59f8 00f0 2ee8 00f0 d2ef 00f0
00e0 00f0 f2e6 00f0 6efe 00f0 53ff 00f0 53ff 00f0 a4f0 00f0 c7ef 00f0 d0cc 00c0
f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0
f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0
f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0
f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0 f3ee 00f0
And, DMA write also works. For example, rewriting the buffer
of the demo program. In the below
example, ./dma_write -r 192.168.10.1 -l
192.168.10.3 -b 1b:00 -a 0x827b2f320 -p
modified_by_dma_write -s 22 at the device host
overwrites 0x827b2f320 at the adapter host.
$ sudo ./demo
PHY ADDR 0x827b2f320 test_string
PHY ADDR 0x827b2f320 test_string
PHY ADDR 0x827b2f320 test_string
PHY ADDR 0x827b2f320 modified_by_dma_write
PHY ADDR 0x827b2f320 modified_by_dma_write
PHY ADDR 0x827b2f320 modified_by_dma_write
Note: As expected, this platform easily, unintentionally, and sometimes silently disrupts the system of adapter host. On the other hand, this charcteristic offers powerful adaptability to diverse use cases.
psmem
psmem is a pseudo memory device implementation in software. psmem on the device host pretends a memory region associated with the BAR4 of the NetTLP adapter. TLPs to the BAR4 space of the NetTLP adapter are transmitted to the device host. When psmem receives a MWr TLP, psmem saves the data and the associating address. When psmem receives a MRd TLP, psmem sends CplD TLP(s) with proper data associating the requested address.
First, run psmem on the device host. Note
that -a 0xa0000000 specifies the
BAR4 address of the adapter at the adapter host. Adjust it
to your environment.
# At device host
$ ./psmem -h
./psmem: invalid option -- 'h'
usage
-r remote addr at NetTLP link
-l local addr at NetTLP link
-R remote host addr to get BAR4 start address
or
-a start addess (HEX)
-b bus number, XX:XX
# snipped
$ ./psmem -r 192.168.10.1 -l 192.168.10.3 -a 0xa0000000 -b 1b:00
start psmem callbacks. BAR4 is 0xa0000000, Dev is 0x1b00
nettlp_cb_thread: start callback on cpu 0, port 12288
nettlp_cb_thread: start callback on cpu 1, port 12289
nettlp_cb_thread: start callback on cpu 2, port 12290
nettlp_cb_thread: start callback on cpu 3, port 12291
nettlp_cb_thread: start callback on cpu 4, port 12292
nettlp_cb_thread: start callback on cpu 5, port 12293
nettlp_cb_thread: start callback on cpu 6, port 12294
nettlp_cb_thread: start callback on cpu 7, port 12295
nettlp_cb_thread: start callback on cpu 8, port 12296
nettlp_cb_thread: start callback on cpu 9, port 12297
nettlp_cb_thread: start callback on cpu 10, port 12298
nettlp_cb_thread: start callback on cpu 11, port 12299
nettlp_cb_thread: start callback on cpu 12, port 12300
nettlp_cb_thread: start callback on cpu 13, port 12301
nettlp_cb_thread: start callback on cpu 14, port 12302
nettlp_cb_thread: start callback on cpu 15, port 12303
Second, access the BAR4 space of the NetTLP adapter. This repository contains a simple but useful application, called memory, that enables accessing any memory regions through /dev/mem.
# At adapter host
$ sudo ./memory 0xa0000000
Then, the memory command periodically access the region started from 0xa0000000, the BAR4 space of the NetTLP adapter, by 1 byte, and you can see and modify memories through the curses-based user interface. Meanwhile, you see psmem outputs MRd to the region at the device host. TLPs issued by the memory accesses can be seen by the modified tcpdump.
# At device host while running psmem
$ sudo tcpdump -ni eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:22:09.944928 IP 192.168.10.1.12288 > 192.168.10.3.12288: NetTLP: MRd, 3DW, tc 0, flags [none], attrs [none], len 1, requester 16:00, tag 0x00, last 0x0, first 0x2, Addr 0xa00001d0
22:22:09.945003 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x51
22:22:09.945018 IP 192.168.10.1.12288 > 192.168.10.3.12288: NetTLP: MRd, 3DW, tc 0, flags [none], attrs [none], len 1, requester 16:00, tag 0x00, last 0x0, first 0x4, Addr 0xa00001d0
22:22:09.945092 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x52
22:22:09.945107 IP 192.168.10.1.12288 > 192.168.10.3.12288: NetTLP: MRd, 3DW, tc 0, flags [none], attrs [none], len 1, requester 16:00, tag 0x00, last 0x0, first 0x8, Addr 0xa00001d0
22:22:09.945185 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x53
22:22:09.945464 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x56
22:22:09.945551 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x57
22:22:09.945567 IP 192.168.10.1.12288 > 192.168.10.3.12288: NetTLP: MRd, 3DW, tc 0, flags [none], attrs [none], len 1, requester 16:00, tag 0x00, last 0x0, first 0x1, Addr 0xa00001d8
22:22:09.945642 IP 192.168.10.3.12288 > 192.168.10.1.12288: NetTLP: CplD, 3DW, WD, tc 0, flags [none], attrs [none], len 1, completer 1b:00, success, byte count 1, requester 16:00, tag 0x00, lowaddr 0x58
Note: the memory command reads each 1 byte continuously. Such access patterns issue continuous MRd TLPs to LibTLP at a hardware-level speed. As a result, the current LibTLP implementation that uses the traditional socket can not send response CplD TLPs immediately. This means that multiple MRd TLPs would be buffered at NIC queues, network stack, socket layer, etc. The consecutive CplD TLPs on the above tcpdump output indicate this phenomenon.